Subject: variables: open issue a) date variables

From: Kjetil Torgrim Homme <[email-masked]>
Date: 02 May 2003 01:04:16 +0200

there are a few open issues mentioned in the draft.  I'm sure there
should be more listed, so please remind me of what I have forgotten or
glossed over.  in any case, I think it is time to try to get them
resolved.  the first one is quite trivial:

  a) should we include more predefined variables to access the time?
     (weekday, week number (US, EU and/or ISO?), name of month, name
     of weekday, ...)  could use strftime(3c) as a list of what to
     offer, but the names should be in English.

the current list is ${year} ${month} ${day} ${hour} ${minute}
${second} ${timezone}, all numeric and zero-padded.

I don't like adding variables with values in natural language, since
that adds internationalisation problems for little gain.
unfortunately, both weekday and week have localisation problems.

strftime(3c) solves this by supplying many conversion specifiers,
i.e. %U, %V and %W for three methods of computing the week number.  I
feel we would need to select one of these (and then the ISO
definitions is the obvious candidate), after all having one is better
than none.

for the weekday, while 0..6 with 0 meaning Sunday may seem alien to
many users (including me :), at least the information is available in
an unambigious fashion.

so my suggestion is to augment the list with the following:

  ${weekday}   (0..6)       %w: 0 is Sunday
  ${week}      (01..53)     %V: week number according to ISO 8601
  ${isoyear}   (0000..9999) %G: year number according to ISO 8601.  
                                needed since Jan 1 2000 is in week 52 
                                of 1999

my reasoning is that this is useful information for some people, and
it is easy to implement.

-- 
Kjetil T.

---

From: Marc Mutz <[email-masked]>
Date: Sat, 3 May 2003 00:47:32 +0200

On Friday 02 May 2003 01:04, Kjetil Torgrim Homme wrote:
<snip>
> ${second} ${timezone}, all numeric and zero-padded.

There are two issues I have with this:
1. We define magic variables and don't provide a mechanism for other 
extensions to register their own magic variables.
2. We combine two basically unrelated extensions ("variables" and 
"dates") into one.

Short comment: Define a namespace for magic variables and move the date 
variables out into the to-be-written draft defining the date test.

Long comments:
Let's assume for the moment that "variables" didn't include "dates", but 
defined a syntax for magic variables[1]. "dates" would encompass the 
date test Ned wanted to write an I-D for and - in the presence of the 
"variables" extension - would define certain magic variables "#year", 
"#month" etc.

ad 1: Change the variable-name production to read
  variable-name := num-variable / magic-variable / user-variable
  magic-variable := "#" identifier
  user-variable := identifier
and require every Sieve extension to define their magic-variables and 
which commands change their state and how. This is more like how common 
script languages (bash, awk, perl,...) work and it provides a namespace 
for magic (or "system") variables alike to perl's $<non-alnum> 
variables.

ad 2: The "dates" extension would then use this mechanism to declare 
#year, #month, #day, #hour, #minute, #second and #timezone to be system 
variables that are initialized to their respective values of the time 
of script execution start (local timezone). Since the "setdate" action 
now didn't really set the date variables anymore, it would better be 
named "settimezone". If executed, these system variable values are 
recomputed, based on the new timezone given to "settimezone", but still 
representing the time of script execution start.

To see the benefit of the system variable notion, consider the imapflags 
extension. This would now declare the #imapflags system variable that 
holds the current flags (e.g. as a list of imap flag names separated by 
spaces). addflags, setflags and removeflags would then act on the 
#imapflags variable, but
  set "#imapflags" "${#imapflags} \\Seen"
would be OK, too, of you accept the fact that you can cause problems if 
you forget the space before "\Seen". Basically, if you assign to system 
variables, it's your own fault. Alternatively, assignment to system 
variables could be disallowed. I think would be the the best way to 
handle it, since it also prevents users from defining variables in the 
system namespace (which wouldn't be too problematic, given that system 
variables spring into life only if their extension is require'd).
Fileinto would then use the value of #imapflags to determine the flags 
to set on (e.g.) IMAP APPEND.

Comments?

Marc

[1] Those that are either implicitly changed by commands or are 
automatic in the sense that the environment provides them w/o explicit 
instruction (with can be rephrased as: "are set implicitely by the 
require command requesting the corresponding extension"). For 
concreteness, let's assume they must all start with a hash '#'.

-- 
If this were a dictatorship, it'd be a heck of a lot easier...just as
long as I'm the dictator...
-- George W. Bush, Washington, DC, Dec 18, 2000,
   during his first trip to Washington as President-Elect

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 03 May 2003 02:06:41 +0200

>   extensions to register their own magic variables.

they are not really magic.  you can do

  SETDATE;
  SET "year" "${year} AD";

unless you call SETDATE, no variables in your name space will be
touched.

>   2. We combine two basically unrelated extensions ("variables" and

yes.  but do we gain anything by splitting the two?  "dates" would
require "variables", and the implementation impact of "dates" is quite
low, especially compared to "variables".

>   Short comment: Define a namespace for magic variables and move the
>   date variables out into the to-be-written draft defining the date
>   test.

I don't think we need magic variables.  I had them in my very first
draft, but threw them out since I couldn't see a good use for them.
we'd need a registry for known magic variable names, since extensions
could conflict.  actions updating variables doesn't need that -- if
they stomp on each others variables, the user will know from the spec
and take whatever action is necessary to save aside any data needed.

>   To see the benefit of the system variable notion, consider the
>   imapflags extension. This would now declare the #imapflags system
>   variable that holds the current flags (e.g. as a list of imap flag
>   names separated by spaces).

nice, but you can get the same capability by adding an action
IMAPFLAGS which updates the value of ${imapflag}.  (this might be a
good idea, actually.)

>   addflags, setflags and removeflags would then act on the
>   #imapflags variable, but
>
>   would be OK, too, of you accept the fact that you can cause
>   problems if you forget the space before "\Seen".

that's just _too_ magic :-)

>   Basically, if you assign to system variables, it's your own
>   fault. Alternatively, assignment to system variables could be
>   disallowed.  I think would be the the best way to handle it, since
>   it also prevents users from defining variables in the system
>   namespace (which wouldn't be too problematic, given that system

yes.

-- 
Kjetil T.

---

From: [email-masked]
Date: Fri, 02 May 2003 17:06:57 -0700 (PDT)

> variables out into the to-be-written draft defining the date test.

I have no problem with defining a magic variable namespace. In fact
in some way I'd prefer it. That being said, however, I have a big
problem with defining it and not using it in the same document. All
too often this leads to issues being left open and problems not being
spotted until too late.

> for magic (or "system") variables alike to perl's $<non-alnum>

I'd somewhat prefer to use $ for the prefix, but it works either way.

> representing the time of script execution start.

Works for me.

> to set on (e.g.) IMAP APPEND.

I kind of like the idea of not being able to assign to system variables.

				Ned

---

From: [email-masked]
Date: Fri, 02 May 2003 17:13:53 -0700 (PDT)

> ${second} ${timezone}, all numeric and zero-padded.

There definitely needs to be a way to specify a day of the week. I really don't
care if it is 1/2/3/4 or mon/tue/wed or whatever, but it needs to be there
because people want to take certain actions based on the day of week.

FWIW, RFC 2445 uses

weekday    = "SU" / "MO" / "TU" / "WE" / "TH" / "FR" / "SA"

Beyond that, an iso8601 option would be useful for checking boundaries
in a single operation. RFC 3339 can be referenced for this.

> unfortunately, both weekday and week have localisation problems.

You've already bought in to the localization issue to some extent by using
Gregorian months. I think worrying about the localization issues for
days of the week is silly.

The same issue was faced in calshd. Not to put too fine a point on it, but if
users of other calendars want their weeks, months and so on represtented they
are free to write up a draft specifying such things. 

> definitions is the obvious candidate), after all having one is better

Assuming we even need week number, I'd suggest %V. That's what ISO 8601
uses and what RFC 2445 picked.

> many users (including me :), at least the information is available in

If you use 1/2/3 you _are_ using the Islamic days of the week ;-)

>   ${week}      (01..53)     %V: week number according to ISO 8601

These are both fine as far as I'm concerned.

>                                 needed since Jan 1 2000 is in week 52

The last doesn't necessarily follow. If you're using week numbers you may not
care about the specific year.

> my reasoning is that this is useful information for some people, and

Sure. Doesn't everyone have a copy of _Calendrical Calculation_ on their desk?
I certainly do.

				Ned

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 03 May 2003 05:36:53 +0200

[Ned Freed]:
>
>   There definitely needs to be a way to specify a day of the week. I
>   really don't care if it is 1/2/3/4 or mon/tue/wed or whatever, but
>   it needs to be there because people want to take certain actions
>   based on the day of week.
>   
>   FWIW, RFC 2445 uses
>   

numeric values seem more versatile, but textual values make for more
readable scripts.

>   Beyond that, an iso8601 option would be useful for checking
>   boundaries in a single operation. RFC 3339 can be referenced for
>   this.

thanks, I hadn't read that RFC (shame on me), but I don't quite get
the reference?  do you mean a variable containing the complete
"2003-05-03T05:03:12.00+02:00" ?  (I think I would leave out the time
zone for this application, actually.)

>   > for the weekday, while 0..6 with 0 meaning Sunday may seem alien
>   > to many users (including me :), at least the information is
>   > available in an unambigious fashion.
>   

well, I'm a member of the The Net Atheists, but on the seventh day He
resteth or something like that, so it always struck me as odd to
number Sunday as the first day of the week.  but I digress ;-)

>   These are both fine as far as I'm concerned.

now I'm not so sure.  RFC 3339 follows ISO 8601 and uses

   date-wday       =  DIGIT  ; 1-7  ; 1 is Monday, 7 is Sunday

>   The last doesn't necessarily follow. If you're using week numbers
>   you may not care about the specific year.

I see two main uses for ${week}.  one is vacation message: "I'll be
out of town during week 45 and 46".  the other is for fileinto, and
then we need ${isoyear} to get a strict ordering.

the problem with the former use is that US weeks doesn't match ISO
weeks.  we Europeans will be fine (I think -- Norwegians for sure ;-).

-- 
Kjetil T.

---

From: Marc Mutz <[email-masked]>
Date: Sat, 3 May 2003 12:27:53 +0200

On Saturday 03 May 2003 02:06, [email-masked] wrote:
<snip>
<snip>

We'd have the date and imapflags (and notify) extensions to go along 
with variables. The variables draft also defines the special 
num-variable "system variables" for the regex extension and :matches.

<snip>

I don't like $ being the prefix since it's also the variable special 
character. "${$year}" smells like resolving references (or pointers if 
you're C) to a perl programmer. Also, the variables draft mentions this 
example:

>         "${company}" => "ACME"
>         "${President, ${Company} Inc.}"
>                      => "${President, ACME Inc.}"

Which makes (if kept, I'm not sure it should be, personally, I think it 
should give an error) the interpretation of

set "year" "2000";
fileinto "archive.${$year}"

ambiguous (could either file into "archive.${2000}" or "archive.2003" if 
2003 was the current year).

Marc

-- 
We notice things that don't work. We don't notice things that do.
We notice computers, we don't notice pennies.
We notice e-book readers, we don't notice books.
                                          -- Douglas Adams

---

From: [email-masked]
Date: Sat, 03 May 2003 08:04:19 -0700 (PDT)

> zone for this application, actually.)

Yes, that's what I meant.

>    date-wday       =  DIGIT  ; 1-7  ; 1 is Monday, 7 is Sunday

Then that's what we'd better use.

> weeks.  we Europeans will be fine (I think -- Norwegians for sure ;-).

It isn't clear to me that weeks are referred to by number enough to matter.
At least not in the US.

				Ned

---

From: [email-masked]
Date: Sat, 03 May 2003 08:08:17 -0700 (PDT)

> you're C) to a perl programmer. Also, the variables draft mentions this

That's sufficient reason, I guess. I do note, however, that nobody else is
jumping in to support this prefix idea. I'm ambivalent about it myself.

> 2003 was the current year).

I don't see any ambiguity here. This is clearly a reference to $year, not
year, and the result will be 2003. The brackets are mandatory in variable
references, not optional.

				Ned

---

From: Marc Mutz <[email-masked]>
Date: Sat, 3 May 2003 22:27:20 +0200

On Saturday 03 May 2003 17:08, [email-masked] wrote:
<snip>
> I do note, however, that nobody
> else is jumping in to support this prefix idea. I'm ambivalent about
> it myself.
<snip>

It's weekend ;-)

As we _have_ magic variables in the current draft (year, month, etc) 
that are implicitly changed by commands (as opposed to be set with 
"set"), no matter if we call them thus or not, it would be a good idea 
to make that explicit and to enable other extensions to define their 
own magic/system variables.

The reasons against magic/system variables Kjetil mentioned were:

1. Year, month, etc. are not really magic, you can write them via set
   and they only spring into life on setdate.

I think we agree that writing to those variables is not a good idea, 
since it's not needed, makes program flow analysis more complex and 
produces potentially invalid system variable content that leads to a 
new class of runtime errors not present with read-only system 
variables.

Furthermore, that they spring into life as a side-effect of a command is 
proof, not disproof, of their magicness.

2. There's no good use for them.

Well, the fact that they are being defined (whether called system 
variables or not) in itself is proof of their usefulness. imapflags 
could also make good use of system variables. In fact, IIRC, the 
imapflags extension was was what prompted the variables extension in 
the first place.

3. We'd need a registry for them, since extensions could conflict.

They can still conflict if we don't make system variables explicit in 
the variables extension. Prefixing them at least avoids conflicts with 
user variables.

As to requiring a registry. Sure, that would be nice, but isn't a 
necessity. We don't have a registry for command or test identifiers 
either, even though they could also conflict between extensions. 
Instead, we believe the extension authors and the review process to 
avoid these conflicts.

Marc

-- 
We do not believe it draws the proper balance in a democratic society
for the activities of government to be concealed from public scrutiny
while the private activities of citizens are made open to government.
       -- EPIC letter to the President of the EU Council of Ministers

---

From: Marc Mutz <[email-masked]>
Date: Sat, 3 May 2003 22:38:16 +0200

On Saturday 03 May 2003 02:06, Kjetil Torgrim Homme wrote:
<snip>
<snip>
<snip>
<snip>
> > Alternatively, assignment to system variables could be
> > disallowed.  I think would be the the best way to handle it,
> > since it also prevents users from defining variables in the system

So you say that
  set "year" "${year} AD";
is possible, but that
  set "imapflags" "${imapflags} \\Seen";
is too magic and writing to system variables should be disallowed?

I agree with the latter, but not with the former. I don't see the 
usefulness of
  set "year" "${year} AD";
since
1. It can lead to hard-to-find bugs if, in a large script, you alter the
   value of "year" and way down the script test against ${year}.
and more importantly,
2. What does the above solve that can't be solved by
     set "myyear" "${year} AD";

Marc

-- 
Military justice is to justice what military music is to music.
                                                  -- Groucho Marx

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 04 May 2003 00:15:24 +0200

[Marc Mutz]:
>
>   [Ned Freed]:
>   > I do note, however, that nobody else is jumping in to support
>   > this prefix idea. I'm ambivalent about it myself.
>   
>   It's weekend ;-)

weekends seem to be more active than workdays on this list.

if you want a prefix character, use "_".  it's part of the identifier
grammar, and has seen similar use in other contexts.

>   I think we agree that writing to those variables is not a good
>   idea, since it's not needed, makes program flow analysis more
>   complex

really?  I think an explicit action to set them is easier to handle
than a multitude of variables which can change as side-effects.  we've
already seen that this is a tricky issue with the match back
references.  there the gain for easy access seems great enough to be
worth it.  SETDATE can be seen as a macro for 

  SET "hour" "${_internal_startup_hour}";
  SET "minute" "${_internal_startup_minute}";
  SET "second" "${_internal_startup_second}";
  ...

pretty straightforward to analyse, IMO.

what actions can influence ${_imapflags}?  how will its value change?

>   and produces potentially invalid system variable content that
>   leads to a new class of runtime errors not present with read-only
>   system variables.

it's not a system variable.  it will not lead to more errors than we
get from making changable variables available to the user.

>   Furthermore, that they spring into life as a side-effect of a
>   command is proof, not disproof, of their magicness.

see macro comment.

>   variables or not) in itself is proof of their
>   usefulness. imapflags could also make good use of system
>   variables. In fact, IIRC, the imapflags extension was was what
>   prompted the variables extension in the first place.

I wasn't around then.  my motivation for writing the draft was to
enable match back references in the argument to fileinto.

>   They can still conflict if we don't make system variables explicit
>   in the variables extension. Prefixing them at least avoids
>   conflicts with user variables.

no, there can be no conflict, two extensions can use the same variable
name and co-exist peacefully.  e.g., you could have two actions
putting its return value in ${result}.  the user would just have to do
a SET to save it in another variable before doing the next action, and
he only has to do that if he needs to use the two actions in the same
Sieve run.  with your truly magic variables, this can't be done.

>   necessity. We don't have a registry for command or test
>   identifiers either, even though they could also conflict between
>   extensions.  Instead, we believe the extension authors and the
>   review process to avoid these conflicts.

fair enough.

-- 
Kjetil T.

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 04 May 2003 00:57:12 +0200

>   is too magic and writing to system variables should be disallowed?

I don't like that writing to some variables has side effects.  it
seems to me that you want this SET to actually change the global IMAP
flags?  I want to require an explicit action to do so.  side effects
must be kept to a minimum.

>   I agree with the latter, but not with the former. I don't see the 

me neither :-)

perhaps if you want to change ${year} to use the Mayan calendar?
(that'd be fun -- lots of regexp matching should do it :-)

>   since
>   1. It can lead to hard-to-find bugs if, in a large script, you 
>      alter the value of "year" and way down the script test against
>      ${year}.

don't do that then.

my point is that the user can use ${year} himself if he doesn't use
SETDATE.  e.g.,

    if header :matches "Date" "*, * * * *:*:* *" {
        set "day" "${2}";
        set "year" "${4}";
        # ...
    }

this is particularily important since SETDATE is part of the
"variables" extension, but I don't like to add "reserved variables"
when we don't need to, and I'm not convinced we do.

>        set "myyear" "${year} AD";

nothing.

-- 
Kjetil T.

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 04 May 2003 02:06:30 +0200

[Ned Freed]:
>
>   > do you mean a variable containing the complete
>   > "2003-05-03T05:03:12.00+02:00" ?  (I think I would leave out the
>   > time zone for this application, actually.)
>   

okay, here's what I got now:

-----
   The action setdate initialises a few variables:

      ${year}     => the year, "0000" .. "9999"
      ${month}    => the month, "01" .. "12"
      ${day}      => the day, "01" .. "31"
      ${hour}     => the hour, "00" .. "23"
      ${minute}   => the minute, "00" .. "59"
|     ${second}   => the second, "00" .. "60",
+                    60 is only used for leap seconds, see [TIMESTAMPS].
+     ${weekday}  => the day of week, "1" .. "7",
+                    1 is Monday
+     ${week}     => the week, "01" .. "53",
+                    numbered according to [ISO8601]
+     ${weekyear} => the year ${week} is part of, "0000" .. "9999",
+                    according to [ISO8601].  (e.g, 2000-01-01 is
+                    in week 52 of 1999)
+     ${date}     => a lexically ordered timestamp.  Using the above
+                    definitions, it is set to the expansion of
+                    "${year}-${month}-${day}T${hour}:${minute}:${second}".
      ${timezone} => the time zone in use.  If the user specified a
                     time zone which was recognised, ${timezone} will
                     contain the name given.  Otherwise, the value
                     MUST be the server's default time zone in offset
                     format.
-----

I changed the name of "isoyear" to "weekyear", I think that's more
accurate.  after all, the year in ISO 8601 is the same as ${year}.
it's only in week context it's different.

>   It isn't clear to me that weeks are referred to by number enough
>   to matter.  At least not in the US.

week numbers are used quite a bit here in Europe.

-- 
Kjetil T.

---

From: Marc Mutz <[email-masked]>
Date: Sun, 4 May 2003 14:41:03 +0200

On Sunday 04 May 2003 00:15, Kjetil Torgrim Homme wrote:
<snip>
> grammar, and has seen similar use in other contexts.

"_" is as good as "#". I've chosen "#" b/c of the #namespace convention 
of IMAP and #... would be a kind of system namespace, like std:: in 
c++.

I don't see why variable names have to match the identifier production. 
num-variable certainly doesn't, either.

> >   1. Year, month, etc. are not really magic, you can write them via
> > set and they only spring into life on setdate.
> >
> really?

Come one. I thought we agreed that writing to system variables is a bad 
idea...

> I think an explicit action to set them is easier to handle 
> than a multitude of variables which can change as side-effects.

But they _do_ change as side effects. You can't go and pretend they 
don't.

> we've already seen that this is a tricky issue with the match back

So what should ${_internal_startup_hour} be other than a system 
variable? Also, how do the equivalent set commands look like for
  setdate "CEST";
? ;-)

<snip>

${#imapflags} is (if resolved) a space-separated list of imap flags. The 
order of flags in ${#imapflags} is undefined (it's conceptually a set).

addflag
- adds the flags given as arg to #imapflags then
- removes all duplicates.

removeflag
- removes all flags given as arg from #imapflags

setflag
- clears #imapflags
- executes addflag <args>

mark
- equivalent to addflag "\\Flagged"

unmark
- equivalent to removeflag "\\Flagged"

> >   command is proof, not disproof, of their magicness.

The macro comment only shifts the problem from ${year} to 
${_internal_startup_year}. Instead of ${year}, you make 
${_internal_startup_year} magic.

> >   They can still conflict if we don't make system variables
> > explicit in the variables extension. Prefixing them at least avoids
> no, there can be no conflict, two extensions can use the same
> variable name and co-exist peacefully. e.g., you could have two 
> actions putting its return value in ${result}.  the user would just
> have to do a SET to save it in another variable before doing the next
> action, and he only has to do that if he needs to use the two actions
> in the same Sieve run.  with your truly magic variables, this can't
> be done.
<snip>

Granted, but where's the usefulness in allowing that in the first place? 
I don't see why such behaviour is needed, the more so as it's very 
inconvenient for the user (oh - and GUI editors!) to have to save away 
${result} themselves.

Summary:
There are two issues, let's separate them:
1. Should set be allowed to act on system variables?
   I hope to have made it clear that that's a bad idea.
2. Should system variables be prefixed?
   Well, why _not_? If it makes set's job easier to determine if
   something is read-only or not? If it makes system variable references
   more explicit?

Marc

-- 
I am Bush of USA. You will be pacified. Resistance is futile.

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 05 May 2003 00:19:13 +0200

[Marc Mutz]:
>
>   Come on. I thought we agreed that writing to system variables is a
>   bad idea...

I don't agree that ${year} is a system variable.

>   But they _do_ change as side effects. You can't go and pretend
>   they don't.

I wouldn't say it's a side effect, SETDATE is quite explicit.  nothing
else will change the variables.

>   > we've already seen that this is a tricky issue with the match
>   > back references.  there the gain for easy access seems great
>   > enough to be worth it.  SETDATE can be seen as a macro for
>   >
>   variable?

${_internal_startup_hour} doesn't exist...

>   ${#imapflags} is (if resolved) a space-separated list of imap
>   flags. The order of flags in ${#imapflags} is undefined (it's
>   conceptually a set).

so

  if string :matches "${#imapflags}" "*\\Answered*\\Flagged*" {
    ...
  }

may or may not match even if both flags are set.  (easily fixed, just
specify lexical sorting of the "set" at all times.)

>   - removes all flags given as arg from #imapflags

wouldn't it be better to make generic functions for this?
    addflag "flag" "\\Flagged";
changes ${flag}.  you'd now use
    keep :flags "${flag}";
rather than
    keep :globalflags;
and
    keep :flags "${flag} \\Flagged";
instead of
    keep :globalflags_plus "\\Flagged";

some scripts may become more verbose, but the complexity of the
IMAPFLAGS is decreased, IMO.  (ADDFLAG and REMOVEFLAG could move to a
separate extension.)

>   - executes addflag <args>

what would SETFLAG "${#imapflags}" do?  just one of those things which
need to be clarified.

>   - equivalent to removeflag "\\Flagged"

what is the rationale for MARK and UNMARK?  according to the draft
these actions are _not_ equivalent to ADDFLAG "\\Flagged".

>      I hope to have made it clear that that's a bad idea.

I strongly agree!

>      something is read-only or not? If it makes system variable
>      references more explicit?

I don't want to introduce system variables at all.  I haven't seen a
compelling case for adding them.

-- 
Kjetil T.

---

From: Tony Hansen <[email-masked]>
Date: Mon, 05 May 2003 23:27:13 -0400

No one but Kjetil responded on this the last time I brought it up. I 
suggested that names be allowed to be compound, and the DATE extension 
put everything into the "time." namespace. So, instead of "${year}", 
we'd have "${time.year}".

	Tony Hansen
	[email-masked]

---

From: [email-masked]
Date: Mon, 05 May 2003 23:21:04 -0700 (PDT)

> we'd have "${time.year}".

I actually prefer a system variable scheme or something similar to this.

				Ned

---

From: Marc Mutz <[email-masked]>
Date: Tue, 6 May 2003 14:27:55 +0200

On Monday 05 May 2003 00:19, Kjetil Torgrim Homme wrote:
<snip>
> I don't agree that ${year} is a system variable.

Granted it's not a clean-cut one, but see below.

> I wouldn't say it's a side effect, SETDATE is quite explicit. 
> nothing else will change the variables.
<snip>

It's a side-effect of executing setdate. I call this a side-effect, 
since it's not an explicit "set".

<snip>
> ${_internal_startup_hour} doesn't exist...

So how can SET "hour" ${_internal_startup_hour}" work?

> may or may not match even if both flags are set.

Correct.

> (easily fixed, just 

I always wondered why imapflags didn't include a flag test. It should, 
esp. if one considers the include extension where a included script 
doesn't know what it's parent has already done and might want to know.

> IMAPFLAGS is decreased, IMO.  (ADDFLAG and REMOVEFLAG could move to a

One could always add the feature to accept multiple flags per string 
argument, so you can use

set "myflags" "${#imapflags}" # save away
# ...
setflags "${myflags}" # restore

I fact, you already used that feature in the code above. Also, keep in 
mind that imapflags implementations already exist and you can imagine 
what a confusion would arise if addflags' et al. syntax would be 
changed.

> what would SETFLAG "${#imapflags}" do?  just one of those things
> which need to be clarified.

With the above feature, it would be a no-op.

> these actions are _not_ equivalent to ADDFLAG "\\Flagged".

But IIRC, it's suggested they are. However, this is nitpicking again, 
since it's clear that the exact (un)mark semantics are 
implementation-defined. I've just chosen the obvious one.

> >      I hope to have made it clear that that's a bad idea.

:-)

> compelling case for adding them.

Ok, here's what I think are properties of system variables, why they are 
needed and why they should be prefixed:

1. System variables are implicitly changed by certain commands and vice
   versa: the value of system variables affects certain commands.

   Examples: ${#imapflags} is changed by {add,remove,set}flags,
             {un,}mark. It implicitly affects keep, fileinto and a
             potential "flags" test.

             A ${#currentdate} variable may affect the to-be-defined
             date test.

   This is the most visible property of system variables, though in
   itself alone it doesn't require system variables to be handled
   specially.

2. They are read-only to "set".

   Examples: ${#imapflags} should be written to only by the imapflags
             actions.
             ${#currentdate} should never be written to.

   So the set action needs to raise an error if the script author tries
   to assign to a system variable. That begs the question of how set
   should know what is a system variable and what isn't. You could keep
   a list of system variables, which is complicated by the fact that (as
   you said) it should be perfectly OK to use the name of a system
   variable as long as the corresponding extension is not "required".
   But apart from the fact that you forgot about the include extension
   (where variables have global, but require'd extensions local scope),
   it would be much easier (for user _and_ engine) if system variables
   had a common prefix to detect them.

3. They may represent concepts that are not naturally represented as
   strings.

   Examples: ${#imapflags} is a set of imap flags, not a string.
             ${#currentdate} is a number or struct tm or whatever, not a
             string.

   There are many issues if you allow ${#imapflags} to be "set" by the
   user (duplicate or invalid flags, ordering issues). However, it's a
   variable (remember that imapflags was what prompted a variables
   extension in the first place, even if you were not personally driven
   by that) and there was objection to define extensions that have any
   kind of "internal variable" (as imapflags once called it) without
   first having variables for Sieve in general.

   It is, however, potentially very useful to allow interpolation into
   strings, and a system variable declaration must specify the string
   representation to use. I've already given an example for
   ${#imapflags}, ${#currentdate} could be interpolated in rfc2822
   format.

   So system variables could be handled internally in their natural
   representation, with a string representation being required only on
   expansion into strings, which should be pretty rare.

4. Their values may not be computed with Sieve.

   You said setdate would be a macro for a list of set's. But it isn't.
   You can't represent setdate "+0200" like that.

   E.g. a hypothetical ${#envelope_sender} system variable would not be
   a good idea, since you can always obtain that by using the
   matches-set idiom:
     if :envelope :all :matches "from" "*" {
       set "myenvelopesender" "${1}";
     }

5. Their values may not be available from the message.

   Example: ${#currentdate}

I now agree that ${year} as specified is not a proper system variable.

Here's a proposal for a date test (which is important to have in it own 
right) that makes setdate superfluous by using the matches-set idiom to 
explicitly set the setdate variables:

if date :current :timezone "+0200" :hour :matches "*" {
  set "${hour}" "${1}";
}

where the date test syntax would be:

date [COMPARATOR] WHICH [TIMEZONE] [DATE-PART] [MATCH-TYPE]
     <key-list: string-list>

WHICH := ":current" / ":received" / ":sent"
         / ( ":header" <headerfield: string-list> )
DATE-PART := ":weekday" / ":day" / ":month" / ":year"
         / ":hour" / ":minute" / ":second" / ":numericzone"
         / ":all" / ... ; ":all" is default
TIMEZONE := ":timezone" <timezone: string>
     ; server's local timezone is default for :current,
       timezone specified in the header is default for :received, :sent
       and :header.

Together with the relational extension and a rfc822-date comparator, you 
could check for dates:

  if allof( date :comparator "rfc822-date"
      :current :value "ge" "Tue, 06 May 2003 14:02:33 +0200",
            date :comparator "rfc822-date"
      :current :all :value "le" "Tue, 13 May 2003 14:04:05 +0200" )
  {
    vacation "blah, blubb";
  }

And for those that want "old" setdate:

  set "setdate_timezone" "CEST"
  include :global "setdate";

where "setdate" includes the matches-set idioms:

--begin datevariables--
require [ "variables", "dates" ];

if date :current :timezone "${datevariables_get_timezone}"
   :year :matches "*" {
  set "year" "${1}";
}
if date ... # and so on for any other setdate variable
--end datevariables--

In this scenario, ${#currentdate} would be a system variable with all 
properties as defined above: It affects the date :current test, is 
read-only, it's natural representation is not a string, it's value 
can't be computed with Sieve and it can't be extracted from the 
message.

Marc

-- 
Ein Grundrecht auf Sicherheit steht bewusst nicht in der Verfassung.
  -- Sabine Leutheusser-Schnarrenberger (ehem. Bundesjustizministerin)

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Wed, 07 May 2003 22:59:17 +0200

[Marc Mutz]:
>
>   It's a side-effect of executing setdate. I call this a
>   side-effect, since it's not an explicit "set".

alright.

>> (easily fixed, just specify lexical sorting of the "set" at all
>> times.)
>
>   I always wondered why imapflags didn't include a flag test. It
>   should, esp. if one considers the include extension where a
>   included script doesn't know what it's parent has already done and
>   might want to know.

agreed.  the current method is to use a :matches, which will work, but
is a bit roundabout.

>   I fact, you already used that feature in the code above.

yep.

>   Also, keep in mind that imapflags implementations already exist
>   and you can imagine what a confusion would arise if addflags' et
>   al. syntax would be changed.

I don't think this is something we should worry too much about.
deploying features before they are standardised is a gamble.

>   But IIRC, it's suggested they are. However, this is nitpicking
>   again, since it's clear that the exact (un)mark semantics are

I was just wondering why their effect is allowed to be implementation
defined.  I can see that MARK/UNMARK are useful shorthands.

>   Ok, here's what I think are properties of system variables, why
>   they are needed and why they should be prefixed:

I agree they need to be recognisable by syntax alone.  one type of
prefix could be "date." or "imap.", ie. the distinguishing feature is
the presence of one or more periods.

>   3. They may represent concepts that are not naturally represented
>      as strings.
>
>      There are many issues if you allow ${#imapflags} to be "set" by
>      the user (duplicate or invalid flags, ordering
>      issues). However, it's a variable (remember that imapflags was
>      what prompted a variables extension in the first place, even if
>      you were not personally driven by that) and there was objection
>      to define extensions that have any kind of "internal variable"
>      (as imapflags once called it) without first having variables
>      for Sieve in general.

I don't understand this reasoning.  you want "internal variables" with
properties very different from those of the general variables.  where
is the connection between the features?

>      It is, however, potentially very useful to allow interpolation
>      into strings, and a system variable declaration must specify
>      the string representation to use. I've already given an example
>      for ${#imapflags}, ${#currentdate} could be interpolated in
>      rfc2822 format.
>
>      So system variables could be handled internally in their
>      natural representation, with a string representation being
>      required only on expansion into strings, which should be pretty
>      rare.

ok.  I honestly don't know what's the best way to handle the IMAP
flags.

1)
part of me wants to add a list variable type.  the main sticky point
is the interpolation.  my suggestion was:

  set "foo" [ "one", "two", "three" ];
  set "bar" [ "${foo}", "four" ];
  set "zot" [ "woo${foo}hoo" ];

now ${bar} is [ "one", "two", "three", "four" ] and ${zot} is [ "woo",
"one", "two", "three", "four", "hoo" ].  this makes "${foo}${bar}"
equivalent with [ "${foo}", "${bar}" ].

to go with this, we'd need a prune command to remove an element from a
list.  actually we need two, one by position, and one by value.  for
imapflags, an append command would be useful as well (it's not worth
worrying about duplicates.)

one problem with this is run-time syntax errors, like

  set "folder" [ "foo", "bar" ];
  fileinto "${folder}";

this could be solved by making making a new list variable syntax,
e.g., @{folder} becomes a list, while ${folder} is becomes a single
string with a single space as delimiter or something.

2)
part of me thinks strings will work OK.  it just needs careful
programming to keep everything padded with spaces.

  set "flags" "${flags} \\Flagged ";  # note trailing space
  if string :matches "${flags}" "* \\Flagged *" {
    set "flags" "${1} ${2}";
  }

3)
part of me thinks an intrinsic variable is the solution which appears
cleanest to the user.

why do you need variables at all for IMAP flags?  to keep state, it
seems.

  if something {
     addflag "\\Flagged";
     if somethingelse {
        addflag "Big";
        keep :globalflags;
     }
  }
  # now we want to restore the flags to the values before the if.
  # or do we?

sorry, I can't come up with a credible example.

>      You said setdate would be a macro for a list of set's. But it
>      isn't.  You can't represent setdate "+0200" like that.

I meant as seen from an optimising Sieve parser, not from a user
perspective.  it wasn't a good comparison, anyway.

>      E.g. a hypothetical ${#envelope_sender} system variable would
>      not be a good idea, since you can always obtain that by using
>      the matches-set idiom:

it seems to me Sieve scripts will soon consist of mostly matches-set
idioms ... ;-)

>   Here's a proposal for a date test (which is important to have in
>   it own right) that makes setdate superfluous by using the
>   matches-set idiom to explicitly set the setdate variables:
>

you won't get daylight savings time handled correctly this way.
compare with

  setdate "Europe/Oslo";
  if not string "${timezone}" "Europe/Oslo" {
    setdate "+0100";
  }

ie., use Europe/Oslo if the server recognises the time zone, otherwise
fall back to +0100.

>          timezone specified in the header is default for :received, :sent

I think this is nice.  but for a user simply wanting to save his mail
in different folders depending on month would write

  setdate;
  fileinto "INBOX.${year}-${month}";

which now becomes

  if date :current :year :matches "*" {
     set "year" "${1}";
  }
  if date :current :month :matches "*" {
     fileinfo "INBOX.${year}-${1}";
  }

>   Together with the relational extension and a rfc822-date
>   comparator,

please make it an ISO 8601 date comparator ...

:-)

the existence of a global "setdate" script is not guaranteed.  isn't
it better to make the support for this certain?  I think your date
test and SETDATE can co-exist.

>   In this scenario, ${#currentdate} would be a system variable with
>   all properties as defined above: It affects the date :current
>   test, is read-only, it's natural representation is not a string,
>   it's value can't be computed with Sieve and it can't be extracted
>   from the message.

but is this a good thing, or do we want to make generic mechanisms
instead?

-- 
Kjetil T.			|  read and make up your own mind
				|  http://www.cactus48.com/truth.html

---

From: Marc Mutz <[email-masked]>
Date: Thu, 8 May 2003 21:46:40 +0200

On Wednesday 07 May 2003 22:59, Kjetil Torgrim Homme wrote:
<snip>
> I agree [system variables] need to be recognisable by syntax alone.
> one type of prefix could be "date." or "imap.", ie. the distinguishing
> feature is the presence of one or more periods.
<snip>

That was my first reaction, too, but I thought that would be too verbose 
and I remembered there was opposition to "named namespaces", so I chose 
a single character prefix.

> > there was objection
> > to define extensions that have any kind of "internal variable" (as
> > imapflags once called it) without first having variables for Sieve
> > in general.
>
> I don't understand this reasoning.  you want "internal variables"
> with properties very different from those of the general variables. 
> where is the connection between the features?
<snip>

I want a single syntactical element for a common concept instead of 
every extension brewing it's own thing. (Notify: $subject$ etc 
substitution in the :message text; Imapflags: :(global)flags; 
Variables: setdate).

The connection is the general variables syntax: ${}.

> now ${bar} is [ "one", "two", "three", "four" ] and ${zot} is [
> "woo", "one", "two", "three", "four", "hoo" ].  this makes
> "${foo}${bar}" equivalent with [ "${foo}", "${bar}" ].
<snip failure discussion>
> string with a single space as delimiter or something.

This might work - for imapflags. What about ${#currentdate}, ${#size} 
(see below). And what about initial content?
<sidenote>
I don't understand why imapflags requires it's internal set to be empty 
on startup. Wouldn't it be more useful to allow implementations to 
preload the set with the current flags? E.g. when applying scripts to a 
selection of messages from an IMAP folder or in sequential script 
("multiscript" environment) execution?
</sidenote>

>   if string :matches "${flags}" "* \\Flagged *" {

Ugh. Too subtle. It's a open invitation for bugs in scripts.

> why do you need variables at all for IMAP flags?  to keep state, it

Yes. And I want to use a system variable for that because that's the 
_concept_ behind it. A new imapflags draft revision can then say:
"fileinto uses the current content of the ${#imapflags} system variable 

<snip>

Won't confuse anyone anymore, since the set of flags is scoped like any 
other variable (ie. globally - currently). If you want to restore the 
imapflags from before the first if, save them away in a user variable 
and restore them after the fact:

set "myimapflags" "${#imapflags}";
if .... {
  ....
}
setflags "${myimapflags}";

> ie., use Europe/Oslo if the server recognises the time zone,
> otherwise fall back to +0100.

My feeling is that trying to use an unknown timezone should flag an 
error.

I used :numericzone to work around the tagged argument name clash. Maybe  
instead of the :timezone parameter there should be a ${#timezone} 
variable that can be altered by a settimezone action?

settimezone "Europe/Berlin";
if not string "${#timezone}" "Europe/Berlin" {
  settimezone "+0100";
}

That introduces the question of which date tests should honour the 
timezone? Maybe all, since you can always say:

set "mytimezone" "${#timezone}";
settimezone ""; # use local (:current) or implied (else) timezone
if date :comparator "i;ascii-numeric" :sent :hour :value "lt" "6" {
  set "prefix" "Oh, my. Nightshift again? ";
}
vacation
    "${prefix}I'm on vacation (Hawaii :-P) until the end of the month.";
settimezone "${mytimezone}";

> >          timezone specified in the header is default for :received,
> > :sent and :header.
>
<snip>

But rfc822 messages have rfc822 dates? Or is it the engine's job to 
convert the string representation of the dates into ISO 8601 strings? I 
think not, since it's also not the engine's job to convert strings to 
numbers before handing them off to i;ascii-numeric.

> it better to make the support for this certain?

It would be cute to have a "sieve stdlib". However, just providing this 
script on the homepage or as an appendix to the variables rfc would 
probably suffice. Those that want it can either include it or 
cut'n'paste it into their scripts.

> I think your date 
<snip>

I don't like setdate polluting the "user namespace" with unused 
variables. How many people will need ${second} compared to ${year} and 
${month}? If setdate remains, I'd like to see it write to ${#year} etc, 
but I'd really prefer to have it's functionality go into a date test, 
else why do we have a setdate and no setenvelope, setaddress, 
setheader, setsize[1]? With a date test, all but the last are equally 
superfluous. Should every foo test have a corresponding setfoo action 
that optimizes away the otherwise necessary matches-set idioms?

[1] This might even be useful, since the relational extension failed to 
redefine the size test to take a [MATCH-TYPE] parameter, so you 
_cannot_ say
  if size :matches "*" {
    set "size" "${1}";
  }
So ${#size} perfectly fits the definition of a system variable :-):

It affects the size test, ...
> >   is read-only, it's natural representation is not a string,
> but is this a good thing,

It's simply needed. Size is just another example[2]. Whether it's good 
or not that there are such variables, they're (implicitly) there even 
in the Sieve base spec.

[2] And one which the variables spec needs to define, so it addresses 
Ned's concerns about defining a concept and not instantly making use of 
it.

> or do we want to make generic mechanisms 
> instead?

What do you mean with "generic mechanism"? Isn't "system variable" such 
a generic mechanism?

Marc

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Sat, 10 May 2003 04:14:45 +0200

[Marc Mutz]:
>
>   I want a single syntactical element for a common concept instead
>   of every extension brewing it's own thing. (Notify: $subject$ etc

I see.

>   > e.g., @{folder} becomes a list, while ${folder} becomes a single
>   This might work - for imapflags. What about ${#currentdate},
>   ${#size} (see below).

these aren't suitable for a list type, are they?  list variables are
useful in other contexts as well, like

   set "hdrs" ["To", "Cc"];
   set "domains" ["njus.no", "usenet.no", "uio.no"];
   if allof (address :localpart ["@{hdrs}"] "kjetilho",
             address :domain ["@{hdrs}"] ["@{domains}"]) { ... }
   if allof (address :localpart ["@{hdrs}"] "postmaster",
             address :domain ["@{hdrs}"] ["@{domains}"]) { ... }

("@" may not be the best character, but I'll just leave it for now as
an example.)

>   And what about initial content?

an empty list seems best.  interestingly, since this is an error:
   set "foo" [ ];
you'd need to
   set "foo" [ "@{undef}" ];
to reset foo to the empty list.

>   ("multiscript" environment) execution?

your view sounds reasonable to me.

>   Ugh. Too subtle. It's a open invitation for bugs in scripts.

possibly.  if this code is given in the documentation, that would
reduce the error rate quite a bit.  btw, it's wrong to blindly add a
flag, you need to make sure there are no duplicates (or else removing
flags becomes very difficult).  shame on me ...

>   >   # now we want to restore the flags to the values before the if.

you snipped my "sorry, I can't come up with a credible example", which
is what "or do we" is about.  why would we ever need to restore the
flags?

>   My feeling is that trying to use an unknown timezone should flag
>   an error.

perhaps that is best.  in the usual case, the argument to SETDATE is a
static string, and the error can be flagged at upload, which is nice.
I'll add that as a SHOULD in my draft.

>   But rfc822 messages have rfc822 dates? Or is it the engine's job
>   to convert the string representation of the dates into ISO 8601
>   strings?

it is the engine's job to normalise the string representation into one
common format.  but forget it, an ISO 8601 date comparator is not
needed, since a normal string compare is enough.

>   > I think your date test and SETDATE can co-exist.
>

that's the big question, isn't it?  the matches-set idiom seems
clunky, a hack.  I don't want to base the design of the language on
it.

>   [2] And one which the variables spec needs to define, so it
>   addresses Ned's concerns about defining a concept and not
>   instantly making use of it.

which concept?

>   What do you mean with "generic mechanism"? Isn't "system variable"
>   such a generic mechanism?

every system variable will behave differently.  I don't think that's
"generic".

---

From: Marc Mutz <[email-masked]>
Date: Sat, 10 May 2003 21:46:25 +0200

On Saturday 10 May 2003 04:14, Kjetil Torgrim Homme wrote:
<snip>
> these aren't suitable for a list type, are they?
<snip>

That's not the point. The point is that list variables solve this 
particular problem (if at all, see the duplicates problem). I've seen 
no other example of their usefulness outside the "nice to have" 
department.

System variables solve all "internal variable" problems, not only 
imapflags. And with a common concept.

<snip>

If at all, I'd prefer an unset action for that.

> reduce the error rate quite a bit. 
<snip>

Unlike the matches-set idiom, this is not generally applicable and I 
still maintain the view that it's too fragile.

> >   >   # now we want to restore the flags to the values before the
> >   > if. # or do we?
> >
> >   <snip>
>
> you snipped my "sorry, I can't come up with a credible example",
> which is what "or do we" is about.  why would we ever need to restore
> the flags?
<snip>

To get rid of the numerous fileinto tagged arguments that imapflags 
defines. There would be only :globalflags semantics left. "local" flags 
can be had with

set "saveflags" "${#imapflags}";
setflags "\\Seen";
fileinto "low-priority";
setflags "${saveflags}";

This is more flexible than the multitude of :flags +-= :globalflags +-= 
tags, which are unified into the common concept of manipulating 
variables.

> an ISO 8601 date comparator is not

Please explain why "a normal string compare" is enough to determine 
which of the two dates
  Sat, 10 May 2003 21:18:44 +0200
  Sat, 10 May 2003 20:18:44 +0000
is earlier.

> > Should every foo test
> > have a corresponding setfoo action that optimizes away the
> > otherwise necessary matches-set idioms?
>
<snip>

The problem is again the already defined tests. It's one thing to let 
the variables extension add system variables for basic Sieve or the 
extensions already implemented/out. It's another thing to let it add a 
load of _actions_, whose availability is determined by a combination of 
extensions.

> >   instantly making use of it.

That of system variables.

> >   What do you mean with "generic mechanism"? Isn't "system
> > variable" such a generic mechanism?
>

Define "behavior". Different system variables behave as differently as 
different actions do, as different tests do. Still, you wouldn't say 
that "action" or "test" is not a generic concept, or would you?

Marc

-- 
Eternal vigilance is the price of liberty   -- Thomas Jefferson

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Sun, 11 May 2003 04:14:18 +0200

I'll try to summarise in a new thread, so just a simple answer here.

[Marc Mutz]:
>
>   > an ISO 8601 date comparator is not needed, since a normal string
>   > compare is enough.
>

those aren't on ISO 8601 format.  still, I forgot about time zones,
you'd need to normalise those before doing the string compare.

-- 
Kjetil T.

---

From: Simon Josefsson <[email-masked]>
Date: Sun, 11 May 2003 11:45:20 +0200

> you'd need to normalise those before doing the string compare.

So a ISO 8601 date comparator is needed?

If I recall correctly, ISO 8601 has some quirks that may argue for a
ISO 8601 date comparator in Sieve.  E.g., 00:00 == 24:00; negative
time fields -10:30, time zones 00:00 == 24:00 (but different days?),
double negative time fields --30, fractional fields.  RFC 3339
disallow some of these though.