Subject: Re: dynamic naming & creation of mailboxes

From: Kjetil Torgrim Homme <[email-masked]>
Date: 10 Jan 2003 21:19:11 +0100

I got some very good feedback from Jutta Degener, and so quite a bit
has changed (for the better, I hope ;).  comments are very welcome.

(the new draft has been submitted to IETF.)

-- 
Kjetil T.			|  read and make up your own mind
				|  http://www.cactus48.com/truth.html

---

From: "Nigel Swinson" <[email-masked]>
Date: Tue, 14 Jan 2003 00:25:06 -0000

> has changed (for the better, I hope ;).  comments are very welcome.

Hiya,

5.  Action setdate

<snip>

   These variables SHOULD reference the time when execution of the Sieve
   script reaches the statement.  The local timezone SHOULD be used.

The local timezone is going to be the timezone of the server, which may not
be the "local" timezone for the user.  I would suggest that this be a UTC
time, as then there will only be one time conversion to do, rather than
potentially two.  ie I check my mail from the US East coast, the server is
US West coast.  "localtime" US West coast isn't so useful to me.

Also part of the point of Sieve is that your filter will do the same thing
on any server.  So if you import your script into a different mailbox, that
runs on a server in a different timezone, then you will have to modify the
times in the sieve script.

And then what if you run an ISP with mailservers in several timezones.  If
the script uses the local timezone, then this means that you can't move
users to different timezones without modifying their sieve script.

   a) Should we include more predefined variables to access the time?
   (hour, minutes, timezone, weekday, week number (US, EU og ISO?), name
   of month, name of weekday, ...)  Could use strftime(3c) as a list of
   what to offer, but the names should be in English.

Don't really follow why we need ${dollar} at all.  Most of the time $ won't
be followed by { and if we want it to then \${ or $\{ will escape it for us.
Incidentally how should we behave if the "}" is missing?  Presumably
treating it as illegal.  Perhaps there should be an example about this...

hour, minutes, name etc are probably more useful predefined variables than
the dollar, and mean you don't have to call setdate.  In fact given how
cheap it is to do a "set date" action, it might actually be better to just
do the setdate in a lazy fashion, when a reference to a time is requested,
rather than have to bother calling setdate.

   b) This extension is particularily useful if fileinto is extended to
   create new folders on demand.

   Example:
            setdate
            fileinto :makefolder "archive.${year}-${month}";

   It is not clear if this feature should be included here or left for a
   separate extension.

Separate extension.

   c) Should the modifier syntax be prefix or postfix?  ${var.lower} may
   be more natural than ${lower:var}.

Prefix.  More "function call" like

   d) Should references to unknown variables be left in place (rather
   than replaced by the empty string) to ease debugging scripts with
   typos?

Set to null, as perhaps a "set =" is only executed in one of the if blocks,
so it's not an "error" that needs debugged if we didn't go through the if.

Cheers

Nigel

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 14 Jan 2003 04:00:50 +0100

[Nigel Swinson]:
>
>   5.  Action setdate
>   
>   <snip>
>   
>      These variables SHOULD reference the time when execution of the
>      Sieve script reaches the statement.  The local timezone SHOULD
>      be used.
>   
>   The local timezone is going to be the timezone of the server,
>   which may not be the "local" timezone for the user.  I would
>   suggest that this be a UTC time, as then there will only be one
>   time conversion to do, rather than potentially two.  ie I check my
>   mail from the US East coast, the server is US West coast.
>   "localtime" US West coast isn't so useful to me.

heh, yeah.  I guess this is different for people in the US and other
large countries which span several timezones.  however, I think it
will be confusing and annoying for users to have their mail archive
switch to a new folder seven hours before midnight.  perhaps allow the
user to specify timezone explicitly?  but then there's the issue of
daylight savings time.  ugh.  AFAIK, there is no IANA registry for
timezone information.  we could use POSIX syntax, but it's pretty
cryptic, e.g., in Central Europe the correct value is
"MET1METDST2,M3.5.0,M10.5.0".  this is OK if your OS supports it, but
I'd hate to write the code myself!

how about adding an optional argument of the form +HHMM or -HHMM, and
punt on the daylight savings issue.  if the argument is left out, use
localtime.  this way small ISPs, universities etc. can offer the
"natural" behaviour without requiring their users to jump through
hoops.  if you want UTC, use setdate +0000.

>   Also part of the point of Sieve is that your filter will do the
>   same thing on any server.

is that a requirement, or just a Good Thing?

>   And then what if you run an ISP with mailservers in several
>   timezones.  If the script uses the local timezone, then this means
>   that you can't move users to different timezones without modifying
>   their sieve script.

that was the reason for SHOULD.  a multinational ISP would probably
use UTC.

>   Don't really follow why we need ${dollar} at all.  Most of the
>   time $ won't be followed by {

true.  I included it to have a simple way to quote arbitrary text for
inclusion in a script without parsing them, like s,\$,\${dollar},g

>   and if we want it to then \${ or $\{ will escape it for us.

no, it won't.  backslashes are resolved before variables are
substituted.  I didn't want to change that part of the original Sieve
specification.  but, the user can actually

   set dollar "$"
   set text "regarding ${dollar}{dollar}"

if the need arises.  so it is really superfluous.  I'll fix the draft,
thanks!

>   Incidentally how should we behave if the "}" is missing?
>   Presumably treating it as illegal.  Perhaps there should be an
>   example about this...

yes.  the parser should terminate its search for } as soon as an
illegal character has been found [^a-z0-9_] (most likely a space
character), and copy the syntax error(?) verbatim.

   set var "something"
   set text "hi ${junk ${var}}"

text now holds the value "hi ${junk something}".

>   hour, minutes, name etc are probably more useful predefined
>   variables than the dollar, and mean you don't have to call
>   setdate.

my first version of the draft had them as builtins.  I used
case-sensitivity to mark them as reserved names (if the first letter
was uppercase, assignment was illegal).  but case sensitivity is a bad
idea, and I think it is cleaner to have no magic reserved words,
unless the user explicitly asks for them.  it also allows the user to
influence the values, cf. timezone discussion above.

>   In fact given how cheap it is to do a "set date" action, it might
>   actually be better to just do the setdate in a lazy fashion, when
>   a reference to a time is requested, rather than have to bother
>   calling setdate.

it is important not to be too lazy.  you want all the time variables
to be set at the same time, or else you risk mail falling between
cracks.

  if string "${hour}" "00" {
  } else if string "${hour}" "23" {
  }

if the mail arrives at just before midnight, the first test fails.
then the clock ticks, and the second test fails also.  (with setdate,
the user is allowed to shoot himself in the foot.)

>      It is not clear if this feature should be included here or left
>      for a separate extension.
>   
>   Separate extension.

ok.  I agree, it just that it's such a tiny extension :-)

>   Prefix.  More "function call" like

yeah.  to clarify "more natural": I used prefix since that is what I'm
used to from bash programming (which has horrific syntax, btw :), but
thought of postfix when I had to write that the modifiers are applied
right to left (which is the natural way of doing it).  with postfix,
the order of the modifiers matches the syntax.

>   Set to null, as perhaps a "set =" is only executed in one of the
>   if blocks, so it's not an "error" that needs debugged if we didn't
>   go through the if.

>   Cheers

thank you for your feedback!

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 14 Jan 2003 10:32:00 +0100

Kjetil Torgrim Homme writes:
> large countries which span several timezones.

It's a company thing, not a country thing. I knew people working in 
Dublin whose servers were somewhere on the West Coast. They complained 
about slow connections and so on - timezone seemed a minor issue ;)

> however, I think it will be confusing and annoying for users to have 
> their mail archive switch to a new folder seven hours before 
> midnight. perhaps allow the user to specify timezone explicitly?

Simplicity is better.

Most people leave work in the afternoon and arrive in the morning. The 
folder will have switched sometime during the interval. Does the exact 
time really matter?

The IMAP users in Petrapavlovsk and those of us who regularly work near 
midnight might have a problem, but after the first couple of months, we 
learn. The broken threads caused by the change is a much worse problem.

--Arnt

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 14 Jan 2003 15:46:50 +0100

[Arnt Gulbrandsen]:
>   > however, I think it will be confusing and annoying for users to
>   > have their mail archive switch to a new folder seven hours
>   > before midnight. perhaps allow the user to specify timezone
>   > explicitly?
>   
>   Simplicity is better.
>   
>   Most people leave work in the afternoon and arrive in the
>   morning. The folder will have switched sometime during the
>   interval. Does the exact time really matter?

and people in New Zealand?  as a user, I would complain loudly about
the broken mail server which didn't even keep a useful time.  when I
ask for ${hour}, I (still as a naive user) would expect it to return
my local time.  never mind that the server has no way of knowing your
local time...  that's why I think the principle of least surprise is
to use the server's timezone.

anyway, SETDATE is tangential to the proposal, and can be left out.

To: [email-masked]
Subject: Re: dynamic naming & creation of mailboxes
From: Kjetil Torgrim Homme <[email-masked]>
Date: 14 Jan 2003 15:46:50 +0100
Lines: 26
Sender: [email-masked]

[Arnt Gulbrandsen]:
>   > however, I think it will be confusing and annoying for users to
>   > have their mail archive switch to a new folder seven hours
>   > before midnight. perhaps allow the user to specify timezone
>   > explicitly?
>   
>   Simplicity is better.
>   
>   Most people leave work in the afternoon and arrive in the
>   morning. The folder will have switched sometime during the
>   interval. Does the exact time really matter?

and people in New Zealand?  as a user, I would complain loudly about
the broken mail server which didn't even keep a useful time.  when I
ask for ${hour}, I (still as a naive user) would expect it to return
my local time.  never mind that the server has no way of knowing your
local time...  that's why I think the principle of least surprise is
to use the server's timezone.

anyway, SETDATE is tangential to the proposal, and can be left out.

From: Arnt Gulbrandsen <[email-masked]>
To: Kjetil Torgrim Homme <[email-masked]>
Subject: Re: dynamic naming & creation of mailboxes
Cc: [email-masked]
Date: Tue, 14 Jan 2003 10:32:00 +0100
Sender: [email-masked]

Simplicity is better.

--Arnt

To: [email-masked]
Subject: Re: dynamic naming & creation of mailboxes
From: Kjetil Torgrim Homme <[email-masked]>
Date: 14 Jan 2003 04:00:50 +0100
Lines: 146
Sender: [email-masked]

   set dollar "$"

   set var "something"

From: "Nigel Swinson" <[email-masked]>
To: <[email-masked]>, "Kjetil Torgrim Homme" <[email-masked]>
Subject: Re: dynamic naming & creation of mailboxes
Date: Tue, 14 Jan 2003 00:25:06 -0000
Sender: [email-masked]

Hiya,

5.  Action setdate

<snip>

   Example:
            setdate

Separate extension.

Cheers

Nigel

To: <[email-masked]>
Subject: Re: dynamic naming & creation of mailboxes
From: Kjetil Torgrim Homme <[email-masked]>
Date: 10 Jan 2003 21:19:11 +0100
Lines: 11
Sender: [email-masked]

--=-=-=

--=-=-=
Content-Disposition: attachment;
  filename=draft-homme-sieve-variables-01.txt

Network Working Group                                        K. T. Homme
Document: draft-homme-sieve-variables-00.txt          University of Oslo
Expires July 10, 2003                                        10 Jan 2003

                      Sieve -- Variables Extension

Status of this Memo

   This document is an Internet-Draft and is subject to all provisions
   of Section 10 of RFC2026.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as Internet-
   Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/1id-abstracts.txt

   To view the list Internet-Draft Shadow Directories, see
   http://www.ietf.org/shadow.html.

   Distribution of this memo is unlimited.

Abstract

   In advanced filtering rule sets, it is useful to keep state or
   configuration details across rules.  This extension adds an action to
   store data in variables, changes the interpolation of double quoted
   strings, and supplies a new test so that the value of a string can be
   examined.

0     Meta-information on this draft

   This information is intended to facilitate discussion.  It will be
   removed when this document leaves the Internet-Draft stage.

0.1   Discussion

   This draft is intended to be an extension to the Sieve mail filtering
   language, available from the RFC repository as

Homme                                                           [Page 1]

Internet Draft        Sieve -- Variables Extension           10 Jan 2003

   <ftp://ftp.ietf.org/rfc/rfc3028.txt>.

   This draft and the Sieve language itself are being discussed on the
   MTA Filters mailing list at <[email-masked]>.  Subscription
   requests can be sent to <[email-masked]> (send an
   email message with the word "subscribe" in the body).  More
   information on the mailing list along with a WWW archive of back
   messages is available at <http://www.imc.org/ietf-mta-filters/>.

0.2   Noted Changes

   a) We don't change the existing syntax, but modify the semantics of
   strings instead.

   b) Variable names are now case-insensitive.

   c) Previously reserved variable names are now initialised by the user
   through extra actions.

   d) Modifiers for the variable values have been added.  The primary
   motivation for this is fileinto "INBOX.lists.${lower:listname}",
   where listname has been determined dynamically from the headers of
   the e-mail.

0.3   Open Issues

   Example:
            setdate

Homme                                                           [Page 2]

Internet Draft        Sieve -- Variables Extension           10 Jan 2003

1.  Introduction

   This is an extension to the Sieve language defined by [SIEVE].  It
   adds support for storing and referencing data in string variables.

   Conventions for notations are as in [SIEVE] section 1.1, including
   use of [KEYWORDS].

2.  Capability Identifier

   The capability string associated with the extension defined in this
   document is "variables".

3.  Interpretation of strings

   This extension changes the semantics of quoted-string, multi-line-
   literal and multi-line-dotstuff found in [SIEVE] to enable the
   inclusion of the value of variables.  The syntax follows [ABNF].

              variable-ref  =  "${" *variable-modifier variable-name "}"
         variable-modifier  =  1*ALPHA ":"
             variable-name  =  num-variable / user-variable
              num-variable  =  1*DIGIT
             user-variable  =  ALPHA *(ALPHA / DIGIT / "_")

   When the string is evaluated, substrings matching variable-ref shall
   be replaced by the value of variable-name.  Variable names are case
   insensitive.  Unknown variables are replaced by the empty string.  As
   per the grammar, illegal variable names leaves the would-be variable-
   ref verbatim, since it doesn't match the variable-ref syntax.

   Examples:
           "&%${}!"    => unchanged, as the empty string is an illegal
                          identifier.
           "${doh!}"   => unchanged, as "!" is illegal in identifiers.
           "${argggh}" => the empty string, if "argggh" is an unknown
                    variable, or else the value of the variable.

   Strings SHOULD be evaluated every time control reaches the statement
   it is a part of, to make sure the expanded variable values are up-to-
   date.

3.1 Interaction with quoting

   The semantics of quoting using backslash are not changed.  Backslash
   quoting is resolved before doing variable substitution.

Homme                                                           [Page 3]

Internet Draft        Sieve -- Variables Extension           10 Jan 2003

   Examples:
           "${fo\o}"  => ${foo}  => the expansion of variable foo.
           "${fo\\o}" => ${fo\o} => illegal identifier => left verbatim.
           "\${foo}"  => ${foo}  => the expansion of variable foo.
           "\\${foo}" => \${foo} => a backslash character followed by the
                                    expansion of variable foo.

3.2 Reserved variables

3.2.1 Numeric variables

   Unless the extension "regex" is in use, all variables whose names are
   all digits MUST evaluate to the empty string.

   The decimal value of the numeric variable name will index the list of
   matching strings from the each of the group operators in the latest
   regular expression match.  The first pair of grouping parentheses has
   index 1.  If the index is out of range, the empty string will be
   substituted.  Index 0 returns the number of matching groups.

   Example:
           require [ "fileinto", "regex", "variables", "vacation" ];

           if header :regex "List-ID" "<(.*)@" {
             fileinto "lists.${1}"; stop;
           }

           if header :regex "From" "^
             set greeting "Dear ${1},";
           }
           if header :regex "Subject" "(.*)" {
                set subject " on ${1}";
           }

        vacation text:
           ${greeting}

           Thank you for your mail${subject}.
           I'm away from office for a few days, but I will get back to
           you as soon as possible.
           .
           ;

3.2.2 Dollar

   The value of ${dollar} is always "$".  This means the literal value
   of "$" can be escaped by entering it as "${dollar}".

Homme                                                           [Page 4]

Internet Draft        Sieve -- Variables Extension           10 Jan 2003

   Example:
           "make ${dollar}${dollar}${dollar} fast" => "make $$$ fast"

   Note: it is not required to escape "$$$".

3.3  Modifiers

   Modifiers change the expansion of the variable.  More than one can be
   specified, in which case they are applied from right to left.
   Modifier names are case insensitive.  Unknown modifiers MUST yield a
   syntax error.

   Examples:
           set string "juMBlEd lETteRS"
           "${length:string}" => "15"
           "${lower:string}" => "jumbled letters"
           "${upperfirst:string}" => "JuMBlEd lETteRS"
           "${upperfirst:lower:string}" => "Jumbled letters"

3.3.1 Modifier "length:"

   The expansion of the variable is replaced by the length of the
   expansion.

3.3.2 Case modifiers

   These modifiers change the letters of the text from upper to lower
   case or vice versa.  The implementation MUST support US-ASCII, but
   it's not required to handle the entire Unicode repertoire.

   3.3.2.1 Modifier "upper:"

   All lower case letters are converted to their upper case counterpart.

   3.3.2.2 Modifier "lower:"

   All upper case letters are converted to their lower case counterpart.

   3.3.2.3 Modifier "upperfirst:"

   The first character of the expansion is converted to upper case if it
   is lower case.  The rest of the expansion is left unchanged.

   3.3.2.4 Modifier "lowerfirst:"

   The first character of the expansion is converted to lower case if it
   is upper case.  The rest of the expansion is left unchanged.

Homme                                                           [Page 5]

Internet Draft        Sieve -- Variables Extension           10 Jan 2003

4.  Action set

   Syntax:   set <name: user-variable> <value: string>

   The "set" action stores the specified value in the variable.
   Assignment to a reserved variable name or an illegal identifier MUST
   cause a syntax error.  All variables have global scope.  Variable
   names are case insensitive.

   Example:
           set honorific  "Mr";
           set first_name "Wile";
           set last_name  "Coyote";
           set vacation text:
           Dear ${HONORIFIC} ${last_name},
           I'm out, please leave a message after the meep.
           .
           ;

   "set" does not affect the implicit keep.

5.  Action setdate

   the action setdate initialises a few variables:

          ${year}   => the current year, "0000" .. "9999"
          ${month}  => the current month, "01" .. "12"
          ${day}    => the current day, "01" .. "31"
          ${hour}   => the current hour, "00" .. "23"
          ${minute} => the current hour, "00" .. "59"
          ${second} => the current second, "00" .. "59"

6.  Test string

       Syntax:  string [MATCH-TYPE] [COMPARATOR]
                <source: string-list> <key-list: string-list>

   The "string" test evaluates to true if any of the strings matches any
   key.  The type of match defaults to ":is".

7.  Security Considerations

   When combined with the regex extension, strings can contain arbitrary
   values controlled by the sender of the e-mail if the author of the

Homme                                                           [Page 6]

Internet Draft        Sieve -- Variables Extension           10 Jan 2003

   script isn't careful.

   The introduction of variables makes advanced decision making easier
   to write, but since no looping construct is provided, all Sieve
   scripts will terminate orderly.

8.  Acknowledgments

   Thanks to Peder Stray for inspiration, and Jutta Degener for valuable
   feedback.

8.  Author's Address

   Kjetil T. Homme
   Frydens g 5B
   0564 Oslo, Norway

   Phone: +47 9366 0091
   E-mail: [email-masked]

Appendix A.  References

   [ABNF] D. Crocker, Ed., "Augmented BNF for Syntax Specifications:
       ABNF", Internet Mail Consortium, RFC 2234, November 1997

   [KEYWORDS] Bradner, S., "Key words for use in RFCs to Indicate
       Requirement Levels", Harvard University, RFC 2119, March 1997.

   [SIEVE] Showalter, T., "Sieve: A Mail Filtering Language", Mirapoint,
       RFC 3028, January 2001.

Appendix B.  Full Copyright Statement

   Copyright (C) The Internet Society 2002. All Rights Reserved.

   This document and translations of it may be copied and furnished to
   others, and derivative works that comment on or otherwise explain it
   or assist in its implementation may be prepared, copied, published
   and distributed, in whole or in part, without restriction of any
   kind, provided that the above copyright notice and this paragraph are
   included on all such copies and derivative works.  However, this
   document itself may not be modified in any way, such as by removing
   the copyright notice or references to the Internet Society or other
   Internet organizations, except as needed for the purpose of develop-
   ing Internet standards in which case the procedures for copyrights
   defined in the Internet Standards process must be followed, or as
   required to translate it into languages other than English.

Homme                                                           [Page 7]

Internet Draft        Sieve -- Variables Extension           10 Jan 2003

   The limited permissions granted above are perpetual and will not be
   revoked by the Internet Society or its successors or assigns.

   This document and the information contained herein is provided on an
   "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
   TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
   BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
   HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
   MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Homme                                                           [Page 8]

--=-=-=

--=-=-=--