Subject: Sieve regex extension

From: Ken Murchison <[email-masked]>
Date: Wed, 09 Feb 2000 15:17:13 -0500

I have implemented a regular expression extension to CMU Sieve, which
I'm told will be included in the upcoming version of Cyrus Imapd.  In
order to promote some discussion on a possible specification for this
extension, I have written a *very* rough draft as a starting point.

An issue which I encountered while implementing the code is that of
escaped meta-characters.  Should they be single-escaped (ie, "\+") or
double-escaped (ie, "\\+")?  Section 2.4.2 of [SIEVE] is sort of vague
on this topic, only saying that "Other escape sequences may be permitted
depending on context."

CMU Sieve currently strips all single backslashes in quoted strings,
which seems to be one reasonable implementation based on the working in
the spec.  This behavior requires that any escaped meta-character in a
regex must be preceded by two backslashes.  This behavior also effects
the :matches match-type in the base spec and imapflags as defined in
draft-melnikov-sieve-imapflags-02.

I am not up to speed on the discussions that led to the current wording
in the spec, but I'd like to see all escape sequences passed through
untouched, or perhaps be context-sensitive as hinted at in the spec. 
But, "this is just my opinion, and I could be wrong."

Regards,
Ken
-- 
Kenneth Murchison     Oceana Matrix Ltd.
Software Engineer     21 Princeton Place
716-662-8973 x26      Orchard Park, NY 14127
--PGP Public Key--    http://www.oceana.com/~ken/ksm.pgp

Network Working Group                                      Ken Murchison
Document: draft-murchison-sieve-regex-00.txt          Oceana Matrix Ltd.
Expires August 14, 2000                                  9 February 2000

                   Sieve -- Regular Expression Extension

Status of this Memo

    This document is an Internet-Draft and is in full conformance with
    all provisions of Section 10 of RFC2026.  Internet-Drafts are
    working documents of the Internet Engineering Task Force (IETF), its
    areas, and its working groups.  Note that other groups may also
    distribute working documents as Internet-Drafts.

    Internet-Drafts are draft documents valid for a maximum of six
    months and may be updated, replaced, or obsoleted by other documents
    at any time.  It is inappropriate to use Internet- Drafts as
    reference material or to cite them other than as "work in progress."

    To view the list Internet-Draft Shadow Directories, see
    http://www.ietf.org/shadow.html.

    Distribution of this memo is unlimited.

Copyright Notice

    Copyright (C) The Internet Society 2000. All Rights Reserved.

Abstract

    In some cases, it is desireable to have a string matching mechanism
    which is more powerful than a simple exact match, a substring match
    or a glob-style wildcard match.  The regular expression matching
    mechanism defined in this draft should allow users to isolate just
    about any string or address in a message header or envelope.

Expires August 14, 2000       Murchison                         [Page 1]

Internet Draft          Sieve -- Regex Extension        February 9, 2000

                           Tablee of Contents

Status of this Memo  . . . . . . . . . . . . . . . . . . . . . . . .   1

Copyright Notice . . . . . . . . . . . . . . . . . . . . . . . . . .   1

Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   1

1.     Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3

2.     Capability Identifier . . . . . . . . . . . . . . . . . . . .   3

3.     Regex Match Type  . . . . . . . . . . . . . . . . . . . . . .   3

4.     Security Considerations . . . . . . . . . . . . . . . . . . .   3

5.     Acknowledgments . . . . . . . . . . . . . . . . . . . . . . .   3

6.     Author's Address  . . . . . . . . . . . . . . . . . . . . . .   3

Appendix A.  References  . . . . . . . . . . . . . . . . . . . . . .   4

Appendix B.  Full Copyright Statement  . . . . . . . . . . . . . . .   4

Expires August 14, 2000       Murchison                         [Page 2]

Internet Draft          Sieve -- Regex Extension        February 9, 2000

1.  Introduction

    This is an extension to the Sieve language defined by [SIEVE] for
    comparing strings to regular expressions.

    Conventions for notations are as in [SIEVE] section 1.1, including
    use of [KEYWORDS].

2.  Capability Identifier

    The capability string associated with the extension defined in this
    document is "regex".

3.  Regex Match Type

    Commands that support matching may take the optional tagged argument
    ":regex" to specify that a regular expression match should be
    performed.  The ":regex" match type is subject to the same rules and
    restrictions as the standard match types defined in [SIEVE].

    The ":regex" match type is compatible with both the "i;octet" and
    "i;ascii-casemap" comparators and may be used with them.

    Implementations MUST support extended ("modern") regular expressions
    as defined by [POSIX.2]  Any regular expression not defined by
    [POSIX.2], including [POSIX.2] basic ("obsolete") regular
    expressions are not supported by this extension.

4.  Security Considerations

    Security considerations are discussed in [SIEVE].  It is believed
    that this extension doesn't introduce any additional security
    concerns.

5.  Acknowledgments

    Thanks to ??? for help with this document.

6.  Author's Address

    Ken Murchison
    Oceana Matrix Ltd.
    21 Princeton Place
    Orchard Park, NY 14127

Expires August 14, 2000       Murchison                         [Page 3]

Internet Draft          Sieve -- Regex Extension        February 9, 2000

    Phone: (716) 662-8973
    EMail: [email-masked]

Appendix A.  References

     [KEYWORDS] Bradner, S., "Key words for use in RFCs to Indicate
         Requirement Levels", Harvard University, RFC 2119, March, 1997.

     [SIEVE] Showalter, T., "Sieve: A Mail Filtering Language", Mira-
         point, Inc., Work In Progress.

     [POSIX.2], "Portable Operating System Interface (POSIX). Part 2,
         Shell and utilities", National Institute of Standards and Tech-
         nology (U.S.).

Appendix B.  Full Copyright Statement

    Copyright (C) The Internet Society 2000. All Rights Reserved.

    This document and translations of it may be copied and furnished to
    others, and derivative works that comment on or otherwise explain it
    or assist in its implementation may be prepared, copied, published
    and distributed, in whole or in part, without restriction of any
    kind, provided that the above copyright notice and this paragraph
    are included on all such copies and derivative works.  However, this
    document itself may not be modified in any way, such as by removing
    the copyright notice or references to the Internet Society or other
    Internet organizations, except as needed for the purpose of develop-
    ing Internet standards in which case the procedures for copyrights
    defined in the Internet Standards process must be followed, or as
    required to translate it into languages other than English.

    The limited permissions granted above are perpetual and will not be
    revoked by the Internet Society or its successors or assigns.

    This document and the information contained herein is provided on an
    "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
    TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
    BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
    HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
    MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Expires August 14, 2000       Murchison                         [Page 4]

---

From: Randall Gellens <[email-masked]>
Date: Sun, 27 Feb 2000 16:12:19 -0800

>But, "this is just my opinion, and I could be wrong."

I agree with you.  Two backslashes should mean one literal backslash. 
"\." in a regexp should mean a literal dot.  "\\." should mean a 
literal backslash followed by any character.  Otherwise it's too 
confusing.
-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
Oh, dear, where can the matter be
When it's converted to energy?
There is a slight loss of parity.
Johnny's so long at the fair.

---

From: Tim Showalter <[email-masked]>
Date: 29 Feb 2000 19:13:15 -0800

> literal backslash followed by any character.  Otherwise it's too 

I believe that this is impossible to correctly parse without passing
information between the parser and the lexer in a very unsavory way.
In particular, if you're trying to decide what a backslash means at
the lexer level, you can't, and you have to wait until the string is
used in context.

I'm not happy that the spec says what it does.  It was a mistake.

For consistancy, I believe only \ and \" should be significant in a
string; to escape a metacharacter, the syntax should be "\\?" (i.e.,
match a literal question mark).  The sequence "\\\?" is not
syntactically legal.

Programmers can grasp this (it's what C, Perl, and some Lisp
implementations do) and no one else will understand it, no matter what
we decide.

I would like to change the spec to reflect the behavior implemented in
CMU's Sieve implementation.

My apologies for not getting back to this sooner.

Tim

To: Randall Gellens <[email-masked]>
Cc: Ken Murchison <[email-masked]>, [email-masked]
Subject: Re: Sieve regex extension
From: Tim Showalter <[email-masked]>
Date: 29 Feb 2000 19:13:15 -0800
Lines: 44
Sender: [email-masked]

> literal backslash followed by any character.  Otherwise it's too 

I believe that this is impossible to correctly parse without passing
information between the parser and the lexer in a very unsavory way.
In particular, if you're trying to decide what a backslash means at
the lexer level, you can't, and you have to wait until the string is
used in context.

I'm not happy that the spec says what it does.  It was a mistake.

For consistancy, I believe only \ and \" should be significant in a
string; to escape a metacharacter, the syntax should be "\\?" (i.e.,
match a literal question mark).  The sequence "\\\?" is not
syntactically legal.

Programmers can grasp this (it's what C, Perl, and some Lisp
implementations do) and no one else will understand it, no matter what
we decide.

I would like to change the spec to reflect the behavior implemented in
CMU's Sieve implementation.

My apologies for not getting back to this sooner.

Tim

Date: Sun, 27 Feb 2000 16:12:19 -0800
To: Ken Murchison <[email-masked]>, [email-masked]
From: Randall Gellens <[email-masked]>
Subject: Re: Sieve regex extension
Sender: [email-masked]

-------------- Randomly-selected tag: ---------------

Date: Sun, 27 Feb 2000 14:08:10 -0800
To: [email-masked]
From: Randall Gellens <[email-masked]>
Subject: Re: Sieve ACAP Profile Draft
Cc: [email-masked]
Sender: [email-masked]

Thanks to comments from Alexey, I yet again updated the draft at 
<ftp://ftp.pensive.org/Public/Randy/draft-gellens-acap-sieve-00.txt>.

(Sorry that it has the same name).

This time I fixed some more syntax errors, clarified the capabilities 
attributes and the extension capability attribute, added the 
extension capability attribute to the example session, and clarified 
the change in the initial greeting.
-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
The typewriting machine, when played with expression, is no 
more annoying than the piano when played by a sister or near 
relation.                                      --Oscar Wilde

Date: Sat, 26 Feb 2000 19:36:16 -0800
To: [email-masked]
From: Randall Gellens <[email-masked]>
Subject: Re: Sieve ACAP Profile Draft
Cc: [email-masked]
Sender: [email-masked]

I just updated the draft at 
<ftp://ftp.pensive.org/Public/Randy/draft-gellens-acap-sieve-00.txt>.

(Sorry that it has the same name).

I fixed the ACAP errors, replaced the "~" with a discussion of using 
"sieve" for email servers and actual entries for ACAP servers, fixed 
references to "account", cleaned up section 5, mentioned the 
"email.sieve.capability" attribute (for Sieve extensions), discussed 
the "email.sieve.*" attributes in the "email" entry of the 
"capability" dataset, and added an Acknowledgments section to thank 
Larry for reviewing it and finding the errors.

-------------- Randomly-selected tag: ---------------
Fine day to throw a party.  Throw him as far as you can.

Date: Fri, 25 Feb 2000 20:21:52 -0800
To: [email-masked]
From: Randall Gellens <[email-masked]>
Subject: Sieve ACAP Profile Draft
Cc: [email-masked]
Sender: [email-masked]

I just sent in my ACAP Profile for Sieve draft.  It can also be 
obtained at 
<ftp://ftp.pensive.org/Public/Randy/draft-gellens-acap-sieve-00.txt>.

-------------- Randomly-selected tag: ---------------
Give thought to your reputation.  Consider changing name and moving
to a new town.

Date: Thu, 17 Feb 2000 23:08:19 -0500
From: Matthew Wall <[email-masked]>
To: [email-masked]
cc: Randall Gellens <[email-masked]>, Edward Hibbert <[email-masked]>, "'Lawrence Greenfield'" <[email-masked]>, Barry Leiba <[email-masked]>
Subject: RE: Comments about draft-martin-managesieve-00.txt
Sender: [email-masked]

-- On Tuesday, February 15, 2000 9:36 AM -0800 the entity known as Randall 
Gellens <[email-masked]> wrote:

<begin quote>

> At 9:37 AM +0000 2/15/00, Edward Hibbert wrote:
>
>> ACAP looks as though it's re-inventing another directory service
>> and access protocol
>
> This perception is not uncommon, but the intent of ACAP is not to be a
> directory access protocol at all, but rather a very easy way for clients
> to access data on a central server that is normally kept on the local
> hard drive.  Client configuration and preference data, for the most part.

<end quote>

To which Matthew Wall offers this response on Thu, 17 Feb 2000 23:02:07 
-0500:

I would add somewhat parenthetically is that some kinds of data -- things 
like user email addresses, for example -- tends to be identical but be 
treated in different ways depending on the context of management and user 
access. Directory data, for instance, is a bit different from the way you 
want your personal addressbooks to be handled: the example I usually give 
is you probably want your Aunt Millie's address in your personal 
addressbook and maybe shareable with your wife, but not in the corporate 
directory.

Sieve filter strings I expect will have similar provenance -- on the one 
hand, there will be times when they may best be stored as part of a 
directory, others where they're appropriate to client preference data.

Speaking from the client side of things, from a client that may wish to use 
sieve strings in a disconnected mode (e.g. with no LDAP server available), 
but where some communication with a remote server is nice for resynching 
from time to time, the ACAP model works a lot better. The email client 
program treats a user-based and defined filter as yet another preference, 
in effect. LDAP can be miserable if you try to overload it with too much of 
this stuff.

I do agree that ACAP would be the best management route in the best of all 
possible worlds, but don't really see a problem using LDAP or whatever 
works in a given environment if the deployment warrants it...we've even 
experimented with IMSP 8-).

- mw

PS Larry's comments are well-taken, however, about implementation humps.

Date: Wed, 16 Feb 2000 10:26:17 -0700
From: Alexey Melnikov <[email-masked]>
To: Randall Gellens <[email-masked]>
CC: Lawrence Greenfield <[email-masked]>, [email-masked]
Subject: Re: Comments about draft-martin-managesieve-00.txt
Sender: [email-masked]

Sorry for the lack of response from my part. Just came back from UK.

Randall Gellens wrote:
> 
> At 09:17 PM 2/14/00 -0500, Lawrence Greenfield wrote:
> >However, defining an "ACAP profile" to
> >handle Sieve scripts isn't as easy as it sounds.
> 
> There are always going to be features that are desired in any Sieve access
> protocol.  The goal behind using a Sieve profile is to make it as easy as
> possible to deploy a protocol that meets the absolute minimum, and is
> upwardly extensible (by going to full ACAP).
> 
> >I want my IMAP server to be able to
> >   open contexts on my ACAP server that stores the scripts.  This way I
> >   don't have to do an ACAP search for every message delivery but I
> >   won't use out-of-date Sieve scripts.
> 
> You could also have an active client that opens contexts on the full ACAP
> server in which the Sieve scripts are stored, and when any of them change,
> pushes the change to the Sieve server.  But that's not my point.  My point
> is that people are going to want a lot in a Sieve access protocol, and
> developers want something that is very easy to implement and deploy.  I
> think a minimal ACAP subset meets the second goal, and provides a way to
> reach the first goal.

I fully agree with Randy (In fact we had a private discussion with him
about the subject).

Currently ACAP profile for Sieve (ACAP Email Account Dataset Class) is
missing some of the functionality described in Tim's document.
Do people think it is worthwhile to extend dataset description?

Alexey

Date: Tue, 15 Feb 2000 14:27:44 -0800
To: Lawrence Greenfield <[email-masked]>, [email-masked]
From: Randall Gellens <[email-masked]>
Subject: Re: Comments about draft-martin-managesieve-00.txt
Cc: Randall Gellens <[email-masked]>, Barry Leiba <[email-masked]>
Sender: [email-masked]

At 09:17 PM 2/14/00 -0500, Lawrence Greenfield wrote:
>However, defining an "ACAP profile" to
>handle Sieve scripts isn't as easy as it sounds.

There are always going to be features that are desired in any Sieve access 
protocol.  The goal behind using a Sieve profile is to make it as easy as 
possible to deploy a protocol that meets the absolute minimum, and is 
upwardly extensible (by going to full ACAP).

>I want my IMAP server to be able to
>   open contexts on my ACAP server that stores the scripts.  This way I
>   don't have to do an ACAP search for every message delivery but I
>   won't use out-of-date Sieve scripts.

You could also have an active client that opens contexts on the full ACAP 
server in which the Sieve scripts are stored, and when any of them change, 
pushes the change to the Sieve server.  But that's not my point.  My point 
is that people are going to want a lot in a Sieve access protocol, and 
developers want something that is very easy to implement and deploy.  I 
think a minimal ACAP subset meets the second goal, and provides a way to 
reach the first goal. 

Date: Tue, 15 Feb 2000 09:36:45 -0800
To: Edward Hibbert <[email-masked]>, "'Lawrence Greenfield'" <[email-masked]>, [email-masked]
From: Randall Gellens <[email-masked]>
Subject: RE: Comments about draft-martin-managesieve-00.txt
Cc: Randall Gellens <[email-masked]>, Barry Leiba	 <[email-masked]>
Sender: [email-masked]

At 9:37 AM +0000 2/15/00, Edward Hibbert wrote:

>ACAP looks as though it's re-inventing another directory service
>and access protocol

This perception is not uncommon, but the intent of ACAP is not to be 
a directory access protocol at all, but rather a very easy way for 
clients to access data on a central server that is normally kept on 
the local hard drive.  Client configuration and preference data, for 
the most part.

-------------- Randomly-selected tag: ---------------
How does a project get to be a year behind schedule?
One day at a time.   --Fred Brooks in _The Mythical Man Month_

From: Edward Hibbert <[email-masked]>
To: "'Lawrence Greenfield'" <[email-masked]>, [email-masked]
Cc: Randall Gellens <[email-masked]>, Barry Leiba <[email-masked]>
Subject: RE: Comments about draft-martin-managesieve-00.txt
Date: Tue, 15 Feb 2000 09:37:06 -0000
Sender: [email-masked]

For the sorts of reasons you raise we are going to use our existing X.500
directory service, via LDAP.  To us (but I'm happy to accept not to
everyone), ACAP looks as though it's re-inventing another directory service
and access protocol, and we already have quite enough of those.  We have no
motivation to add ACAP support to our existing directory service.

Edward Hibbert.
DCL.

-----Original Message-----
From: Lawrence Greenfield [mailto:[email-masked]]
Sent: 15 February 2000 02:17
To: [email-masked]
Cc: Randall Gellens; Barry Leiba
Subject: Re: Comments about draft-martin-managesieve-00.txt

Ok, we're all interesting in getting ACAP servers out there.
I'd love to just use ACAP.  However, defining an "ACAP profile" to
handle Sieve scripts isn't as easy as it sounds.

For instance:
  I want to be able to share my Sieve script with my coworkers, but
  not my boss (I filter all of his mail).

For instance:
  Since we don't want the ACAP server and IMAP server necessarily
  sharing the same filesystem, I want my IMAP server to be able to
  open contexts on my ACAP server that stores the scripts.  This way I
  don't have to do an ACAP search for every message delivery but I
  won't use out-of-date Sieve scripts.

Ok, great, now we need ACLs and contexts.  I'm sure I can justify just
about any other ACAP feature you want me to.

We (CMU) have a working ACAP server.  I'm willing to start putting
Sieve scripts on it next week.  I'd be extremely happy to say "we're
going to use ACAP" and see people implement ACAP servers to store
Sieve scripts.

I'm not sure this is going to encourage deployment of Sieve.

Or, rather: is anybody going to develop an ACAP server to deploy Sieve
in their products?

Larry

   Date: Mon, 14 Feb 2000 16:53:01 -0800
   From: Randall Gellens <[email-masked]>

   At 08:55 AM 2/8/00 -0500, Barry Leiba wrote:
   >I have to say that this whole thing seems a little excessive,
considering
   >the statement that this is an interim solution anyway:

   I agree.  I don't think it's a good idea to go the yet-another-protocol 
   route unless it moves us closer to where we want to be.

   My proposal all along has been to use ACAP or a subset.  We have
prototype 
   implementations of an ACAP profile for Sieve (along with a Sieve engine) 
   running on NT and Unix, and a prototype Windows GUI that manages scripts.

   By using an ACAP profile, both client and server support becomes pretty 
   easy, and the client can talk to a full ACAP server just as well as a
mini, 
   Sieve-only one.  So we get rapid deployment and move closer to a real 
   solution (full ACAP servers).

Date: Mon, 14 Feb 2000 21:17:18 -0500 (EST)
From: Lawrence Greenfield <[email-masked]>
To: [email-masked]
Cc: Randall Gellens <[email-masked]>, Barry Leiba <[email-masked]>
Subject: Re: Comments about draft-martin-managesieve-00.txt
Sender: [email-masked]

Ok, we're all interesting in getting ACAP servers out there.
I'd love to just use ACAP.  However, defining an "ACAP profile" to
handle Sieve scripts isn't as easy as it sounds.

For instance:
  I want to be able to share my Sieve script with my coworkers, but
  not my boss (I filter all of his mail).

For instance:
  Since we don't want the ACAP server and IMAP server necessarily
  sharing the same filesystem, I want my IMAP server to be able to
  open contexts on my ACAP server that stores the scripts.  This way I
  don't have to do an ACAP search for every message delivery but I
  won't use out-of-date Sieve scripts.

Ok, great, now we need ACLs and contexts.  I'm sure I can justify just
about any other ACAP feature you want me to.

We (CMU) have a working ACAP server.  I'm willing to start putting
Sieve scripts on it next week.  I'd be extremely happy to say "we're
going to use ACAP" and see people implement ACAP servers to store
Sieve scripts.

I'm not sure this is going to encourage deployment of Sieve.

Or, rather: is anybody going to develop an ACAP server to deploy Sieve
in their products?

Larry

   Date: Mon, 14 Feb 2000 16:53:01 -0800
   From: Randall Gellens <[email-masked]>

   At 08:55 AM 2/8/00 -0500, Barry Leiba wrote:
   >I have to say that this whole thing seems a little excessive, considering
   >the statement that this is an interim solution anyway:

   I agree.  I don't think it's a good idea to go the yet-another-protocol 
   route unless it moves us closer to where we want to be.

   My proposal all along has been to use ACAP or a subset.  We have prototype 
   implementations of an ACAP profile for Sieve (along with a Sieve engine) 
   running on NT and Unix, and a prototype Windows GUI that manages scripts.

   By using an ACAP profile, both client and server support becomes pretty 
   easy, and the client can talk to a full ACAP server just as well as a mini, 
   Sieve-only one.  So we get rapid deployment and move closer to a real 
   solution (full ACAP servers).

Date: Mon, 14 Feb 2000 16:53:01 -0800
To: Barry Leiba <[email-masked]>, [email-masked]
From: Randall Gellens <[email-masked]>
Subject: Re: Comments about draft-martin-managesieve-00.txt
Sender: [email-masked]

At 08:55 AM 2/8/00 -0500, Barry Leiba wrote:
>I have to say that this whole thing seems a little excessive, considering
>the statement that this is an interim solution anyway:

I agree.  I don't think it's a good idea to go the yet-another-protocol 
route unless it moves us closer to where we want to be.

My proposal all along has been to use ACAP or a subset.  We have prototype 
implementations of an ACAP profile for Sieve (along with a Sieve engine) 
running on NT and Unix, and a prototype Windows GUI that manages scripts.

By using an ACAP profile, both client and server support becomes pretty 
easy, and the client can talk to a full ACAP server just as well as a mini, 
Sieve-only one.  So we get rapid deployment and move closer to a real 
solution (full ACAP servers).

Date: Wed, 09 Feb 2000 15:17:13 -0500
From: Ken Murchison <[email-masked]>
Organization: Oceana Matrix Ltd.
To: [email-masked]
Subject: Sieve regex extension
Sender: [email-masked]

This is a multi-part message in MIME format.
--------------331F211DC2B852C11C573F1B
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

Regards,
Ken
-- 
Kenneth Murchison     Oceana Matrix Ltd.
Software Engineer     21 Princeton Place
716-662-8973 x26      Orchard Park, NY 14127
--PGP Public Key--    http://www.oceana.com/~ken/ksm.pgp
--------------331F211DC2B852C11C573F1B
Content-Type: text/plain; charset=us-ascii;
 name="draft-murchison-sieve-regex-00.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
 filename="draft-murchison-sieve-regex-00.txt"

Status of this Memo

Copyright Notice

Abstract



                           Tablee of Contents



1.  Introduction

2.  Capability Identifier

3.  Regex Match Type

5.  Acknowledgments

6.  Author's Address

    Ken Murchison
    Oceana Matrix Ltd.
    21 Princeton Place
    Orchard Park, NY 14127



    Phone: (716) 662-8973
    EMail: [email-masked]

Appendix A.  References



--------------331F211DC2B852C11C573F1B--

Date: Tue, 8 Feb 2000 18:27:15 -0500 (EST)
From: Timothy Martin <[email-masked]>
To: [email-masked]
Subject: Re: Comments about draft-martin-managesieve-00.txt
Sender: [email-masked]

> Date: Tue, 08 Feb 2000 08:55:04 -0500
> From: Barry Leiba <[email-masked]>
> Organization: Internet Messaging Technology; TR2_Xagent=WATSON
> 
> I have to say that this whole thing seems a little excessive, considering
> the statement that this is an interim solution anyway:
> 
>     This an interim measure as it is hoped that eventually Sieve scripts
>     will be stored on ACAP. This document is intended to proceed on the
>     experimental track.
> 
> What we do is to specify a folder that's used to hold Sieve scripts (we 
> use the name _Mail_Filters_; a standard name might be chosen), and the

I disagree with this solution for the same reason we don't store
addressbooks or preferences in IMAP folders. First of all it adds
complexity to server implementations. Also, mail clients that are not
aware of this addition would not be able to ignore it. A user might
see this _Mail_Filter_ folder, notice that the messages in it look
like garbage and delete them.

A better option might be to allow sieve script submission via
SMTP. The problems with this option is that the smtp server one
submits to often isn't the final delivery agent so it might not have
any knowledge of sieve (and therefor can't give the sieve
capabilities). Also managing multiple scripts this way seems
difficult.

My main concern would be that we'd be adding extensions to IMAP that
have little to do with mail access. I know writing yet another parser
is a pain but in IMO it is a lesser evil than adding yet another
addition to IMAP.

-Tim

Date: Tue, 8 Feb 2000 17:58:59 -0500 (EST)
From: Timothy Martin <[email-masked]>
To: [email-masked]
Cc: 
Subject: Re: Comments about draft-martin-managesieve-00.txt
Sender: [email-masked]

> > > > What do you think about adding response codes for various failures?
> > > 
> > > Please consider these.  Adding them would greatly raises the hopes of
> > > intelligent error recovery in client implementations, something that
> > > seems to be very difficult for IMAP clients.  (Fortunately this is
> > > much simpler than IMAP.)
> > 
> > Would using the ACAP response codes be a good idea?
> 
> They're probably the best thing going.  They had several advantages
> over IMAP with no real additional cost.
> 
> By the way, if you're not already using quoted strings for response
> text, please consider that, too.  It makes tokenization easier.

Our implentation always uses quoted strings but currently that isn't
required. It would seem requiring them for responses would make the
tokenization more difficult if anything.

Date: Tue, 08 Feb 2000 17:41:40 -0500
From: Matthew Wall <[email-masked]>
To: [email-masked]
cc: Barry Leiba <[email-masked]>
Subject: Re: Comments about draft-martin-managesieve-00.txt
Sender: [email-masked]

-- On Tuesday, February 8, 2000 8:55 AM -0500 the entity known as Barry 
Leiba <[email-masked]> wrote:

<begin quote>

>
> So the user can use any IMAP client to create the active script, by just
> saving (APPENDing) the message to the _Mail_Filters_ folder, and then
> turning on the \Flagged flag.
>
> This is not the ideal answer, but it *is* simple, and it *is* an interim
> measure.  Think about it.

<end quote>

To which Matthew Wall offers this response on Tue, 08 Feb 2000 17:35:58 
-0500:

Yah, I'd actually prefer using plain old SMTP for the submission protocol 
as the interim solution (assuming SMTP AUTH for authenticity), with a magic 
mail processor at the other end. Users can mess up these "magic" IMAP 
folders all too easily, other mail clients can stomp on them, etc. And I'm 
loathe to tie Sieve support into IMAP per se, since the 'delivery' concept 
is a bit different in the Cyrus IMAP concept than it is in other mailers. 
(Better, yes, but that's beside the point right now 8-).

Think about it: you can use a modified Listserv to parse the Sieve scripts, 
and good old email to return error messages. Users, at least anybody who's 
signed up for a mailing list, can understand this. If you ship the parts 
around using MIME application/sieve, presumably your well-behaved 
Sieve-cooperating MUAs can do some automated processing of results for 
presentation to the user and possible correction. In a worst case, a 
hand-prepared script can be emailed as text/plain. That's no worse than any 
other scheme that sends filters around in this manner.

As an interim hack, awaiting ACAP, LDAP, etc., I can't think of anything 
that's simpler and less problematic for other services, and requires little 
in the way of modification of existing software for Sieve other than the 
engine executing at the appropriate time.

- Matt

Date: Tue, 08 Feb 2000 08:55:04 -0500
From: Barry Leiba <[email-masked]>
Organization: Internet Messaging Technology; TR2_Xagent=WATSON
To: [email-masked]
Subject: Re: Comments about draft-martin-managesieve-00.txt
Sender: [email-masked]

I have to say that this whole thing seems a little excessive, considering
the statement that this is an interim solution anyway:

    This an interim measure as it is hoped that eventually Sieve scripts
    will be stored on ACAP. This document is intended to proceed on the
    experimental track.

What we do is to specify a folder that's used to hold Sieve scripts (we 
use the name _Mail_Filters_; a standard name might be chosen), and the
contents (that is, what one would get from an IMAP "fetch body[]") of the
most recent message in that folder with the \Flagged flag set is used.
If there's an error in the script, the Sieve processor sends e-mail to 
the user and turns the \Flagged flag off and the \Draft flag on.

So the user can use any IMAP client to create the active script, by just
saving (APPENDing) the message to the _Mail_Filters_ folder, and then 
turning on the \Flagged flag.

This is not the ideal answer, but it *is* simple, and it *is* an interim
measure.  Think about it.

Barry Leiba, Internet Messaging Technology  ([email-masked])
http://www.research.ibm.com/people/l/leiba

Cc: [email-masked]
To: Timothy Martin <[email-masked]>
Subject: Re: Comments about draft-martin-managesieve-00.txt
From: Tim Showalter <[email-masked]>
Date: 07 Feb 2000 15:57:07 -0800
Lines: 39
Sender: [email-masked]

Timothy Martin <[email-masked]> writes:

> > > >    [SASL-ANON]. SASL mechanisms which use plaintext passwords [..]
> > This paragraph seems to be unnecessary.  [...]
> Agreed. I'm not sure why I didn't listen to you the first time :)

I said that before?  At least I'm consistant.

> > This has the same race condition as HAVESPACE except it adds
> > synchronizing literals.
> 
> It doesn't have to. Once the server agrees to the syncronization it 
> can reserve that space because it knows the client will send a
> script. However, this could possibly lead to denial of service attacks 
> and greatly complicates implementations.

HAVESPACE can also reserve the space.  It moves the burden of doing so 
elsewhere, which may not be the best thing to do.

But the command still has to be able to fail.  What makes you think
that write(2) is going to succeed anyway, even if you've got the
space?

> > > What do you think about adding response codes for various failures?
> > 
> > Please consider these.  Adding them would greatly raises the hopes of
> > intelligent error recovery in client implementations, something that
> > seems to be very difficult for IMAP clients.  (Fortunately this is
> > much simpler than IMAP.)
> 
> Would using the ACAP response codes be a good idea?

They're probably the best thing going.  They had several advantages
over IMAP with no real additional cost.

By the way, if you're not already using quoted strings for response
text, please consider that, too.  It makes tokenization easier.

Tim

Date: Mon, 7 Feb 2000 18:10:07 -0500 (EST)
From: Timothy Martin <[email-masked]>
To: [email-masked]
Cc: 
Subject: Re: Comments about draft-martin-managesieve-00.txt
Sender: [email-masked]

> > >    [SASL-ANON]. SASL mechanisms which use plaintext passwords
> > >    (including the PLAIN mechanism [PLAIN]) MUST NOT be used unless a
> > >    security layer is active or backwards compatibility dictates other
> > >    wise.
> 
> This paragraph seems to be unnecessary.  PLAIN already states a
> stronger requirement, and you can't weaken it here by pleading to
> backwards compatibility.  Implementations that implement cleartext
> PLAIN are in violation of the protocol despite the best of intentions.

Agreed. I'm not sure why I didn't listen to you the first time :)

> > I prefer IMAP approach: use synchronizing literals in PUTSCRIPT. If
> > server respond '+ ...', then it will accept the script,
> > otherwise it will return NO.
> 
> This has the same race condition as HAVESPACE except it adds
> synchronizing literals.

It doesn't have to. Once the server agrees to the syncronization it 
can reserve that space because it knows the client will send a
script. However, this could possibly lead to denial of service attacks 
and greatly complicates implementations.

> PUTSCRIPT command has to be able to fail... or we can have a two-phase
> commit.  :-)
> 
> > What do you think about adding response codes for various failures?
> 
> Please consider these.  Adding them would greatly raises the hopes of
> intelligent error recovery in client implementations, something that
> seems to be very difficult for IMAP clients.  (Fortunately this is
> much simpler than IMAP.)

Would using the ACAP response codes be a good idea?

Thanks,
Tim

Date: Mon, 7 Feb 2000 17:58:54 -0500 (EST)
From: Timothy Martin <[email-masked]>
To: [email-masked]
CC: 
Subject: Re: Comments about draft-martin-managesieve-00.txt
Sender: [email-masked]

> >1.6.  Script Names
> >
> >    Sieve script names may contain any valid UTF8 characters, but ***names
> >    must be at least one character long***. ...
>                
> When one reads the draft for the first time it is not clear why you have
> such restriction.
> It is better to explain that empty name has different semantics when
> SETACTIVE is used.

Good point. I'll try to phrase it better.

> >2.1.  AUTHENTICATE Command
> >
> ...
> >
> >    The service name specified by this protocol's profile of SASL is
> >    "imap" since implementations are generally tied to an IMAP
> >    installation.
> 
> I object because the service name is an element used in several SASL
> mechanisms (at least KERBEROS_V4, GSSAPI & DIGEST-MD5)

I agree with this objection. Are there any objections to the use of
the service name "sieve"?

> >    [SASL-ANON]. SASL mechanisms which use plaintext passwords
> >    (including the PLAIN mechanism [PLAIN]) MUST NOT be used unless a
> >    security layer is active or backwards compatibility dictates other
> >    wise.
> 
> I don't see the case of "backward compatibility" for the protocol. The
> situation is different from POP3/IMAP4 where PLAIN is widely deployed.

As Tim points out, I was referring to the large number of sites that
refuse to use strong authentication and insist on storing their
plaintext passwords in questionable places.

> >2.2.  STARTTLS Command
> 
> How client/server should behave if TLS negotiation fails?
> There was some discussions in TLS Mailing List
> (<[email-masked]>) regarding TLS and SMTP.

I have not been keeping up with that list. If there is a way to deal
with TLS negotiation failing will it be a generic way that all
protocols will be able to inherit or will something protocol specific
need to be written?

> >2.5.  HAVESPACE Command
> >
> >    Arguments:
> >         String - name
> >         Number - size
> >
> >    The HAVESPACE command is used to query the server for available
> >    space. Clients specifiy the name wished to save the script as and
> >    it's size in octets.
> 
> Why do you want to pass name to HAVESPACE, i.e. does it make any
> difference whether name is the name of an existing script or  not?

Yes since it is overwriting the script.

Consider the example where you have a 10K limit and one script "foo"
of size 6K on the server. 'HAVESPACE "bar" 7000' would fail but
'HAVESPACE "foo" 7000' would succeed.

> Also, we was discussing HAVESPACE with Lyndon and he pointed out that
> there is a race condition in using HAVESPACE (imagine two clients wants
> to store script for the same user. There is enough space for any of them
> but not for both).
> 
> I prefer IMAP approach: use synchronizing literals in PUTSCRIPT. If
> server respond '+ ...', then it will accept the script,
> otherwise it will return NO.

The grammer is simpler with the HAVESPACE command. PUTSCRIPT still
needs to check quota but a good client should be able to avoid sending 
the script data and then getting a quota violation 99% of the time.

> General:
> 
> What do you think about adding response codes for various failures?
> 

I didn't because I wasn't sure of a good/simple way of adding them?
Any suggestions?

Thanks,
Tim

To: Timothy L Martin <[email-masked]>
Cc: Alexey Melnikov <[email-masked]>, [email-masked]
Subject: Re: Comments about draft-martin-managesieve-00.txt
From: Tim Showalter <[email-masked]>
Date: 07 Feb 2000 12:58:45 -0800
Lines: 49
Sender: [email-masked]

I have a couple comments on Alexey's comments.  I don't think I've
read the draft in detail, so please pardon my ignorance.

Alexey Melnikov <[email-masked]> writes:

> >    The service name specified by this protocol's profile of SASL is
> >    "imap" since implementations are generally tied to an IMAP
> >    installation.
> 
> I object because the service name is an element used in several SASL
> mechanisms (at least KERBEROS_V4, GSSAPI & DIGEST-MD5)

I object as well.  The service name matters.

> >    [SASL-ANON]. SASL mechanisms which use plaintext passwords
> >    (including the PLAIN mechanism [PLAIN]) MUST NOT be used unless a
> >    security layer is active or backwards compatibility dictates other
> >    wise.

This paragraph seems to be unnecessary.  PLAIN already states a
stronger requirement, and you can't weaken it here by pleading to
backwards compatibility.  Implementations that implement cleartext
PLAIN are in violation of the protocol despite the best of intentions.

> I don't see the case of "backward compatibility" for the protocol. The
> situation is different from POP3/IMAP4 where PLAIN is widely deployed.

Backwards compatibility is an issue.  If a system is authenticating
against a Unix passwd database, plaintext is the only thing that
works.

> I prefer IMAP approach: use synchronizing literals in PUTSCRIPT. If
> server respond '+ ...', then it will accept the script,
> otherwise it will return NO.

This has the same race condition as HAVESPACE except it adds
synchronizing literals.

PUTSCRIPT command has to be able to fail... or we can have a two-phase
commit.  :-)

> What do you think about adding response codes for various failures?

Please consider these.  Adding them would greatly raises the hopes of
intelligent error recovery in client implementations, something that
seems to be very difficult for IMAP clients.  (Fortunately this is
much simpler than IMAP.)

Tim

Date: Sat, 05 Feb 2000 01:33:06 -0700
From: Alexey Melnikov <[email-masked]>
To: Timothy L Martin <[email-masked]>, [email-masked]
Subject: Re: Comments about draft-martin-managesieve-00.txt
Sender: [email-masked]

Some additions:

>2.1.  AUTHENTICATE Command
...
>    Server implementations SHOULD support proxying so that an
>    administrator can administer a user's scripts. Proxying is when a
>    user authenticates as himself but logs in as another user.

Authorization is much better term, IMHO.

>3.  Formal Syntax
    SAFE-UTF8-CHAR        = SAFE-CHAR / UTF8-2 / UTF8-3 / UTF8-4 /
                            UTF8-5 / UTF8-6

SAFE-CHAR and others are not defined in the ABNF

Alexey

Date: Sat, 05 Feb 2000 00:56:33 -0700
From: Alexey Melnikov <[email-masked]>
To: Timothy L Martin <[email-masked]>
CC: [email-masked]
Subject: Comments about draft-martin-managesieve-00.txt
Sender: [email-masked]

>1.1.  Changes
>
>    Changes since 00
>
>    -dropped syncronized literals. added HAVESPACE command

The ABNF grammar still allows synchronizing literals. Update grammar or
leave it (see HAVESPACE comment)

>1.6.  Script Names
>
>    Sieve script names may contain any valid UTF8 characters, but ***names
>    must be at least one character long***. ...
               
When one reads the draft for the first time it is not clear why you have
such restriction.
It is better to explain that empty name has different semantics when
SETACTIVE is used.

>2.  Commands
>
>    The following commands are valid. Prior to successful authentication
>    only the AUTHENTICATE, CAPABILITY, STARTTLS, and LOOGOUT commands
                                                      ^^^^^^^
Typo in LOGOUT

>2.1.  AUTHENTICATE Command
>
...
>
>    The service name specified by this protocol's profile of SASL is
>    "imap" since implementations are generally tied to an IMAP
>    installation.

I object because the service name is an element used in several SASL
mechanisms (at least KERBEROS_V4, GSSAPI & DIGEST-MD5)

>    [SASL-ANON]. SASL mechanisms which use plaintext passwords
>    (including the PLAIN mechanism [PLAIN]) MUST NOT be used unless a
>    security layer is active or backwards compatibility dictates other
>    wise.

I don't see the case of "backward compatibility" for the protocol. The
situation is different from POP3/IMAP4 where PLAIN is widely deployed.

>2.2.  STARTTLS Command

How client/server should behave if TLS negotiation fails?
There was some discussions in TLS Mailing List
(<[email-masked]>) regarding TLS and SMTP.

>2.5.  HAVESPACE Command
>
>    Arguments:
>         String - name
>         Number - size
>
>    The HAVESPACE command is used to query the server for available
>    space. Clients specifiy the name wished to save the script as and
>    it's size in octets.

Why do you want to pass name to HAVESPACE, i.e. does it make any
difference whether name is the name of an existing script or  not?

Also, we was discussing HAVESPACE with Lyndon and he pointed out that
there is a race condition in using HAVESPACE (imagine two clients wants
to store script for the same user. There is enough space for any of them
but not for both).

I prefer IMAP approach: use synchronizing literals in PUTSCRIPT. If
server respond '+ ...', then it will accept the script,
otherwise it will return NO.

>2.6.  PUTSCRIPT Command
>
>    Arguments:
>         String - Script name
>         String - Script content
>
>    The PUTSCRIPT command is used by the client to submit a Sieve script
>    to the server.
>
>    If the script already exists upon success the old script will be
>    overwritten. The old script MUST NOT be overwritten if PUTSCRIPT
>    fails in any way.

I suggest adding "A script of the empty length erases script from the
server" (i.e. it will not show up in LISTSCRIPTS anymore)

>3.  Formal Syntax
...
>
>    response-authenticate = *(string CRLF / response-okno / response-special-ok)

I think response-authenticate should look like:

response-authenticate = *(string CRLF) (response-okno /
response-special-ok)

i.e. any number of string CRLF followed by OK or NO

General:

What do you think about adding response codes for various failures?

Alexey