Subject: Status of Sieve vacation extension?

From: Peter Svanberg <[email-masked]>
Date: Tue, 25 Jun 2002 13:01:45 +0200

What is the status of the vacation extension? The Internet draft 
has expired. Should it be replaced by something else, or what is 
the plan?

    Peter Svanberg
    KTH, Stockholm

Date: Tue, 25 Jun 2002 13:01:45 +0200
From: Peter Svanberg <[email-masked]>
To: [email-masked]
Subject: Status of Sieve vacation extension?
Sender: [email-masked]

What is the status of the vacation extension? The Internet draft 
has expired. Should it be replaced by something else, or what is 
the plan?

    Peter Svanberg
    KTH, Stockholm

Subject: Re: "body" extension
From: Tim Showalter <[email-masked]>
To: [email-masked]
Date: 17 Jun 2002 16:53:59 -0600
Sender: [email-masked]

On Mon, 2002-06-17 at 09:16, Nigel Swinson wrote:
> The most common usage seems to be as an attachment blocker.  And they have
> to implement this by using all sorts of fancy regexes which very quickly
> become a fairly sizable performance problem.  Which is what they are
> beginning to moan about just now.  There are also folks trying to do some
> spam filtering and then as a very early stage virus filter before the new
> virus definitions are out.
> 
> To reduce the performance burden we were thinking of implementing :matches
> for the x_body given that it is all they really need for their attachment
> blocker, and then also implementing a mime test that I was hoping we'd all
> design together so that we could get a standard for the syntax.  Glad to see
> this discussion coming up now :o)

:matches can (and will) get fooled about what does and doesn't match--a
message describing the symptoms, or a email of a Sieve filter that
matches the offending messages, will probably trigger the filter.

Although it's a lot easier to implement O(n) :matches than O(n)
:regex...

I'm definitely in favor of trying to provide MIME filtering capabilities
(i.e., mimepart :filename :matches "*.vbs in particular, but also types,
etc.).  Doing this with x_body can cause problems and false positives. 
There are real horror stories to this.

> I really like the idea of having arguments such as :raw or :mime that allow
> implementations to "build up" the extension by just supporting :raw then
> adding :mime and so forth.  Perhaps we should have arguments that describe
> what cte decoding should be done too?

I suspect CTE decoding is all-or-nothing, and it's almost always all. 
Where I can see raw matching being more useful is matching against
headers.

Basically my problem with not decoding CTEs is that if you want to match
against a base64 string, you want the raw form because if the words
aren't quite aligned right, weirdness happens.

We could just document x_body as is.  There's nothing in Sieve that
documents the usual X- convention, but the folks who know what X- means
will get the convenient, unsettling feeling that they're playing with
fire at a gas station.

Tim

Subject: Re: "body" extension
From: Tim Showalter <[email-masked]>
To: [email-masked]
Date: 17 Jun 2002 16:42:25 -0600
Sender: [email-masked]

On Sun, 2002-06-16 at 10:57, Cyrus Daboo wrote:
> What I would like to see are generic 'spamtest' and 'virustest' extensions 
> in sieve that would allow sieve implementations to take better advantage of 
> third-party tools in a transparent manner. For example, some spam tools 
> insert specific headers with the header content indicating the level of 
> spam. Right now you have to code a sieve script to recognise these headers 
> for each type of spam tool that you might use. A better approach, IMHO, is 
> to have a generic 'spamtest' command that returns a number in the range, 
> say, 0 to 10, that can be used to trigger an appropriate action.

This seems like a good idea to me.  Having these as external fields
helps; in some cases it can greatly reduce the amount of I/O necessary
(adding a header could require rewriting a lot of the message).

Tim

From: "Nigel Swinson" <[email-masked]>
To: <[email-masked]>
Subject: Re: "body" extension
Date: Mon, 17 Jun 2002 16:16:59 +0100
Sender: [email-masked]

Hi folks.  Rockliffe's Mailsite has the "x_body" extension that keeps
getting mentioned in this thread.  Here's how it works:

It matches exactly as Jutta's extension describes.  regex is supported,
matches is not.  No mime decoding, cte decoding, charset decoding no
nothing.  Just plain simple character comparison.  If fact, it takes the
utf-8 string from the Sieve script and compares against the body without
even removing the utf-8 encoding.  :o/

So the syntax is:

x_body [COMPARATOR] [MATCH-TYPE] <key-list: string-list>

These comparators are supported: ["i;octet" | "i;ascii-casemap"]
These match types are supported: [:contains | :regex]

Mailsite supports Sieve scripts at the server, domain and mailbox level, but
I think that the x_body extension is most commonly used in the server level
scripts rather than at mailbox/domain level.

So yeah in many ways it's grubbier than a grubby thing.  But still very
useful and heavily used by our customers.  The fact that it is so heavily
used does imply that even at this "basic" level it's worth implementing, and
then leaving the decision of whether or not to take the performance hit up
to the customer.

The most common usage seems to be as an attachment blocker.  And they have
to implement this by using all sorts of fancy regexes which very quickly
become a fairly sizable performance problem.  Which is what they are
beginning to moan about just now.  There are also folks trying to do some
spam filtering and then as a very early stage virus filter before the new
virus definitions are out.

To reduce the performance burden we were thinking of implementing :matches
for the x_body given that it is all they really need for their attachment
blocker, and then also implementing a mime test that I was hoping we'd all
design together so that we could get a standard for the syntax.  Glad to see
this discussion coming up now :o)

We called it "x_body" rather than "body" because we knew that eventually
there would be a body extension that we would want to support, and didn't
want to conflict with it.   I agree with Tim in that the "body" keyword is
too valuable to polute with anything less than the real McCoy.

I really like the idea of having arguments such as :raw or :mime that allow
implementations to "build up" the extension by just supporting :raw then
adding :mime and so forth.  Perhaps we should have arguments that describe
what cte decoding should be done too?

Also tests that allow access to the size of an attachment, or the count of
the attachments would be useful too.  Perhaps we are looking at a "mime"
test as well as a "body" test?

Here's some scribblings to ponder over...

body: [MIME-BODY-PART] [CONTENT-TRANSFER-DECODING]
  [COMPARATOR] [MATCH-TYPE] <key-list: string-list>

CONTENT-TRANSFER-DECODING: ":none" ":all" / ":base64" / "quoted-printable"
MIME-BODY-PART: ":mime-header" [COMPARATOR] [MATCH-TYPE]
    <header-names: string-list> <key-list: string-list>

Now we could look for addresses in the headers of a body part
   Syntax:   address [MIME-BODY-PART] [ADDRESS-PART] [COMPARATOR]
[MATCH-TYPE]
             <header-list: string-list> <key-list: string-list>

Or if a particular header exists in one of the body parts
   Syntax:   exists [MIME-BODY-PART] <header-names: string-list>

Or if a header contains a string in one of the body part headers
   Syntax:   header [MIME-BODY-PART] [COMPARATOR] [MATCH-TYPE]
             <header-names: string-list> <key-list: string-list>

The size of a body part matching a particular criteria.
   Syntax:   size [MIME-BODY-PART] <":over" / ":under"> <limit: number>

So if you do a body test with no mime-body-part selector, and a
content-transfer-decoding of :none, then you would end up with the x_body
that we have implemented.

Nigel

Date: Sun, 16 Jun 2002 12:57:20 -0400
From: Cyrus Daboo <[email-masked]>
To: Tim Showalter <[email-masked]>, [email-masked]
Subject: Re: "body" extension
Sender: [email-masked]

Hi,

--On  Tim Showalter <[email-masked]> wrote:

| I'm not convinced Sieve is the right place to do virus scanning

I agree - most of the examples given for body filtering are down to the 
need to do virus and spam filtering. I don't think the need for virus and 
spam filtering alone should justify the presence of body.

What I would like to see are generic 'spamtest' and 'virustest' extensions 
in sieve that would allow sieve implementations to take better advantage of 
third-party tools in a transparent manner. For example, some spam tools 
insert specific headers with the header content indicating the level of 
spam. Right now you have to code a sieve script to recognise these headers 
for each type of spam tool that you might use. A better approach, IMHO, is 
to have a generic 'spamtest' command that returns a number in the range, 
say, 0 to 10, that can be used to trigger an appropriate action.

The sieve implementation would determine exactly what spamtest does. In the 
case of sieve sitting behind one of those spam tools, the spamtest command 
would implicitly look for the relevant headers and have some form of 
normalising those results into its 0..10 range. If a sieve implementation 
had a spam detection system builtin, it could utilise that directly.

The benefit of this is that you don't have to keep rewriting scripts for 
different kinds of spam or virus tools that may be in use - each sieve 
implementation would deal with the specifics of the spam/virus tool itself, 
hiding complexity from the end user. If a user really wanted a lower level 
of control, they would be free to not use spamtest, and instead do the 
header/body tests directly.

-- 
Cyrus Daboo

Subject: Re: "body" extension
From: Tim Showalter <[email-masked]>
To: [email-masked]
Date: 15 Jun 2002 22:26:35 -0700
Sender: [email-masked]

On Fri, 2002-06-14 at 17:33, Jutta Degener wrote:
> 
> On Fri, Jun 14, 2002 at 03:09:18PM -0600, Tim Showalter wrote:
> > Doing this right (charset decoding with comparators) is hard, but is the
> > right thing.
> 
> I think that's the right thing for some applications,
> but not for others.

I want to address this but I want to answer everything else first.

> I'm thinking about the application you describe in the
> form of another extension I'm calling "text".  (You're not
> the only person who wants this to be part of "body" and
> triggered with a flag or by its absence; I'm resisting this
> mostly because I want to get "body" out to closely resemble
> existing practice, and I haven't yet heard that this kind
> of behavior is existing practice.  Please set me straight if
> you know differently!)

Codifying existing practice would be fine with me, but my problem is
that the particular piece of real estate that would be the "body"
keyword seems valuable to me.  What we're describing here is the
uncommon case where you want to go mucking with the raw data of the
message.  It's easy to implement, but when a user goes and tries to use
it in practice, they're not going to be able to decode CTEs.

We have a "header" keyword; it searches decoded headers.  The keyword to
search CTE-decoded bodies ought to be "body".

I don't know what anyone's x_body extension does, unfortunately, but I'd
be a lot more comfortable if we called this rawbody or body :raw.

> (There could be a third in the canon, working-title "content",
> which transfer-encodes, but doesn't go any further.  That's
> probably what you want if you are trying to implement a virus
> scanner that looks for signature strings in binary files.)

That's an interesting idea as well.  I'm not convinced Sieve is the
right place to do virus scanning, especially given the prevalence of
Microsoft Word macros and deliberately obfuscated code, though.

> > I do not believe this is the right form for "body", unless there's a
> > required tagged argument ("body :raw").  If someone asks for "body",
> > this probably isn't what they want.
> 
> What I would like to know - and I think you have that information,
> but you haven't put it into your reply - is what's _actually implemented
> right now_.  What are the syntax and semantics of your existing
> "body" command?

I don't have a Sieve implementation, so that's not all that important. 
But what I implemented has very similar syntax to what you described. 
It's really easy to implement.  I got away with it for a few releases. 
But it will be difficult to change the semantics of body from "search
raw CTEs and MIME headers" to "search decoded CTEs and skip MIME
headers", as we won't have a suitable replacement.

Actually I don't even know which implementations do x_body, but I have
seen it used in example scripts.

> I think that's the right thing for some applications,
> but not for others.

I agree a raw body search is useful.  It's even easy to implement in a
lot of environments (mmap makes this just trivial), and I think a lot of
implementations would immediately start supporting it for those reasons.

I think there are three applications for a body search: (1) text
searching in text parts; (2) searching raw MIME because Sieve doesn't
provide the tools for it; (3) searching non-text parts for non-text
data, what you're calling "content".  I hadn't thought about (3) before.

A raw body search is insufficient for all of these, (1) and (3) because
of CTE issues, and (2) because it can be fooled too easily.

There are places where a raw body search will be useful, but I'd like to
see it called something else.  Supporting uses (1) and (3) well would be
valuable.  Use (2) would be good, but I think the right way to do it is
to provide MIME-matching functions.

If there are other specific applications I'd like to know about them.  I
am not sure that there's a good way to generalize this stuff in terms of
the interface.

Tim

Date: Fri, 14 Jun 2002 16:33:10 -0700
From: Jutta Degener <[email-masked]>
To: [email-masked]
Subject: Re: "body" extension
Sender: [email-masked]

On Fri, Jun 14, 2002 at 03:09:18PM -0600, Tim Showalter wrote:
> Doing this right (charset decoding with comparators) is hard, but is the
> right thing.

I think that's the right thing for some applications,
but not for others.

I'm thinking about the application you describe in the
form of another extension I'm calling "text".  (You're not
the only person who wants this to be part of "body" and
triggered with a flag or by its absence; I'm resisting this
mostly because I want to get "body" out to closely resemble
existing practice, and I haven't yet heard that this kind
of behavior is existing practice.  Please set me straight if
you know differently!)

The "text" extension decodes transfer-encodings and knows
about charsets.  It translates as much as the server can
(quality of service issue here) into plain text, then searches
that plain text for a given search string.  If you want to
prevent your employees from receiving e-mail that contains
the words "willing teenage girls", that's where you go.

(There could be a third in the canon, working-title "content",
which transfer-encodes, but doesn't go any further.  That's
probably what you want if you are trying to implement a virus
scanner that looks for signature strings in binary files.)

> Finally, body searching really is expensive.  It doesn't take a very
> naive implementation of regex to make body matching slow; :matches is
> actually quite sufficient if it's recursive.

Good point; I'll add "matches" to the list of warnings.

> \n is not a valid sequence in quoted strings (nor is it a line delimiter
> in an RFC822 message)

Ouch.  Thanks.  That whole example is a bit too specific for a
real draft -- I doubt that people will still understand it in a 
few years.

> I do not believe this is the right form for "body", unless there's a
> required tagged argument ("body :raw").  If someone asks for "body",
> this probably isn't what they want.

What I would like to know - and I think you have that information,
but you haven't put it into your reply - is what's _actually implemented
right now_.  What are the syntax and semantics of your existing
"body" command?

Jutta

Subject: Re: "body" extension
From: Tim Showalter <[email-masked]>
To: [email-masked]
Date: 14 Jun 2002 15:09:18 -0600
Sender: [email-masked]

Sorry about that--I'm just getting used to the fact that ctrl-return
sends messages in this mailer...

On Fri, 2002-06-14 at 11:39, Simon Josefsson wrote:
> 
> Jutta Degener <[email-masked]> writes:
> 
> > Below is a strawman Internet draft that describes the easiest,
> > most general form of "body" I could imagine -- a simple match
> > against the text of an e-mail message that is not the header,
> > without content-decoding of any sort.
> 
> A future extension of this could be to even MIME decode data before
> matching it as well -- international users need this to match
> non-ASCII in bodies.

So anyway, speaking only for myself:

I've implemented that in a different filtering system, and it turns out
to be kind of useful.  But the gotchas are severe.

(1) people will try to use it to parse MIME.
(2) people will complain that it doesn't do i18n.
(3) it's useless outside of ASCII, because the messages are invariably
not in Unicode, but the input messages are.

Doing this right (charset decoding with comparators) is hard, but is the
right thing.  The quick and dirty body extension--especially if it uses
the "body" identifier for labeling--will eventually, if not immediately,
have to be obsoleted by a more general and complete mechanism.

I've seen several mailers recently that send perfectly reasonable 7bit
parts as base64 as well.

I believe that if we're going to go down this path, we might as well do
it right and specify MIME handling.

Finally, body searching really is expensive.  It doesn't take a very
naive implementation of regex to make body matching slow; :matches is
actually quite sufficient if it's recursive.

Specific comments on the draft:

I do not believe this is the right form for "body", unless there's a
required tagged argument ("body :raw").  If someone asks for "body",
this probably isn't what they want.

\n is not a valid sequence in quoted strings (nor is it a line delimiter
in an RFC822 message)

It doesn't take a very naive implementation of regex to make body
matching slow; :matches is actually quite sufficient if it's recursive.

Tim

Subject: Re: "body" extension
From: Tim Showalter <[email-masked]>
To: Simon Josefsson <[email-masked]>
Cc: Jutta Degener <[email-masked]>, [email-masked]
Date: 14 Jun 2002 14:57:55 -0600
Sender: [email-masked]

On Fri, 2002-06-14 at 11:39, Simon Josefsson wrote:
> 
> Jutta Degener <[email-masked]> writes:
> 
> > Below is a strawman Internet draft that describes the easiest,
> > most general form of "body" I could imagine -- a simple match
> > against the text of an e-mail message that is not the header,
> > without content-decoding of any sort.
> 
> A future extension of this could be to even MIME decode data before
> matching it as well -- international users need this to match
> non-ASCII in bodies.

Mirapoint has implemented a body search in their non-Sieve filtering
(Mirapoint doesn't support Sieve, unfortunately).

(I speak only for myself.)

Tim

To: Jutta Degener <[email-masked]>
Cc: [email-masked]
Subject: Re: "body" extension
From: Simon Josefsson <[email-masked]>
Date: Fri, 14 Jun 2002 19:39:54 +0200
Lines: 30
Sender: [email-masked]

Jutta Degener <[email-masked]> writes:

> Below is a strawman Internet draft that describes the easiest,
> most general form of "body" I could imagine -- a simple match
> against the text of an e-mail message that is not the header,
> without content-decoding of any sort.

A future extension of this could be to even MIME decode data before
matching it as well -- international users need this to match
non-ASCII in bodies.

> 5. Security Considerations
>
>    The system must be sized and restricted in such a manner that even
>    malicious use of body matching does not deny service to other users.
>
>    Matching strings in the body of an e-mail message may be more general
>    than the author of the sieve script intended.  The strings may be
>    matched in headers of included messages; strings that are dangerous 
>    only when used in enclosed MIME headers may be matched in text that
>    is in itself not dangerous, or discusses the very problem that the
>    script is attempting to defend against.

A thought:

Matching strings can also be less general than the author intended.
This is because the same data can be expressed in several ways using,
e.g., quoted-printable and base64 encodings.  Thus trying to filter on
a specific strings in order to redirect certain data (viruses, certain
key words) can be worked around by the sender.

Date: Fri, 14 Jun 2002 09:54:59 -0700
From: Jutta Degener <[email-masked]>
To: [email-masked]
Subject: "body" extension
Sender: [email-masked]

Below is a strawman Internet draft that describes the easiest,
most general form of "body" I could imagine -- a simple match
against the text of an e-mail message that is not the header,
without content-decoding of any sort.  I plan to submit this
in earnest about a week from now, in time for Yokohama, and am
looking for initial comments.

My goal is not to create the catch-all of message body operations,
but just to arrive at a draft that has the simplest and/or most
common features of the "body" as actually implemented and used
so far.

There are multiple "body" implementations (I'm counting "x_body"
as one) out there; I don't know enough about their details, and
am interested in responses from people who have implemented
or used some form of "body" -- does this match the implementations
you know about, and if not, how do the implementations you know
differ from it?

Thanks!,

Jutta <[email-masked]>
--
INTERNET DRAFT: Sieve -- "body" extension                  Jutta Degener
Document: draft-degener-sieve-body-00.txt                 Sendmail, Inc.
Expires: December 2002                                         June 2002

                      Sieve -- "body" extension

Status of this memo

   This document is an Internet-Draft and is subject to all
   provisions of Section 10 of RFC2026.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as
   Internet-Drafts.

   Internet-Drafts are draft documents valid for a maximum of six
   months and may be updated, replaced, or obsoleted by other
   documents at any time.  It is inappropriate to use Internet-
   Drafts as reference material or to cite them other than as
   "work in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/1id-abstracts.html

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html

Abstract

   This document defines a new test for the Sieve language that
   tests the occurrence of a string in the uninterpreted body
   of an e-mail message.

1. Introduction

   The "body" test, initially proposed for the [SIEVE] base document,
   has later been removed from it because it was held to be too costly
   to implement.  Nevertheless, several server vendors have implemented
   some form of the "body" command.  This document describes a simple
   version of such a command in the hopes of providing a common ground
   for users and implementors of the existing practice.

2. Conventions used.

   Conventions for notations are as in [SIEVE] section 1.1, including
   use of [KEYWORDS] and "Syntax:" label for the definition of action
   and tagged arguments syntax.

   The capability string associated with extension defined in this
   document is "body".

3. Test body

   Syntax:   "body" [COMPARATOR] [MATCH-TYPE] <key-list: string-list>

   The body test matches text in the body of an e-mail message,
   that is, anything following the first empty line after the header.
   (The empty line itself, if present, is not considered to be part
   of the body.)

   For the purposes of this test, the actual MIME structure of the
   body is irrelevant.  The server MUST NOT remove transfer encoding
   from the message.

   (This is done in part to save work for the executing server,
   in part to allow matches against syntactically incorrect
   messages.)

   If a message consists of a header only, not followed by an
   empty line, all "body" tests fail, including that for an
   empty string.

   If a message consists of a header followed only by an empty
   line with no body lines following it, the message is considered
   to have an empty string as a body.

   Example:  require "body";
	     # ...
   	     # filter out some "Sircam" virus messages
   	     if body :contains "Hi! How are you?\nI send you this file" {
                discard;
	     }

4. Interaction with Other Sieve Extensions

   Any extension that extends the grammar for the COMPARATOR or
   MATCH-TYPE nonterminals will also affect the implementation of
   "body".
   
   Especially when used in conjunction with a naive implementation of
   [REGEX], the body extension can place a considerable load on a
   system that may affect all users of the system.

5. Security Considerations

   The system must be sized and restricted in such a manner that even
   malicious use of body matching does not deny service to other users.

   Matching strings in the body of an e-mail message may be more general
   than the author of the sieve script intended.  The strings may be
   matched in headers of included messages; strings that are dangerous 
   only when used in enclosed MIME headers may be matched in text that
   is in itself not dangerous, or discusses the very problem that the
   script is attempting to defend against.

6.  Acknowledgments

    This document will be revised in part based on comments and
    discussions which will take place on and off the SIEVE mailing list.
    Thanks in advance for the help of those who will take the time to review
    this draft and make suggestions.

7. Author's Address

    Jutta Degener
    Sendmail, Inc.
    6425 Christie Ave, 4th Floor
    Emeryville, CA 94608

    Email: [email-masked]

Appendices

Appendix A.  References

   [KEYWORDS] Bradner, S., "Key words for use in RFCs to Indicate
   Requirement Levels", RFC 2119, March 1997.

   [REGEX] Murchison, K., "Sieve -- Regular Expression Extension",
   draft-murchison-sieve-regex-04.txt, August 2001.

   [SIEVE] Showalter, T.,  "Sieve: A Mail Filtering Language", RFC 3028,
   January 2001.

Appendix B. Full Copyright Statement

    Copyright (C) The Internet Society 2002. All Rights Reserved.

    This document and translations of it may be copied and furnished to
    others, and derivative works that comment on or otherwise explain it
    or assist in its implementation may be prepared, copied, published
    and distributed, in whole or in part, without restriction of any
    kind, provided that the above copyright notice and this paragraph
    are included on all such copies and derivative works.  However, this
    document itself may not be modified in any way, such as by removing
    the copyright notice or references to the Internet Society or other
    Internet organizations, except as needed for the purpose of
    developing Internet standards in which case the procedures for
    copyrights defined in the Internet Standards process must be
    followed, or as required to translate it into languages other than
    English.

    The limited permissions granted above are perpetual and will not be
    revoked by the Internet Society or its successors or assigns.

    This document and the information contained herein is provided on an
    "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
    TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
    BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
    HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
    MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.