Subject: Regex matching complete headers.

From: "Nigel Swinson" <[email-masked]>
Date: Wed, 3 Jan 2001 13:35:06 -0000

2 points and some suggestions about the Sieve Home page

--------------------------------------------------------------------------------

In the regex draft we have in the example:

            # or the subject is all uppercase (no lowercase)
            header :regex :comparator "i;octet" "subject"
              "^[^:lower:]*$" ) {

What if the Subject is mutliline and one of the lines contains uppercase letters, while the other contains only lowercase letters?  ie.

Subject: this is the first line that contains only lowercase
  this is a contination of the Subject header but it contains UPPERCASE LETTERS

The regular expression "^[^:lower:]*$" is going to match the first line, and therefore give us a match, but this isn't what we intended.

I have a customer who quite sensibly would like to filter all messages that have either no from header, or an empty from header.  Our exists test will pass if the header exists but is empty, so we need a regex test too that tests to say if the header is completely empty.  Initial thoughts would be something along the lines of "^[[:space:]]*$ but then this will match the following from header:

From:
 [email-masked]
 [email-masked]

as the very first line contains nothing but spaces.

Can we currently do this with Sieve at present?  If not perhaps we need to be able to use :regex with :is or :contains?  I see it says in the Sieve draft that we can't give more than one match type, but then perhaps this is a suitable occasion?

--------------------------------------------------------------------------------

Could we also allow:
  a.. \w in place of [:word:]
  b.. \s in place of [:space:]
  c.. \d in place of [:digit:]
  d.. \l in place of [:lower:]
  e.. \u in place of [:upper:]
I find them really quite useful.  Or are we trying to stick to POSIX rigidly. (Still haven't got a copy of this.  Bloomin have to bloomin pay for it. Grrr)

--------------------------------------------------------------------------------

Finally, whoever is responsible for http://www.cyrusoft.com/sieve/ could they please update the broken link that points to http://search.ietf.org/internet-drafts/draft-melnikov-sieve-imapflags-03.txt instead of http://search.ietf.org/internet-drafts/draft-melnikov-sieve-imapflags-04.txt

Additionally should there not be a link to the regex work at http://search.ietf.org/internet-drafts/draft-murchison-sieve-regex-02.txt it seems relevant in the Internet Drafts section

--------------------------------------------------------------------------------

That's all for now.

Cheers

Nigel

---

From: Ken Murchison <[email-masked]>
Date: Wed, 03 Jan 2001 10:27:15 -0500

> Nigel Swinson wrote:
> ----------------------------------------------------------------------
> What if the Subject is mutliline and one of the lines contains
> uppercase letters, while the other contains only lowercase letters?
> ie.
> 
> The regular expression "^[^:lower:]*$" is going to match the first
> line, and therefore give us a match, but this isn't what we intended.

Actually, you've misread the regex.  It is looking for subjects that DO
NOT contain any lowercase chars.  This being the case, :regex should
work for multiline headers, just like the other comparators.  In fact, I
tested it with my regex implementation in cmu-sieve and it works fine. 
Using this script:

require "regex";

if header :regex :comparator "i;octet" "subject" "^[^:lower:]*$" {
	discard;
}

A subject of:

Subject: FIRST LINE
	SECOND LINE

will be discarded

and a subject of:

Subject: FIRST LINE
	Second LINE

will be kept

It is my understanding that all of the comparators work on the the
ENTIRE header contents, regardless of the number of lines, ie the script

if header :contains "subject" "second" {
	discard;
}

will discard either of the examples above.

> I have a customer who quite sensibly would like to filter all messages
> that have either no from header, or an empty from header.  Our exists
> test will pass if the header exists but is empty, so we need a regex
> test too that tests to say if the header is completely empty.

Try:

if anyof (not exists "from", header :regex "from" "^$")

Any good header parser should gobble up leading whitespace, so even if
the from: header exists an contains nothing but whitespace, the test
above should work.

> ----------------------------------------------------------------------
> 
> Could we also allow:
> 
>    * \w in place of [:word:]
>    * \s in place of [:space:]
>    * \d in place of [:digit:]
>    * \l in place of [:lower:]
>    * \u in place of [:upper:]
> 
> I find them really quite useful.  Or are we trying to stick to POSIX
> rigidly.

I went with POSIX simply because it is a defined spec and I didn't want
to describe exactly what a regex should be (with all the ifs/ands/buts,
etc).  If there is consensus that shortcuts like these are desired, I'm
not opposed to adding them to the draft.

Regards,
Ken
-- 
Kenneth Murchison     Oceana Matrix Ltd.
Software Engineer     21 Princeton Place
716-662-8973 x26      Orchard Park, NY 14127
--PGP Public Key--    http://www.oceana.com/~ken/ksm.pgp

---

From: [email-masked]
Date: Wed, 03 Jan 2001 07:30:39 -0700 (PDT)

> What if the Subject is mutliline and one of the lines contains uppercase
> letters, while the other contains only lowercase letters?  ie.

>   this is a contination of the Subject header but it contains UPPERCASE LETTERS

This is not an example of multiple subject lines, it is an example of a single
folded subject line. Folding is removed prior to comparison; it is arbitrary
and can change during message transit. Unfortunately, the sieve specification
doesn't make this clear, and it should. (The reason it doesn't is probably that
the handling of folding is considered to be such a fundamental characteristic
of email headers that nobody thought to point it out.)

> The regular expression "^[^:lower:]*$" is going to match the first line, and
> therefore give us a match, but this isn't what we intended.

No, it will apply to the entire unfolded line, so it won't match.

> I have a customer who quite sensibly would like to filter all messages that
> have either no from header, or an empty from header.  Our exists test will pass
> if the header exists but is empty, so we need a regex test too that tests to
> say if the header is completely empty.  Initial thoughts would be something
> along the lines of "^[[:space:]]*$ but then this will match the following from
> header:

> as the very first line contains nothing but spaces.

Again, this is incorrect. There is no empty From: field here.

> Can we currently do this with Sieve at present?

Yes you can.

> If not perhaps we need to be
> able to use :regex with :is or :contains?  I see it says in the Sieve draft
> that we can't give more than one match type, but then perhaps this is a
> suitable occasion?

Nope.

> I find them really quite useful.  Or are we trying to stick to POSIX rigidly.
> (Still haven't got a copy of this.  Bloomin have to bloomin pay for it. Grrr)

Personally, I think there are real advantages to sticking to the POSIX
specification and not introducing shortcuts. For example, it enables use of an
OOTB POSIX regex facility.

> Finally, whoever is responsible for http://www.cyrusoft.com/sieve/ could they
> please update the broken link that points to
> http://search.ietf.org/internet-drafts/draft-melnikov-sieve-imapflags-03.txt
> instead of
> http://search.ietf.org/internet-drafts/draft-melnikov-sieve-imapflags-04.txt

> Additionally should there not be a link to the regex work at
> http://search.ietf.org/internet-drafts/draft-murchison-sieve-regex-02.txt it
> seems relevant in the Internet Drafts section

I suggest you send mail to the site maintainer, and if that's not obvious
to the postmaster. This isn't appropriate for this mailing list.

				Ned

---

From: Tim Showalter <[email-masked]>
Date: 03 Jan 2001 13:24:33 -0800

[email-masked] writes:

> This is not an example of multiple subject lines, it is an example of
> a single folded subject line. Folding is removed prior to comparison;
> it is arbitrary and can change during message transit. Unfortunately,
> the sieve specification doesn't make this clear, and it should. (The
> reason it doesn't is probably that the handling of folding is
> considered to be such a fundamental characteristic of email headers
> that nobody thought to point it out.)

Right.  Actually I think there are clearer folding rules in the
forthcoming 822bis document anyway, and I was thinking about those when
I forgot about this.

I'll try and add a sentence saying this if/when I can.

Tim

---

From: [email-masked]
Date: Wed, 03 Jan 2001 13:25:47 -0700 (PDT)

> I'll try and add a sentence saying this if/when I can.

Good.

				Ned