Subject: 'header' test and whitespace

From: Philip Guenther <[email-masked]>
Date: Mon, 21 Nov 2005 16:17:22 -0800

First of all, my apologies to the list for failing to get out the
updated 3028bis I-D by last Friday as I promised.

Part of the hold up has been in determining the exact wording to
insert on a couple points.  In particular, Barry Leiba argued
strongly in Vancouver that the 'header' test should _at_least_
normalize to a single space all whitespace that appears at the
beginning or end of each line of a header field value.  That is,
the header field unfolding algorithm would not only remove the CRLF
pair but also compress the whitespace around the CRLF into a single
space.

Alternatively, that same compression could be applied to *all* the
embedded whitespace in the field value, such that the 'header' test
would never 'see' tabs or consecutive spaces.

So, there are three choices:
1) just compress whitespace at fold points
2) compress all whitespace
3) no compression

Note that we had previously agreed that leading and trailing
whitespace on the field value in total should be completely ignored;
that would not be affected by the compression described above.

Philip Guenther

---

From: Ned Freed <[email-masked]>
Date: Mon, 21 Nov 2005 16:40:10 -0800 (PST)

> updated 3028bis I-D by last Friday as I promised.

Haven't gotten the revised vacation draft out either ;-(

> 2) compress all whitespace

I strongly object to (1) and (2). Making the header test not respect interior
spaces is a _really_ bad idea. Like it or not, there are communities that are
very sensitive to spacing issues (many Japanese users seem to qualify) and
these communites need to be able to test and see exactly what sorts of spaces
are present in fields (and possibly modify them).

Additionally, compression down to a single space is actually insufficient in
some cases - what's needed is for spaces to be removed, not compressed. But
this can, and should, be done through an appropriately defined comparator. It
should not be a built in part of the header test.

> that would not be affected by the compression described above.

I have no problem with removing whitespace at the beginning of the field - in
fact I prefer it. I think removing trailing whitespace has a chance of causing
problems, but the benefits probably equal if not exceed the costs.

				Ned

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Tue, 22 Nov 2005 13:18:45 +0100

> 2) compress all whitespace

the unfold logic is too complex for most authors of e-mail software, it
seems, and a "compress whitespace" rule might make life easier for them.
however, they will probably still screw up

Subject: This RFC 2047 thingy is =?UTF-8?Q?p?=
 =?UTF-8?Q?icky?=

so I don't think it matters much in practice.  so my conclusion is:
don't confuse matters even more by adding Sieve specific unfolding
rules.

> that would not be affected by the compression described above.

I don't see why we need to strip trailing whitespace, but I don't feel
strongly about it.
-- 
Kjetil T.

---

From: Michael Haardt <[email-masked]>
Date: Tue, 22 Nov 2005 14:03:13 +0100

> are present in fields (and possibly modify them).

I think the same logic that led to the consensus of stripping trailing white
space might help:

  How does the MUA present the header?

MUAs do not compress all whitespace.  If Sieve did, comparisons would not
work as the user expects them to.  To me, that's a second reason against
(2).

Both UCB mail and mutt do compress whitespace at fold points for the
overview and show all whitespace at fold points when displaying the
message, including the newline.  Both surprises me, and I do not like it.
So much for logic. ;-)

Not looking at MIME, (1) might remove white spaces that were originally
part of the message, but happened to be at the folding position.
I dislike the thought of not seeing the original content.

As a result, I vote for (3).

Michael

---

From: Barry Leiba <[email-masked]>
Date: Tue, 22 Nov 2005 12:27:38 -0500

Ned says...
> are present in fields (and possibly modify them).

I proposed (1), and am not fond of (2); given Ned's statement and the comments
from Michael and Kjetil, I think it's clear that (2) is out.

> problems, but the benefits probably equal if not exceed the costs.

I agree.

 > message, including the newline.  Both surprises me, and I do not like it.

It's that inconsistency that prompts this suggestion.  Behaviour is so varied
that I'd rather see us define something predictable.

 > don't confuse matters even more by adding Sieve specific unfolding

There's definitely something to be said for us to just point to the RFC2822
unfolding rule, and not define our own.

I'll be happy with whatever decision we make about the whitespace at the fold
points, but I prefer compressing it to one space there.  I will NOT be happy
if we don't strip leading and trailing whitespace on the overall unfolded
field value.  Moreover, I think another important point in what I suggested
is that this stripping be done ANY TIME Sieve touches a header field value,
NOT JUST IN COMPARISONS.  If we stick the value into a variable and then
compare against the variable, the behaviour must be the same as if we
compare against the header field value directly, which means that the
leading and trailing whitespace must be stripped when the value is put into
the variable in the first place.

Barry

--
Barry Leiba, Pervasive Computing Technology  ([email-masked])
http://www.research.ibm.com/people/l/leiba
http://www.research.ibm.com/spam

---

From: Michael Haardt <[email-masked]>
Date: Wed, 23 Nov 2005 23:55:10 +0100

> that I'd rather see us define something predictable.

"Varied" is a nice word for what I consider a "bug".  Unfolding as
specified in RFC 2822 is easy to understand and logical to me, yet
two MUAs decide to break it in two different ways.  I agree we should
say something about it.

> unfolding rule, and not define our own.

You have my vote for that.

> the variable in the first place.

You lost me somewhere.  Do you mean:

Headers accessed by Sieve are always: Unfolded, stripped of heading and
trailing white space and MIME-decoded?

If so, that would make a raw header test/comparison impossible.  I suggest
that specific tests and actions define if they do the above, and I have
no problems if variable assignments do, but a requirement in the base spec
that all operations on headers must work that way sounds limiting to me.

Michael

---

From: Barry Leiba <[email-masked]>
Date: Wed, 23 Nov 2005 22:26:25 -0500

I see from Alexey's minutes that I was supposed to post the text that
I proposed.  I didn't remember that; sorry.  Philip started the
discussion, but didn't post the text.  I gave Philip XML, and here's
what I suggested:

<t>
Because the meaning of leading and trailing whitespace characters in header
fields is ambiguous, and their survival in message transport and processing
is inconsistent, ALL handling of message headers in Sieve MUST normalize
the header field values.  The normalization is similar to, but not the same
as, unfolding (see RFC2822), and is done as follows:
<list style="number">
   <t>
     Remove leading and trailing whitespace characters from each line of
     the header field (multiple lines, in the case of multi-line continuation).
   </t>
   <t>
     Remove the delimiting CRLF from each line.
   </t>
   <t>
     Catenate the lines in order, inserting one ASCII space character (0x20)
     between each pair.
   </t>
</list>
</t>

<t>
To show how this normalization works, we use the character "~" (tilde) to
represent the ASCII space character in the following example.
This normalization will result in all of the following normalizing to the
same value for the subject field, "a~b~~~c~d":
<list style="empty">
   <t>
     Subject:~a~b~~~c~d
   </t>
   <t>
     Subject:a~b~~~c~d~
   </t>
   <t>
     Subject:a~b~~~c~
     <vspace/>
     ~~~~d~~
   </t>
   <t>
     Subject:~~~~~a
     <vspace/>
     ~~~~b~~~c~~~~~
     <vspace/>
     ~~~~d
   </t>
   <t>
     Subject:~a
     <vspace/>
     ~b~~~c
     <vspace/>
     ~d
   </t>
</list>
</t>

Note that I didn't suggest RFC2047-decoding, but I think that's a reasonable
addition to this.  Alternatively, we could specify that strings be decoded
in comparisons (perhaps specified by an option like ":decode" or ":raw").

Barry

---

From: Barry Leiba <[email-masked]>
Date: Wed, 23 Nov 2005 22:20:31 -0500

> trailing white space and MIME-decoded?

I didn't lose you; you got it, yes.

> If so, that would make a raw header test/comparison impossible.

OK; I'm happy to have an explicit "leave it raw" function.  Lacking that,
I would like to see them normalized.

Barry

---

From: Ned Freed <[email-masked]>
Date: Thu, 24 Nov 2005 17:30:47 -0800 (PST)

> as, unfolding (see RFC2822),

I think the unfolding part needs to be the same as what's in RFC 2822.

>      Catenate the lines in order, inserting one ASCII space character (0x20)

This makes the unfolding different from what's in RFC 2822. I think this is a
mistake. There should not be any "insert space" operation here - RFC 2822
section 2.2.3 simply calls for CRLFs to be removed.

> same value for the subject field, "a~b~~~c~d":

I didn't bother going through all this.

> in comparisons (perhaps specified by an option like ":decode" or ":raw").

A :raw option makes sense. I actually have no problem with adding it to the
base specification but others may disagree.

				Ned

---

From: Tony Hansen <[email-masked]>
Date: Thu, 24 Nov 2005 21:37:28 -0500

Ned Freed wrote:
> 
>> Because the meaning of leading and trailing whitespace characters
>> in header fields is ambiguous, and their survival in message
>> transport and processing is inconsistent, ALL handling of message
>> headers in Sieve MUST normalize the header field values. The
>> normalization is similar to, but not the same as, unfolding (see
>> RFC2822),
> 

ditto

	Tony Hansen
	[email-masked]

---

From: Alexey Melnikov <[email-masked]>
Date: Fri, 25 Nov 2005 13:32:44 +0000

I agree.

>>      the header field (multiple lines, in the case of multi-line 
This step is actually not mentioned in RFC 2822.

> section 2.2.3 simply calls for CRLFs to be removed.

[...]

> base specification but others may disagree.

I am personally Ok with adding :raw to the base spec, but do we need a 
new capability?

---

From: Ned Freed <[email-masked]>
Date: Fri, 25 Nov 2005 08:41:55 -0800 (PST)

> This step is actually not mentioned in RFC 2822.

Yes. I don't view this as part of the unfolding algorithm, however. But now
that you mention it, a better way to describe this is to do the unfolding
part first and then remove leading and trailing spaces. So the steps become:

(1) Remove all CRLFs.

(2) Remove leading and trailing spaces.

(3) Decode RFC 2047 and convert to utf-8.

The last two steps should be skipped in :raw mode. (The first step is retained
because folding points aren't supposed to have any semantics and are known to
change unpredictably. The same cannot be said for spaces or encoded words.)

> I am personally Ok with adding :raw to the base spec, but do we need a

Yes, I'm afraid so. Perhaps something like rawheader? (I believe the header
test is the only one for which :raw makes sense.)

				Ned

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Fri, 25 Nov 2005 23:56:45 +0100

> (3) Decode RFC 2047 and convert to utf-8.

if we're going to mention 2047, we should also mention that RFC 2231
applies for some headers.  although it seems to me that using a
bastardised 2047 is the deployed norm for Content-Disposition and
friends, that's not correct according to the specifications, and being
more explicit about it might help conformance to spec.

> change unpredictably. The same cannot be said for spaces or encoded words.)

the exact method used of RFC 2047 encoding doesn't have any semantics,
either.  a discussion thread might have a Subject which switches back
and forth between Q and B encoding, or in the length of encoded-words
and hence the number of lines.  this is very apparent for us
non-USans :-)

so I don't think it's appropriate to make :raw be :halfraw.  the CR LF
is actually a part of the raw header.

while we're at it, what do we do about headers which can have multiple
values, e.g. "Cc"?  (multiple headers is deprecated in 2822, but must be
supported.)  I don't have a good suggestion.  the naive approach is to
concatenate them as if they were simply separated by CR LF, but for "Cc"
you would really want to include a comma in the delimiter as well.  the
other option is to say that the first matching header is used.  this
sits well with short-circuiting logic, but means it's impossible to
capture the complete value of the header.

> test is the only one for which :raw makes sense.)

what happens if you add a :raw argument and upload to today's
implementations?  will they reject during upload?  will they ignore it
during runtime?  will they bomb during runtime?

IMHO it's okay as long as it doesn't cause a runtime error.  (Cyrus 2.2
will reject upload, I haven't checked others.)

the behaviour of "header" is under-specified in the current spec, and
deployed implementations probably deviating behaviour.  the more
detailed specs we're discussing here might make many of them obviously
non-conforming where they were only arguably non-conforming before (i.e.
not performing according to intent).  but is it important?  the number
of capabilities makes life harder for both server and client
implementors, and we should try to limit the number of them as much as
possible.

-- 
Kjetil T.

---

From: Ned Freed <[email-masked]>
Date: Fri, 25 Nov 2005 15:53:28 -0800 (PST)

> either.

In the abstract, maybe. But in practice it does. For example, it happens to
make perfect sense for me to filter out all messages that use the koi8-r 
charset. I receive maybe 300 such messages every day, and they are without
exception spam.

There are also cases where language can be inferred with reasonable reliability
from charset, and this can be very useful in some applications. (There often
isn't enough text in a header to analyze to determine the langauge being used.)

> a discussion thread might have a Subject which switches back

The encoding probably doesn't matter, especially since it sometimes changes in
transit. Charset changes are much rarer.

> is actually a part of the raw header.

I disagree. Folding points change all the time, they are specifically defined
not to have semantics, and I don't believe I've ever seen a case where I wanted
a test that is sensitive to CRLFs. The same cannot be said for encoded words
and trailing spaces.

> capture the complete value of the header.

Simple: You test all the values as separate fields. Sieve tests already allow for
a list of fields but there's an implicit list even when there's only a single
field name specified.

> during runtime?  will they bomb during runtime?

Our implementation will return a runtime error. I believe this is the correct
behavior. We don't check during upload - our sieves are typically provisioned
via LDAP and we have no control over what tools are used to insert them into
the directory.

And yes, this makes error reporting a real challenge.

> will reject upload, I haven't checked others.)

This is just another extension, so I don't see why causing a runtime
error is a problem.

> implementors, and we should try to limit the number of them as much as

I disagree with this as well. Additional new capabilities are nowhere as big
a problem as nailing things down in ways that break current behavior. To be
blunt, if this specification evolves to a point where our deployed sieve
scripts are incompatible with it, I'll have no choice but to stop supporting
sieve as defined by the IETF.

				Ned

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Sat, 26 Nov 2005 09:08:50 +0100

> charset. I receive maybe 300 such messages every day, and they are without

that will work until the spammers are savvy enough to switch to UTF-8 to
avoid such filtering.  the correct test would check for code points, not
encodings.

> isn't enough text in a header to analyze to determine the langauge being used.)

yes, although I have received messages in Norwegian using GB2312
encoding and messages in English using KOI8-R encoding, it's a safe bet
that the sender of these messages understand Chinese and Russian
respectively since they've set up their e-mail program to use these
encodings.

> transit. Charset changes are much rarer.

not in this neck of the woods.  different users will prefer UTF-8, ISO
8859-1 and ISO 8859-15, and the Subject will switch back and forth.

> a test that is sensitive to CRLFs. The same cannot be said for encoded words

okay, fine.  I was thinking abut cases where you want to repeat it back
at the sender, e.g.

  vacation :subject text:
Auto: I'm away from office
 (was: ${subject} )
.
;

where the fact that the CRLFs are included in ${subject} makes it
unnecessary to worry about folding of overlong subjects.  the vacation
draft should perhaps be explicit in that Subject must be folded
appropriately?

> a list of fields but there's an implicit list even when there's only a single

yes, "but [this] means it's impossible to capture the complete value of
the header", "*" will only fetch the contents of the first one.  perhaps
a loop construct for the address test would be a more appropriate
solution to that problem?

  for.every.address ["To", "Cc"] { block }

this can of course wait until we see the need.  for the base
specification, I would like a tiny change to make this a little more
apparent:

   For instance, the test `header :contains ["To", "Cc"]
-  ["[email-masked]", "[email-masked]"]' is true if either the
+  ["[email-masked]", "[email-masked]"]' is true if either a
   To header or Cc header of the input message contains either of the
   email addresses "[email-masked]" or "[email-masked]".

> > > test is the only one for which :raw makes sense.)

I'd like it for address, too.  this is the difference between
"xn--srlandslaget-vjb.no" and "sørlandslaget.no".  okay, so this isn't
in the spec today, but we might as well be ready for it.

> This is just another extension, so I don't see why causing a runtime

right, that was my point.  if you didn't raise a runtime error, we might
get away with not declaring it an extension.  but you do, so we don't.

-- 
Kjetil T.

---

From: Ned Freed <[email-masked]>
Date: Sat, 26 Nov 2005 08:45:35 -0800 (PST)

> avoid such filtering.  the correct test would check for code points, not

The same can be said for every other check you can think of - eventually
the spammers will adapt. That doesn't mean it isn't useful now, and it doesn't
mean other similar tests won't be useful in the future.

This is not, and should not be, a search for the FUSSP.

> 8859-1 and ISO 8859-15, and the Subject will switch back and forth.

That's quite true here but beside the point. I'm talking about changes in
transit, not the use of various different stuff when messages are composed.
Indeed, the reason that you see such mixtures regularly (as do I) is because
such changes _aren't_ being made by composing and transfer agents - they leave
the original source alone.

> draft should perhaps be explicit in that Subject must be folded

Wow, I had never even considered putting explicit folding points into a
vacation subject argument. Given that there can be substitutions and encoded
word generation and who knows what else happening it seems like a really bad
idea for a script to do this explicitly, but I suppose the vacation draft needs
to say that (a) The subjects and other headers in generated vacation responses
have to be properly folded and (b) CRLFs in subject arguments need to be
handled sensibly. In particular, the case of
<text-with-no-space><CRLF><text-with-no-space> has to be handled in a way that
doesn't cause an illegal header to be generated. I think I'll say something
like "CRLFs in :subject arguments MAY simply be removed but MUST be handled
in a fashion that prevents an illegal header field from being produced."

> the header", "*" will only fetch the contents of the first one.

Yep.

> perhaps

>   for.every.address ["To", "Cc"] { block }

Yes, seems reasonable. In fact I once argued that the concept of a result
generator might make sense in sieve - the Icon language uses this to great
advantage in a bunch of different ways.

>    email addresses "[email-masked]" or "[email-masked]".

Seems like a good idea to me.

> in the spec today, but we might as well be ready for it.

Agreed, and it may be closer than you think given how the meeting at the last
IETF on i18n stuff in email went.

> get away with not declaring it an extension.  but you do, so we don't.

OK, seems like we're in agreement then.

				Ned

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Tue, 29 Nov 2005 12:28:57 +0100

> > for the base specification, I would like a tiny change to make this
>> a little more apparent:
> 

I'm repeating it once more so Philip may take note of it :-).

also in the latest draft is this:

   The preferred way to test whether a given header is either empty or
   absent is to combine an "exists" test and a "header" test:

           anyof (header :is "Cc" "", not exists "Cc")

this will evaluate to true for:

  Cc: [email-masked]
  Cc:

it may not be worth considering this rather degenerate case.  it seems
to me Bcc is a more likely candidate for the stated test.  however, one
way of stating the test correctly is:

  not (header :matches "Cc" "?*")

it's not much more convoluted, so I think it is worthwhile to change the
"preferred way".
-- 
Kjetil T.

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Tue, 29 Nov 2005 14:00:37 +0100

On Tue, 2005-11-29 at 12:28 +0100, Kjetil Torgrim Homme wrote:
> [...] one way of stating the test correctly is:
> 

sorry, those parentheses are superfluous and should be left out.

  not header :matches "Cc" "?*"

-- 
Kjetil T.

---

From: "Mark E. Mallett" <[email-masked]>
Date: Tue, 29 Nov 2005 12:23:50 -0500

> it's not much more convoluted, so I think it is worthwhile to change the

Parentheses aside... in my implementation at least, the "header" test
would return true (because of the non-empty 'cc'), and be notted to
return false.  So this wouldn't work.  In order for it to work, the
interior 'header' test would have to be influenced by the 'not' somehow,
so that it rearranged the logic to be:

   header :not :matches "Cc" "?*"

which would depend on how it treated the lack of a header field at all.

But I think you could say

   not allof (exists "cc", header :matches "cc" "?*")

mm

---

From: Philip Guenther <[email-masked]>
Date: Tue, 29 Nov 2005 15:15:01 -0800

>I'm repeating it once more so Philip may take note of it :-).

Okay, okay, it's in now.

>it's not much more convoluted, so I think it is worthwhile to change the

Ick.  I agree that your version matches the "plain meaning" of the
description better than the one currently in the doc, but it is an
imprecise description.  Hmm.  How about:

    Testing whether a given header is either absent or doesn't contain
    any non-whitespace characters can be done using a negated "header"
    test:
	not header :matches "Cc" "?*"

Philip Guenther

---

From: Philip Guenther <[email-masked]>
Date: Tue, 29 Nov 2005 15:25:43 -0800

"Mark E. Mallett" <[email-masked]> writes:
>return false.  So this wouldn't work.

His point is that _is_ the desired behavior.  Given the two line message
header above, you would say that the Cc: header is either missing or
empty?  It certainly isn't missing and while one occurrence is empty, I
would not call "the Cc: header" empty, so returning 'false' is the Right
Thing.

Anyway, I've replaced the text in the I-D with this:
    Testing whether a given header is either absent or doesn't contain
    any non-whitespace characters can be done using a negated "header"
    test:
        not header :matches "Cc" "?*"

...
>   not allof (exists "cc", header :matches "cc" "?*")

That's equivalent to what's in the -05 I-D.

Philip Guenther

---

From: "Mark E. Mallett" <[email-masked]>
Date: Tue, 29 Nov 2005 20:46:25 -0500

> would not call "the Cc: header" empty, so returning 'false' is the Right

Oh.. I misunderstood the remarks then (and then assUmeD too much about
the code snippet not working).  I nearly added some words about how I
wasn't absolutely sure from the textual description what the test was
looking for (in the presence of of the "degenerate" example), but I
didn't since it seemed clear enough to me -- but in the opposite of the
intended meaning, apparently.  In my sight, yes, there is a given header
field there that is empty, and thus the test should return true (to
match the narrative).  I don't really see "the Cc: header" -- I see two
"Cc: header fields" and I would indeed view one of their bodies as
empty.  It's hard to account for the way different people look at
things, especially when English is involved.

>         not header :matches "Cc" "?*"

I would still come up with the same (opposite of what was intended)
interpretation of the new phrasing, although I'd probably be less sure
of it :).  There's still the ambiguity between "all" and "any" (or
"some" and "none", etc) that can lead to different interpretations.  And
I think replacing "empty" with a more complex phrase obscures the
meaning-- "empty" or "blank" should be pretty clear in the narrative.  I
would find this a little clearer:

    Testing whether there is no appearance of a non-empty field body for
    a given header name can be done using a negated header test:

although still clunky.

> That's equivalent to what's in the -05 I-D.

Hah.  Yes, indeed.

mm

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Wed, 30 Nov 2005 17:01:04 +0100

> >I'm repeating it once more so Philip may take note of it :-).

thanks!  (sorry, didn't mean to imply you weren't paying diligent
attention.)

> 	not header :matches "Cc" "?*"

this seems spurious in the spec text, IMHO, since there is no rationale
for why the text was added.  how about:

        The following test will return false when the header is absent:

        header :is "Cc" ""

        Testing whether a given header is either absent or contains
        nothing but whitespace can be done using a negated "header"
        test: