Subject: more 3028bis comments

From: "Mark E. Mallett" <[email-masked]>
Date: Tue, 5 Jul 2005 16:52:00 -0400

Here's a few more comments about 3028bis (3028bis-02 specifically,
which is the latest one that I have seen).  Sorry if any of this
has already been addressed.

2.2.     Whitespace

   > Whitespace is used to separate tokens.  Whitespace is made up of
   > tabs, newlines (CRLF, never just CR or LF), and the space character.
   > The amount of whitespace used is not significant.

This is specifically about whitespace in the Sieve language.  So... how
many implementations violate this?  i.e., does everyone generate an
error if a script's whitespace contains a naked CR or LF ?

2.4.2.4. MIME Parts

   > In a few places, [MIME] body parts are represented as strings.  These
   > parts include MIME headers and the body.  This provides a way of
   > embedding typed data within a Sieve script so that, among other
   > things, character sets other than UTF-8 can be used for output
   > messages.

This still confuses me.  Where is it referenced?

2.8.     Blocks

   > A control structure is a command that happens to take a test and a
   > block as one of its arguments; depending on the result of the test
   > supplied as another argument, it runs the code in the block some
   > number of times.

This is inconsistent since "require" and "stop" are defined to be 
control structures, and neither of them take a block as an argument.
Either reword the above, or change 3.2 and 3.3 so that they don't
use the word "structure" (with perhaps an intro in '3' defining
the use of "control structure" vs just "control").

2.9.     Commands

   ...

   > A control command is similar, but it takes a test as an argument, and
   > ends with a block instead of a semicolon.

same sort of thing as previous comment.  Maybe more like:

   A control command is a command that affects the flow of execution of
   the Sieve script in some way.  One form of a control command is a
   "control structure" which ends with a block instead of a semicolon.

   > A test command is used as part of a control command.  It is used to
   > specify whether or not the block of code given to the control command
   > is executed.

similarly, maybe say "control structure command"

2.10.5.  Extensions and Optional Features

   > Because of the differing capabilities of many mail systems, several
   > features of this specification are optional.  Before any of these
   > extensions can be executed, they must be declared with the "require"
   > action.

"require" is defined to be a control.  So... '"require" control' rather
than '"require" action'.

2.10.6.  Errors

   > When an error happens, implementations MUST notify the user that an
   > error occurred, which actions (if any) were taken, and do an implicit
   > keep.

Probably extremely nit-picky, but I've always wondered when I read
this "what user, and how do we notify them?"  Some users have no
access to anything other than their mailboxes.  I suspect that many
implementations will do some system-wide logging, which notifies the
admin, but not the user.

3.      Control Commands
3.2.     Control Structure Require
3.3.     Control Structure Stop

The whole "structure" thing mentioned above.

3.1.     Control Structure If

   > Syntax:   if <test1: test> <block1: block>
   >
   > Syntax:   elsif <test2: test> <block2: block>
   >
   > Syntax:   else <block>

Shouldn't this be "else <block3: block> ?

5.1.     Test address

   > Example:  if address :is :all "from" "[email-masked]" {
   >              discard;

recommend adding the  closing "}" here 

5.8.     Test not

   > Syntax:   not <test>

Shouldn't this be something like  "not <test1: test>"  ?

Other random comments:

The spec provides no way to access the "From_" line (which is the
"From<sp>" line with no colon that is added by some mail software.
While it's not part of RFC2822, and admittedly a minor concern at
best, that 'header' is often there, but inaccessible.  I have no
solution, other than perhaps making "From_" a special header name.

There was some discussion about adding character escapes (\u etc);
this probably falls under the "substantive changes or additions"
prohibition, but maybe not.

The 'variables' extension has to specify greedy/non-greedy ":matches" ;
this really ought to be in 3028 so that extensions can consistently
follow whatever rule there is; although, again, it might fail the
"non-substantive" test.

Yours,
-mm-

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Wed, 06 Jul 2005 00:48:12 +0200

> error if a script's whitespace contains a naked CR or LF ?

does it matter?  this text is unchanged from RFC 3028, so it will not
affect conformance.

> This still confuses me.  Where is it referenced?

I think it's only "reject" in the original spec.  I agree the text is
confusingly placed even in RFC 3028, and now that reject has been moved
to a separate document, the above text should follow along.

> implementations will do some system-wide logging, which notifies the

good question.  delivering an error report to INBOX in addition to the
message is one way of solving this, but does anyone do this?

> solution, other than perhaps making "From_" a special header name.

we have envelope "from" for the e-mail address.  we don't have anything
for the delivery date, but I suspect Ned is considering that for his
"date" extension.

> prohibition, but maybe not.

Philip has tried to pave the way for it:

   A quoted string starts and ends with a single double quote (the <">
   character, ASCII 34).  A backslash ("\", ASCII 92) inside of a quoted
   string is followed by either another backslash or a double quote.
   This two-character sequence represents a single backslash or double-
   quote within the string, respectively.

   Scripts SHOULD NOT escape other characters with a backslash.

   An undefined escape sequence (such as "\a" in a context where "a" has
   no special meaning) is interpreted as if there were no backslash (in
   this case, "\a" is just "a").

this allows future extensions to define a special meaning for "\a".

(you can find the discussion on this around 2005-03-08)

> follow whatever rule there is; although, again, it might fail the

I don't think this behaviour description is relevant for the main spec.
it can not be observed without the variables extension.
-- 
Kjetil T.

---

From: Ned Freed <[email-masked]>
Date: Tue, 05 Jul 2005 17:46:18 -0700 (PDT)

> does it matter?  this text is unchanged from RFC 3028, so it will not

I see no "MUST generate an error" clause here, so as far as I'm concerned the
handling of bare CR and bare LF means you have a nonconformant script, but how
implementations handle this particular form of nonconformance is undefined.

Attempting to nail this down to a "MUST generate an error" would be a huge
mistake IMO, given the use of bare LF and bare CR as the default line
terminators on many systems. And I really don't want to repeat the whole
"canonical format versus local storage format" discussion again, thank you very
much. So let's let this particular sleeping dog lie, OK?

> message is one way of solving this, but does anyone do this?

We do. Our implementation has the concept of a "responsible address" that's
attached to each sieve script. When an error occurs this address is notified of
the problem. For user sieves this is the user's own address, system sieves have
the main system postmaster as the responsible address, and so on.

Of course the delivery of such notifications has to be done in a way
that doesn't go through sieve processing and generate an exponentially
growing loop.

> for the delivery date, but I suspect Ned is considering that for his

Actually I'm not. The current time, yes, but since the time of sieve evalaution
isn't exactly specified in terms of its relationship to final delivery it
doesn't make sense to have this as part of a general date extension. It would
at best be another, separate extension.

> it can not be observed without the variables extension.

Yes, exactly so. This could easily come back to bite us when we need to move to
draft.

				Ned

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Wed, 06 Jul 2005 08:14:23 +0200

> at best be another, separate extension.

true, it would be best to make it a separate extension so that "date"
deployment is not hampered.  the "deliverydate" extension could probably
be specified by adding a couple of paragraphs to the "date" document,
though.  whether that feels natural we'll see later.

> Yes, exactly so. This could easily come back to bite us when we need to move to

sorry, I'm dense.  are you agreeing with Mark or with me?

I don't worry about this issue since I trust we will be able to avoid
publishing extensions with incompatible behaviours in the future, even
when the extensions don't depend on each other.  there will always pop
up be cases such as this, and we can't get all such behaviours described
in the base spec.
-- 
Kjetil T.

---

From: Ned Freed <[email-masked]>
Date: Wed, 06 Jul 2005 12:31:40 -0700 (PDT)

> sorry, I'm dense.  are you agreeing with Mark or with me?

With you.

My worry here is that requiring non-greedy in the base specification is, as you
put it, requiring nonobservable behavior that cannot be externally verified.
When documents go to draft we at a minimum will have to write up a list of
conformance criteria and then make sure we have enough conforming
implementations. Including non-observable criteria on the list means that
the only person who can submit information about a given implementation
is someone able to see the code for that implementation. This would have
been a problem for several past conformance list exercises.

				Ned

---

From: "Mark E. Mallett" <[email-masked]>
Date: Wed, 6 Jul 2005 17:22:36 -0400

> put it, requiring nonobservable behavior that cannot be externally verified.

The issue I had in mind was that of setting precedent; now that we know
that an extension wants to define the greediness behaviour, it seemed to
me that it would have been best to have that precedent set in the base
document.  Otherwise, unrelated extensions could simply state what
behaviour they require, with one contradicting another.  I imagine that
in practice this would be unlikely; precedent set by one extension is
likely to be followed by another.  Nevertheless, since we know that this
definition is needed by an extension, it feels much better to me to
specify it in the base document.  So I'm looking for neatness more than
anything.  Also, having it there, even if the behaviour is not visible,
makes it easier for an implementation coded to that specification to be
able to support extensions that do make it visible.

This may or may not be outside the charter; again it's just something I
wanted to bring up while it's possible to talk about it.

> been a problem for several past conformance list exercises.

I too am probably being dense... but in another message you said, regarding

> implementations handle this particular form of nonconformance is undefined.

which seems to be somewhat opposite reasoning, justifying the
specification of something entirely on the basis that nonconformance
can't be observed.  I'll probably reply to that separately, tho.

mm

---

From: "Mark E. Mallett" <[email-masked]>
Date: Wed, 6 Jul 2005 17:37:56 -0400

> much. So let's let this particular sleeping dog lie, OK?

Oh- I had forgotten that you asked that, when I wrote my other response,
sorry.  I have a hard time letting go that the spec mandates something
that is probably impossible to follow, and likely derives from something
that's not relevant (i.e., wire format).  But other than making this
side comment, I will indeed let it go :-)

> > message is one way of solving this, but does anyone do this?

I do something like that, in a way that's not very clean and I'm not
very proud of, which is one reason I asked the question..  I file any
error message text along with the same message that caused it.  (We also
have a personal logfile facility, but that's not guaranteed to be accessible
by the end user, nor is the end user likely to check it very often.)

> the main system postmaster as the responsible address, and so on.

That sounds like an interesting idea and is probably worth pursuing as
an alternative.  Thanks..

> > for the delivery date, but I suspect Ned is considering that for his

I was talking about accessing the header field in same ways that others
can be accessed, not other ways of getting at the specific parts of that
field that we might predict people might want.  I've seen occasions
where script writers either want to suppress the line if supplied, or
add one if not, e.g.:

    if exists "From_" { deleteheader "From_" };

as a very simple case.  Knowing that you can't access that (admittedly
fake) is a thing by itself, without having to say all the things that
you might want to do with it.  Anyway it was simply on my collected list
of known issues that I was looking at while making the posting.

-mm-

---

From: Philip Guenther <[email-masked]>
Date: Wed, 06 Jul 2005 15:53:31 -0700

...
>> We do. Our implementation has the concept of a "responsible
>> address" that's attached to each sieve script. When an error
>> occurs this address is notified of the problem. For user sieves
>> this is the user's own address, system sieves have the main
>> system postmaster as the responsible address, and so on.

Sendmail's implementation does that too.

>field that we might predict people might want.

The "From " line is neither a header field nor permitted by RFC 822
or RFC 2822.  It's simply part of a mailbox format that dates back
to V7 UNIX.  Indeed, it is the bane of UNIX mail software to deal
with that wart and treating it as part of the message header is,
IMHO, a mistake as it complicates handling of other mailbox formats
or even of resending message.

For example, procmail _usually_ (but not always!) inserts or updates
a "From " line before processing any rcfiles.  It also tries to
strip that line before forwarding the message.  However, it fails
to remove it when saving the message in mailboxes whose format does
not call for (or permit!) a "From " line, such as MH or maildir.
It also fails to enforce the presence of a "From " line when saving
to V7-format mailboxes, meaning that a poorly written rcfile can
remove or munge it, resulting in a broken mailbox.

Confusing framing info with data is a mistake.  Providing access
to incoming framing info is one thing, as is providing a means to
set it on output (in those cases where it is used), but that should
not be done by the backdoor of an invalid field name.

>I've seen occasions
>as a very simple case.

All the cases I've seen of people deleting "From " lines fall into
three groups:

1) the @($* software isn't removing the "From " line when saving
   the message in other formats or when feeding the message to
   another program
2) the *#@( software is letting through a bad "From " line from
   'upstream' and has to be forced to generate a new, correct one
   manually by deleting the bogus one
3) the user is confused as ends up breaking their V7 mailboxes

All three of those cases are best handled by fixing the softwware
to strip "From " lines on input, perhaps after taking a stab at
extracting an address and date from them for later use, and then
only generating a new "From " line in output when writing directly
to a V7 mailbox or when otherwise directed to.

Philip Guenther

---

From: Ned Freed <[email-masked]>
Date: Thu, 07 Jul 2005 12:25:54 -0700 (PDT)

> Sendmail's implementation does that too.

Good to hear.

> or even of resending message.

Agreed. It's what X.400 would call "delivery info", a remnant of what was in
the envelope at the time of final delivery. So, to the extent it contains
useful information, it should be treated as envelope material, not header
material.

> not be done by the backdoor of an invalid field name.

Exactly so.

				Ned

---

From: Ned Freed <[email-masked]>
Date: Thu, 07 Jul 2005 13:09:06 -0700 (PDT)

> document.

I disagree. It is far more important that specifications specify the minumum
necessary and not specify stuff that cannot be verified externally.

> Otherwise, unrelated extensions could simply state what
> likely to be followed by another.

And to the extent it possible for one specification to contradict another, it
is also possible for an extension to unwittingly contradict the base
specification. Irrespective of the liklihood of such a contradiction occurring,
the increased likihood of it happening because of the location of one of the
contradictory statements is bound to be marginal.

> Nevertheless, since we know that this

But it may make it harder to advance the base to draft. And advancement to
draft is what we're supposed to be focused on per the charter.

> can't be observed.  I'll probably reply to that separately, tho.

First, it is axiomatic that conformance criteria only apply to defined
behavior. Undefined behavior cannot be something you test conformance to.
Second, advancement to draft involves checking for sieve implementation
compliance; the existance or nonexistance of incompliant scripts in the wild
isn't particularly relevant unless, I suppose, it could be shown that all
scripts out there are incompliant.

				Ned

---

From: Philip Guenther <[email-masked]>
Date: Fri, 08 Jul 2005 00:08:56 -0700

>This still confuses me.  Where is it referenced?

I don't think it is.  Indeed, it don't think it was in RFC 3028 either.
The 'vacation' extension uses something like this with its ':mime'
keyword, but it has enough of a definition there that I don't think the
text in 3028bis is needed.  Everyone say "bye" to section 2.4.2.4...

>the use of "control structure" vs just "control").

Yeah, calling just the subset of control commands that have a block
argument "control structures" seems the way to go.  I'll make the
wording tweaks.

...
...
...
>Shouldn't this be something like  "not <test1: test>"  ?

I agree.  Done.

>solution, other than perhaps making "From_" a special header name.

See my other reply for why I disagree with this.

>prohibition, but maybe not.

Right.  The discussion on the list following the Minneapolis meeting
concluded that such a change should be left an extension, if anyone
wants to define one, and that the base spec should simply recommend that
scripts avoid superfluous backslashes.

>follow whatever rule there is; although, again, it might fail the

I agree with Kjetil and Ned: this is only observable in the presence of
such extensions, so it should be left to them.  Indeed, two extensions
could define the opposite semantics if they had separate means of
observing the results and while it would be a pain to implement, it
wouldn't be impossible.

(I somehow doubt the latter such extension would pass its last-call if
it didn't have a superb justification for the different choice.  The
base-spec doesn't have to ban annoying extensions to keep them from
happening; alternatively, given the efficacy of the previous ban on
side-effects-in-tests, we should be disinclined from going that path
again.  A ban is neither necessary, nor sufficient.)

Philip Guenther