Subject: Re: draft-degener-sieve-editheader-01.txt

From: "Mark E. Mallett" <[email-masked]>
Date: Wed, 7 Jan 2004 18:03:04 -0500

Two very interesting changes in this draft:

>    	"addheader" [":last"] <name: string> <value: string>
> 
>    By default, the header field is inserted at the beginning of
>    the existing header.  If the optional flag ":last" is
>    specified, it is appended at the end.

What's the rationale?  So far (i.e., without hearing more), I liked it
better the way it was: the default being to add the header at the end,
not the beginning.  I would not mind seeing a ":first" to alter that
behaviour though.

and:

>    A message modified by addheader or deleteheader MUST NOT
>    be considered the same as the original message unless it
>    matches the original message exactly.
>    
>    For example, the following code fragment
> 
>    	keep;
> 	addheader "X-Flavor: vanilla"
> 	keep;
> 
>    files two messages, not one.

I see where that is coming from.  If you do a "keep" after changing
the header, it's assumed that you really meant it; and if weeding out
duplicate "keep" operations prevents that, then you have lost some
change.  However: you have not lost the message unless some prior
action had deliberately caused it; any weeding out represents a script
error rather than loss of the message, so I am not sure this is an
improvement.

On the flip side, anyone who has depended on duplicate "keep"
operations being weeded out will be surprised by having two messages
filed.  Likely if you have done a "keep" in the same code area as the
addheader, you probably do want the second copy.  But in another
case?  Not so clear, especially if you have done something else with
the message after changing the header.

One alternative: make a note that the message has changed, and if so,
inhibit any subsequent de-duplication logic on the next action only.
So the next "keep" would succeed unless (e.g.) "redirect" were done
first.  This would introduce a new "changed" flag, but this new text
introduces that concept anyway.  That's not as perfect as, say, views,
but it's no worse than the suggested one.

BTW, in the absence of a "changed" flag, this new philosophy might also
imply that implicit keep is once again turned on (if it had been
turned off).  Whether it is or is not is now ambiguous.

And.. what about replaceheader?  I would assume that any change made
by replaceheader would also cause the message to be considered
"different" -- but that new text explicitly calls out "editheader" and
"deleteheader" without mentioning "replaceheader," so it seems
deliberate.

Yours,
mm

---

From: Jutta Degener <[email-masked]>
Date: Wed, 7 Jan 2004 16:53:37 -0800

> not the beginning.  I would not mind seeing a ":first" to alter that

It's much easier to insert a header field at the beginning of
a header (one simply sends that header field, then the text of
a message).  Perhaps as a result of that, the fields at the
beginning of the message have taken on the character of a
delivery "trace", while the fields at the end of the block
typically were inserted by the client.

Some people who talked to me felt very strongly that additions
to the trace section in the right place were more in keeping with
what's considered allowable header modifications, while inserting
in the middle, with no way of tracking who did that, should only
happen if explicitly requested.  As one of the people who occasionally
has to decipher the traces to figure out where some problem originates,
that made intuitive sense to me.

> error rather than loss of the message, so I am not sure this is an

In an ambiguous and possibly erroneous situation, I prefer to err
on the side of shorter explanations and more saved information.

> inhibit any subsequent de-duplication logic on the next action only.

That seems difficult to explain.  I don't see why the first of
any number of duplicate fileinto's etc. should receive preferential
treatment.

> turned off).  Whether it is or is not is now ambiguous.

It is my intention that the implicit keep should _not_ be restored
after a header modification.  I'll weaken the section on weeding out
duplicates to (hopefully) be less likely to inspire that idea.

> And.. what about replaceheader?

As mentioned in section 9.1, replaceheader is no longer in the draft.
My implementation still has it, but it's a vendor-specific extension
after encountering determined opposition from some members of the
work group.  If you implement it, the rules with respect to duplication
would be the same as for add- and deleteheader.

Jutta <[email-masked]>

---

From: "Nigel Swinson" <[email-masked]>
Date: Thu, 8 Jan 2004 02:59:11 -0000

> > >    files two messages, not one.

The above represents a problem for those of us who have read section 2.10.6
of 3028 and decided to do a full execution of the script, then apply all
actions.... if we come across a script like this:

 keep;
 addheader "X-Flavor: vanilla"
 keep;
 addheader "X-Flavor: strawberry"
 keep;
 addheader "X-Flavor: chocolate"
 keep;
 addheader "X-Flavor: pistachio"
 keep;
 addheader "X-Flavor: bannana"
 keep;

And we are filtering a large message (Say 20MB), then the simple and obvious
implementation is going to have to freeze a copy of the message every time
it sees fileinto/redirect/keep and then if it sees addheader/removeheader it
will need to create a copy of the message.  This means that with the above
script we'll end up with 6 different copies of the message = 120MB of
memory/storage.  Sounds like a potential security hole.

So it seems that as it is written, editheader is putting strong bias on
"take actions as you go" implementations.  Do we want do do this?

I can't see that the above functionality is really all that useful, so at
the moment I'm inclined to say that we move more towards section "2.10.3
Message Uniqueness in a Mailbox" of 3028 which says:

   Implementations SHOULD NOT deliver a message to the same folder more
   than once, even if a script explicitly asks for a message to be
   written to a mailbox twice.

I think it makes it too complicated to consider there to be any more than
one "message" in context when processing the script.

So what should the keep/addheader/keep script do?  I recon it files one copy
of the message into the inbox, and the implementation is free to either add
the version with or without the header.  If the user really wants to keep a
copy that does NOT have the header, then they do keep/stop.  If they want
one with the header they do addheader/keep.

I know some will probably not like the above, but then 3028 already permits
implementations to operate differently WRT to some situations like catching
errors.  ie an "action-as-you-go" with a fileinto/reject/reject script may
file the message before producing an error, so I think it would be ok for
this slightly peculiar case to operate in an implementation specific way.

Nigel

---

From: Jutta Degener <[email-masked]>
Date: Wed, 7 Jan 2004 20:03:23 -0800

On Thu, Jan 08, 2004 at 02:59:11AM -0000, Nigel Swinson wrote:
> "take actions as you go" implementations.

What was intended was merely a strong bias for a "take
actions as you go" *model*.  I think that's easier to explain
and remember for users.

As an implementor, you can postpone header edits in the same
way you postpone other commands; you just need to keep track of
what the header looks like right now, and remember which
deliveries picked up which modifications.

Jutta

---

From: "Mark E. Mallett" <[email-masked]>
Date: Thu, 15 Jan 2004 18:55:35 -0500

On Wed, Jan 07, 2004 at 04:53:37PM -0800, Jutta Degener wrote:

Hi-  thanks for the responses.  Sorry it's been over a week.

> typically were inserted by the client.

I don't look at it that way, myself.  Not to mention that I don't
think "ease" has anything to do with it in a sieve application: that
application is manipulating the message, and so putting the header one
place is no more difficult than putting it another.  I do follow that
you are explaining why adding at the beginning is a precedent.  However
it's a precedent for a different context.

> happen if explicitly requested.

I would disagree with those people :-)  We're not talking MTA here nor
are we talking about trace fields: we're talking about manipulating a
message where there is the presumption that one has the right to do
so, in the same way that one can go into a text editor and manipulate
it.  Admittedly there is a quasi- area where a sieve filter can be
applied in a forwarding context.  But in the (more common) case where
the filter is operating on behalf of the recipient, I would want to be
able to do the changes I want done.  (kind of a tautology there.)

> on the side of shorter explanations and more saved information.

I think the shortest explanation would be not to introduce the
"changed message" concept.  One might say that if one has done a "keep"
then changed the message then did a "keep" again, an error can occur no
matter how you look at it, so keeping the traditional de-duplication
logic is better.  I find this new potential for duplicate filings
disturbing.

> any number of duplicate fileinto's etc. should receive preferential

Because of what I said in the original message:

     On the flip side, anyone who has depended on duplicate "keep"
     operations being weeded out will be surprised by having two
     messages filed.  Likely if you have done a "keep" in the same
     code area as the addheader, you probably do want the second
     copy.  But in another case?  Not so clear, especially if you have
     done something else with the message after changing the header.

I guess I wasn't clear with that.  I was trying to say that it's likely
that if you do an immediate "keep" right after the addheader, you
probably meant to keep the changed message.  But if there's a "keep"
far away further in the script, you probably want the de-duplication
to happen (i.e., supress that other far-away "keep").  I was trying to
invent some reason why you would really want the second keep to file
the message again (after a change), and contrast that with what I
consider to be the standard case where you don't want it to happen, and
then come up with a generic rule to infer intent as expressed in the
language.  Perhaps a better inference would be:  if there is a header
modification followed by an action within the same code block (block
enclosed by curly braces), ignore the de-duplication and perform the
action anyway.

The bottom line is that I don't like the part where normal de-duplication
is inhibited.  I'd vote for getting rid of the new stuff about the changed
message, and if that prevents the filing of a changed message, fix the
script :-)

> would be the same as for add- and deleteheader.

Hmm- certainly seems like replaceheader is a convenient shortcut
for "deleteheader" and "replaceheader" -- except that you lose
position, which I consider a loss (for reasons given above).
(As I recall, earlier discussion included some finer control over
header placement; again, losing that is, it seems to me, a loss.)

This response brings up a couple new questions too, which can only
be explained by my ignorance I'm afraid, but here goes:

Is there a separate work group for "editheader"?  I had thought from
the archives that it open for discussion here.  What other work groups
are there?

How does a vendor-specific extension get specified vis-a-vis "require" ?
And in that light, could not the rules for duplication also become
"vendor-specific" ?

Yours,
mm

---

From: Jutta Degener <[email-masked]>
Date: Thu, 15 Jan 2004 16:50:14 -0800

> But in the (more common) case where

I don't think we have enough data to judge which use of
this proposed extension is common.

> position, which I consider a loss (for reasons given above).

You lose count, too; if multiple headers of the same sort are present,
you can't generally rename them (e.g., Cc to Resent-Cc) because you
can't loop in sieve.

> Is there a separate work group for "editheader"?

There's no IETF work group (anymore) for sieve -- so all extensions are
formally private submissions, although they do tend to refernce to this
mailing list as a point of discussion.

> How does a vendor-specific extension get specified vis-a-vis "require" ?

RFC 3028, section 6.1.  Their name starts with "vnd."

> And in that light, could not the rules for duplication also become

In general, I don't think that's a good way of dealing with conflicting
opinions about details of an implementation.

Jutta

---

From: "Mark E. Mallett" <[email-masked]>
Date: Thu, 15 Jan 2004 21:52:44 -0500

> this proposed extension is common.

I was referring to common use of sieve, per its stated design as a
final delivery agent.  However, I did allude to the possibility that a
sieve filter *could* be used in a pass-through mode, since you were
the one who was talking about MTA-like properties providing precedent
for header placement, and I was making some conjectures about where
that perspective was coming from.  My contention is that in a local
delivery mode, one ought to be able to do to the mail what one wants
to do to it.  As I said, I could see where there could be more
constraints on a filter that was operating in other than local
delivery mode.  However I think it would be a shame to cripple a local
delivery agent for that reason:  it would be better to define an
operational mode that had more constraints if one wanted.

> you can't generally rename them (e.g., Cc to Resent-Cc) because you

While that is true (no loops), I'm not sure it's a real answer.  There
are ways to address that concern, a couple of which you illustrated in
a previous draft (nicely showing its usefulness).  Rename them all,
rename the nth one, rename the ones matching a pattern, per prior
draft.  There certainly could be situations where replaceheader can't
do what you want: but that's no reason to remove it.

> RFC 3028, section 6.1.  Their name starts with "vnd."

That's not really what I was getting at.  I'd want to see
"replaceheader" retained in the draft, even if it's as a "MAY" with a
separate capability name (a la the "envelope" precedent) rather than
punted like that with no chance for uniform implementation of optional
facilities.  Certainly anything can be incorporated in a private
implementation, but that's not this discussion.

> opinions about details of an implementation.

My point too :-)

mm

---

From: Philip Guenther <[email-masked]>
Date: Thu, 15 Jan 2004 20:35:51 -0800

"Mark E. Mallett" <[email-masked]> writes:
... [regarding default placement of added header fields]
>My contention is that in a local

Given that the draft *does* provide a means to do what you want
(place the added header field at the end of the header instead of
the beginning), calling an agent that follows the draft 'crippled'
seems like pure hyperbole.  Can you refute the previously provided
reasons for defaulting to prepending or otherwise provide a
countervailing argument?

... [regarding replaceheader]
>There certainly could be situations where replaceheader can't

It wasn't removed because it didn't do what was wanteded; it was
removed because people felt it too complicated and had various
hairy problems (for example, involving rfc2047-encoded words).

...
>implementation, but that's not this discussion.

The question isn't whether replaceheader should be described in an
rfc.  The question is whether addheader and deleteheader should be
held up while the people try to reach consensus on replaceheader.
It's one thing to combine separable extensions in a single document
when they're all ready to go at the same time, but that isn't true
here.

Philip Guenther
[email-masked]

---

From: "Mark E. Mallett" <[email-masked]>
Date: Fri, 16 Jan 2004 01:46:09 -0500

On Thu, Jan 15, 2004 at 08:35:51PM -0800, Philip Guenther wrote:
> seems like pure hyperbole.

I didn't say that last phrase, so it's hard to respond to it.  An agent
that follows this draft in either of its last two forms is not
crippled.  I didn't even say (or intend to say, though I think I can
see how it got inferred) that the draft is "crippled" by the default
being the beginning rather than the end.  I was talking about the line
of reasoning leading up to this point:  i.e., that there is a
precedent set by MTAs prepending headers, and MUAs appending them.  I
said that a SIEVE filter (certainly by one definition) can commonly be
a local agent acting on behalf of the mailbox owner.  I said that I
could imagine another mode where it was acting more MTA-like, and that
perhaps a mode could be defined where the filter was not allowed such
a free hand (since it would be operating on mail *other* than that
owned by the person directing it).  (In fact I think it would be
useful to make that role distinction when talking about what a filter
can and can not do.)  I then said that that hypothetical mode should
not be the one that dictated the way drafts (or agents) were written;
if it were, it would be restrictive (i.e., crippling) for the LDA
role.  It's a few steps back from that philosophical observation to
the specific item in the draft, sorry about the apparent connection.

> Can you refute the previously provided

Not "refute" -- no.  Disagree with, yes, but really only mildly, and
I've outlined why a few times.  If I wasn't clear I can try again, but
I doubt that people need to have me belabor it.  For reasons I've
already given I feel that appending is better for a final delivery
agent.  A draft stage is where one puts in these opinions so I am:
obviously other(s) felt similarly motivated, though in the other
direction, between the last two drafts.  I would even suggest having
the default be implementation-dependent, and if one wanted to be
explicit one would use :first or :last .

> when they're all ready to go at the same time, but that isn't true

Yeah- that's one reason I was wondering where this discussion had
taken place up to now (between the last two drafts, that is).  If
you've been involved in discussing it you have a different perspective
than I, who only knows that it is gone and wonders why, and wonders
what will become of it.  I certainly understand the goal of pragmatism,
especially if there's conflict over "replaceheader".  But it's also
a good goal to have an extension that encapsulates all related functions.

Yours,
-mm-

---

From: "Matthew Elvey (FM)" <[email-masked]>
Date: Tue, 20 Jan 2004 11:49:11 -0800

My belated two cents - just that; I don't feel strongly; no responses 
necessary.

Let's keep it simple.  Wow, I'm not aware there's good precedent for 
MUAs appending instead of prepending.  I thought they prepended, like 
good MTAs do. I'd like to see addheader put the added header at the top, 
where new headers are *supposed* to go (or so I thought), anyone have a 
compelling real case where :last is needed?

>    matches the original message exactly.

I'd like to see replaceheader reconsidered for the spec, or at least any 
objections to it made public; the standards-setting process must be public.

This does apply the KISS principle because it'll mean an otherwise 
imminent vendor-specific implementation will be unnecessary.
******
FYI: Progress is being made on the draft for the "refuse" action; Alexey 
and I should post a draft soon; we've worked through a bunch of issues 
already.

---

From: Cyrus Daboo <[email-masked]>
Date: Tue, 20 Jan 2004 15:10:59 -0500

Hi Matthew,

--On Tuesday, January 20, 2004 11:49 -0800 "Matthew Elvey (FM)" 
<[email-masked]> wrote:

| Let's keep it simple.  Wow, I'm not aware there's good precedent for MUAs
| appending instead of prepending.  I thought they prepended, like good
| MTAs do. I'd like to see addheader put the added header at the top, where
| new headers are *supposed* to go (or so I thought), anyone have a
| compelling real case where :last is needed?

Most of the mailing list software I have come across appends List- headers 
rather than prepends them. Same thing for anti-spam/anti-virus warnings 
etc. One can argue whether those types of tools are MUAs or MTAs by some 
definition, but I think its reasonable that append is the default.

-- 
Cyrus Daboo

---

From: Cyrus Daboo <[email-masked]>
Date: Tue, 20 Jan 2004 15:31:38 -0500

Hi Cyrus,

--On Tuesday, January 20, 2004 15:10 -0500 Cyrus Daboo <[email-masked]> 
wrote:

| but I think its reasonable that append is the default.

I meant to say that I think it is reasonable that append (:last) is 
available as an option - I don't want to argue that it should be the 
default.

I personally would likely always use :last as I frequently have to examine 
the chain of Received headers for debugging purposes and having other 
headers interspersed would be annoying. Right now the CMU sieve 
implementation does add an X-CMU-Sieve header and that does appear before 
the final delivery Received header as their SIEVE implementation sits 
between SMTP and LMPT.

-- 
Cyrus Daboo

---

From: Jutta Degener <[email-masked]>
Date: Tue, 20 Jan 2004 13:53:28 -0800

> >   matches the original message exactly.

At this point, I'd like to throw this out as well.  Anyone have
a compelling reason not to?

Personally, I'd like it to be replaced by an explicit statement
saying that these added headers do *not* make the message different.

Should we say that
[ ] the first fileinto wins, say that
[ ] any one fileinto may win (but it has to be one of
    the actual fileintos - can't have just half the
    headers or additional headers),
[ ] the last fileinto wins,
[ ] or leave it completely open?

I'd like one of the first three -- I think if it is one of the
fourth, it starts weakening the connection between message modification
and message use, and I'd like that connection to be a strong one.

> objections to it made public; the standards-setting process must be public.

As I remember it, Ned Freed was vocal and clear on the list about his
intention to do everything in his powers to stop replaceheader, to
general mumbling approval.  Maybe I misinterpreted something?

Jutta

---

From: "Nigel Swinson" <[email-masked]>
Date: Wed, 21 Jan 2004 00:27:45 -0000

> saying that these added headers do *not* make the message different.

Great!  I'd like it thrown too, but it does sounds a little bizzare to do put in the counter statement, as we all know that it isn't actually the same message.   If we are throwing it out, doesn't it make any of the following 4 options correct?
 
> and message use, and I'd like that connection to be a strong one.

Well action-as-you-go implementations are going to vote for 1, actions-all-at-the-end impelmentations are going to vote for 3, and I think we need to make it equally feasable to implement either way which means I think we have to have either option 2 or 4.  So that makes me vote for option 2?

Nigel

---

From: "Mark E. Mallett" <[email-masked]>
Date: Wed, 21 Jan 2004 10:53:59 -0500

On Wed, Jan 21, 2004 at 12:27:45AM -0000, Nigel Swinson wrote:
> Well action-as-you-go implementations are going to vote for 1,
> actions-all-at-the-end impelmentations are going to vote for 3, and I
> think we need to make it equally feasable to implement either way
> which means I think we have to have either option 2 or 4.  So that
> makes me vote for option 2?

That seems logical.

mm