Subject: Sieve include, 'global scripts' and managesieve

From: Aaron Stone <[email-masked]>
Date: Sun, 26 Nov 2006 13:46:44 -0800

Hi Cyrus and list,

One of the issues raised in San Diego was how to upload scripts meant to
be included with 'include :global'. A request that's been made more than
a few times on the DBMail mailing list is to have sieve scripts that are
always run during message delivery, controlled by the mail administrator
rather than individual users.

Has this idea been discussed before? Does anybody have an implementation
that runs 'global scripts' in this way?

I am inclined to believe that this is a good idea, and that basically I
need a dummy postmaster user to own the global scripts and mark one as
active, to be executed prior to the user's delivery scripts.

The special considerations I can think of are that the implicit keep
does not cross between the postmaster script and the user's script,
except in the case of a reject. That is, if the postmaster does a
fileinto, discard, keep, redirect, whatever, this affects only the
"postmaster's copy" of the message. A reject should trash the message
and cancel the user's delivery, however.

Hmm. Can of worms? Can we tease out a consistent behavior? Would
recommending a dummy postmaster user solve the managesieve question for
global scripts?

Thanks,
Aaron

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Sun, 26 Nov 2006 23:07:23 +0100

> recommending a dummy postmaster user solve the managesieve question for

sounds to me like you want to use this to duplicate e-mail, for SarbOx
archival purposes or whatever.  I don't think this is relevant for
Sieve.  make the copy in your MTA so that the archival account is a
firstclass citizen as far as Sieve is concerned.

-- 
Kjetil T.

---

From: Ned Freed <[email-masked]>
Date: Sun, 26 Nov 2006 14:10:53 -0800 (PST)

> Has this idea been discussed before?

Yes. There was even a draft (by Jutta if memory serves) describing how
to deal with multiple sieve results that apply to the same message.

> Does anybody have an implementation

Yes, our implementation does this. In our implementation there can be many
sieves associated with different subsets of the recipients of a message.
Basically it ends up being an inverted tree with each branch representing one
or more recipients. The so-called "system" sieve sits at the root, "channel"
sieves above that, moving on out to user sieves on the leaves.

After all the sieves are evaluated the result that ends up attaching to each
address is basically the first sieve along the path to the root that said to do
something besides an implicit keep. This way a user can override a system-level
action like discard, e.g. in the case of a per-user whitelist.

There are exceptions. We've found it necessary to implement some nonstandard
actions that can be used in system-level sives which then cannot be overridden
by users.

> active, to be executed prior to the user's delivery scripts.

We don't implement managesieve currently. Our user sieves are typically stored
in LDAP. System and channel sieves tend to live in files, although they can
also be placed in LDAP.

The main problem in the provisioning space is that the stuff you tend to do in
system sieves is somewhat differnet from what you do in user sieves.

> recommending a dummy postmaster user solve the managesieve question for

The approach Jutta documented was based on having multiple sieves, not on
having a single sieve that incorporates other sieves through some sort of
include mechanism. Our implementation ended up being semantically quite
close to what she described even through we didn't collaborate on the
details.

Whether or not this effort is worth reviving is another matter. It is far too
late for us to consider making any sort of significant changes to multiple
script semantics. Too many customers depend on things continuing to work
the way they do now, and who cam blame them?

				Ned

---

From: Tony Hansen <[email-masked]>
Date: Sun, 26 Nov 2006 18:16:22 -0500

I have a bunch of ditto's to add to Ned. Our implementation also has
global scripts, stored in LDAP, that are run very similarly to what
Jutta's draft specifies.

I always thought it would be worth publishing Jutta's spec, even if it's
only done as experimental or informational, so that people reinventing
this particular wheel will have past experience to fall back on.

	Tony Hansen
	[email-masked]

---

From: Aaron Stone <[email-masked]>
Date: Sun, 26 Nov 2006 15:28:46 -0800

> to deal with multiple sieve results that apply to the same message.

Ok, I just read draft-degener-sieve-multiscript-00, and it exactly what
I was looking for with respect to the 'system scripts'.

> action like discard, e.g. in the case of a per-user whitelist.

Ok, this makes sense.

> system sieves is somewhat differnet from what you do in user sieves.

Hence the non-standard actions that you mentioned? Could you give a few
examples of these actions?

> close to what she described even through we didn't collaborate on the

Right, there's two issues here, and they're sort of sharing the same
terminology. How about 'system scripts' and 'global scripts' where the
system scripts are scripts that are run during delivery, as you and
Jutta have described.

I'm thinking that if there's a user who owns the system scripts, then
that user might also be the owner of the scripts in Cyrus'
include :general namespace. Better than trying to create a #Public
namespace for Managesieve uploads :-P

> the way they do now, and who cam blame them?

You mean not changing your implementation of multiple script
semantics :-P That's ok if you don't want to change it, but I'm
interested in looking into it a bit more and documenting the execution
path if it generally useful to anybody else out there. Reviving Jutta's
multiscript draft certainly looks like an easy way to start working on
this again.

Aaron

---

From: Aaron Stone <[email-masked]>
Date: Mon, 27 Nov 2006 01:59:09 -0800

> firstclass citizen as far as Sieve is concerned.

I'm sure some of my users have this in mind, and others have other
things in mind. If I provide multiscript functionality, that's one
feature for me to write and many purposes they can use it for.

I do agree that the specific case of keeping a second copy of mail is
probably better done by an MTA hook, if possible, but I don't see any
reason to prevent a Sieve script from being used for this purpose. Let's
not hang the whole discussion on this one use case, however -- given
that Ned and Tony have both said that their implementations have some
form of multiscript, I think we can assume that it is useful enough to
look into some more.

Aaron

---

From: "Nigel Swinson" <[email-masked]>
Date: Mon, 27 Nov 2006 14:28:04 -0000

> global scripts, stored in LDAP, that are run very similarly to what

Likewise our product currently permits 5 different styles of sieve script.  One owned by the server administrator run before the message data is accepted through SMTP, one when the message reaches the domain, and the other 3 before the message data is written to the end users mailbox.  

Some actions operate differently depending on which script they are used in, for example fileinto in the server script files to the file system, not to a mail folder.

We found we needed a set of security related rules that the user could not opt out of, and a set of policy rules that they could, so the domain administrator controls a sieve script run immediately before, and one immediately after the end users' own script.  draft-degener-sieve-multiscript-00 documents a useful discussion of the complexities between "stop processing ANY more scripts" as opposed to "stop processing this script" which is certainly useful and might be worth keeping.

Nigel

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Mon, 27 Nov 2006 15:38:33 +0100

> form of multiscript, I think we can assume that it is useful enough to

but Jutta's draft is very different, if a fileinto is taken, the next
script will not be processed (there is no longer an implicit or explicit
keep).  the draft suggests that fileinto is disallowed to solve this
problem.  also note that a discard can not be overruled by the user.
(I'm a bit mystified by Ned saying his system is similar to Jutta's)

> and cancel the user's delivery, however.

you seem to want to change the semantics of fileinto so that it does not
cancel implicit keep, unless when run as the last script.  this sounds
very complex and confusing to me, and is the reason I suggest the
postmaster script works on a separate copy of the message.

hmm.  what *is* the prescribed behaviour of

   fileinto "INBOX.list";
   keep;

I couldn't find anything definite in the draft.  will this store two
copies?  if so, this idiom can be used to make a copy without cancelling
the keep.  how about

   fileinto "INBOX";
   keep;

clearly there will be only one copy due to duplicate suppression for a
single folder, but will it stay in an explicit keep state?  (a discard
will still have no effect, since it only cancel an implicit keep.)
-- 
Kjetil T.

---

From: Ned Freed <[email-masked]>
Date: Mon, 27 Nov 2006 08:39:50 -0800 (PST)

> examples of these actions?

They tend to be non-overrideable variants of existing actions. For example,
we have a jettison action that's like discard except that it cannot be
overridden by user sieves.

In retrospect I wish we'd done this as a :system or :nooverride parameter, but
we initially didn't see a reason for wanting this sort of behavior with more
than one action. But it too late for us to change now - the best we could do is
add support for an alternate approach.

> system scripts are scripts that are run during delivery, as you and

We already use the term "system scripts" this way, so this works for me.

> namespace for Managesieve uploads :-P

I don't have strong feelings about this either way, but I agree that a
naming convention would be useful.

> multiscript draft certainly looks like an easy way to start working on

I'm with Tony on this - the goal should be to document this space so
implementors have a place to start, not necessarily to have a standards-track
specification for it. Even if there weren't existing implementations,
implementations specifics are likely to make the exact semantics hard
to nail down.

				Ned

---

From: Aaron Stone <[email-masked]>
Date: Mon, 27 Nov 2006 09:56:09 -0800

> (I'm a bit mystified by Ned saying his system is similar to Jutta's)

I didn't read it the same way, but I'll look again to see this angle. It
doesn't make sense, so IMHO an updated draft would not specify the
confusement that you've identified.

> postmaster script works on a separate copy of the message.

I want the postmaster to be able to do a fileinto without cancelling the
keep *for the next script*. Ned has said that he has a reject variant
that includes the semantics of my paragraph above. I am not sure if a
separate flag or action is needed in the case of reject, but I think
it's an open issue.

> will still have no effect, since it only cancel an implicit keep.)

The system script would have some other set of mailboxes -- owned by the
system user. As Nigel mentions, he uses the filesystem for his system
scripts. The system script runs as the postmaster user. So this also
raises a question about how to do envelope matching, for example, with
Relational which prohibits matching against other people's envelopes to
discover who else has received the message.

Aaron

---

From: Ned Freed <[email-masked]>
Date: Mon, 27 Nov 2006 08:38:24 -0800 (PST)

> this particular wheel will have past experience to fall back on.

Agreed - experimental or informational would be great for something like this.

				Ned

---

From: "Nigel Swinson" <[email-masked]>
Date: Mon, 27 Nov 2006 19:02:17 -0000

> discover who else has received the message.

Most of the sieve specs so far are very targetted at the end user scripts.  For server administrator scripts, I see no reason why they shouldn't have access to the complete envelope.

So yes, it is an important point to note, that the "envelope" needs to be refined as the message progresses through the system.  So in our implementation, the server script has the full SMTP envelope to this server, but the end user only sees an envelope with a single recipient, as the envelope has at that stage been broken down into lots of smaller envelopes, one per recipient.

Nigel

---

From: Aaron Stone <[email-masked]>
Date: Tue, 28 Nov 2006 05:04:21 -0800

> So yes, it is an important point to note, that the "envelope" needs to be refined as the message progresses through the system.  So in our implementation, the server script has the full SMTP envelo…

This seems like the right approach. Ned, have you done the same thing?

Aaron

---

From: Ned Freed <[email-masked]>
Date: Tue, 28 Nov 2006 08:06:00 -0800 (PST)

> This seems like the right approach. Ned, have you done the same thing?

For user scripts yes, for system scripts no. The issue is how to handle
a system sieve that does an envelope recipient test - in this case
evaluation has to allow for the system sieve to return different results
for different recipients. We handle this by evaluating system sieves that
perform envelope tests once per recipient. In each of those evaluation it
appears that there's a single recipient address.

The alternative is to have a single result for system sieves that applies to
all recipients. If you did this it would make sense to expose the entire
recipient list to the script at once, with the result that :count on  envelope
"to" becomes meaningful and so on. But when we looked at how customers used
envelope tests in system sieves, they clearly wanted per-recipient results more
than the ability to look at the entire envelope at once, so that's how we
implemented it.

				Ned