Subject: Re: Sieve extension to expire mails?

From: Michael Haardt <[email-masked]>
Date: Thu, 11 Mar 2004 12:12:24 +0100

> This is usually a folder property that a mailstore process removes messages
> based on. Do you want this to be a message property? I really see no reason
> why a sieve extension is needed for this, if you mark a message and the user
> prefers to remove such messages after x hours/days then it should be a
> function
> in the delivery process and mailstore cleanup?
>
> Or is sieve now more than a mta filter language?

The idea of a folder property versus a message property is interesting,
but indeed I thought about a folder property to be specified by Sieve.
Expiring mails on delivery is way more efficient than daily jobs, because
on many systems (e.g. maildir) the folder contents are already in the page
cache.  Sieve, as MDA filter, controls local delivery on my systems and if
local delivery includes expiration, it feels natural to control that, too.

Michael

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 11 Mar 2004 14:08:52 +0100

Michael Haardt writes:
> The idea of a folder property versus a message property is 
> interesting, but indeed I thought about a folder property to be 
> specified by Sieve. Expiring mails on delivery is way more efficient 
> than daily jobs, because on many systems (e.g. maildir) the folder 
> contents are already in the page cache. Sieve, as MDA filter, 
> controls local delivery on my systems and if local delivery includes 
> expiration, it feels natural to control that, too.

Sieve, as a language/protocol, must be held to stricter standards than 
any single implementation.

If sieve makes the promise that a a mailbox should contain only the past 
seven days' mail, users might reasonably rely on that. If, then, no 
mail is delivered in seven days, the mailbox should STILL contain the 
past seven days' mail (that is, it should be empty). If expiry happens 
as part of delivery, older mail will still be retrievable.

If your implementation confuses delivery and expiry, that's okay. You 
have a privacy/security issue, that's all. But if the sieve language 
confuses them, all implementations do. Much worse.

FWIW, I also don't like the idea of sieve managing folder properties. 
That seems to be out of scope for sieve.

Arnt

---

From: [email-masked]
Date: Thu, 11 Mar 2004 06:42:24 -0800 (PST)

> any single implementation.

I agree 100%

> as part of delivery, older mail will still be retrievable.

I guess I have no problem per se with having per-folder or per-mailbox
expirations. But sieve operates per-message, not per-folder or per-mailbox.
Supporting per-message expirations is considerably more difficult than
per-folder or per-mailbox, and I doubt if the benefits are worth the cost.

But regardless of the expiration model, this proposal seems to me to be
putting the cart before the horse - we'd need to define an IMAP expiration
extension before it is reasonable to extend sieve to support it. And such a
sieve extension only makes sense if the model chosen for expirations is
per-message.

> That seems to be out of scope for sieve.

Exactly right. If the expiration model is per-folder or per-mailbox sieve
has no business trying to muck with it.

				Ned

---

From: Rick Block <[email-masked]>
Date: Thu, 11 Mar 2004 08:05:27 -0700

> sieve extension only makes sense if the model chosen for expirations is

Expiration based on message age is common in voicemail systems.  Some
such systems warn users about messages about to expire and allow users
to "renew" messages, so although the expiration interval is typically
per mailbox/folder the expiry process must be driven by a writable
per message property (one of effective date or expiration date).

-Rick

---

From: Rob Siemborski <[email-masked]>
Date: Thu, 11 Mar 2004 10:24:55 -0500 (EST)

> sieve extension only makes sense if the model chosen for expirations is

There's nothing that says we need an IMAP extension for expiration before
defining a sieve extension for expiration.  Sieve can work in enviornments
other than IMAP.

You could also have a sieve implementation that ran against the current
contents of a mailbox iteratively, and threw out (or refiled) all messages
older than the expiration date, for example [though I suppose this relies
more on a date comparison extension than any particular "Expire"
extension].

-Rob

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Rob Siemborski | Andrew Systems Group * Research Systems Programmer
PGP:0x5CE32FCC | Cyert Hall 207 * [email-masked] * 412.268.7456
-----BEGIN GEEK CODE BLOCK----
Version: 3.12
GCS/IT/CM/PA d- s+: a-- C++++$ ULS++++$ P+++$ L+++ E W+ N(-) o? K- w-- O-
M-- V-- PS+ PE+ Y+ PGP+ t+@ 5+++ X- R@ tv-- b+ DI+++ D++ G e++ h+ r- y?
------END GEEK CODE BLOCK-----

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 11 Mar 2004 16:53:37 +0100

Rob Siemborski writes:
>>  But regardless of the expiration model, this proposal seems to me to 
>>  be putting the cart before the horse - we'd need to define an IMAP 
>>  expiration extension before it is reasonable to extend sieve to 
>>  support it. And such a sieve extension only makes sense if the 
>>  model chosen for expirations is per-message.
>
> There's nothing that says we need an IMAP extension for expiration 
> before defining a sieve extension for expiration. Sieve can work in 
> enviornments other than IMAP.

It can. But since the two tend to go together in practice, I'd say that 
extensions should too, if only to lessen the chance of future gotchas. 
I would hate to stumble into another maze like ACL/2.

As I read the ANNOTATE draft, Freenet is free to say that 
/vendor/freenet/expiry is an integer containing some number of days, 
counting from the INTERNALDATE. No extension needed.

Arnt

---

From: [email-masked]
Date: Thu, 11 Mar 2004 07:36:32 -0800 (PST)

> per message property (one of effective date or expiration date).

First, the fact that some (but by no means all) voice mail systems work this
way doesn't make it a good idea for email systems to work this way. My
voicemail is seriously stressed by my having 20 stored messages in it. In
contrast, my email handles more than 2,000,000 stored messages without
incident, and I routinely put more than 20,000 messages in a single folder.
Per-message expirations could easily have a deleterious effect on this sort of
scalability.

Second, even assuming per-message expiries are a good idea doesn't change the
fact that this proposal is being made in the wrong venue. An IMAP extension
needs to be specified first. Only after that is in place does it make sense to
consider whether or not a sieve extension makes sense. And this in turn
breaks down into three cases:

(1) Expiration is done per-folder or per-mailbox. Sieve deals with messages,
    not folders or mailboxes, so an extension is inappropriate.

(2) Expiration is done per-message using annotations. A sieve extension
    to deal with annotations makes sense in this case; a sieve extension
    to deal with expiration specifically does not.

(3) Expiration is done separately from annotations. A sieve extension may
    make sense in this case, but the odds of this being the way expirations
    get implemented are very, very low.

				Ned

---

From: [email-masked]
Date: Thu, 11 Mar 2004 09:06:31 -0800 (PST)

> defining a sieve extension for expiration.  Sieve can work in enviornments

Perhaps, but if we decide on this course it will be the first time we've
created an action in sieve that isn't bound to a well understand protocol
semantic defined elsewhere. Every existing action (keep, reject, fileinto,
discard, redirect, and even addheader and refuse) depends normatively
on these sorts of semantics.

For example, folders exist independent of IMAP, but having IMAP there to define
the essential semantics of folders is very useful. Failing to capture the
semantics of message expiration outside the context of sieve would IMO be a
mistake.

> more on a date comparison extension than any particular "Expire"

Sure, and I'm reasonably comfortable with this, although to be fair it does
extent the applicability of sieve past the use on or around the time of
delivery.

				Ned

---

From: "Mark E. Mallett" <[email-masked]>
Date: Thu, 11 Mar 2004 18:37:35 -0500

> per-folder or per-mailbox, and I doubt if the benefits are worth the cost.

Just to back up a little bit: I've got a similar item on my own
todo/wish list, so I do support a related concept.  However I don't
look at it as a sieve agent managing mailbox expirations.  Pure mailbox
expirations are, I would agree, more of an external maintenance function.
The thing that I would like to see is more of limiting the volume
of a folder, either in terms of total data size or of number of messages.
So either

    fileinto :keep-maximum-messages 100 "somefolder";
or
    fileinto :keep-maximum-storage 100k "somefolder";
(or both).

One comment raised was along the lines of "sieve deals with messages,
not with folders."  I would contend that sieve is a language that can
operate inside of many contexts, one of which is local delivery.  With
local delivery, the distinction is blurred.  I own my incoming mail,
and I own my stored folders, and I want to be able to tell my delivery
agent how to manage both of them and resolve a storage/quota issue in
a reasonably understandable way.  This sort of thing (which I think of
as "rolling mailboxes") would be a nice facility for quarantining
incoming messages which, without the facility, one might otherwise
simply delete.

Also, I understand the philosophy that says that this sort of thing
should be an attribute of a folder.  However I don't agree that every
operation that you might want to do on a folder should be expressed as
a folder attribute, especially one with magic side effects.  There's
really nothing wrong with expressing folder limits as a control over
delivery; the distinction is where and how you express it.  Expressing
it inside a delivery script is as much setting a folder/delivery
attribute as is expressing it in some parameter attached to the
folder.  BTW I have nothing against either.  (This is kind of like the
way that various OSes differ in their treatment of files: in some, one
configures files to behave in a certain way; in others, one writes
programs to treat files in different ways.)

> sieve extension only makes sense if the model chosen for expirations is

I don't really see why IMAP should drive SIEVE (then again, I'm no
longer talking about "expiration" so this may not be a disagreement.)
SIEVE is a filtering language, IMAP is a folder access protocol.  They
are not the same, although I understand that for some, they are
intertwined.  But I wouldn't use that subset of circumstances as a
driver for SIEVE functionality.  (If I'm misunderstanding, I'm happy
to be educated.)

Bottom line: I don't (necessarily) support sieve dictating folder
expiration, but would look forward to a "rolling mailbox" sort of
facility.

Yours,
-mm-

---

From: [email-masked]
Date: Thu, 11 Mar 2004 17:34:36 -0800 (PST)

>     fileinto :keep-maximum-storage 100k "somefolder";

Or how about limits on an entire folder tree? Or limits expressed in terms of
messages with different characteristics? (These are not theoretical -- I've
seen requests for both of these.) The list goes on and on and on.

Like it or not, this is digging deep into the general area of message store
semantics and brings up any number of fairly subtle issues, e.g., are the
criteria met by deleting old messages, deleting the largest messages, holding
new messages, bouncing new messages, etc. etc.

I will again repeat that this is an area that sieve has previously succeeded in
avoiding, and given that we have at least two active working groups (IMAPEXT
and LEMONADE) playing in this general area, plus one  inactive one that has
come fairly close to this space (ACAP) I think it is unlikely in the extreme
that sieve work on such things will be uncontroversial. Remember, we made an
intentional choice not to be an IETF working group, and that choice  has
consequences, both good and bad.

> SIEVE is a filtering language, IMAP is a folder access protocol.

The reason IMAP should drive sieve in this area is because IMAP is,
in addition to being a protocol, the way we've chosen to capture
a consistent set of message store semantics.

> They
> intertwined.

The implementation intertwining of the two is largely irrelevant as far as I'm
concerned. My concern is that this group isn't the right place to define
significant extensions to message store semantics. Even if we do it and do it
right I suspect the IESG won't like it one bit. (I base this assessment on the
LEMONADE chartering experience, where the IESG required explicit language
restricting the group's activities to IMAP and also called for close
coordination with IMAPEXT.)

> But I wouldn't use that subset of circumstances as a

> expiration, but would look forward to a "rolling mailbox" sort of

Well, as others have previously pointed out, one way to get this sort of
functionality is through the specification of an extension that hooks into
the IMAP ANNOTATE or ANNOTATEMORE extensions. Nothing would prevent someone
from using such an extension in a non-IMAP environment.

				Ned

---

From: Rob Siemborski <[email-masked]>
Date: Fri, 12 Mar 2004 00:11:19 -0500 (EST)

>     fileinto :keep-maximum-storage 100k "somefolder";

In addition to Ned's comments (which I tend to agree with), does
implementing this sort of folder semantic on a per-message basis even
make any sense?

Let's presume the script:

if envelope :all :contains "to" "rjs3+a@" {
        fileinto "INBOX.foo";
} elsif envelope :all :contains "to" "rjs3+b@" {
        fileinto :keep-maximum-storage 100k "INBOX.foo";
}

Does this mean the script only enforces the limit when an "rjs3+b" message
arrives?  If so, is that even useful?

Its entirely unclear to me that folder-wide attributes are useful to
employ in a sieve script.  Per-message expiration dates (and even generic
annotation-setting) seems fine, but folder-wide attributes just don't seem
like a good fit.

-Rob

-----BEGIN GEEK CODE BLOCK----
Version: 3.12
------END GEEK CODE BLOCK-----

---

From: "Mark E. Mallett" <[email-masked]>
Date: Fri, 12 Mar 2004 15:41:03 -0500

> arrives?  If so, is that even useful?

That does look like that is what that script would do, and no, it's
probably not useful.  I would call that a bad script.  When one
provides facilities in a language, one is not personally expected to
debug every script that can be written in it.

There's always a conflict between exposing and encapsulating language
facilities.  For example, the original poster had the notion of adding
a folder definition that would ensure that all operations into that
folder would operate the same way.  I wouldn't want that: I generally
go for the low-level granularity that lets the script writer have
complete control (and responsibility for their own errors).

(In my own implementation I have other ways of dealing with that
conflict anyway.)

mm

---

From: "Mark E. Mallett" <[email-masked]>
Date: Fri, 12 Mar 2004 15:31:42 -0500

> seen requests for both of these.) The list goes on and on and on.

One can always theorize extreme tangents for any discussion.  Even so,
it could be useful to have fundamental access to operations on
existing mailstore when storing a new message.

> new messages, bouncing new messages, etc. etc.

Obviously we disagree, because (re "like it or not") I do
like it :-)  In the case of a rolling mailbox, the issues are not
subtle.  One has a sliding window of the mailbox, where old
messages are dropped to make room for new ones.  This is really
not an outlandish idea, nor does it introduce any new folder
semantics.  If it sparks other ideas or other ways to access mail
folders, so much the better-- any way to give owner of a mailstore
a little more control at delivery time.  That's one of the things
SIEVE is for.

mm

---

From: Rob Siemborski <[email-masked]>
Date: Fri, 12 Mar 2004 16:50:56 -0500 (EST)

I suppose that's what I get for hitting the send key at the wrong time...

Let's try this again.

On Fri, 12 Mar 2004, Mark E. Mallett wrote:

> probably not useful.  I would call that a bad script.

Well-defined syntax shouldn't let you write a "bad script".  I'd argue
that this particular addition to sieve is likely to be extraordinarly
confusing to users.  If I have 10 places in a script where I can file into
a particular mailbox, I shouldn't need to remember that in each of those
places I need to set the *same* max storage limit.

> complete control (and responsibility for their own errors).

If such an overriding defaults system existed, this problem wouldn't
exist.  You could even have both a default value, *and* allow people to
override it.

Though, in the end, I think this is something best left as a folder
attribute.

-Rob

-----BEGIN GEEK CODE BLOCK----
Version: 3.12
------END GEEK CODE BLOCK-----

---

From: Rob Siemborski <[email-masked]>
Date: Fri, 12 Mar 2004 16:48:16 -0500 (EST)

> probably not useful.  I would call that a bad script.

Well-defined syntax shouldn't let you write a "bad script".

> When one

-----BEGIN GEEK CODE BLOCK----
Version: 3.12
------END GEEK CODE BLOCK-----

---

From: "Mark E. Mallett" <[email-masked]>
Date: Fri, 12 Mar 2004 17:12:19 -0500

> Well-defined syntax shouldn't let you write a "bad script".

I would have to flat-out disagree with that.  But that's a whole
nother conversation.

>  I'd argue

I can sympathize with those comments even if I don't necessarily have
the same perspective as you.  Mine is to give the power to the script
writer, not protect them from that power.  On a related note: do people
who will be confused by languages write scripts by hand, anyway?  I
imagine that those people use some interface that generates scripts
for them, and "power users" will write scripts by hand.  Neither of
these groups should object to having powerful underlying facilities.

BTW I am not pushing this as a proposal; note that I am commenting in
a thread started by someone else.  But I do believe it is a good idea
that could either be implemented as-is or incorporated into a more
complete extension somewhere down the line.

Yours,
-mm-

---

From: Rob Siemborski <[email-masked]>
Date: Fri, 12 Mar 2004 17:16:59 -0500 (EST)

> I would have to flat-out disagree with that.  But that's a whole

Semantics.  It's a syntacticly correct script that even has a valid
interpretation, which is logical, even if its not necessaraly what the
user would expect.  That does not at all make it a "bad script", as you
say.

> On a related note: do people who will be confused by languages write
> scripts by hand, anyway?

Its possible to write code in any language that is confusing, the point is
to minimize the confusion.  Even experienced users might not see what is
going on in a script that is a thousand lines long, with fileintos
into the same mailbox separated by a few hundred lines.

> I imagine that those people use some interface that generates scripts

But they might object to the language making it easy to make mistakes.

-Rob

-----BEGIN GEEK CODE BLOCK----
Version: 3.12
------END GEEK CODE BLOCK-----

---

From: [email-masked]
Date: Fri, 12 Mar 2004 15:13:51 -0800 (PST)

> One can always theorize extreme tangents for any discussion.

But these weren't extremes. They are both quite reasonable requests.

> Even so,

Again, it depends on how those operations are organized. And the way to make
sure they are properly organized is to work out the semantics in a venue
appropriate for mail store semantic discussions. This isn't that venue.

> semantics.

Sure it does. Folders normally retain messages until they are deleted.
This changes that semantics rather completely, don't you think?

				Ned

---

From: "Mark E. Mallett" <[email-masked]>
Date: Fri, 12 Mar 2004 18:49:55 -0500

> This changes that semantics rather completely, don't you think?

I don't think so, and that's an opinion that I was trying to convey.
Perhaps there's a distinction centering around "deleted by whom?"  My
attitude is that the owner of a mailbox could legitimately instruct a
delivery agent to manipulate that mailbox as part of delivery.  One of
those instructions could be "delete the oldest message [under some
condition]".  I see no philosophical difference between that
instruction and manually hitting the 'd' key.  This introduces no new
storage semantics nor any new operational semantics.  It does wrap up
a few primitive operations into a higher level function, but that
doesn't make those operations new.  (Making them more granular would
actually be to my liking.)

Just me, I guess?

-mm-

---

From: [email-masked]
Date: Fri, 12 Mar 2004 16:04:41 -0800 (PST)

> Perhaps there's a distinction centering around "deleted by whom?"

Yes, and it is a very significant one. If I have a specialized client that
prowls through some set of folders and deletes things based on some criteria or
other, its operational parameters are by definition a client concern.  Clients
the semantics and so the issue of consistent semantics for store-based deletion
doesn't arise.

If, on the other hand, a store implements some sort of message expiration
system, it almost certainly will be provided to multiple users who will  want
to use multiple clients to control and manipulate the facility on multiple
servers. Failure to nail down the semantics properly is likely to lead to
problems -- messages won't be deleted when they should have been, and worse,
messages might be deleted when they should not have been. One of the reasons we
have standards is to try and prevent this sort of thing from happening.

It is entirely reasonable to talk about using sieve to control either
a client or server side expiration facility. But a client side facility
would logically be controlled through the use of private headers or
annotations or something similar. The only thing that would need to
be standardized in this case is the means to add headers or annotations
or whatever in sieve.

A server side facility, on the other hand, needs to have the server
side semantics nailed down first. I have alredy explained why I believe
this is the case in considerable detail.

> My

> Just me, I guess?

It certainly is not a view I share, nor is it a view that I feel is
likely to lead to a successful sieve extension. Again, I have
explained why I think this is the case.

				Ned