Subject: Expiring/deleting old mail

From: Michael Haardt <[email-masked]>
Date: Tue, 16 Mar 2004 14:22:36 +0100

Hello,

let me summarise the recent discussion.  I hope this reflects most
views expressed on the topic:

Some mail systems allow the concept of "rolling/automatically expiring"
mailboxes/folders which automatically remove old mail, which is useful
for mailing lists and junk mail folders.  It is important that this
function is not performed by the client, because clients usually don't
clean up during vacation.

For that reason, I suggest a Sieve extension, which allows the mail
system to remove mail.  There are several ways to address the problem:

Folder Attributes

Sieve focuses on message delivery, so using sieve to control folder
attributes would be using it for a purpose it was not intended for.

Message Attributes

This fits well into Sieve, it allows the richest set of features, but
message expiration attributes are also the most complicated approach.
Further, users might expect expiration dates to be met exactly and
not be pleased about lazy expiration, adding further complexity for
the implementation.  IMAP ANNOTATE may be the way to go for message
attributes.

Message removal at delivery

This approach is easy to understand for users and easy to implement: It
keeps removing mail, starting from the oldest, until the given criteria
are matched.  It could be realised by adding an option to "fileinto"
and probably "keep":

  fileinto :max-days 7 "inbox.mailinglist";
  fileinto :max-messages 100 "inbox.mailinglist";
  fileinto :max-storage 1M "inbox.mailinglist";

It is still open how the last action should behave when delivering
a 2M message.  Perhaps there is a better name instead of "max-".
This mechanism should not be called expiration, because people expect
a different behaviour by that name.

Michael

---

From: Rob Siemborski <[email-masked]>
Date: Tue, 16 Mar 2004 11:46:09 -0500 (EST)

> a different behaviour by that name.

I'd say that discussion is also needed about how to treat multiple,
conflicting, fileinto commands.

-Rob

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Rob Siemborski | Andrew Systems Group * Research Systems Programmer
PGP:0x5CE32FCC | Cyert Hall 207 * [email-masked] * 412.268.7456
-----BEGIN GEEK CODE BLOCK----
Version: 3.12
GCS/IT/CM/PA d- s+: a-- C++++$ ULS++++$ P+++$ L+++ E W+ N(-) o? K- w-- O-
M-- V-- PS+ PE+ Y+ PGP+ t+@ 5+++ X- R@ tv-- b+ DI+++ D++ G e++ h+ r- y?
------END GEEK CODE BLOCK-----

---

From: Cyrus Daboo <[email-masked]>
Date: Tue, 16 Mar 2004 12:11:03 -0500

Hi Michael,

--On Tuesday, March 16, 2004 2:22 PM +0100 Michael Haardt 
<[email-masked]> wrote:

| Some mail systems allow the concept of "rolling/automatically expiring"
| mailboxes/folders which automatically remove old mail, which is useful
| for mailing lists and junk mail folders.  It is important that this
| function is not performed by the client, because clients usually don't
| clean up during vacation.
|
| For that reason, I suggest a Sieve extension, which allows the mail
| system to remove mail.  There are several ways to address the problem:

I'm afraid I still think expiration like this belongs in the mailstore not 
the delivery agent. There are other ways for messages to be delivered into 
mailboxes other than sieve and expiration policy for those is just as 
relevant as it is for mailboxes with sieve delivery occurring. e.g. 
Mailboxes that are targets of IMAP COPY and APPEND commands should be able 
to have expiration behaviour as you describe.

So I think the right approach is to define an IMAP extension for setting 
expiration policies on mailboxes, utilizing ANNOTEMORE for controlling the 
per-mailbox settings. The expiration test/action would be triggered by any 
action that could change the mailbox (COPY, APPEND, SELECT and external 
delivery) or, if a server chooses to implement it, an automated timed 
expiry check at an interval specified by the user or dictated by the system.

Note that one could be quite sophisticated about the expiry checks - e.g. 
IMAP SEARCH criteria could be used in the annotation to specify which 
messages are to be expired - so I could setup criteria to exclude messages 
that have the important (\Flagged) flag set etc.

-- 
Cyrus Daboo

---

From: [email-masked]
Date: Tue, 16 Mar 2004 17:12:04 -0800 (PST)

> system to remove mail.  There are several ways to address the problem:

Including nailing down the semantics of this as an IMAP extension first,
and once that's done worrying the sieve issues if appropriate.

Note that semantics specified as part of an IMAP extension can be used outside
of IMAP.

> a different behaviour by that name.

I'm sorry, but these are simply folder-level attributes crammed in at the
message level because that's the mechanism sieve provides. But calling them
message atributes doesn't change their nature.

I'm opposed to all of these approaches and strongly opposed to the last one. If
you want to do this, do it right - specify an IMAP extension and then, if it
makes sense, specify a sieve extension that can be used to control the
mechanism.

				Ned

---

From: Rob Siemborski <[email-masked]>
Date: Wed, 17 Mar 2004 09:17:52 -0500 (EST)

> message atributes doesn't change their nature.

I agree on the last one -- but message annotations can also be "this
message expires on date xyz" which is not necessaraly related to when the
next message in the mailbox expires.

Though I also agree that the sanest way to do this is just with a
folder-wide attribute in IMAP, and leave sieve out of it.

-Rob

-----BEGIN GEEK CODE BLOCK----
Version: 3.12
------END GEEK CODE BLOCK-----

---

From: Rob Siemborski <[email-masked]>
Date: Wed, 17 Mar 2004 09:53:06 -0500 (EST)

On Wed, 17 Mar 2004 [email-masked] wrote:

> In other words, there are two fairly different models for how to handle
> expirations in a message store, and we need to pick the one that makes the most
> sense. Doesn't this sound like something store implementors might want to have
> a wee bit of say in?

Yes -- I agree that sieve is not the place to do this.  Certainly not at
first, anyway.

-Rob

-----BEGIN GEEK CODE BLOCK----
Version: 3.12
------END GEEK CODE BLOCK-----

---

From: [email-masked]
Date: Wed, 17 Mar 2004 06:48:16 -0800 (PST)

> next message in the mailbox expires.

Of course they can, but this is just the per-message way of doing it,
rather than the per-folder way.

				Ned

---

From: "Hector Santos" <[email-masked]>
Date: Wed, 17 Mar 2004 10:38:03 -0500

Maybe these can offer some insight. These are our mail server mail policy
attributes (per conference AKA folder).  Our mail server automatically
maintains the mail once per day between 1am and 5am (adjustable).

o Maximum Messages:
o Message Number to Trigger Renumber:
o Days to Keep Read Private Mail:
o Days to Keep Unread Private or Public Mail:
o Days to keep external user mail : (outgoing mail created by your users).
o Delete external user mail once delivered by SMTP:

Other folder/user profile group based options  that can have an effect on
the mail tracking/maintance.

o Allow Mail Previewing (Non-POP3 devices)
o Mark downloaded (POP3) Mail as Received:  Yes/No

In general, users have no say in our product mail policy design.  POP3/WEB
mail access has changed that over the years with users now having some
"leave on server" options in their MUA to help them with roaming.   However,
tracking, mail maintenance, mail receipt request, etc (Subscribers can't
claim they didn't get an expiration notice) is still important so our design
compromise was to have two Receive bits. One that is always recorded for
internal purposes and one that is used for the user.  This allows them to
Preview mail on lets say the Web without marking it read, and then via MUA
download the mail with direct mail
pointers retrieving the "unread user mail."    Same idea with the MUA
(POP3), option to not mark receive allows them to roam to around to other
machines.   However, the internal received bit still prevails for the normal
mail packing/maintenance options (Days to Key UnRead/Read Private Mail
options).   Users have no say on how long mail stays in storage, unless of
course, it is shorter than the server limits, in which case,  the MUA can
delete the message.

I guess my point with the note, that whole concept present a very difficult
PR/system policy issue where MUA clients were demanding new behaviors
conflict with some of the product's traditional strong mail security,
tracking and maintenance policies.  The #1 being subscription notices could
not be bypass with user's previews or non-marking downloads, and the 2nd was
the increase mail storage requirements in large systems because the triggers
were being bypassed.

Hope this provide some engineering insights.

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

---

From: "Mark E. Mallett" <[email-masked]>
Date: Wed, 17 Mar 2004 15:52:28 -0500

On Tue, Mar 16, 2004 at 12:11:03PM -0500, Cyrus Daboo wrote:
> the delivery agent.

Unfortunately I can see both perspectives :-)

I do understand yours, or at least I think I do.  You want to have
mailstore attributes that control this particular combination of mail
folder operations.

The other one is that the agent acts just like I do, only I'm not
sitting at the keyboard.  I can sit down at my computer and access a
folder and make sure I have less than 100 messages in it when I
leave.  I do this using existing primitive access functions.
Similarly, I could tell my sieve-enabled LDA to follow this same
methodology.  The issue of "what if some other access access doesn't
do this" is a red herring.  I can manually open up the folder using a
mail client and treat the folder in a different way... so?  It's the
same thing: it's me doing the access whether I've told an agent to do
it, whether I access using my keyboard, or whether I told another
agent to do something else.  If I do conflicting things it's my
choice, and it may or may not be an error.  I don't think I need an
IMAP extension to allow primitive operations to be combined in these
ways.

Personally I wouldn't want to use a system that required me to
configure my folders for each kind of access I wanted to do, or each
way that I wanted to use them.  Nor would I want to use mail folders
that had side-effects when I accessed them.  I can respect that others
might want this, though-- it's just not my mindset.

I can also respect an opinion that SIEVE only operates on a single
message, and yeah, I see that point.  But if SIEVE is the filtering
language in my LDA of choice (and it is), and I would like to have
some interaction with an existing mailstore (and I do), some level of
access to an existing store is not really out of place.

Anyway, I'm not trying to repeat myself here, just providing what
I hope is a little more light.  I doubt anyone who has the other
perspective will be convinced, though.

Yours,
-mm-