Subject: [imap5] ERASE command as part of MOVE

From: Bron Gondwana <[email-masked]>
Date: Sat, 2 Jun 2012 00:23:11 +0200

While looking over Cyrus's excellent review of the MOVE
draft it struck me that MOVE actually has another command
hidden inside its definition.

MOVE is defined as COPY + STORE + "UID EXPUNGE".

But it would be very useful to have the second half of that
available as a separate command in its own right, "ERASE"
(I would say DELETE, but that's already stolen by a different
namespace)

This probably shits the purists as much as anything else.
But I can tell you for sure that the FastMail web interface
does its "Delete Permanently" as: 

    $Res = $Self->store($Uids, "+flags", "(\\seen \\deleted)")
         && $Self->uidexpunge($Uids)
         && $Self->refresh_count('');

Which is an extra roundtrip to the server and an extra lock and
parse of the mailbox index.  If this was implemented as:

    $Res = $Self->erase($Uids)
         && $Self->refresh_count('');

It would make everything more efficient.  And we wouldn't have to
touch the \Seen flag just in case the intermediate step failed.

So Arnt, does it make sense to either propose both commands as
part of this draft and implement one in terms of the other, or
to do an ERASE first and then a separate MOVE referencing ERASE?

Bron.

---

From: Timo Sirainen <[email-masked]>
Date: Sat, 2 Jun 2012 01:27:58 +0300

On 2.6.2012, at 1.23, Bron Gondwana wrote:

> parse of the mailbox index.

It doesn't need to be an extra roundtrip. You can run uidexpunge even if store fails and it doesn't do anything bad. By avoiding the extra roundtrip it is possible for the server to optimize the STORE+EXPUNGE into equivalent of ERASE. (Dovecot does halfway that - it for example doesn't rename maildir files on STORE stage, just unlinks them on EXPUNGE.)

---

From: Barry Leiba <[email-masked]>
Date: Fri, 1 Jun 2012 21:54:54 -0400

> available as a separate command in its own right, "ERASE"
...
> to do an ERASE first and then a separate MOVE referencing ERASE?

As Timo says, you can pipeline the STORE and UID EXPUNGE.  Unless you
can come up with some reason that an atomic ERASE command would save
something significant in the message store -- something I can't
imagine to be the case -- there's nowhere near the use for this as
there is for MOVE (where it's not just the round trips that are saved,
because it allows message-store optimization in some cases).

But more to the point: you'll note that the proposed charter is VERY
tight on this, and will not be changed... there is exactly ONE thing
chartered here, and an ERASE command is not it.  This is quite
intentional, to head off tangents and feature creep, and to keep us to
the thing that's being demanded by, we're told, a significant number
of implementors.

Please do not try to add anything more to this very tightly focused effort.

Barry, Applications AD

---

From: Bron Gondwana <[email-masked]>
Date: Sat, 2 Jun 2012 06:12:21 +0200

> the thing that's being demanded by, we're told, a significant number

It just happens to include an ERASE command.
 
> Please do not try to add anything more to this very tightly focused effort.

Fair enough.  I'm sure there's a good reason for being this hard-assed
about not changing things.  It's hardly the first time MOVE has been
suggested, and somehow it's managed to not make the difference every
other time (which is probably why there's an incompatible AOL one
already.  I'm kinda suprised gmail didn't do one as well - though I
guess they work around it with keyword fiddling instead if they know
they're talking to their own server)

I'm still going to write it locally because it's a single instruction
which describes the exact intention to the server.

Bron.

---

From: Bron Gondwana <[email-masked]>
Date: Sat, 2 Jun 2012 06:24:09 +0200

> because it allows message-store optimization in some cases).

OK, a REAL atomic erase (no intermediate state - guaranteed) gives you
this:

S1: UID MOVE 1:* FolderA
S2: UID MOVE 1:* FolderB

A guarantee that at the end, the messages will be in FolderA OR FolderB,
but not in both.

Bron.

---

From: Bron Gondwana <[email-masked]>
Date: Sat, 2 Jun 2012 06:27:51 +0200

> It doesn't need to be an extra roundtrip. You can run uidexpunge even if store fails and it doesn't do anything bad. By avoiding the extra roundtrip it is possible for the server to optimize the ST…

Not so easy if you want to delete the messages by sequence number of
course.

TAG ERASE 1:10

Isn't implementable as a pipelined command, it would have to be

TAG1 STORE 1:10 +Flags \Deleted
TAG2 UID EXPUNGE UID1,UID2,UID3,UID4,UID5..UID10

So how does Dovecot handle a crash or filesystem error during that
intermediate stage?  Replay an action log?  What if the file can't
be renamed, but it's already replied to a client saying "yes, your
flag store succeeded"?

Bron.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Sat, 2 Jun 2012 11:10:34 +0200

Yeah, well... but MOVE is something people have been asking for and 
suggesting every year for at least a decade, and ERASE isn't...

Arnt

---

From: Bron Gondwana <[email-masked]>
Date: Sat, 2 Jun 2012 13:06:46 +0200

> suggesting every year for at least a decade, and ERASE isn't...

True.  I'll butt out :)

Bron.

---

From: Timo Sirainen <[email-masked]>
Date: Sat, 2 Jun 2012 14:39:09 +0300

> TAG2 UID EXPUNGE UID1,UID2,UID3,UID4,UID5..UID10

It can still be pipelined. You would of course need to know the UIDs.

> intermediate stage?  Replay an action log?  

Yes.

> What if the file can't

The message gets marked as having "dirty flags" meaning the flags in index file should be used instead of the flags in maildir file.