Subject: [imapext] MOVE - A few late comments

From: Zoltan Ordogh <[email-masked]>
Date: Tue, 25 Sep 2012 19:21:44 +0000

Hi all,
I might be a little late, I'll send it for your consideration anyway.
Comments are prompt as I have to type it up quickly; let me know if it's not verbose enough.

General:
There are a good deal of abbreviations that are not spelled out on their first use (nor later on in the document).

Section 1: "Formal syntax is defined by [RFC5234<http://tools.ietf.org/html/rfc5234>]."

I guess what you mean to say is that the formal syntax is defined using the ABNF notation defined in RFC5234.

Section 2: "This document defines an IMAP extension to move messages from one mailbox to another."

This document defines new IMAP commands to move messages?

Section 2: "This function (very common in MUA UIs) is not provided by stock IMAP ..."

What is "stock imap"? One particular spec/product?

Section 2: "and cope with partial failures and side effects"
What side effects? If you have a reference that discusses those, add it. If you don't, remove it.

Section 3: "each message SHOULD either be moved or unaffected"

I'd say MUST; there's no third option.

Section 3: "The server SHOULD NOT leave a message in neither or both mailboxes afterwards"

I'd say MUST NOT; leave no room for fuzzy state.

Section 3:

"   UID MOVE is the same as the three-command sequence in all other

   respects, which implies that extensions which affect the three-

   command sequence also affect UID MOVE, and that response codes such

   as COPYUID, TRYCREATE and so on should be sent as appropriate."

Isn't there a less complicated way of saying this? For example:

Since the UID MOVE command consist of three individual commands (COPY, STORE, EXPUGE), it also triggers execution of the extensions (if any) to these commands on the server-side. Consequently, the UID MOVE command returns response codes of each command the their extensions (i.e. COPYUID, TRYCREATE, etc.).

Section 3:

"Implementers will need to read [RFC4315<http://tools.ietf.org/html/rfc4315>] to understand what UID

   EXPUNGE does. Implementing [RFC4315<http://tools.ietf.org/html/rfc4315>] is not necessary."

That sounds very awkward. Does it really make UIDPLUS a normative reference?

Section 3:

"   Note that moving messages to the current mailbox is well-defined, so

   that MOVE is defined for all the cases where the COPY/STORE/EXPUNGE

   sequence is."

This is not good enough, especially when ACLs are in use.

Considering RFC4314 permissions:

-    COPY requires that I have 'r' right at the source and 'i' right at the destination.

-    STORE requires that I have 't' right at the source.

-    EXPUNGE requires that I have 'e' right at the source.

Consequently, I must have 'i' at the destination and 'r'+'t'+'e' at the source.

Section 3:

"   The MOVE command performs the same actions UID MOVE. Notably, the

   server may send EXPUNGE (or VANISHED) messages before the tagged

   response, so the client cannot safely send more commands with MSN

   arguments while the server is processing MOVE."

All right, but does it trigger extensions the same way UID MOVE does?

Some extensions do not operate with UIDs.

Also, a server implementation could translate MSNs to UIDs during execution - which would trigger extensions hooked onto UID, too (even though MSNs were used in the request). I.e. If UIDPLUS were to fire when MSNs were used in the MOVE command, would that break the client?

Any thoughts?

Section 4.

This is a nice idea, but not quite future-proof.

Were all extensions (at least those that are found in RFCs) enumerated to date?

Section 4.2:

"it requires the same ACL rights as the union of those three commands"

The union of three commands is a command-triplet, not access rights.

I think what you want to say is that the UID MOVE requires that all permissions that those three commands require if executed one-by-one.

What about the MSN MOVE command? (this is clearly missing from this section)

Section 5:

"[RFC3501<http://tools.ietf.org/html/rfc3501>] defines the non-terminals "capability", "command", "set" and "mailbox"."

You might want to change "command" to "command-select" (or, add command-select).

Section 5:

"     capability     =/ "MOVE"

      command-select =/ ["UID "] "MOVE" SP set SP mailbox"

RFC3501 does not define "set". Do you mean sequence-set? That's not good for UIDs.

I guess a new non-terminal to represent a message reference is in order?

Any thoughts on this?

   reference           = reference-type SP reference-value

   reference-type      = "UID" / "SEQ" / "URLAUTH" / reference-type-ext

   reference-type-ext  = atom

                         ; It is not required that new reference types

                         ; begin with "X-", see [XDASH]

   reference-value     = uniqueid / \         ; see [IMAP]

                         sequence-set / \     ; see [IMAP]

                         authorized-url / \   ; see [URLAUTH]

                         reference-value-ext

   authorized-url      = authimapurlfull / \  ; see [URLAUTH]

                         authimapurlrump      ; see [URLAUTH]

   reference-value-ext = atom

                         ; New reference values MUST correspond to

                         ; reference-type-ext

Section 6:

"   This document is believed to add no security problems."

Provided that the server does verify that all required permissions are in place.

Section 6:

"Problems with these algorithms can lead to mail loss."

You mean bad implementations of good algorithms, or just badly designed algorithms, right?

Section 9:

I'd think RFC4315 is not a normative reference; after all, one can always go with MSNs.

I guess it depends on how the MSN to UID translation turns out.

Thank you for considering.

---------------------------------------------------------------------
This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful.

---

From: Bron Gondwana <[email-masked]>
Date: Tue, 25 Sep 2012 21:49:47 +0200

> This document defines new IMAP commands to move messages?

If it adds a CAPABILITY, it's an EXTENSION, right?

> What is "stock imap"? One particular spec/product?

RFC3501

> I'd say MUST NOT; leave no room for fuzzy state.

I can imagine failure modes on a server which doesn't have global
transactional state where the message is committed to the destination
mailbox but not yet removed from the source mailbox when something dies.
Hence MUST NOT makes it significantly harder to implement.

I'm happy with 

  "MUST NOT leave a message neither mailbox, and SHOULD
  NOT leave a message in both mailboxes."

> Since the UID MOVE command consist of three individual commands (COPY, STORE, EXPUGE), it also triggers execution of the extensions (if any) to these commands on the server-side. Consequently, the…

Now are you sure you've read all the discussion leading up to this?
Because we've been leaning more the other way, de-emphasising the
individual commands.  I'm not sure that your wording is less
complicated.

That said, the original wording is pretty fuzzy, and contains a lower
case "should".

Perhaps this?

  "If the server supports extentions to COPY, STORE or EXPUNGE then it
   SHOULD apply those same extensions to MOVE and return the same
   responses (i.e. COPYUID, TRYCREATE, etc.) as if it was running the
   three-command sequence in order"

> That sounds very awkward. Does it really make UIDPLUS a normative reference?

There's no non-UID version of EXPUNGE which allows you to expunge only
a subset of messages.

> Consequently, I must have 'i' at the destination and 'r'+'t'+'e' at the source.

Sorry, what's your problem here?  If you don't have permissions, then
the command fails with a permission denied response (possibly
EEXIST if you don't even have rights to list the destination)

And in this case, since source and destination are the same...

   "moving messages to the current mailbox"

You obviously must have "irte" on the current mailbox to succeed.

> Also, a server implementation could translate MSNs to UIDs during execution - which would trigger extensions hooked onto UID, too (even though MSNs were used in the request). I.e. If UIDPLUS were t…

You have a point here.  Hmm.

. copy 1:2 inbox.sub
. OK [COPYUID 1348602306 1:2 1:2] Completed

Looks like I used MSNs and got a COPYUID response from Cyrus on my
local testbed.  We have all sorts of clients talk to our servers,
and I've never seen a bug report that COPYUID confused any of them.

Do you know any clients that would be confused?

> What about the MSN MOVE command? (this is clearly missing from this section)

Are there any MSN commands which require different ACLs than their UID
equivalents?

> Provided that the server does verify that all required permissions are in place.

Well, yeah - duh.  If you don't verify ACLs as specified in the other
parts of the document, then you'll have a security problem.

Bron.
-- 
  Bron Gondwana
  [email-masked]

---

From: Timo Sirainen <[email-masked]>
Date: Tue, 25 Sep 2012 23:05:52 +0300

> Do you know any clients that would be confused?

This works as specified by RFC 4315. There's no problem with sending COPYUID response to non-UID COPY, and I don't think there should be any problem with sending COPYUID response to non-UID MOVE.

---

From: Jan Kundrát <[email-masked]>
Date: Wed, 26 Sep 2012 09:12:03 +0200

>    three-command sequence in order"

One should be very careful with the wording here, as saying "return the
same responses" is not true in the current draft where the COPYUID
response code is present in an untagged response -- that's different
from RFC4315.

With kind regards,
Jan
-- 
Trojita, a fast e-mail client -- http://trojita.flaska.net/

---

From: Barry Leiba <[email-masked]>
Date: Wed, 26 Sep 2012 11:55:26 -0400

> individual commands.

Indeed, and in a number of ways, Zoltan's review makes it clear that
we need an updated draft that addresses the comments so far, so that
people aren't reviewing what's basically old text.  Arnt, can you do
this soon, so we can get fresh reviews?

Barry

---

From: Zoltan Ordogh <[email-masked]>
Date: Wed, 26 Sep 2012 16:11:36 +0000

> -----Original Message-----
> From: Bron Gondwana [mailto:[email-masked]]
> Sent: Tuesday, September 25, 2012 15:50
> To: Zoltan Ordogh; [email-masked]
> Subject: Re: [imapext] MOVE - A few late comments
> 
[ZÖr] Right. However, and extension does not "do" anything (does not move messages); the commands defined in that extension do.

> > What is "stock imap"? One particular spec/product?
[ZÖr] Then put that there.

>   NOT leave a message in both mailboxes."
[ZÖr] Here's an extension that moves your message on success, fails, fails with copying the message or fails with the message disappearing. That's just not acceptable. It is either successful and the message is moved, or it fails with an error and nothing happens in the storage. Failing is ok. Introducing duplicates and disappearing messages are not ok. If the command  does not work in a predictable manner, then this exercise is moot. 

> > Since the UID MOVE command consist of three individual commands
> (COPY, STORE, EXPUGE), it also triggers execution of the extensions (if any)
> to these commands on the server-side. Consequently, the UID MOVE
> command returns response codes of each command the their extensions
> (i.e. COPYUID, TRYCREATE, etc.).
> 
> individual commands.  I'm not sure that your wording is less complicated.
> 
> That said, the original wording is pretty fuzzy, and contains a lower case
> "should".
> 
> Perhaps this?
> 
[ZÖr] I understand that drafts (that are about to become RFCs) should leave no room for misinterpretation and should not require following any discussion on any mailing list. If it does, then it means that the draft is not ready yet. Having said that, if there's a reason for doing this, it should be stated in the draft.

Your proposal is better than the original, but, again, I do not understand:
 - why is it a "SHOULD"? Can the server hand-pick extensions it likes and execute only those? No. All supported extensions should be executed (i.e. it's a MUST) - unless of course there was an error so it makes no sense to execute the rest.
 - It is not clear what responses to expect in case of an error. IMHO, only the response from the command that failed is relevant.

> > "Implementers will need to read
> [RFC4315<http://tools.ietf.org/html/rfc4315>] to understand what UID
> >    EXPUNGE does. Implementing
> [RFC4315<http://tools.ietf.org/html/rfc4315>] is not necessary."
> >
> There's no non-UID version of EXPUNGE which allows you to expunge only a
> subset of messages.
[ZÖr] I guess my IMAP is getting rusty. It's been a while, sorry. In any case, I have to say that I am a bit lost now.
I mean, if the MSN will have to be turned into a UID every time (for the sake of EXPUNGE and its response), then how come implementing RFC4315 is not required? Just what kind of response would the server generate and how would the related extensions trigger without supporting RFC4315?

> Sorry, what's your problem here?  If you don't have permissions, then the
> command fails with a permission denied response (possibly EEXIST if you
> don't even have rights to list the destination)
> 
[ZÖr] The problem is the sentence. I was under the impression that it aims to clarify the circumstances under which the MOVE command can be used.
Without ACLs, the meaning of this sentence is unclear.
With ACLs, it is just as unclear, but at least there's a section later that could be referenced.
So, I'd like to see this sentence improved and in case ACL is in use, a cross-reference to section 4.2 would be nice.

> > Also, a server implementation could translate MSNs to UIDs during
> execution - which would trigger extensions hooked onto UID, too (even
> though MSNs were used in the request). I.e. If UIDPLUS were to fire when
> MSNs were used in the MOVE command, would that break the client?
> >
> > Any thoughts?
> 
> Looks like I used MSNs and got a COPYUID response from Cyrus on my local
> testbed.  We have all sorts of clients talk to our servers, and I've never seen a
> bug report that COPYUID confused any of them.
> 
[ZÖr] Thanks for running a quick test.
Timo's response explained this, so I guess this is a non-issue. Kiitos, Timo!
Would it be of value to capture Timo's response in the draft?

> > I think what you want to say is that the UID MOVE requires that all
> permissions that those three commands require if executed one-by-one.
> >
[ZÖr] That's not the point I was trying to make. Let me try again.
There are two points actually.
The first point is a request for improving the language because the result of the "union of those three commands" is not an access right. The result is a command-triplet. I am guessing that the text intends to say that to execute the MOVE command, all permissions need to be in place, just as if those commands were executed one after the other by the client.

Onto the second point. The extension includes two commands: UID MOVE and MOVE.
The latter is not addressed in section 4.2.
I guess removing 'UID' from 'UID MOVE' would address this.

> Well, yeah - duh.  If you don't verify ACLs as specified in the other parts of
> the document, then you'll have a security problem.
> 
[ZÖr] I think you misunderstood because I was not clear. Perhaps this is clearer:
The server verifies all required permissions for all three commands. I.e. it does not just check the permission for COPY and executes the rest, but it also verifies the permission for STORE and EXPUNGE as well. Ideally, the server should verify all these permission prior to executing the command so it would not run into a scenario when a user can COPY, STORE but not EXPUGE; the server would have to roll back the whole thing.

> Bron.
[ZÖr] Thank you Bron, I appreciate taking your time and responding an these issues.

---

From: Barry Leiba <[email-masked]>
Date: Wed, 26 Sep 2012 12:21:33 -0400

>> Hence MUST NOT makes it significantly harder to implement.
...
> [ZÖr] Here's an extension that moves your message on success, fails, fails with
> copying the message or fails with the message disappearing. That's just not
> acceptable. It is either successful and the message is moved, or it fails with an
> error and nothing happens in the storage. Failing is ok. Introducing duplicates and
> disappearing messages are not ok. If the command  does not work in a predictable
> manner, then this exercise is moot.

I have to say that I agree with Zoltan here.  We don't allow other
IMAP commands that leave things in indeterminate states, and I don't
think we should define this one that way.  COPY is clear: if it
succeeds, you have two copies of the message.  If it fails, nothing
has changed.  It would be odd to say that if it succeeds, a copy
SHOULD exist in the target mailbox.  No, if it succeeds, a copy MUST
exist there.

If a server, with a particular back-end mailstore, can't reliably
implement MOVE so that it works reliably (if it succeeds, the message
is moved and exists only in the target; if it fails, the message is
not moved and exists only in the source), then it simply shouldn't
implement this extension.  It may, of course, choose to implement it
anyway, knowing that it might sometimes be in violation of this MUST,
but the protocol shouldn't waffle about it.

Barry

---

From: Brandon Long <[email-masked]>
Date: Wed, 26 Sep 2012 09:30:06 -0700

> but the protocol shouldn't waffle about it.

At the very least, it should be MUST that a message shouldn't be in neither.

Brandon

---

From: Bron Gondwana <[email-masked]>
Date: Wed, 26 Sep 2012 22:44:22 +0200

> > SHOULD exist in the target mailbox.  No, if it succeeds, a copy MUST

Sorry - I think I got crossed wires here.  Of course if the command
succeeds it MUST exist in the destination.  No ifs, no buts.  If it
fails, then it SHOULD be in just one, and MUST be in at least one.

> At the very least, it should be MUST that a message shouldn't be in neither.

Absolutely!

Bron.
-- 
  Bron Gondwana
  [email-masked]

---

From: Barry Leiba <[email-masked]>
Date: Wed, 26 Sep 2012 18:22:40 -0400

> fails, then it SHOULD be in just one, and MUST be in at least one.

No, I don't think that's sufficient.  The command has to work reliably. 
That means that if it *fails*, the result has to be well defined.  The
only reasonable well-defined failure mode is that the message MUST exist
only in the source.

As I said, your implementation can choose to violate that if it has to,
using the alternative that you give above.  But the *protocol* needs to
specify it cleanly, I think.

Barry

---

From: Bron Gondwana <[email-masked]>
Date: Thu, 27 Sep 2012 00:39:54 +0200

> specify it cleanly, I think.

You are right.  I'm now convinced that the spec should say MUST.

Bron ( if you don't get either answer from the server the usual
       network gremlin lack of determinism applies )

-- 
  Bron Gondwana
  [email-masked]

---

From: Timo Sirainen <[email-masked]>
Date: Thu, 27 Sep 2012 01:47:08 +0300

> specify it cleanly, I think.

For each individual message ok. But should all the messages be either moved or none moved? Mainly should it fail and abort if some of the messages were already expunged? COPY is defined to either copy all or nothing. My MOVE implementation also moves all or nothing. But some people wanted to use rename() and such that would make reverting in the middle difficult.

---

From: Barry Leiba <[email-masked]>
Date: Wed, 26 Sep 2012 18:53:46 -0400

> For each individual message ok. But should all the messages be either moved or none moved? Mainly should it fail and abort if some of the messages were already expunged? COPY is defined to either c…
I'm OK with saying that the action for each message must be well
defined, but allow for a group MOVE to fail in the middle.  We should
verify the consensus on that point, though.

Barry

---

From: Brandon Long <[email-masked]>
Date: Wed, 26 Sep 2012 16:04:43 -0700

On Wed, Sep 26, 2012 at 3:53 PM, Barry Leiba <[email-masked]>wrote:

>
> > For each individual message ok. But should all the messages be either
> moved or none moved? Mainly should it fail and abort if some of the
> messages were already expunged? COPY is defined to either copy all or
> nothing. My MOVE implementation also moves all or nothing. But some people
> wanted to use rename() and such that would make reverting in the middle
> difficult.

I'd rather not have yet another IMAP spec require transactional guarantee
on an arbitrarily large number of messages.  We can have 4B messages in a
folder, I think its pretty insane to think that a MOVE of 4B messages
either happened or didn't.

Brandon

---

From: Zoltan Ordogh <[email-masked]>
Date: Thu, 27 Sep 2012 13:31:03 +0000

> From: Barry Leiba [mailto:[email-masked]] On Behalf Of Barry Leiba
> Sent: Wednesday, September 26, 2012 18:54
> To: Timo Sirainen
> Cc: Bron Gondwana; Zoltan Ordogh; [email-masked]
> nothing. My MOVE implementation also moves all or nothing. But some
> people wanted to use rename() and such that would make reverting in the
> middle difficult.
> I'm OK with saying that the action for each message must be well defined,
> but allow for a group MOVE to fail in the middle.  We should verify the
> consensus on that point, though.
> 
[ZÖr] I also think it's ok to fail in the middle of MOVE, but the last message - when the failure occurred - should stay the way it was before the failure.
I say it's ok because I don't think the MOVE command is likely to fail in the middle. It is likely to fail at the first message:
 - permission problems
 - destination mailbox does not exist
 - first message does not exist
I guess it can fail in the middle when:
 - a large message is encountered that triggers quota limit - though this can be countered.
 - an extension of COPY/STORE/EXPUNGE triggers an error
 - a server error occurs (not related to the command itself but something that forces the command to be stopped).

I do have a question however about a likely scenario:
Is it considered an error if a message that was requested to be moved no longer exists by the time it is about to be moved?
For example.
C: MOVE 1:2000 test
The server begins moving the messages one by one, starting from the end or the beginning.
In the meantime, another client deletes message 666.
Would the MOVE command that is in progress on the server stop with an error at message 666 or would the server just skip it?
IMHO, it could skip it without an error.

Thank you!

---

From: Bron Gondwana <[email-masked]>
Date: Thu, 27 Sep 2012 15:37:49 +0200

> IMHO, it could skip it without an error.

Cyrus would hold a lock, so it doesn't matter - but if the message
doesn't exist at the start, it just doesn't move it (possible if it
was expunged in another parallel session)

And that's fine.

Bron.
-- 
  Bron Gondwana
  [email-masked]