Subject: Re: [imap5] Should unsolicited EXPUNGE responses be returned during UID MOVE?

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 1 Jun 2012 09:51:50 +0200

Sorry about my bad temper earlier this morning.

On 05/31/2012 10:58 PM, Adrien W. de Croy wrote:
> Even though we have to send EXPUNGEs to any other connected
> client on that mailbox (that didn't do the move), the approach is a bit
> wasteful of bandwidth etc.
>
> If you issue a command, you want to know when it's done and how it
> turned out.  If the MOVE completes in its entirety without issue (99.99%
> of the time), then  a simple "yeah it worked" response is the most
> efficient.  We can inform of exceptions, e.g. "something broke", or
> "these messages didn't work".  In the end, if there's a failure so what
> if the client has to resynch indices.

IMAP is a bit wasteful of bandwidth in many ways. We have a goodish 
solution for that, RFC 4978, and it's fairly well deployed. Of course we 
might add Microsoft-like special rules to lessen the pain by each 
particular instance of bandwidth waste, but since we already have a 
goodish solution for all of them, why bother?

I think that if you really want to not send EXPUNGE, you should provide 
some numbers to argue your case. Take the log files from some server, 
analyse the moves. How many messages are affected, and how many bytes 
would be sent on average for those commands when a) EXPUNGE is used b) 
EXPUNBGE+COMPRESS=DEFLATE c) VANISHED+C=D and d) nothing.

I suspect that b and c will be very, very close to d.

(FWIW, the easiest way to simulate b/c is to massage an IMAP session so 
you get a file containing just what the server sends, and then just 
'gzip -1' the prefix up until VANISHED/COMPRESS and until just after 
that, and look at the size difference. It won't be exact, but fairly 
good. Or 'gzip -4' or even -9.)

Arnt

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Fri, 01 Jun 2012 08:15:19 +0000

Hi
  
You may consider this to be merely a documentation bug. It was that 
until the first implementations were rolled out the door. It's no 
longer that. 
  
WinGate doesn't send EXPUNGEs (it explicitly suppresses them for this). 
  Presumably there are other servers, which is what caused Mozilla to 
implement MOVE in Thunderbird (AOL? Fastmail?).  So there are millions 
of clients deployed.
  
All I'm saying is there are deployed servers and clients that follow 
the current I-D.
  
Therefore If you change the behaviour without changing the method name 
/ advertisement or any discernable behaviour from the client then a 
server won't know if a client expects EXPUNGE responses or not, and / 
or whether an EXPUNGE response will break it.  And there will be 
clients out there who expect one or the other behaviour, and older 
servers who do it one way, and maybe newer ones that do it the new way.
  
And the expunge is completely redundant from an information POV.  You 
do a UID MOVE 1:1000 and you really want to get 1000 expunges back?   
What does that really tell you that you couldn't deduce from the final 
OK result (no error case)?
  
We could very easily add the EXPUNGES back in.  But that may cause a 
nightmare of support.  We can't force everyone to upgrade their server. 
  We can't force everyone to upgrade it in synch with all their email 
clients that use it.
  
Now this all may be moot if we know of no clients that can't handle 
either behaviour (e.g. existing deployed clients that will break if 
they get an EXPUNGE back from a MOVE).  If  there are no such clients, 
is it even broken then? Why do we need the EXPUNGE responses?  It 
appears to work fine without them.
  
It should just be a question of whether the EXPUNGE reponses solve some 
outstanding problem / hole in the currently deployed method.
  
Otherwise this might be a good case for a new name, like MOVEX or 
something.
  
Adrien
  
----
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com
WinGate 7 is released! - http://www.wingate.com/getlatest/

------ Original Message ------
From: "Arnt Gulbrandsen" <[email-masked]>
To: "Adrien W. de Croy" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Sent: 1/06/2012 6:03:26 p.m.
Subject: Re: [imap5] Should unsolicited EXPUNGE responses be returned 
during UID MOVE?
>Listen closely. The missing EXPUNGEs are a document bug, not 
>representative of any servers I know. 
>
>Arnt

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Fri, 01 Jun 2012 08:25:04 +0000

Hi
  
sorry, saw this after my last email

------ Original Message ------
>solution for that, RFC 4978, and it's fairly well deployed. Of course 
>we might add Microsoft-like special rules to lessen the pain by each 
  
we support 4978 and it's great for sure.  There are a lot of clients 
that don't support it though.
  
And it has a non-zero additional CPU cost which therefore must affect 
scalability.

But sure, if you're already compressed, then those EXPUNGEs will 
compress to nearly nothing.
  
They should add 4978 to the LEMONADE profile.  It does a better job for 
mobile clients than some of the other ones.
  
>
>
>I think that if you really want to not send EXPUNGE, you should 
>provide some numbers to argue your case. Take the log files from some 
>server, analyse the moves. How many messages are affected, and how 
>many bytes would be sent on average for those commands when a) EXPUNGE 
>is used b) EXPUNBGE+COMPRESS=DEFLATE c) VANISHED+C=D and d) nothing. 
>
I'm sure you're correct.
  
My main issue is with interop since we have a legacy issue due to the 
current I-D being in the wild now.
  
Can we just test maybe to see if any existing clients will break on a 
MOVE that sends EXPUNGEs?  If not, then this may be a non-issue.
  
Adrien
>
>
>(FWIW, the easiest way to simulate b/c is to massage an IMAP session 
>so you get a file containing just what the server sends, and then just 
>_______________________________________________ 
>imap5 mailing list 
>[email-masked] 
>https://www.ietf.org/mailman/listinfo/imap5

---

From: Jan Kundrát <[email-masked]>
Date: Fri, 01 Jun 2012 11:09:12 +0200

On 06/01/12 10:15, Adrien W. de Croy wrote:
> until the first implementations were rolled out the door. It's no longer
> that.  

Hi Adrien,
what servers and clients actually support the "UID MOVE" command right
now, in contrast to non-standard stuff like "UID XMOVE", "x-aolmove"
etc? Which of these implementations are designed to not send/expect the
EXPUNGE/VANISHED?

Honestly, I'm asking, I haven't seen any hard data about the existing
support in this thread, and hence I don't know.

Cheers,
Jan

-- 
Trojita, a fast e-mail client -- http://trojita.flaska.net/

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Fri, 01 Jun 2012 10:00:34 +0000

Sorry, I thought I was clear... WinGate (IMAP server) does this.

Adrien

----

------ Original Message ------
From: "Jan Kundrát" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Sent: 1/06/2012 9:09:12 p.m.
Subject: Re: [imap5] Should unsolicited EXPUNGE responses be returned 
during UID MOVE?

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 1 Jun 2012 14:26:32 +0200

I'm shocked. The -00 draft is only two months old, and there are already 
implementations that I don't know about?

I suppose this is great news. But to me it's more shocking than anything 
else.

Arnt

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Fri, 01 Jun 2012 19:47:49 +0000

our implementation of MOVE is over a year old
  
there's a previous draft.
http://tools.ietf.org/html/draft-krecicki-imap-move
  
----

------ Original Message ------
From: "Arnt Gulbrandsen" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Sent: 2/06/2012 12:26:32 a.m.
Subject: Re: [imap5] Should unsolicited EXPUNGE responses be returned 
during UID MOVE?
>I'm shocked. The -00 draft is only two months old, and there are 
>already implementations that I don't know about? 
>
>I suppose this is great news. But to me it's more shocking than 
>anything else. 
>
>Arnt

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Fri, 01 Jun 2012 20:02:31 +0000

I just re-read the draft, and double-checked our implementation.
  
appears I have a fair amount of egg on my face, apologies to all.
  
51439 1574952 2/06/2012 8:00:12.781 219.89.218.74 Adrien W. de Croy 
IMAP4 Server 2032 11893 Debug 0 C=>: 10 uid move 657428,657449 "Trash"
51440 1574953 2/06/2012 8:00:12.812 219.89.218.74 Adrien W. de Croy 
IMAP4 Server 2032 11893 Debug 0 <=S: * 1789 EXPUNGE
51441 1574954 2/06/2012 8:00:12.812 219.89.218.74 Adrien W. de Croy 
IMAP4 Server 2032 11893 Debug 0 <=S: * 1789 EXPUNGE
51443 1574956 2/06/2012 8:00:12.859 219.89.218.74 Adrien W. de Croy 
IMAP4 Server 2032 11893 Debug 0 <=S: 10 OK [COPYUID 1333493640 
657428,657449 7758:7759] command completed
  
It does clearly state EXPUNGE is required, and it even looks like we 
implemented it that way.
  
Sorry all I should have double checked.  I don't know where I got the 
idea they were suppressed from (inspection of code which obviously I 
got wrong).
  
Regards
  
Adrien

----

------ Original Message ------
From: "Adrien W. de Croy" <[email-masked]>
To: "Arnt Gulbrandsen" <[email-masked]>;"[email-masked]" 
<[email-masked]>
Sent: 2/06/2012 7:47:49 a.m.
Subject: Re: [imap5] Should unsolicited EXPUNGE responses be returned 
during UID MOVE?
>>Arnt _______________________________________________ imap5 mailing 
>>list [email-masked] https://www.ietf.org/mailman/listinfo/imap5 
>

---

From: Bron Gondwana <[email-masked]>
Date: Fri, 1 Jun 2012 23:51:59 +0200

On Fri, Jun 01, 2012 at 08:02:31PM +0000, Adrien W. de Croy wrote:
>   It does clearly state EXPUNGE is required, and it even looks like
> we implemented it that way.

I'm just going to double check mine as well:

. fetch 1:* uid
* 1 FETCH (UID 322)
* 2 FETCH (UID 332)
* 3 FETCH (UID 333)
* 4 FETCH (UID 334)
* 5 FETCH (UID 336)
* 6 FETCH (UID 338)
* 7 FETCH (UID 340)
* 8 FETCH (UID 341)
* 9 FETCH (UID 342)
* 10 FETCH (UID 343)
* 11 FETCH (UID 344)
* 12 FETCH (UID 345)
* 13 FETCH (UID 346)
* 14 FETCH (UID 347)
. OK Completed (0.000 sec)
. xmove 1:2 inbox.sub
* 1 EXPUNGE
* 1 EXPUNGE
. OK [COPYUID 1338587236 322,332 1:2] Completed
. uid xmove 340 inbox.sub
* 5 EXPUNGE
. OK [COPYUID 1338587236 340 3] Completed
. enable qresync
* ENABLED CONDSTORE QRESYNC
. OK Completed
. xmove 1:* inbox.sub
* VANISHED 333:334,336,338,341:347
. OK [COPYUID 1338587236 333:334,336,338,341:347 4:14] Completed

Looks like it supports both with and without UID.

I would argue that since MOVE implies COPY + STORE + EXPUNGE, then
using the non-UID version of MOVE implies that you have to follow
the pipelining rules as if you had issued that COPY, that STORE and
that EXPUNGE.  Which means you can't pipeline multiple non-UID moves,
because the EXPUNGE command can't be pipelined with other actions
until it's been completed.

I'll go back and respond to that particular point in fact.

Bron.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Sat, 2 Jun 2012 17:11:52 +0200

This discussion is SO much better than the horrors we've had in the past.

Arnt