Subject: [imapext] Proposal for a new IMAP Working Group to revise CONDSTORE & QRESYNC

From: Alexey Melnikov <[email-masked]>
Date: Tue, 05 Mar 2013 14:57:25 +0000

Hi,
I've noticed some surge in activity related to implementing CONDSTORE 
(RFC 4551)/QRESYNC (RFC 5162) in both clients and servers, so I drafted 
a new WG charter proposal to update/clarify these:

-----------
The Internet Message Access Protocol (IMAP), defined in RFC 3501,
specifies a protocol for accessing email messages on a server
that implements a message store from a client. It also includes
commands for manipulating the message store -- creating, deleting, and
renaming mailboxes, adding a message to a mailbox, and copying
messages from one mailbox to another.

Base IMAP as described in RFC 3501 requires that in order to discover
flag changes and expunged messages in a mailbox, the client has to
fetch flags for all messages it knows in the mailbox and compare returned
results with its own state. This can generate a significant amount of
traffic for big mailboxes.

The IMAP CONDSTORE extension (RFC 4551) provides a facility for IMAP
clients to quickly resynchronize mailbox flag changes in a mailbox.
The IMAP QRESYNC extension (RFC 5162) extended CONDSTORE to also
cover expunged messages and reduced the number of round trips needed
to resynchronize by extending the SELECT/EXAMINE command.
Both extensions seen deployment in both clients and servers.
These deployments has exposed errors and clarity issues in the
specifications, which need correcting.

The IMAP QRESYNC extension (imapqresync) working group has the task
of updating CONDSTORE and QRESYNC extensions as Proposed Standards.
The working group might produce one (combined) or two separate
documents (as now) updating these extensions. The working group will
review errata and update the documents as needed to incorporate those,
and will correct significant errors and inconsistencies, but will keep
changes at this stage to a minimum. In the event that implementation
experience would dictate an incompatible change to one (or both) of
these extensions, the corresponding IMAP capability identifiers would
be changed.

No other IMAP extension work is in scope for this working group.
---------

So please comment on whether you think this work is worth doing and on 
the proposed charter text.

I've also posted a new version of RFC 5162bis 
(http://tools.ietf.org/html/draft-melnikov-5162bis-01), which addressed 
most (but not all) of existing errata on the document. If a new WG is to 
be formed, I propose that the document is used as one of the initial 
documents.

Thank you,
Alexey

---

From: Timo Sirainen <[email-masked]>
Date: Tue, 5 Mar 2013 21:01:53 +0200

On 5.3.2013, at 16.57, Alexey Melnikov <[email-masked]> wrote:

> I've noticed some surge in activity related to implementing CONDSTORE (RFC 4551)/QRESYNC (RFC 5162) in both clients and servers, so I drafted a new WG charter proposal to update/clarify these:
..
> So please comment on whether you think this work is worth doing and on the proposed charter text.

Can we keep it on this mailing list? I've too many mailing lists already. :)

> I've also posted a new version of RFC 5162bis (http://tools.ietf.org/html/draft-melnikov-5162bis-01), which addressed most (but not all) of existing errata on the document. If a new WG is to be formed, I propose that the document is used as one of the initial documents.

Sounds like a good idea. I didn't read it yet, but some QRESYNC/CONDSTORE things that come to my mind:

 - command length limits as discussed by Michael
 - don't allow server to send VANISHED duplicates
 - algorithm how server uses the seq/uid mapping could be better to reduce how much data to send (I was planning on thinking this through but never got around to it)
 - \recent flag changes highestmodseq? probably nobody cares and makes it more difficult for server.
 - condstore: is anyone really using per-flag modseq counters?

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 05 Mar 2013 19:09:39 +0000

> Can we keep it on this mailing list? I've too many mailing lists already. :)
If our ADs are agreeable, the idea is to reuse [email-masked].
>   - don't allow server to send VANISHED duplicates
I think I dealt with this in the most recent draft. But yes, this needs 
to be fixed.
>   - \recent flag changes highestmodseq?
No.
> probably nobody cares and makes it more difficult for server.

---

From: Dave Cridland <[email-masked]>
Date: Tue, 5 Mar 2013 19:14:21 +0000

> I've noticed some surge in activity related to implementing CONDSTORE
(RFC 4551)/QRESYNC (RFC 5162) in both clients and servers, so I drafted a
new WG charter proposal to update/clarify these:

For the record, I'm keen to get this working group chartered, and I'm in a
position to dedicate a reasonable amount of time to it, with the support of
my employer.

In particular I'm able to volunteer to co-edit either or both
specifications as needed.

(For those who haven't noticed, my employer has changed recently; I'm back
in the client game)

Dave.

---

From: Barry Leiba <[email-masked]>
Date: Tue, 5 Mar 2013 14:35:45 -0500

> If our ADs are agreeable, the idea is to reuse [email-masked].

Yes, that's my plan as well.  It worked well for imapmove.

Barry, one of our ADs

---

From: Pete Maclean <[email-masked]>
Date: Tue, 05 Mar 2013 20:47:31 -0500

I am at the point of almost completing a QRESYNC implementation in my 
server and I would very much like to see some clarifications in the 
spec.  I am sorry to say this to an audience that includes its 
authors but making sense of RFC 5162 proved to be very taxing on my 
brain.  Even now, a colleague and I have differing interpretations of 
parts of it that we have yet to definitely resolve.

Pete Maclean

---

From: Barry Leiba <[email-masked]>
Date: Tue, 5 Mar 2013 21:06:38 -0500

> I am at the point of almost completing a QRESYNC implementation in my server
> and I would very much like to see some clarifications in the spec.  I am
> sorry to say this to an audience that includes its authors but making sense
> of RFC 5162 proved to be very taxing on my brain.  Even now, a colleague and
> I have differing interpretations of parts of it that we have yet to
> definitely resolve.

Please don't be "sorry" to say it: the authors, as much as (or more
than) anyone, want to get the spec right.  Your recent implementation
experience with certainly be critical in determining what's wrong and
what needs clarification.  Please stick with us, and help fix it.

In particular, while we're going through a formal chartering process
now, there's no reason to wait for that before posting a list of what
parts you had problems with.  Alexey's actively working on an update,
and he'll start incorporating fixes for your issues right away.

Barry

---

From: Michael M Slusarz <[email-masked]>
Date: Tue, 05 Mar 2013 19:54:55 -0700

Quoting Timo Sirainen <[email-masked]>:

>> I've noticed some surge in activity related to implementing  
>> CONDSTORE (RFC 4551)/QRESYNC (RFC 5162) in both clients and  
>> servers, so I drafted a new WG charter proposal to update/clarify  
>> these:
> ..
>> (http://tools.ietf.org/html/draft-melnikov-5162bis-01), which  
>> addressed most (but not all) of existing errata on the document. If  
>> a new WG is to be formed, I propose that the document is used as  
>> one of the initial documents.
>
> Sounds like a good idea. I didn't read it yet, but some  
> QRESYNC/CONDSTORE things that come to my mind:
>
>  - algorithm how server uses the seq/uid mapping could be better to  
> reduce how much data to send (I was planning on thinking this  
> through but never got around to it)

Adding to Timo's list of things to possibly discuss/consider:

- Make VANISHED responses mandatory - or at least an option to allow  
for this (I think I saw some placeholder text in the draft regarding  
this)

- There was discussion a few weeks ago about whether a server should  
return "ENABLED QRESYNC" or "ENABLED QRESYNC CONDSTORE" to an "ENABLE  
QRESYNC" command

- Editorial suggestion: A better discussion, or at least a more open  
discussion, that in practical use servers don't tend to "remember"  
expunged messages for long.  It was a bit confusing and tripped me up  
as a client author since this implementation detail is sort of buried  
in the server implementation sections.

E.g. Section 3.2 could probably use an example of a server that only  
remembers expunges over the last 100 modseqs and has consolidated all  
expunges before that into a single checkpointed stored value.  So  
people aren't as surprised, as I was, the first time they see a  
VANISHED FETCH of (HIGHESTMODSEQ - 50) that somehow returns tens of  
thousands of UIDs when in reality only a few were actually expunged in  
that modseq timeframe.

- condstore: this might be more of an implementation tip, but this is  
valid response in CONDSTORE that is not necessarily intuitive and  
needs to be handled:

a SELECT INBOX (CONDSTORE)
[...]
* 2 EXISTS
* OK [HIGHESTMODSEQ 5] Highest
a OK [READ-WRITE] SELECT completed.
b UID FETCH 5 (UID)
* 1 FETCH (UID 5)
* 2 FETCH (FLAGS (\Seen) MODSEQ (6))
b OK FETCH completed.

michael

___________________________________
Michael Slusarz [[email-masked]]

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 6 Mar 2013 10:01:27 +0000

Hi Pete,
I am very interested to see your list of issues. Please send it either here or directly to me (the former is preferred).

Best Regards,
Alexey

On 6 Mar 2013, at 01:47, Pete Maclean <[email-masked]> wrote:

> I am at the point of almost completing a QRESYNC implementation in my server and I would very much like to see some clarifications in the spec.  I am sorry to say this to an audience that includes its authors but making sense of RFC 5162 proved to be very taxing on my brain.  Even now, a colleague and I have differing interpretations of parts of it that we have yet to definitely resolve.
> 
> Pete Maclean

---

From: Cyrus Daboo <[email-masked]>
Date: Wed, 06 Mar 2013 10:57:47 -0500

Hi Alexey,

> be formed, I propose that the document is used as one of the initial

What about also doing an update to RFC4549?

I think the big thing client implementors want is a single clear 
specification of extensions that describe how to accurately do mailbox 
synchronization. To that end I would even argue that we should be looking 
at pieces to take out to achieve that goal. For example, how many clients 
actually make use of the condition store piece of CONDSTORE, as opposed to 
just the SELECT/SEARCH sync operations? Same goes for QRESYNC - what pieces 
of it have actually been used by clients? Are there bits that could be 
removed?

So I'd like to see more than a simple rev of the documents to fix errata - 
or at least lets not rule that out right now.

-- 
Cyrus Daboo

---

From: Dave Cridland <[email-masked]>
Date: Wed, 6 Mar 2013 17:43:42 +0000

On Mar 6, 2013 10:57 AM, "Cyrus Daboo" <[email-masked]> wrote:
>
> Hi Alexey,
>
>
> --On March 5, 2013 at 2:57:25 PM +0000 Alexey Melnikov <
>
> So I'd like to see more than a simple rev of the documents to fix errata
- or at least lets not rule that out right now.

There's certainly unused pieces, like the complex per flag modseq, that can
be dropped. I'm not totally sure that general conditional store is
something that's so onerous that our needs dropping, though.

I'm in ready agreement that the QRESYNC document could use some serious
improvements to its prose, too. It was hard to write effectively, not least
because the technique was relatively new, and the reviewers were almost
entirely server side as I recall. I'm hopeful that with many more client
side reviewers we can write a clearer description of how it all works.

Dave.

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 06 Mar 2013 18:10:04 +0000

> What about also doing an update to RFC4549?

I am not against revising RFC 4549, for example to fold sections from 
CONDSTORE & QRESYNC into it. But I am somewhat worried to volunteer for 
something that might drag beyond the end of 2013. I think I would like 
to hear what other people think about this.

> synchronization. To that end I would even argue that we should be 
> looking at pieces to take out to achieve that goal. For example, how 
> many clients actually make use of the condition store piece of 
> CONDSTORE, as opposed to just the SELECT/SEARCH sync operations? Same 
> goes for QRESYNC - what pieces of it have actually been used by 
> clients? Are there bits that could be removed?

Per flag modseq is an interesting question to discuss.
Another thing I would like to obsolete is the old SEARCH MODSEQ response 
syntax, I would really like to move to ESEARCH.

But these are just some ideas to think about and discuss.

> So I'd like to see more than a simple rev of the documents to fix 
> errata - or at least lets not rule that out right now.

Ack.

---

From: Michael M Slusarz <[email-masked]>
Date: Wed, 06 Mar 2013 13:26:39 -0700

Quoting Dave Cridland <[email-masked]>:

>> For example, how many clients
>> just the SELECT/SEARCH sync operations?
>

FWIW, we use it:

C: 8 UID STORE 46041 (UNCHANGEDSINCE 70609) +FLAGS \deleted
S: * 12 FETCH (UID 46041 MODSEQ (70610) FLAGS (\Deleted \Seen $junk))
S: 8 OK Store completed.

michael

---

From: Cyrus Daboo <[email-masked]>
Date: Wed, 06 Mar 2013 15:30:40 -0500

Hi Michael,

--On March 6, 2013 at 1:26:39 PM -0700 Michael M Slusarz 
<[email-masked]> wrote:

>> There's certainly unused pieces, like the complex per flag modseq, that
>> can be dropped. I'm not totally sure that general conditional store is

What do you do when there is a failure due to a conflicting change?

-- 
Cyrus Daboo

---

From: Michael M Slusarz <[email-masked]>
Date: Wed, 06 Mar 2013 13:40:05 -0700

> What do you do when there is a failure due to a conflicting change?

Refresh mailbox/message view with new flag state; generic error  
message to user to the effect of "[mailbox actions: at least one  
message flag/message actions: the flags] could not be set as another  
process has changed the message flags on the server".  Probably a bit  
techie in terms of language, but this is also not a common occurrence  
so I haven't really felt the need to tweak.

I think we ignore for delete actions, but I can't recall for sure.

michael

---

From: Timo Sirainen <[email-masked]>
Date: Wed, 6 Mar 2013 22:47:54 +0200

On 6.3.2013, at 22.40, Michael M Slusarz <[email-masked]> wrote:

> Refresh mailbox/message view with new flag state; generic error message to user to the effect of "[mailbox actions: at least one message flag/message actions: the flags] could not be set as another process has changed the message flags on the server".  Probably a bit techie in terms of language, but this is also not a common occurrence so I haven't really felt the need to tweak.
> 

I don't really see any point in using UNCHANGEDSINCE for any system flags. For \Seen you've clearly seen it even if someone else has done something with the message. For \Deleted you've clearly decided it's not worth having the message. For \Answered you've again clearly answered the message regardless of what someone else may have done. \Draft probably should be only used in APPEND. \Flagged is probably usually also something you don't care if someone else has changed the flags, but depending on use I guess there could be a reason..

Now, for custom keywords there can be a ton of possibilities for UNCHANGEDSINCE. So I'm not interested in removing that feature. The per-flag modseqs seem quite pointless though (I still haven't really understood how they even work, not that I've really tried to either).

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Wed, 06 Mar 2013 20:51:23 +0000

I've also been researching on and off support for QRESYNC.  It's fairly 
involved, especially relating to storing historic data about expunged 
records.  So any clarification that would make implementation easier I'd 
be keen on.

We've had CONDSTORE for quite some time.

We didn't implement per-flag mod sequences, I didn't see any point.

Do any clients use it?

I'd be very interested to know which clients support QRESYNC.

Regards

Adrien

------ Original Message ------
From: "Timo Sirainen" <[email-masked]>
To: "Michael M Slusarz" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Sent: 7/03/2013 9:47:54 a.m.
Subject: Re: [imapext] Proposal for a new IMAP Working Group to revise 
CONDSTORE & QRESYNC
>I don't really see any point in using UNCHANGEDSINCE for any system 
>flags. For \Seen you've clearly seen it even if someone else has done 
>something with the message. For \Deleted you've clearly decided it's 
>not worth having the message. For \Answered you've again clearly 
>answered the message regardless of what someone else may have done. 
>\Draft probably should be only used in APPEND. \Flagged is probably 
>usually also something you don't care if someone else has changed the 
>flags, but depending on use I guess there could be a reason..
>
>Now, for custom keywords there can be a ton of possibilities for 
>UNCHANGEDSINCE. So I'm not interested in removing that feature. The 
>per-flag modseqs seem quite pointless though (I still haven't really 
>understood how they even work, not that I've really tried to either).
>
>_______________________________________________
>imapext mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/imapext

---

From: Jan Kundrát <[email-masked]>
Date: Thu, 07 Mar 2013 15:31:38 +0100

> I'd be very interested to know which clients support QRESYNC.

I looked at some mobile clients in my thesis last summer [1]. The results can be summarized roughly like this:

- Trojita supports QRESYNC, of course.

- QRESYNC is supported by Modest/Tinymail (Nokia N900) and the QMF (Nokia N9, N950 -- the MeeGo Harmattan series).

- CONDSTORE (only) is supported by iOS.

- Android's stock client doesn't support anything; K9 doesn't have any support for CONDSTORE/QRESYNC, either.

- Thunderbird's source code contains references to CONDSTORE, so it looks like it supports it, but not QRESYNC.

Hope this helps,
Jan

[1] http://trojita.flaska.net/msc-thesis.pdf

-- 
Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Thu, 07 Mar 2013 23:45:43 +0000

Thanks for that Jan

did you check recent versions of iOS?

anyone else know of any common desktop clients supporting QRESYNC?

I did a wee audit a while ago and got

* Horde_Imap_Client
* RoundCubeMail (0.7-beta onwards)
* em Client
Adrien

------ Original Message ------
From: "Jan Kundrát" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Sent: 8/03/2013 3:31:38 a.m.
Subject: [imapext] Re: Proposal for a new IMAP Working Group to revise 
CONDSTORE & QRESYNC
>I looked at some mobile clients in my thesis last summer [1]. The 
>results can be summarized roughly like this:
>
>- QRESYNC is supported by Modest/Tinymail (Nokia N900) and the QMF 
>(Nokia N9, N950 -- the MeeGo Harmattan series).
>
>- Thunderbird's source code contains references to CONDSTORE, so it 
>looks like it supports it, but not QRESYNC.
>
>Hope this helps,
>Jan
>
>-- Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/

---

From: Jan Kundrát <[email-masked]>
Date: Fri, 08 Mar 2013 10:25:52 +0100

> did you check recent versions of iOS?

What I did was looking at the publicly available information in their changelogs, release notes and developer information. The CONDSTORE was documented as being added in some version, but there was no such information about the QRESYNC.

With kind regards,
Jan

---

From: Dave Cridland <[email-masked]>
Date: Fri, 8 Mar 2013 09:18:39 -0500

On 8 Mar 2013 04:26, "Jan Kundrát" <[email-masked]> wrote:
> What I did was looking at the publicly available information in their
changelogs, release notes and developer information. The CONDSTORE was
documented as being added in some version, but there was no such
information about the QRESYNC.

I have to admit I thought QRESYNC was in there - is anyone with an iPhone
and a QRESYNC-supporting mailserver able to check?

Dave.

---

From: Ashley Clark <[email-masked]>
Date: Fri, 8 Mar 2013 06:50:21 -0800

iOS added support for CONDSTORE in 4.0. It does not support QRESYNC.

–Ashley

On Mar 8, 2013, at 6:18 AM, Dave Cridland <[email-masked]> wrote:

> I have to admit I thought QRESYNC was in there - is anyone with an iPhone and a QRESYNC-supporting mailserver able to check?
> 
> Dave.
>