Subject: Re: [imapext] CONDSTORE / QRESYNC

From: Bron Gondwana <[email-masked]>
Date: Sun, 07 Jul 2013 22:29:51 +1000

On Wed, Jun 12, 2013, at 05:13 AM, Alexey Melnikov wrote:

On 11/06/2013 14:49, Dave Cridland wrote:

On Tue, Jun 11, 2013 at 1:38 PM, Timo Sirainen <[1][email-masked]> wrote:

On 11.6.2013, at 13.39, Dave Cridland <[2][email-masked]> wrote:

> On the other hand, I cannot think of a case when I'd actually want
per-metadata MODSEQ - it's also not clear to me that any extant servers
actually support this anyway. In the absence of per-metadata MODSEQ,
then a client can add a flag if the message is still unseen (for
example), which seems to me to be a more useful case.

> Does any server actually implement per-metadata MODSEQ? Does anyone
want this?

Perhaps it could be a useful difference for message flag changes vs.
annotation changes. But I haven't implemented annotation support yet,
so I'm not really sure.

One possible option there would be to call this out as out of scope,
and require the annotation extension to have its own modification
sequence.

If this is a question of what to say in CONDSTORE, I would be Ok with
that. For example saying something like "CONDSTORE requires one modseq
per all flags/keywords. Other IMAP extensions might add additional
modseqs for other metadata (or might reuse the flag one)." would be Ok.
I would be less happy with modifying existing syntax (apart from
removing something which is already optional) or requiring other
extensions to use their own per-metadata modseqs.

I agree with this.  At the moment we don't store separate modseqs with
the annotations/metadata.  I've been considering it.  It would be nice
not to pay the full metadata cost (at the moment it's a 200ish byte
preview) during replication for every flag change.

Bron.

--
Bron Gondwana
[email-masked]

References

1. mailto:[email-masked]
2. mailto:[email-masked]

---

From: Dave Cridland <[email-masked]>
Date: Sun, 07 Jul 2013 15:34:57 -0000

Bron Gondwana wrote:

            If this is a question of what to say in CONDSTORE, I would
            be Ok with that. For example saying something like
            "CONDSTORE requires one modseq per all flags/keywords.
            Other IMAP extensions might add additional modseqs for
            other metadata (or might reuse the flag one)." would be
            Ok. I would be less happy with modifying existing syntax
            (apart from removing something which is already optional)
            or requiring other extensions to use their own per-
            metadata modseqs.

Do you have any specific text you'd like to see here?

---

From: Bron Gondwana <[email-masked]>
Date: Mon, 08 Jul 2013 22:42:29 +1000

On Mon, Jul 8, 2013, at 01:34 AM, Dave Cridland wrote:

Bron Gondwana wrote:

Thinking about it...

There's another thing I'm trying to get into coherent words now.  There
are two very different things that happen with qresync selects:

1) the server has full information, and sends a tiny bit of data.
2) the server doesn't have full information, and dumps a potentially
large amount of information.

A client may want case 1, but not want case 2.  A server might want to
generate case 1, but not all the data for case 2.

So it makes sense to distingish between the two cases.  A way to say
"tell me what's changed if you know, or just tell me there's not enough
information and I'll query for the data I need".

Anyway... that's what's going through my head today.  We do something
similar in the Cyrus sync protocol, where if the DELETEDMODSEQ
(highmark of forgotten data) is less than last time, we can just do a
delta sync, otherwise we have to do a full metadata sync.

Bron.

--
Bron Gondwana
[email-masked]

---

From: Dave Cridland <[email-masked]>
Date: Mon, 08 Jul 2013 13:40:58 -0000

Bron Gondwana wrote:

In theory, that can't happen. In principle, the protocol's designed
such that a server has all the information to minimize the transfer.

There's two discrete information dumps that happen:

a) FLAGS (and in principle other metadata) changed since the given
MODSEQ. This grows proportionally to the amount of changes, but it
will never transfer more data than is needed. Note that a client can
reduce the scope here by claiming only a particular range is important
- Polymer did this, as I recall, by requesting only "old" messages.

b) VANISHED messages since the MODSEQ given. If a server doesn't store
the deleted MODSEQs, then this could grow, but a client has the
ability to control this by sending the mapping data samples which can
limit it. In addition, this compresses well, of course, in the set
representation.

So a really poor client might cause a lot of work for a server, but I
don't think that's particularly limited to QRESYNC - a client not
using QRESYNC at all and doing the classic "UID FETCH 1:* (FLAGS)"
will never cause more work than a bad QRESYNC client, will it?
(Outside of pathological cases like a client asking for everything,
with a MODSEQ of 1 - even then it's not significantly more data).

Dave.