Subject: [Extra] New draft: IMAP UNIQUEID

From: Bron Gondwana <[email-masked]>
Date: Tue, 20 Mar 2018 05:10:54 +1100

https://datatracker.ietf.org/doc/draft-gondwana-imap-uniqueid/

This is a pretty rough first draft, so the wording will need some
smithing, but the implementation looks good in Cyrus IMAP:
brong@zula:~/src/cyrus-imapd$ git show | diffstat
imapd.c       |   43 +++++++++++++++++++++++++++++---------
imapd.h       |   15 ++++++++-----
imapparse.c   |   29 ++++++++++++++++++++++++++
index.c       |   26 +++++++++++++++++++++++
mailbox.h     |    3 +-
search_expr.c |   64
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++6 files changed, 162 insertions(+), 18 deletions(-)

That implements the entire spec.  Of course, we already have the unique
IDs stored internally, and we've already implemented all the JMAP
support for IDs, so it makes sense that it was easy for me.
Open questions:
* bytestring 1..255 for everything?
* more "prior art" discussion for X-GM work?

Cheers,

Bron.

--
  Bron Gondwana, CEO, FastMail Pty Ltd
  [email-masked]

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 20 Mar 2018 10:45:17 +0000

Bron Gondwana writes:
> This is a pretty rough first draft, so the wording will need 
> some smithing, but the implementation looks good in Cyrus IMAP:

I see you didn't do any thread id.

A thread ID has been a very convenient part of gmail's extensions, most 
importantly to fetch and display a thread without having to engage with the 
entire mailbox, which can be very, very big on gmail.

> * bytestring 1..255 for everything?

I bet the cyrus implementation doesn't use that. It uses something that 
needs no quoting anywhere. Right?

Specs that say "anything goes" and are implemented with a nice restricted 
repertoire are trouble magnets. Clients will use e.g. JSON and forget to 
quote, and later when the first "\/ comes along, there's a bug. Needless.

Better to specify a repertoire that's reasonably well filled by running 
code. We've discussed this before and you know my opinion, but really, 
anything is okay so long as the the spec is mostly tested by running code.

> * more "prior art" discussion for X-GM work?

Doesn't matter IMO.

Arnt

---

From: Bron Gondwana <[email-masked]>
Date: Tue, 20 Mar 2018 22:56:15 +1100

> I see you didn't do any thread id.

Yes, THRID is exactly a thread id.

> A thread ID has been a very convenient part of gmail's
> extensions, most> importantly to fetch and display a thread without having to engage
> with the> entire mailbox, which can be very, very big on gmail.
> 
> I bet the cyrus implementation doesn't use that. It uses
> something that> needs no quoting anywhere. Right?

Kinda. Cyrus uses a guid for mailbox uniqueid which may need quoting.
The emailid from jmap and thread id I've copied exactly into the Cyrus
implementation of this draft.
In particular, the message id is the first 24 characters of the hex
representation of the full digest.sha1, and the thread id is the hex of
a 64 bit number which is derived from the sha1 of the first seen message
in that thread.
Yes, I do plan to move away from sha1 eventually!

> Specs that say "anything goes" and are implemented with a nice
> restricted> repertoire are trouble magnets. Clients will use e.g. JSON and
> forget to> quote, and later when the first "\/ comes along, there's a bug.
> Needless.> 
> Better to specify a repertoire that's reasonably well filled by
> running> code. We've discussed this before and you know my opinion, but really,> anything is okay so long as the the spec is mostly tested by
> running code.
Something that can be an atom would be nice for sure. Easy enough to
convert Cyrus over, not sure about everything else.  We'll have to bring
jmap along with whatever it is, because compatibility is important.
> _________________________________________________
> Extra mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/extra

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 20 Mar 2018 12:10:45 +0000

Just one comment.

> Yes, THRID is exactly a thread id.

Optional, ie. not reliably there. Gmail's is reliable. You write one path 
of code and that works and is fast, and that's what makes it convenient.

Arnt

---

From: Bron Gondwana <[email-masked]>
Date: Tue, 20 Mar 2018 23:23:16 +1100

Fallback is to treat each message as its own thread. I could also define
it to return the msgid if the server doesn't support threading, but then
the client can't know.
I guess that's a legit question to the list though, what would
people prefer as the behaviour for a server which doesn't have
thread data stored?
I've been very careful to make this standard implementable in a way that
doesn't need any additional data stored in the server, albeit with
reduced usefulness in that case.
Bron.

On Tue, 20 Mar 2018, at 23:10, Arnt Gulbrandsen wrote:
> Optional, ie. not reliably there. Gmail's is reliable. You write
> one path> of code and that works and is fast, and that's what makes it
> convenient.> 
> Arnt
>

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 20 Mar 2018 12:40:29 +0000

Bron Gondwana writes:
> Fallback is to treat each message as its own thread. I could 
> also define it to return the msgid if the server doesn't support 
> threading, but then the client can't know.
>

You could get both by saying that if a message's own ID and its thread ID 
are the same, that means the message cannot be threaded by the server, 
either because the server doesn't thread, because the message lacks a 
message-id, or for some other reason.

This permits the clients that prefer simplicity use one code path, while 
clients that want to can detect the situation. The downside is that a 
compliant server must somehow make sure that its thread and message IDs 
don't clash.

Arnt

---

From: Bron Gondwana <[email-masked]>
Date: Wed, 21 Mar 2018 00:40:01 +1100

On Tue, 20 Mar 2018, at 23:40, Arnt Gulbrandsen wrote:
> You could get both by saying that if a message's own ID and its
> thread ID> are the same, that means the message cannot be threaded by the server,> either because the server doesn't thread, because the message lacks a> message-id, or for some other reason.
> 
> This permits the clients that prefer simplicity use one code
> path, while> clients that want to can detect the situation. The downside is that a> compliant server must somehow make sure that its thread and
> message IDs> don't clash.

Sounds like a possibility.  I'll bring that up on Thursday, and propose
the alternatives to the list in a separate message afterwards to
collect opinions.
Bron.

---

From: Brandon Long <[email-masked]>
Date: Tue, 20 Mar 2018 17:37:52 +0000

Note that in Gmail, the threadid is the msgid of the first message on the
thread, so our impl wouldn't be compatible.

Brandon

On Tue, Mar 20, 2018, 6:40 AM Bron Gondwana <[email-masked]> wrote:

> the alternatives to the list in a separate message afterwards to collect
> opinions.
>
> Bron.
>
>
> --
> _______________________________________________
> Extra mailing list
> [email-masked]

---

From: "Chris Newman" <[email-masked]>
Date: Wed, 21 Mar 2018 14:43:01 +0000

We already have a protocol element called "unique identifer" in 
base-spec IMAP. Adding a new protocol element called "UNIQUEID" with 
different semantics from "unique identifier" is confusing. I suggest 
FOLDERID or MAILBOXID as alternatives that are less confusing. FYI, 
we've implemented this protocol element (right now it's called 
VENDOR.ORACLE-FOLDERID and the argument is an integer).

Adding a new IMAP protocol element called "MSGID" with different 
semantics from the existing RFC 5322 protocol element "message id" is 
confusing. I suggest EID or EMAILID as an alternative.

I'm not sure THRID is a good name. It looks a lot like a misspelling of 
THIRD which may make the protocol harder to read. I suggest either TID 
or THREADID as alternatives.

The specification needs ABNF.

The specification does not follow the syntax model in RFC 4466 for 
STATUS. You would have to make the folder id an integer or put an extra 
layer of () around the folderid string to follow that syntax model. I'm 
not convinced it's important to follow the RFC 4466 syntax model for 
STATUS, but I thought I'd point out the issue so it can be discussed.

Other than those issues, I support this proposal and am interested in 
implementing the parts of it we have not already implemented (EID & 
THRID).

Our implementation also has a VENDOR.ORACLE-FOLDERMODSEQ status item. 
This is updated when mailbox metadata (RFC 5464) changes. Please 
consider if that feature would be a useful part of this feature bundle. 
The two status items (folder id & folder modseq) make it easier to build 
a stateless activesync gateway to IMAP.

		- Chris

---

From: Bron Gondwana <[email-masked]>
Date: Thu, 22 Mar 2018 02:45:24 +1100

> VENDOR.ORACLE-FOLDERID and the argument is an integer).

I'm happy with MAILBOXID to be compatible with JMAP.  In Cyrus
it's a UUID.
> semantics from the existing RFC 5322 protocol element "message id" is> confusing. I suggest EID or EMAILID as an alternative.

Let's go with EMAILID to be compatible.

> I'm not sure THRID is a good name. It looks a lot like a
> misspelling of> THIRD which may make the protocol harder to read. I suggest either TID> or THREADID as alternatives.

Likewise THREADID.

> The specification needs ABNF.

Yes, I'll get there!  

> STATUS. You would have to make the folder id an integer or put
> an extra> layer of () around the folderid string to follow that syntax
> model. I'm> not convinced it's important to follow the RFC 4466 syntax model for
I'm happy to get guidance from the group on this.  Probably depends what
deployed parsers are doing.
> Other than those issues, I support this proposal and am interested in> implementing the parts of it we have not already implemented (EID &
> THRID).
> 
> Our implementation also has a VENDOR.ORACLE-FOLDERMODSEQ status item.> This is updated when mailbox metadata (RFC 5464) changes. Please
> consider if that feature would be a useful part of this feature
> bundle.> The two status items (folder id & folder modseq) make it easier
> to build> a stateless activesync gateway to IMAP.

Tell me more about this and how it's semantically different from
HIGHESTMODSEQ.  We have something similar in cyrus (mbentry->modseq)
which isn't used for anything yet, but is planned to be used for
replication.  And of course we'll probably be using it for JMAP to know
that only counts have changed.
However - I wouldn't include that in THIS specification.  This
specification is about things which don't change!
Bron.

---

From: Josef 'Jeff' Sipek <[email-masked]>
Date: Wed, 21 Mar 2018 12:34:52 -0400

On Wed, Mar 21, 2018 at 14:43:01 +0000, Chris Newman wrote:
> We already have a protocol element called "unique identifer" in base-spec
> IMAP. Adding a new protocol element called "UNIQUEID" with different
> semantics from "unique identifier" is confusing. I suggest FOLDERID or
> MAILBOXID as alternatives that are less confusing.

100% agreed.

> FYI, we've implemented
> this protocol element (right now it's called VENDOR.ORACLE-FOLDERID and the
> argument is an integer).

FWIW, we have X-GUID in Dovecot:

a status inbox (uidvalidity x-guid)
* STATUS inbox (UIDVALIDITY 1521206682 X-GUID 704283249ac5ab5ac7190000807e0fe7)

a fetch 1 (uid x-guid)
* 1 FETCH (UID 1749 X-GUID 90a39608dec8ab5aea330000807e0fe7)

I don't think there is a way to get an email's thread-id right now.

> Adding a new IMAP protocol element called "MSGID" with different semantics
> from the existing RFC 5322 protocol element "message id" is confusing. I
> suggest EID or EMAILID as an alternative.
>
> THIRD which may make the protocol harder to read. I suggest either TID or
> THREADID as alternatives.

It took me 4 passes over one of the emails in this thread to notices that it
doesn't say THIRD.

I think I like EMAILID & THREADID more than the EID/TID.

All 3 items (mailbox, mail, and thread ids) are useful, and it's good to see
this codified.

Jeff.

> The specification does not follow the syntax model in RFC 4466 for STATUS.
> You would have to make the folder id an integer or put an extra layer of ()
> around the folderid string to follow that syntax model. I'm not convinced
> it's important to follow the RFC 4466 syntax model for STATUS, but I thought
> I'd point out the issue so it can be discussed.
> 
> implementing the parts of it we have not already implemented (EID & THRID).
> 
> Our implementation also has a VENDOR.ORACLE-FOLDERMODSEQ status item. This
> is updated when mailbox metadata (RFC 5464) changes. Please consider if that
> feature would be a useful part of this feature bundle. The two status items
> (folder id & folder modseq) make it easier to build a stateless activesync
> gateway to IMAP.
> 
> 		- Chris
> 

-- 
I have always wished for my computer to be as easy to use as my telephone;
my wish has come true because I can no longer figure out how to use my
telephone.
		- Bjarne Stroustrup

---

From: Bron Gondwana <[email-masked]>
Date: Thu, 22 Mar 2018 04:19:29 +1100

>   704283249ac5ab5ac7190000807e0fe7)
Thanks Josef - does this look like an argument in favour of ignoring
4466's grammar?  I think maybe it does!  You've clearly taken the same
path I did of not requiring the STATUS value to be a number or a list...
> * 1 FETCH (UID 1749 X-GUID 90a39608dec8ab5aea330000807e0fe7)

Definitely looks like another reason that the choice of string rather
than number is right for EMAILID and THREADID (I've already updated
the draft)
> I don't think there is a way to get an email's thread-id right now.

Are you storing some kind of thread correlation key?  I only have a
pretty old version of the dovecot source lying around on my laptop (from
last time I build imaptest from source!) and it looks at a glance like
maybe it's building threads from first principles each time.
Mostly I'm interested in "if you are, will this spec work for it?".

Cheers,

Bron.

---

From: "Chris Newman" <[email-masked]>
Date: Wed, 21 Mar 2018 20:34:42 +0000

> path I did of not requiring the STATUS value to be a number or a 

One of the authors of RFC 4466 will be at the meeting (and is on this 
list). Perhaps he recalls why the STATUS grammar was restricted to a 
number or a list?

		- Chris

---

From: Josef 'Jeff' Sipek <[email-masked]>
Date: Wed, 21 Mar 2018 20:05:51 -0400

> path I did of not requiring the STATUS value to be a number or a list...

Both FETCH X-GUID and STATUS X-GUID have been in Dovecot for almost 10
years, so I suspect that the reasoning behind why it was done this way is
long lost.

It looks like STATUS X-GUID is the only non-integer STATUS we return.

About 4466: I don't really think that's an argument either way.  We'll have
to add the new names anyway and it is easy enough to add them either way.
(At least in Dovecot, for STATUS is it literally a format string change and
for FETCH it is only marginally harder.)

> than number is right for EMAILID and THREADID (I've already updated

Yeah, I think a string is less constraining on the implementation.

I agree with Arnt that allowing any bytes could lead to ugly escaping issues
(especially for IMAP/JMAP gateways), and so I think it would be worth it to
restrict the string to something sane - printable ASCII?

Obviously, Dovecot would be ok with 0-9 and a-f, but I think that'd be
unnecessarily over-constrained.

> maybe it's building threads from first principles each time.

Dovecot keeps a per-mailbox index file (dovecot.index.thread) which
maintains a list of CRC32s (or at least that's what the field is called) and
indexes into the main index file.  Together, I think the two index files
contain enough info construct THREAD responses.

It isn't always clear how the values flow between the in-memory structures
and the index & cache files on disk in Dovecot.  It is also possible that
there are opportunities to optimize this in Dovecot, most clients seem to
ignore THREAD and just build their own threading info.  We'll want to
revisit threading in Dovecot anyway to figure out how to exactly handle
jmap's mailbox-less approach to threads.

> Mostly I'm interested in "if you are, will this spec work for it?".

I haven't really had a need to understand that code, so I'm not sure
how exactly it works.  I should look into it more to see what we do about
supporting multiple THREAD algos and how that fits the proposed THREADID.
Naively, it would seem to be that THREADIDs depend on the threading algo one
selects - but in IMAP the threading algo is per THREAD command and not a
server config.

All I meant by my previous comment was that we don't have an existing X-FOO
to FETCH this thread-id.  

Jeff.

-- 
Ready; T=0.01/0.01 16:31:03