Subject: Re: [Extra] IMAP4rev2: how to signal canonical mailbox name after CREATE

From: Alexey Melnikov <[email-masked]>
Date: Mon, 6 Jul 2020 18:38:06 +0100

On 17/06/2020 13:27, Bron Gondwana wrote:

> On Wed, Jun 17, 2020, at 22:15, Timo Sirainen wrote:
>>>
>>> 1) abandon use of this response code (and use a new response 
>>> instead), e.g.
>>>
>>> * CREATED "a123" "Denormalized name" MAILBOXID "123456"
>>>
>>>  This would require clients to ignore an extra response, if they 
>>> don't recognize it.
>>>
>>
>> Somehow I've a feeling that there are going to be clients that will 
>> break if they see these kind of unknown untagged replies. But that's 
>> just a guess, so maybe not. We could of course send it only when 
>> client has used ENABLE IMAP4REV2 and then there wouldn't be any problems.
>
> I like this.  Only send if IMAP4REV2 is enabled and all[tm] your 
> problems go away.

Following on the suggestion to use LIST extended data items for this

       C: C04 create NonCannonicalName
       S: * LIST () "/" "CannonicalName" ("TAG" ("C04"))
       S: C04 OK done

In the example above imagine that "CannonicalName" is the canonical 
(normalized) version of "NonCannonicalName"

And the corresponding ABNF:

tag-extended-item = "TAG" SP "(" tag-string ")"
    ; Complies with &lt;mbox-list-extended-item&gt;
    ; The value complies with &lt;tagged-ext-val&gt;
    ; The value is the tag of the corresponding
    ; CREATE command.

And a similar extended data item can be used to return MAILBOXID (when 
supported).

Does this work for people?

Best Regards,

Alexey

---

From: "Bron Gondwana" <[email-masked]>
Date: Tue, 07 Jul 2020 09:30:51 +1000

>>>> 1) abandon use of this response code (and use a new response instead), e.g.

>>>>  This would require clients to ignore an extra response, if they don't recognize it.

>>> 
>>> Somehow I've a feeling that there are going to be clients that will break if they see these kind of unknown untagged replies. But that's just a guess, so maybe not. We could of course send it only when client has used ENABLE IMAP4REV2 and then there wouldn't be any problems.
>> 
>> I like this. Only send if IMAP4REV2 is enabled and all[tm] your problems go away.

> In the example above imagine that "CannonicalName" is the canonical (normalized) version of "NonCannonicalName"

> And a similar extended data item can be used to return MAILBOXID (when supported).

Yes, this works for me.

Bron.

--
 Bron Gondwana, CEO, Fastmail Pty Ltd
 [email-masked]

---

From: "Bron Gondwana" <[email-masked]>
Date: Wed, 08 Jul 2020 22:58:38 +1000

I have reviewed all the changes from -12 to -15 and am still mostly happy :) I haven't reviewed -15 in full because it's a big job!

Regarding the changes in -13, I have a couple of small things:

*Regarding unsolicited fetch:**
*

If the server chooses to send unsolicited FETCH responses, they MUST include UID FETCH item. Note that this is a new requirement when compared to RFC 3501.

This could do with a clarification that this only applies after "ENABLED IMAP4REV2".

*Regarding SELECT allowing non-canonical names:**
*

We've already discussed the list response to CREATE. Should SELECT also return the canonical representation of the mailbox name as part of the reply?

*Next steps:**
*

Alexey - are there any other changes still waiting before we do a working group last call?

Thanks,

Bron.

On Fri, Feb 14, 2020, at 10:47, Bron Gondwana wrote:
> Thanks Alexey, I see there's a -12 there now, and I'm happy with all the edits in it :)
> 
> There are still some outstanding points from this thread - are you expecting to issue a -13 for those?
> 
> Thanks,
> 
> Bron.
> 
> On Sat, Feb 8, 2020, at 04:04, Alexey Melnikov wrote:
>> Hi Bron, 

>> Replying to issues raised not covered by my reply to Timo:

>> On 29/01/2020 12:08, Bron Gondwana wrote:
>>> Sorry about the delay in doing this! I was supposed to be doing the shepherding write-up, and so I figured I should read it properly.
>>> 
>>> I've had a full read through over some very many days (I admit, I skimmed through the ABNF) and I have comments. It's long, and hopefully comprehensive - though I'm sure I missed things.
>>> 
>>> 
>>> *2.1 Link level*
>>> 
>>> Is it appropriate to mention TLS on 993 here? It's mentioned deep within the document, but not here.
>> Done.
>> 
>>> *2.3.2 Flags Message Attribute*
>>> 
>>> formatting issue: $Forwarded is duplicated 3 times and $MDNSent is duplicated twice in the plain-text format.
>> This is actually a bug in TXT rendering from XML.

>>  [snip]

>>> *6.3.1 ENABLE command*
>>> 
>>> The example shows a conversation with an imap4rev1 only server.
>> Fixed.
>> 
>>> The examples don't clearly show that only the newly enabled responses are correct, e.g. the sequence:
>>> 
>>> C1 ENABLE CONDSTORE
>>> * ENABLED CONDSTORE
>>> C1 OK foo
>>> C2 ENABLE CONDSTORE QRESYNC
>>> * ENABLED QRESYNC
>>> C2 OK foo
>>> 
>>> The second doesn't say "ENABLED CONDSTORE" because it was already enabled earlier on this connection.
>> Actually, it is perfectly Ok for the server to return in response to C2:

>> * ENABLED QRESYNC CONDSTORE
>> 
>> The text in the document wasn't entirely clear. It was trying to say that the preferred server behaviour is:
>> 
>> L1 ENABLE A B
>> * ENABLED A B
>> L1 OK foo
>> 
>> L2 ENABLE C
>> * ENABLED C
>> L2 OK foo
>> 
>> and not 
>> 
>> L2 ENABLE C
>> * ENABLED A B C
>> L2 OK foo
>> 
>> I clarified this.
>>> 
>>> (AKA: clients MUST remember what they have enabled)
>>> 
>>> *6.3.2 SELECT command*
>>> 
>>> The [UIDNEXT <n>] response is duplicated with two different pieces of text.
>> Again, this is a tool issue.
>> 
>>> *When deselecting a selected mailbox, the server MUST return an untagged OK response with the "[CLOSED]" response code when the currently selected mailbox *is closed (see Paragraph 10). ***- I'm not sure where "Paragraph 10" is.*
>> I will check. I think the section reference for response codes is missing.
>> 
>>> The last paragraph of this section has the word "deprecated" spelled as "depractated".
>> Fixed.
>> 
>>> 
>>> *6.3.7 SUBSCRIBE*
>>> 
>>> Speak of the devil. Lots of sites violate the MUST NOT unilaterally remove rule. Should we weaken it to SHOULD?
>> Done.
>> 
>>>  The justification for the MUST is "the server site might choose to do X" - well, if the server site is doing that, then maybe they SHOULDN'T unilaterally remove the subscriptions, but otherwise it's a better user experience and what people want.
>>> 
>>> (also: by enabling a non-default break the rules config, it's what our server does!)
>>> 
>>> *6.3.8 UNSUBSCRIBE*
>>> 
>>> *This command returns a tagged OK response only if the unsubscription is successful. *- this is not clear on what happens if the mailbox is not currently subscribed - does it silently succeed, or is there an expected error?
>>> 
>>> For what it's worth, our server just says "OK Completed" even if it's already been unsubscrbed - and likewise for subscribe if it's already subscribed it just say "yep, all good".
>> I agree that your server behaviour is sensible. Updated description of both SUBSCRIBE and UNSUBSCRIBE.
>> 
>>> 
>>> . create INBOX.foo
>>> . OK [MAILBOXID (7eca8f55-e233-4458-9d48-af854fb4b279)] Completed
>>> . subscribe INBOX.foo
>>> . OK Completed
>>> . unsubscribe INBOX.foo
>>> . OK Completed
>>> . unsubscribe INBOX.foo
>>> . OK Completed
>>> . subscribe INBOX.foo
>>> . OK Completed
>>> . subscribe INBOX.foo
>>> . OK Completed
>>> 
>>> 
>>> *6.3.9 LIST*
>>> 
>>> There's a weird line wrapping near the top of page 44 where a paragraph is split in half:
>>> 
>>>    The reference and mailbox name arguments are interpreted into a
   canonical form that represents an unambiguous left-to-right
   hierarchy.  The returned mailbox names will be in the interpreted
   form,

   that we call "canonical LIST pattern" later in this document.  To
   define the term "canonical LIST pattern" formally: it refers to the
   canonical pattern constructed internally by the server from the
   reference and mailbox name arguments.
>>> 
>> I will double check, but I think it is another rendering issue to TXT format (there is a comment in bettween 2 parts of the sentence.
>> 
>>> 
>>> *6.3.9.2 LIST Return Options*
>>> 
>>> The CHILDREN item is duplicated with two different pieces of text.
>> Again, a rendering issue. These are actually 2 separate paragraphs under the same item.

>>> *6.4.4 SEARCH*
>>>    For example,
   the criteria DELETED FROM "SMITH" SINCE 1-Feb-1994 refers to all
   deleted messages from Smith that were placed in the mailbox since
   February 1, 1994.
>>> 
>>> 
>>> That's not strictly true - SAVEDATE would be "placed in the mailbox". The wording should say "with INTERNALDATE greater than February 1, 1994" or so.
>> Good point. Fixed.

>> 
>>> *6.4.4.1 SAVE Result*
>>> 
>>> I'm not sure I understand the interaction here with closing and re-opening a mailbox. I would expect that closing a mailbox would wipe the SAVED result, and that trying to use a SAVED result when there isn't one should return an error.
>> I think Timo explained this well in his response.
>> 
>> 
>>  [snip]

>>> 
>>> *7.1 Status Responses*
>>> 
>>> Using 'o' as a tag in the ALREADYEXISTS item is confusing, it looks like an attempt to do dot points!
>> Changed :-).
>> 
>>  [snip]

>> _______________________________________________
>> Extra mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/extra
>> 
> 
> --

---

From: Alexey Melnikov <[email-masked]>
Date: Thu, 9 Jul 2020 11:33:46 +0100

Hi Bron,

On 08/07/2020 13:58, Bron Gondwana wrote:
> I have reviewed all the changes from -12 to -15 and am still mostly 
> happy :)  I haven't reviewed -15 in full because it's a big job!
>
> If the server chooses to send unsolicited FETCH responses, they 
> MUST include UID FETCH item.  Note that this is a new requirement when 
> compared to RFC 3501.
>

Actually, in this one case: I don't think so. It shouldn't break any old 
(IMAP4rev1 only) clients to return this.

But this made me think: Maybe the document needs a new section that list 
everything that the server should do after ENABLED IMAP4rev2?

> We've already discussed the list response to CREATE.  Should SELECT 
> also return the canonical representation of the mailbox name as part 
> of the reply?

This will be again returning an untagged LIST response on SELECT.

I will add it to my to do list.

>>>> Sorry about the delay in doing this!   I was supposed to be doing 
>>>> the shepherding write-up, and so I figured I should read it properly.
>>>>
>>>> I've had a full read through over some very many days (I admit, I 
>>>> skimmed through the ABNF) and I have comments.  It's long, and 
>>>> hopefully comprehensive - though I'm sure I missed things.
>>>>
>>>>
>>>> *2.1 Link level*
>>>>
>>>> formatting issue: $Forwarded is duplicated 3 times and $MDNSent is 
>>>> duplicated twice in the plain-text format.
>>>
>>>> The examples don't clearly show that only the newly enabled 
>>>> responses are correct, e.g. the sequence:
>>>>
>>>> C1 ENABLE CONDSTORE
>>>> * ENABLED CONDSTORE
>>>> C1 OK foo
>>>> /When deselecting a selected mailbox, the server MUST return an 
>>>> untagged OK response with the "[CLOSED]" response code when the 
>>>> currently selected mailbox *is closed (see Paragraph 10). */*- I'm 
>>>> not sure where "Paragraph 10" is.*
>>>> Speak of the devil.  Lots of sites violate the MUST NOT 
>>>> unilaterally remove rule. Should we weaken it to SHOULD?
>>> Done.
>>>
>>>>   The justification for the MUST is "the server site might choose 
>>>> to do X" - well, if the server site is doing that, then maybe they 
>>>> SHOULDN'T unilaterally remove the subscriptions, but otherwise it's 
>>>> a better user experience and what people want.
>>>>
>>>> /This command returns a tagged OK response only if the 
>>>> unsubscription is successful. /- this is not clear on what happens 
>>>> if the mailbox is not currently subscribed - does it silently 
>>>> succeed, or is there an expected error?
>>>>
>>>> For what it's worth, our server just says "OK Completed" even if 
>>>> it's already been unsubscrbed - and likewise for subscribe if it's 
>>>> already subscribed it just say "yep, all good".
>>> I will double check, but I think it is another rendering issue to 
>>> TXT format (there is a comment in bettween 2 parts of the sentence.
>>>
>>>>
>>>> That's not strictly true - SAVEDATE would be "placed in the 
>>>> mailbox".  The wording should say "with INTERNALDATE greater than 
>>>> February 1, 1994" or so.
>>>
>>> Good point. Fixed.
>>>
>>>
>>>> *6.4.4.1 SAVE Result*
>>>>
>>>> I'm not sure I understand the interaction here with closing and 
>>>> re-opening a mailbox.  I would expect that closing a mailbox would 
>>>> wipe the SAVED result, and that trying to use a SAVED result when 
>>>> there isn't one should return an error.
>>> https://www..ietf.org/mailman/listinfo/extra
>>>
>>
>> --

---

From: "Bron Gondwana" <[email-masked]>
Date: Thu, 09 Jul 2020 21:03:14 +1000

>> *Regarding unsolicited fetch:***
>> 
> Actually, in this one case: I don't think so. It shouldn't break any old (IMAP4rev1 only) clients to return this.

I don't even remember which client it bothered, but I remember one of them didn't like it.

Maybe it just didn't like it if you added UID to explicit FETCH when you didn't ask for it, not sure.

> But this made me think: Maybe the document needs a new section that list everything that the server should do after ENABLED IMAP4rev2?

>> *Regarding SELECT allowing non-canonical names:***
>> 

> I will add it to my to do list.

Yeah, that makes sense.

Cheers,

Bron.

---

From: Timo Sirainen <[email-masked]>
Date: Fri, 10 Jul 2020 09:44:48 +0300

> Does this work for people?

OK. But should we return this always or only when canonical name changes? With MAILBOXID at least always. I guess it could be nice to do it always in any case so clients don't forget to support it.

But in SELECT case perhaps only return it on name change?

---

From: Alexey Melnikov <[email-masked]>
Date: Fri, 10 Jul 2020 11:47:29 +0100

On 07/07/2020 00:30, Bron Gondwana wrote:

>>>> break if they see these kind of unknown untagged replies. But 
>>>> that's just a guess, so maybe not. We could of course send it only 
>>>> when client has used ENABLE IMAP4REV2 and then there wouldn't be 
>>>> any problems.
>>>

After thinking a bit more about this, I remembered that we already 
registered OLDNAME extended data item in Section 5.4 of RFC 5465. (We 
can think of canonical mailbox name as the server "renaming" the mailbox.)

Matching to oldname is easier than matching to the tag of the command 
that generated the change.

So unless there are objections, I would go with that instead.

---

From: Timo Sirainen <[email-masked]>
Date: Sat, 11 Jul 2020 11:26:15 +0300

On 10. Jul 2020, at 13.47, Alexey Melnikov <[email-masked]> wrote:
> After thinking a bit more about this, I remembered that we already registered OLDNAME extended data item in Section 5.4 of RFC 5465. (We can think of canonical mailbox name as the server "renaming" the mailbox.)
> 
> Matching to oldname is easier than matching to the tag of the command that generated the change.
> 
> 
> 

Hmm. I guess that could work.. What's the behavior if the mailbox name doesn't change? First I was thinking:

 * MAILBOXID supported: Send LIST () "/" "name" (MAILBOXID "12345")
 * No MAILBOXID supported: Don't send LIST reply at all

But since CREATE already should return MAILBOXID in the tagged reply, it's not very useful to send MAILBOXID in the LIST reply, right? So the LIST reply should be only if the mailbox name changes? If it happens rarely enough, it could be that clients won't actually implement that.

Also, should we require that this name changing only is allowed for normalizing UTF8, or could it be used for "mailbox alias names"? That's been kind of an issue previously where two different folder names (both in LIST reply) point to the same physical folder, but there's no way to tell this to cilents so they cache the mails twice. Although in theory the client could realize this already by seeing that two folders have the same MAILBOXID.

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 13 Jul 2020 16:23:32 +0100

Hi Timo,

> On 10 Jul 2020, at 07:44, Timo Sirainen <[email-masked]> wrote:
> 

(The same answer applies to both TAG and OLDNAME)
I think when it changes (or MAILBOXID is returned), but this doesn’t break if the server returns it always.

> But in SELECT case perhaps only return it on name change?

Yes, save these extra bytes :-).

Best Regards,
Alexey

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 13 Jul 2020 16:29:38 +0100

Hi Timo,

> On 11 Jul 2020, at 09:26, Timo Sirainen <[email-masked]> wrote:
> 
> But since CREATE already should return MAILBOXID in the tagged reply, it's not very useful to send MAILBOXID in the LIST reply, right?
Indeed.

> So the LIST reply should be only if the mailbox name changes? If it happens rarely enough, it could be that clients won't actually implement that.
> 

I am thinking that mailbox alias names can be supported by exact the same mechanism. I had a similar thought about aliases. But I was not sure what, if anything the document should say about them.

Best Regards,
Alexey

---

From: "Bron Gondwana" <[email-masked]>
Date: Tue, 14 Jul 2020 08:24:02 +1000

On Tue, Jul 14, 2020, at 01:23, Alexey Melnikov wrote:
>> On 6.. Jul 2020, at 20.38, Alexey Melnikov <[email-masked]> wrote:
>>> 

>>>>> 
>>>>> Somehow I've a feeling that there are going to be clients that will break if they see these kind of unknown untagged replies. But that's just a guess, so maybe not. We could of course send it only when client has used ENABLE IMAP4REV2 and then there wouldn't be any problems..
>>>> 

> Yes, save these extra bytes :-).

I disagree - I'd rather just have it ALWAYS be in the SELECT/EXAMINE response. It will still fit within a single packet, so it's not going to cost any more on the wire.

Bron.