Subject: [EAI] Comments on draft-ietf-eai-imap-utf8-04.txt

From: Alexey Melnikov <[email-masked]>
Date: Fri, 14 Nov 2008 17:50:30 +0000

Pete,
I've done a detailed review of the latest IMAP draft. You can consider 
it my WGLC-style review.
The document is in a good shape, but I think it would need 1 more 
revision before going to IETF LC.

> 3.1.  IMAP UTF-8 Quoted Strings

 [...]

>    All IMAP servers SHOULD accept UTF-8 in mailbox names and IMAP
>    servers which support the "Mailbox International Naming Convention"
>    described in RFC 3501 section 5.1.3 MUST accept utf8-quoted mailbox
>    names and convert them to the appropriate internal format.  [TBD
>    stringprep for mailbox names?  Can we reuse SASLprep?].

Note that I was faced with a similar issue in the ManageSieve protocol, 
where script names are in UTF-8. After some discussion in the Sieve WG 
we've decided to go with something like this:

   A Sieve script name is a sequence of Unicode characters encoded in
   UTF-8 [UTF-8].  A script name MUST comply with Net-Unicode Definition
   (Section 2 of [NET-UNICODE]), with the following additional
   restrictions:

   o  0000-001F; [CONTROL CHARACTERS]

   o  007F; DELETE

   o  0080-009F; [CONTROL CHARACTERS]

   o  2028; LINE SEPARATOR

   o  2029; PARAGRAPH SEPARATOR

Where:

[NET-UNICODE] Klensin, J. and M. Padlipsky, "Unicode Format for Network
Interchange", RFC 5198, March 2008.

>    IMAP servers MUST NOT accept UTF-8 characters when storing a new
>    message keyword, unless the mailbox is UTF-8 only, in which case IMAP
>    servers SHOULD accept UTF-8 in message keywords.  [TBD stringprep for
>    message keywords?  Can we reuse SASLprep?]

Haven't we decided to leave keywords as ASCII some time ago?
I don't have a strong opinion either way.

But anyway, this should be similar to the above, with some additional 
restrictions imposed by IMAP: spaces and some US-ASCII separator 
characters are not allowed.

> 3.2.  UTF8 Parameter to SELECT and EXAMINE
>
>    The "UTF8" capability also indicates the server supports the UTF8
>    parameter to SELECT and EXAMINE.  When a mailbox is selected with the
>    UTF8 parameter, it alters the behavior of all IMAP commands related
>
>    to message sizes, message headers and MIME body headers so they refer
>    to the message with UTF-8 headers.  If the mailstore is not UTF-8
>    header native and the SELECT or EXAMINE command with UTF-8 header
>    modifier succeeds, then the server MUST return results as if the
>    mailstore was UTF-8 header native with upconversion requirements as
>    described in Section 8.  The server MAY reject the SELECT or EXAMINE
>    command with the [NOT-UTF-8] response code, unless the UTF8=ALL or
>    UTF8=ONLY capability is advertised.
>
>    Servers MAY include mailboxes which can only be selected or examined
>    if the UTF8 parameter is provided.  However, such mailboxes MUST NOT
>    be included in the output of an unextended LIST, LSUB or equivalent
>    command.  If a client attempts to SELECT or EXAMINE such mailboxes
>    without the UTF8 parameter, the server MUST reject the command with a
>    [UTF-8-ONLY] response code.  As a result, such mailboxes will not
>    accessible by IMAP clients written prior to this specification and
>    are discouraged unless the server advertises UTF8=ONLY or the server
>    implements IMAP4 LIST Command Extensions [RFC5258].
>
>    TBD: describe syntax based on draft-melnikov-imap-ext-abnf-05.

I think I can help with that ;-):

   utf8-select-param = "UTF8"
                     ;; Conforms to <select-param> from RFC 4466.

> 3.4.  UTF-8 Interaction with IMAP4 LIST Command Extensions
>
>    When an IMAP server advertises both the "UTF8" capability and the
>    "LIST-EXTENEDED" [RFC5258] capability, the server MUST support the
>    LIST extensions described in this section.  When an IMAP server
>    advertises the UTF8=ONLY capability and the LIST-EXTENDED capability,
>    the server MUST reject these LIST extensions with a BAD response.

Ok, I might be confused here, but I think this contradicts description 
of the UTF8=ONLY capability in section 7:

   This capability permits an IMAP server to advertise that it does not
   support the international mailbox name convention (modified UTF-7),
   and does not permit selection or examination of any mailbox unless
   the UTF8 parameter is provided.

> 3.4.1.  UTF8 and UTF8ONLY LIST Selection Options
>
>    The UTF8 LIST selection option tells the server to include mailboxes
>    that only support UTF-8 headers in the output of the list command.
>    The UTF8ONLY LIST selection option tells the server to include all
>    mailboxes that support UTF-8 headers and to exclude mailboxes that
>    don't support UTF-8 headers.  Note that UTF8ONLY implies UTF8 so it
>    is not necessary for the client to request both.  Use of either
>    selection option will also result in UTF-8 mailbox names in the
>    result as described in Section 3.3.

This need to clarify that either selection option implies the UTF8 
return option described in section 3.4.2,
which in its term means that "\NoUTF8" and "\UTF8Only" mailbox 
attributes need to be returned when necessary.
The only place where this information can be found is in the IANA 
registration itself. I think most readers would miss that.

> 3.4.2.  UTF8 LIST Return Option
>
>    If the client supplies the UTF8 LIST return option, then the server
>    MUST include either the \NoUTF8 or the \UTF8Only mailbox attribute as
>    appropriate.  The \NoUTF8 mailbox attribute indicates an attempt to
>    SELECT or EXAMINE that mailbox with the UTF8 parameter will fail with
>    a [NOT-UTF-8] response code.  The \UTF8Only mailbox attribute
>    indicates an attempt to SELECT or EXAMINE that mailbox without the
>    UTF8 parameter will fail with a [UTF-8-ONLY] response code.  Note
>    that computing this information may be expensive on some server
>    implementations so this return option should not be used unless
>    necessary.
>
>    The ABNF [RFC5234] for these LIST extensions follows:
>
>      list-select-independent-opt =/ "UTF8" / "UTF8ONLY"

This has some implications that you might have not thought about: 
extending this productions means that these options don't interact with 
other options, such as RECURSIVEMATCH. While I think the UTF8 option 
should behave in the same way as the REMOTE option (as it might 
potentially extend the list of returned mailboxes), I think UTF8ONLY 
should be treated more like SUBSCRIBED (as it subsets the list of 
returned mailboxes).

So, if you agree, I suggest changing this to:

     list-select-independent-opt =/ "UTF8"
     list-select-base-opt =/ "UTF8ONLY"

>      mbox-list-oflag             =/ "\NoUTF8" / "\UTF8Only"

I think you've copied typo from an old LIST-EXTENDED draft: please 
change "mbox-list-oflag" to "mbx-list-oflag".
 [...]

> 4.  UTF8=APPEND Capability
>
>    If the UTF8=APPEND capability is advertised, then the server accepts
>    UTF-8 headers in the APPEND command message argument.  A client which
>    sends a message with UTF-8 headers to the server MUST include the
>    UTF8 APPEND parameter.  The ABNF for this APPEND parameter follows:
>
>      append-ext    =/ "UTF8"

This is probably Ok, but note that RFC 4466 required each label to have 
a parameter. Whether this is a bug, or there was a reason for that, I 
currently don't remember.

>    A server which advertises UTF8=APPEND has to comply with the
>    requirements of the IMAP base specification and [RFC5322] for message
>    fetching.  Mechanisms for 7-bit downgrading to help comply with the
>    standards are discussed in Downgrading mechanism for
>    Internationalized eMail Address (IMA) [I-D.ietf-eai-downgrade].
>
>    IMAP servers which do not advertise the UTF8=APPEND or UTF8=ONLY
>    capability SHOULD reject an APPEND command which includes any 8-bit
>    in the message headers with a "NO" response.

Does this need a new response code? I am not entirely sure.

> 8.  Up-Conversion Server Requirements
>
>    When an IMAP4 server uses a traditional mailbox format that includes
>    7-bit headers and it chooses to permit access to that mailbox with
>    the UTF8 parameter, it MUST support minimal up-conversion as
>    described in this section.
>
>    The server MUST support up-conversion of the following address
>    header-fields in the message header: From, Sender, To, CC, Bcc,
>    Resent-From, Resent-Sender, Resent-To, Resent-CC, Resent-Bcc, and
>    Reply-To.  This up-conversion MUST include address local-parts
>    encoded according to [TBD],

Did you mean utf8-local-part defined in section 4.4 of RFC 5335?

>    address domains encoded according to IDNA

>    [RFC3490], and MIME header encoding [RFC2047] of display-names and
>    any [RFC5322] comments.

_______________________________________________
IMA mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ima

---

From: John C Klensin <[email-masked]>
Date: Fri, 14 Nov 2008 14:20:55 -0500

--On Friday, 14 November, 2008 17:50 +0000 Alexey Melnikov
<[email-masked]> wrote:

> Pete,
> I've done a detailed review of the latest IMAP draft. You can
> consider it my WGLC-style review.
> The document is in a good shape, but I think it would need 1
> more revision before going to IETF LC.
> 
>...

> Note that I was faced with a similar issue in the ManageSieve
> protocol, where script names are in UTF-8. After some
> discussion in the Sieve WG we've decided to go with something
> like this:
> 

It would be better if "restrictions" were supplemented with some
words that make it painfully clear that these characters are not
permitted.  I know that is what you mean, but it is possible to
misread this and get confused about it.

> [NET-UNICODE] Klensin, J. and M. Padlipsky, "Unicode Format
> for Network Interchange", RFC 5198, March 2008.

>...

>>    IMAP servers MUST NOT accept UTF-8 characters when storing
>>    a new message keyword, unless the mailbox is UTF-8 only,
>>    in which case IMAP servers SHOULD accept UTF-8 in message
>>    keywords.  [TBD stringprep for message keywords?  Can we
>>    reuse SASLprep?]
> 

We have indeed, at least for header field names.  I believe that
the logic extended to these cases.

I do have a strong opinion that, if non-ASCII keywords are
accepted, you are going to need to address comparison models for
the keywords, including normalization and case-folding issues.
We sort of know how to do that, but I can't recommend a tour of
that rat hole.

     john

---

From: Pete Resnick <[email-masked]>
Date: Thu, 20 Nov 2008 11:54:59 -0600

On 11/14/08 at 5:50 PM +0000, Alexey Melnikov wrote:

>>[TBD stringprep for mailbox names?  Can we reuse SASLprep?].
>
>protocol, where script names are in UTF-8. After some discussion in 
>the Sieve WG we've decided to go with something like this:
>

Works for me.

>>    ...unless the mailbox is UTF-8 only, in which case IMAP
>Haven't we decided to leave keywords as ASCII some time ago? I don't 
>have a strong opinion either way.

Same here. I'll check with Chris as to why he put this in the original version.

>                     ;; Conforms to <select-param> from RFC 4466.

OK.

>Ok, I might be confused here, but I think this contradicts 
>description of the UTF8=ONLY capability in section 7:
>

I think the point here is that if you have UTF8=ONLY, then it is 
pointless to use the UTF8 or UTF8ONLY param to LIST, since all of the 
mailboxes only support UTF-8 headers. But I guess the param will do 
no harm either. Again, I'll ask Chris.

>return option described in section 3.4.2, which in its term means 
>that "\NoUTF8" and "\UTF8Only" mailbox attributes need to be 
>returned when necessary. The only place where this information can 
>be found is in the IANA registration itself. I think most readers 
>would miss that.

I'll repeat it here.

>     list-select-base-opt =/ "UTF8ONLY"

No problem.

>change "mbox-list-oflag" to "mbx-list-oflag".

Done.

>This is probably Ok, but note that RFC 4466 required each label to 
>have a parameter. Whether this is a bug, or there was a reason for 
>that, I currently don't remember.

Crap. That's going to give parsers serious problems. Let's discuss offline.

>Does this need a new response code? I am not entirely sure.

(*Shrug*) I don't think so.

>Did you mean utf8-local-part defined in section 4.4 of RFC 5335?

Thank you.

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
Qualcomm Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102
_______________________________________________
IMA mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ima

---

From: Alexey Melnikov <[email-masked]>
Date: Thu, 20 Nov 2008 13:08:52 -0600

Pete Resnick wrote:

> pointless to use the UTF8 or UTF8ONLY param to LIST,

Pointless for a client that only supports UTF8=ONLY - probably.
However, I think the intent is to encourage client deployments now. In 
short term there would be no UTF8=ONLY servers, so clients would need to 
work when UTF8 and UTF8ONLY are required. I don't think you want to 
require two code paths in such clients.

> since all of the mailboxes only support UTF-8 headers. But I guess the 
> param will do no harm either. Again, I'll ask Chris.

 [...]

> Crap. That's going to give parsers serious problems. Let's discuss 

Sure.

---

From: Joseph Yee <[email-masked]>
Date: Thu, 20 Nov 2008 14:43:15 -0500

>> Interchange", RFC 5198, March 2008.

Not related to Sieve, but current IMAP mailbox names are expected to 
store in modified UTF-7 format.  Any chance to relax it since we are 
i18n-ize (lazy to spell) other parts already.  Don't remember seeing the 
discussion of this online before...

> Same here. I'll check with Chris as to why he put this in the original 

Slightly strong opinion here :)
Pre predefined keywords like "Answered", "Seen" should stay the same, 
but user generated keywords?  Would love to see it available.  But if it 
meant new mechanism, different way to handle depends on CAPA, or diverse 
charset support...I am happy to stay in ASCII for now.

> mailboxes only support UTF-8 headers. But I guess the param will do no 
> harm either. Again, I'll ask Chris.
> 
>> return option described in section 3.4.2, which in its term means that 
>> "\NoUTF8" and "\UTF8Only" mailbox attributes need to be returned when 
>> necessary. The only place where this information can be found is in 
>> the IANA registration itself. I think most readers would miss that.
> 
> I'll repeat it here.
> 

Thanks

Joseph

---

From: Pete Resnick <[email-masked]>
Date: Thu, 20 Nov 2008 13:56:09 -0600

On 11/20/08 at 2:43 PM -0500, Joseph Yee wrote:

>i18n-ize (lazy to spell) other parts already.  Don't remember seeing 
>the discussion of this online before...

Is section 3.3 not sufficient? It says:

"After an IMAP client successfully issues an "ENABLE UTF8" command, 
the server MUST NOT return in LIST results any mailbox names to the 
client following the IMAP4 Mailbox International Naming Convention. 
Instead, the server MUST return any mailbox names with characters 
outside the US-ASCII repertorie using utf8-quoted syntax. (The IMAP4 
Mailbox International Naming Convention has proved problematic in the 
past, so the desire is to make this syntax obsolete as quickly as 
possible.)"

>>>Haven't we decided to leave keywords as ASCII some time ago? I 
>>>don't have a strong opinion either way.
>>
>Pre predefined keywords like "Answered", "Seen" should stay the 
>same, but user generated keywords?  Would love to see it available. 
>But if it meant new mechanism, different way to handle depends on 
>CAPA, or diverse charset support...I am happy to stay in ASCII for 
>now.

You might want to address John Klensin's comments on this. On 
11/14/08 at 2:20 PM -0500, John C Klensin wrote:

>I do have a strong opinion that, if non-ASCII keywords are accepted, 
>you are going to need to address comparison models for the keywords, 
>including normalization and case-folding issues. We sort of know how 
>to do that, but I can't recommend a tour of that rat hole.

pr

---

From: Pete Resnick <[email-masked]>
Date: Thu, 20 Nov 2008 15:10:55 -0600

>>Crap. That's going to give parsers serious problems. Let's discuss offline.

OK, we discussed offline, and the above is wrong. Instead, we think 
we need to replace with:

utf8-literal = "UTF8" SP "(" literal8 ")"
append-data =/ utf8-literal

We will also need to update CATENATE (RFC 4469) by adding:

cat-part =/ utf8-literal

Does that look OK to everyone? (And if you don't have the stomach to 
read 4466, your silence will be taken as acceptance.)

pr

---

From: Chris Newman <[email-masked]>
Date: Thu, 20 Nov 2008 16:14:56 -0600

> Same here. I'll check with Chris as to why he put this in the original

I wrote it before that decision.  Fine to go with ASCII particularly since 
annotations have gotten further.

> harm either. Again, I'll ask Chris.

Yes, that's my intent and yes the text should be fixed so it's not 
contradictory.

> (*Shrug*) I don't think so.

The litmus test for response codes is "can the client do something 
reasonable and different with the additional error than it would with a 
generic error".  I think the answer is "no" in this case as the client is 
busted already if it's sending EAI to a server that won't accept it.

		- Chris

---

From: Alexey Melnikov <[email-masked]>
Date: Thu, 20 Nov 2008 16:22:22 -0600

Chris Newman wrote:

> --On November 20, 2008 11:54:59 -0600 Pete Resnick 
> <[email-masked]> wrote:
>

Just to make sure that everyone is on the page, I would like to discuss 
this quickly during the Minneapolis meeting.