Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
From: Mark Crispin <[email-masked]>
To: Alexey Melnikov <[email-masked]>
Cc: Arnt Gulbrandsen <[email-masked]>, IMAP Extensions <[email-masked]>
Date: Fri, 21 Sep 2007 07:46:51 -0700 (PDT)
List-Id: <ietf-imapext.imc.org>

On Thu, 20 Sep 2007, Alexey Melnikov wrote:
>>>>    This allows i;unicode-casemap to be used for collation
>>>>    in IMAP servers that do not implement IMAP-I18N.
>>> I am not sure I understand why you are trying to say here. By IMAP-I18N I 
>>> assume you've meant the COMPARATOR extension?
>> No, I mean the entire IMAP-I18N document.
> IMAP-I18N effectively consist of 2 parts: the COMPARATOR extension and the 
> LANGUAGE extension. The LANGUAGE extension has nothing to do with searching.

Right, and what I am saying is that all IMAP servers should use (or be 
upgraded to use) i;unicode-casemap today, even if they do not implement 
COMPARATOR and/or LANGUAGE.

I consider both COMPARATOR and LANGUAGE to be frills, but upgrading 
searching to use i;unicode-casemap to be essential.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Date: Thu, 20 Sep 2007 23:43:05 +0100
From: Alexey Melnikov <[email-masked]>
To: Mark Crispin <[email-masked]>
CC: Arnt Gulbrandsen <[email-masked]>, IMAP Extensions <[email-masked]>
Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Sender: [email-masked]

Mark Crispin wrote:

> On Sat, 15 Sep 2007, Alexey Melnikov wrote:
>
>>>    This allows i;unicode-casemap to be used for collation
>>>    in IMAP servers that do not implement IMAP-I18N.
>>
>> I am not sure I understand why you are trying to say here. By 
>> IMAP-I18N I assume you've meant the COMPARATOR extension?
>
> No, I mean the entire IMAP-I18N document.

IMAP-I18N effectively consist of 2 parts: the COMPARATOR extension and 
the LANGUAGE extension. The LANGUAGE extension has nothing to do with 
searching.

>   There's a lot in IMAP-I18N, but the immediate pressing problem is 
> SEARCH comparison and SORT/THREAD collation.
>
> Every server should use i;unicode-casemap instead of something like 
> "convert 'a'-'z' to 'A'-'Z' in both strings, then use strstr() or 
> strcmp() as appropriate".  ALL servers that do the latter should be 
> upgraded to use i;unicode-casecmp *now*, even if full IMAP-I18N 
> support is not going to happen right away.
>
> I'd go so far as to say that this requirement belongs in the base 
> specification and should be in the next update to the base specification.

I agree with the last sentence.

>> And I don't like statements like this in general. This document can't 
>> put any requirements on implementations that don't support any 
>> extension from it.
>
> For now, it can be a should, or a recommendation pending an update in 
> the base specification.

Such statement has to be tied to an extension (such as UNICASEMAP and 
COMPARATOR).
In theory an implementation can say that it complies with RFC XXXX 
(future IMAP-I18N RFC number), but doesn't support any extensions from 
it, but that would be silly.

> Nonetheless, the day when a server could get away with ASCII casemap 
> and strstr()/strcmp() is at an end.
>
> The c-client library has an open source implementation that is easily 
> separable from the rest of c-client.
>
>>> Then, you need to decide if other collations used in IMAP MUST 
>>> follow the charset resolution process laid out in 
>>> i;unicode-casemap.  If so, perhaps you want to make 
>>> i;unicode-casemap normative (if it isn't already) and state that its 
>>> Step 1 is mandatory for other collations.
>>
>> IMAP I18N can't change requirements on collations without changing 
>> RFC 4790. So this is not an option, unless there is WG consensus that 
>> RFC 4790 is seriously broken and need to be replaced.
>
> I was unclear here; this is not a requirement on collations, but 
> rather a requirement on IMAP's use of collations.  If another 
> collation does not have this procedure (which is possible since RFC 
> 4790 does not require it), then IMAP requires that this procedure be 
> performed (as an IMAP-mandated "step 0") prior to the collation action 
> if that collation is used in IMAP.
>
> "You are allowed to use X in Z.  But if X does not do Y as part of its 
> steps, then you must first do Y yourself before doing X in Z."
>
>> However, IMAP I18N can state that the resolution process in IMAP is 
>> the same as Step 1 from i;unicode-casemap.
>
> We may be saying the same thing.

After you clarification above, I agree.

> The basic point here is that you don't have to do the same thing twice 
> if the collation also requires it, but you must do it at least once in 
> IMAP.
>
>>> Another alternative is that IMAP-I18N can specify a different 
>>> resolution procedure.  If i;unicode-casemap only sees well-formed 
>>> UTF-8 strings, then all of its Step 1 becomes a no-op.
>>
>> This is also an option.
>
> That is, however, an option that I hope is not chosen.

Date: Sat, 15 Sep 2007 08:16:36 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: Alexey Melnikov <[email-masked]>
cc: Arnt Gulbrandsen <[email-masked]>, IMAP Extensions <[email-masked]>
Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Sender: [email-masked]

On Sat, 15 Sep 2007, Alexey Melnikov wrote:
>>    This allows i;unicode-casemap to be used for collation
>>    in IMAP servers that do not implement IMAP-I18N.
> I am not sure I understand why you are trying to say here. By IMAP-I18N I 
> assume you've meant the COMPARATOR extension?

No, I mean the entire IMAP-I18N document.  There's a lot in IMAP-I18N, but 
the immediate pressing problem is SEARCH comparison and SORT/THREAD 
collation.

Every server should use i;unicode-casemap instead of something like 
"convert 'a'-'z' to 'A'-'Z' in both strings, then use strstr() or strcmp() 
as appropriate".  ALL servers that do the latter should be upgraded to use 
i;unicode-casecmp *now*, even if full IMAP-I18N support is not going to 
happen right away.

I'd go so far as to say that this requirement belongs in the base 
specification and should be in the next update to the base specification.

> And I don't like statements like this in general. This document can't put any 
> requirements on implementations that don't support any extension from it.

For now, it can be a should, or a recommendation pending an update in the 
base specification.  Nonetheless, the day when a server could get away 
with ASCII casemap and strstr()/strcmp() is at an end.

The c-client library has an open source implementation that is easily 
separable from the rest of c-client.

>> Then, you need to decide if other collations used in IMAP MUST follow the 
>> charset resolution process laid out in i;unicode-casemap.  If so, perhaps 
>> you want to make i;unicode-casemap normative (if it isn't already) and 
>> state that its Step 1 is mandatory for other collations.
> IMAP I18N can't change requirements on collations without changing RFC 4790. 
> So this is not an option, unless there is WG consensus that RFC 4790 is 
> seriously broken and need to be replaced.

I was unclear here; this is not a requirement on collations, but rather a 
requirement on IMAP's use of collations.  If another collation does not 
have this procedure (which is possible since RFC 4790 does not require 
it), then IMAP requires that this procedure be performed (as an 
IMAP-mandated "step 0") prior to the collation action if that collation is 
used in IMAP.

"You are allowed to use X in Z.  But if X does not do Y as part of its 
steps, then you must first do Y yourself before doing X in Z."

> However, IMAP I18N can state that the resolution process in IMAP is the same 
> as Step 1 from i;unicode-casemap.

We may be saying the same thing.  The basic point here is that you don't 
have to do the same thing twice if the collation also requires it, but you 
must do it at least once in IMAP.

>> Another alternative is that IMAP-I18N can specify a different resolution 
>> procedure.  If i;unicode-casemap only sees well-formed UTF-8 strings, then 
>> all of its Step 1 becomes a no-op.
> This is also an option.

That is, however, an option that I hope is not chosen.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Date: Sat, 15 Sep 2007 11:28:27 +0100
From: Alexey Melnikov <[email-masked]>
To: Mark Crispin <[email-masked]>
CC: Arnt Gulbrandsen <[email-masked]>, IMAP Extensions <[email-masked]>
Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Sender: [email-masked]

Mark Crispin wrote:

> On Sun, 9 Sep 2007, Alexey Melnikov wrote:
>
>> What does "convert character encodings compatibly" means? (Compatibly 
>> for what?) A collation defines which character set it operates on. I 
>> suggest we just say so. Or, considering that support for UTF-8 
>> charset is required by this document, we can just say that all 
>> strings are converted to Unicode before invoking the active 
>> comparator. This will match what Sieve spec is saying.
>
> Better would be to lay out the problem.  Something to the effect of:
>
>    IMAP strings may be in arbitrary charsets, which must be resolved
>    against the collation's input domain.

I like this part so far.

>    The i;unicode-casemap collation accepts input in arbitrary charsets,
>    and specifies a charset resolution procedure involving conversion to
>    Unicode and fallback to octet comparison for strings that can not be
>    converted.

This is just summarizing i;unicode-casemap behavior. I don't mind this 
text, as long as it remains informative.

>    This allows i;unicode-casemap to be used for collation
>    in IMAP servers that do not implement IMAP-I18N.

I am not sure I understand why you are trying to say here. By IMAP-I18N 
I assume you've meant the COMPARATOR extension?

And I don't like statements like this in general. This document can't 
put any requirements on implementations that don't support any extension 
from it.

> Then, you need to decide if other collations used in IMAP MUST follow 
> the charset resolution process laid out in i;unicode-casemap.  If so, 
> perhaps you want to make i;unicode-casemap normative (if it isn't 
> already) and state that its Step 1 is mandatory for other collations.

IMAP I18N can't change requirements on collations without changing RFC 
4790. So this is not an option, unless there is WG consensus that RFC 
4790 is seriously broken and need to be replaced.
However, IMAP I18N can state that the resolution process in IMAP is the 
same as Step 1 from i;unicode-casemap.

> Another alternative is that IMAP-I18N can specify a different 
> resolution procedure.  If i;unicode-casemap only sees well-formed 
> UTF-8 strings, then all of its Step 1 becomes a no-op.

This is also an option.

> This is why I fought so hard to keep Step 1; I felt that it was 
> necessary for i;unicode-casemap to be usable without an i18n document 
> for that protocol.  [Put another way: Sieve can override Step 1 if it 
> so desires by feeding i;unicode-casemap only well-formed UTF-8 strings.] 

Date: Wed, 12 Sep 2007 20:26:33 -0500
To: Arnt Gulbrandsen <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Cc: [email-masked]
Sender: [email-masked]

On 9/12/07 at 3:00 PM +0200, Arnt Gulbrandsen wrote:

>(FYI: I've been ill. Still not quite well. My mail backlog will take 
>many days to work off.)

Sorry to hear it. I'll hold off on any action until you're caught up 
and let me know that we're good to go for IETF Last Call.

Feel better soon.

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated

Date: Wed, 12 Sep 2007 15:09:39 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Cc: Alexey Melnikov <[email-masked]>, Pete Resnick <[email-masked]>, Dan Karp <[email-masked]>, Timo Sirainen <[email-masked]>, Dave Cridland <[email-masked]>
Sender: [email-masked]

Dave Cridland writes:
> But you can search for that using "i;unicode-casemap" anyway. It's  
> the fallback position described in step 1(b). There is no invalid  
> input for "i;unicode-casemap".

The SEARCH key doesn't have to not accept invalid input. AFAICT, for a 
command such as "a search charset utf-8 body x", the server is 
permitted to check the UTF-8-ness of x and say BAD if it hates what it 
sees. (Permitted or required, I don't think it matters in this case.)

Arnt

Date: Wed, 12 Sep 2007 15:00:35 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Cc: Alexey Melnikov <[email-masked]>, Pete Resnick <[email-masked]>, Dan Karp <[email-masked]>, Dave Cridland <[email-masked]>
Sender: [email-masked]

(FYI: I've been ill. Still not quite well. My mail backlog will take 
many days to work off.)

Dave Cridland writes:
> On Wed Sep  5 13:14:17 2007, Dan Karp wrote:
>
>> > Well, given that you have to implement "i;octet" anyway for  >  
>> "i;unicode-casemap", I'm not sure it makes any real difference  >  
>> whether we mandate it or not.
>>
>> That's not the case.  It certainly isn't so in my case...
>
> Well, you don't have to expose it, but you do need to implement all  
> of its operations.

Perhaps. I don't do it (or won't do it - the code isn't complete). I'm 
not sure whether my finished implementation should be said to comply 
with i;unicode-casemap, though.

Arnt

Date: Mon, 10 Sep 2007 23:02:33 +0100
From: Alexey Melnikov <[email-masked]>
To: Mark Crispin <[email-masked]>
CC: Pete Resnick <[email-masked]>, Arnt Gulbrandsen <[email-masked]>, IMAP Extensions <[email-masked]>
Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Sender: [email-masked]

Mark Crispin wrote:

>> Also, the document doesn't describe what should happen if removal of 
>> RFC 2047 or RFC 2045 encoding or conversion to the character set 
>> supported by collation fails.
>
> IMHO: a failure in removing RFC 2047 encoding can only happen as a 
> result of a syntax error.  Therefore, the string wasn't RFC 2047, it 
> just kinda looked like RFC 2047.

This is a good argument. Thanks.

 [...]

>>  Sieve treats a comparator result of "undefined" the same as a result
>>  of "no-match". That is, this specification
>>  does not provide any means to directly search for invalid comparator 
>> input.
>
> This effect is doable by the resolution method, without requiring the 
> comparator to have an "undefined result".

Undefined result can be detected by a comparator once resolution is 
already complete. So handling of undefined results has to be specified 
explicitly.

Note that none of the existing comparators ever return "undefined", 
however a future comparator might.

> That is, the resolution method generate strings that can not match.

Date: Sun, 9 Sep 2007 14:20:01 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: Alexey Melnikov <[email-masked]>
cc: Pete Resnick <[email-masked]>, Arnt Gulbrandsen <[email-masked]>, IMAP Extensions <[email-masked]>
Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Sender: [email-masked]

On Sun, 9 Sep 2007, Alexey Melnikov wrote:
> Please correct me if I am wrong, but I don't think that RFC 2047 encoding can 
> be specified in an argument to SEARCH command. So I suggest the text in () to 
> be removed.

You are correct.

> What does "convert character encodings compatibly" means? (Compatibly for 
> what?) A collation defines which character set it operates on. I suggest we 
> just say so. Or, considering that support for UTF-8 charset is required by 
> this document, we can just say that all strings are converted to Unicode 
> before invoking the active comparator. This will match what Sieve spec is 
> saying.

Better would be to lay out the problem.  Something to the effect of:

    IMAP strings may be in arbitrary charsets, which must be resolved
    against the collation's input domain.

    The i;unicode-casemap collation accepts input in arbitrary charsets,
    and specifies a charset resolution procedure involving conversion to
    Unicode and fallback to octet comparison for strings that can not be
    converted.  This allows i;unicode-casemap to be used for collation
    in IMAP servers that do not implement IMAP-I18N.

Then, you need to decide if other collations used in IMAP MUST follow the 
charset resolution process laid out in i;unicode-casemap.  If so, perhaps 
you want to make i;unicode-casemap normative (if it isn't already) and 
state that its Step 1 is mandatory for other collations.

Another alternative is that IMAP-I18N can specify a different resolution 
procedure.  If i;unicode-casemap only sees well-formed UTF-8 strings, then 
all of its Step 1 becomes a no-op.  This is why I fought so hard to keep 
Step 1; I felt that it was necessary for i;unicode-casemap to be usable 
without an i18n document for that protocol.  [Put another way: Sieve can 
override Step 1 if it so desires by feeding i;unicode-casemap only 
well-formed UTF-8 strings.]

> Also, the document doesn't describe what should happen if removal of RFC 2047 
> or RFC 2045 encoding or conversion to the character set supported by 
> collation fails.

IMHO: a failure in removing RFC 2047 encoding can only happen as a result 
of a syntax error.  Therefore, the string wasn't RFC 2047, it just kinda 
looked like RFC 2047.

Handling of charset conversion failure is as stated above.

> Sieve specifies the following:
>     Text that the implementation cannot convert to Unicode for any reason MAY be
>     treated as plain US-ASCII (including any [RFC 2047] syntax) or
>     processed according to local conventions.
> (FYI, the unicasemap collation conforms to the MAY)

Basically, it says "syntax error" => "random ASCII text, not RFC 2047".

>  Sieve treats a comparator result of "undefined" the same as a result
>  of "no-match". That is, this specification
>  does not provide any means to directly search for invalid comparator input.

This effect is doable by the resolution method, without requiring the 
comparator to have an "undefined result".  That is, the resolution method 
generate strings that can not match.

> I believe that was some support on the mailing list in favor of UNICASEMAP 
> extension, which requires the server to only implement i;unicode-casemap + 
> SEARCH/SORT behavior.

I agree.  Something in IMAP-I18N should state that SEARCH SHOULD use 
i;unicode-casemap even if IMAP-I18N is not implemented; and that new 
implementations of the base specification MUST do so.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Date: Sun, 09 Sep 2007 21:46:39 +0100
From: Alexey Melnikov <[email-masked]>
To: Pete Resnick <[email-masked]>, Arnt Gulbrandsen <[email-masked]>
CC: IMAP Extensions <[email-masked]>
Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Sender: [email-masked]

Pete Resnick wrote:

> This fell through the cracks:
>
> This is a Working Group Last Call on draft-ietf-imapext-i18n-12.txt. 
> It has been reviewed quite a bit already, so I think there is no need 
> to go longer than **1 WEEK** for this. If there are no significant 
> issues, I will write this up and submit it for IETF Last Call at COB 
> next Tuesday. If there are issues to be addressed, Arnt can fix them 
> in a -13 and I will write that one up.
>
> Please submit comments NOW.

Note that I am trying to digest information from several documents (RFC 
3501, RFC 4790, draft-ietf-imapext-i18n-12.txt, 
draft-crispin-collation-unicasemap-07.txt, 3028bis), so I am apologizing 
in advance if some of my suggestions are contradictory. So please review 
this carefully.

 >4.2 Comparators and Character Encodings
 >
 >    When SEARCH, SORT, THREAD or another command needs to perform
 >    collation operations on messages (or on the command's arguments),

Please correct me if I am wrong, but I don't think that RFC 2047 
encoding can be specified in an argument to SEARCH command. So I suggest 
the text in () to be removed.

 >    the server MUST remove MIME encoding (see [RFC2047] for headers and
 >    [RFC2045] for bodyparts)

Good so far.

 >    and convert character encodings compatibly
 >    before doing the collation operation.

What does "convert character encodings compatibly" means? (Compatibly 
for what?) A collation defines which character set it operates on. I 
suggest we just say so. Or, considering that support for UTF-8 charset 
is required by this document, we can just say that all strings are 
converted to Unicode before invoking the active comparator. This will 
match what Sieve spec is saying.

Also, the document doesn't describe what should happen if removal of RFC 
2047 or RFC 2045 encoding or conversion to the character set supported 
by collation fails. Sieve specifies the following:

      Text that the implementation cannot convert to Unicode for any 
reason MAY be
      treated as plain US-ASCII (including any [RFC 2047] syntax) or
      processed according to local conventions.

(FYI, the unicasemap collation conforms to the MAY)

 >
 >    Strings encoded using unknown character encodings should sort
 >    together with invalid input (as defined by the active collation) for
 >    the SORT and THREAD commands.

The document also needs to define how collation's "undefined" result 
(i.e. what is returned if at least one of the input strings is "invalid" 
according to the active collation) is to be handled. FYI, Sieve says the 
following:

   Sieve treats a comparator result of "undefined" the same as a result
   of "no-match". That is, this specification
   does not provide any means to directly search for invalid comparator 
input.

RFC 3501, section 6.4.4 says:

      In all search keys that use strings, a message matches the key if
      the string is a substring of the field.  The matching is
      case-insensitive.

This might be obvious, but can draft-ietf-imapext-i18n-12.txt be more 
explicit that it replaces the second sentence with a call to a comparator?

I believe that was some support on the mailing list in favor of 
UNICASEMAP extension, which requires the server to only implement 
i;unicode-casemap + SEARCH/SORT behavior.

Minor nit reported by ID nits:

  -- No information found for draft-daboo-imap- - is the name correct?

I think this is because the draft name was split into 2 lines:

    [METADATA] Daboo, C., "IMAP METADATA Extension", draft-daboo-imap-
               annotatemore-10 (work in progress), November 2006.

I am not sure it is worth fixing, but if you can do this easily, that 
would be great.

To: [email-masked]
Cc: [email-masked]
From: [email-masked]
Subject: I-D ACTION:draft-ietf-imapext-sort-19.txt 
Date: Thu, 06 Sep 2007 15:15:01 -0400
Sender: [email-masked]

--NextPart

A New Internet-Draft is available from the on-line Internet-Drafts 
directories.
This draft is a work item of the Internet Message Access Protocol Extension Working Group of the IETF.

	Title		: INTERNET MESSAGE ACCESS PROTOCOL - SORT AND THREAD EXTENSIONS
	Author(s)	: M. Crispin, K. Murchison
	Filename	: draft-ietf-imapext-sort-19.txt
	Pages		: 0
	Date		: 2007-9-6
	
This document describes the base-level server-based sorting and
    threading extensions to the [IMAP] protocol.  These extensions
    provide substantial performance improvements for IMAP clients which
    offer sorted and threaded views.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-imapext-sort-19.txt

To remove yourself from the I-D Announcement list, send a message to 
[email-masked] with the word unsubscribe in the body of 
the message. 
You can also visit https://www1.ietf.org/mailman/listinfo/I-D-announce 
to change your subscription settings.

Internet-Drafts are also available by anonymous FTP. Login with the 
username "anonymous" and a password of your e-mail address. After 
logging in, type "cd internet-drafts" and then 
"get draft-ietf-imapext-sort-19.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html 
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

Internet-Drafts can also be obtained by e-mail.

Send a message to:
	[email-masked].
In the body type:
	"FILE /internet-drafts/draft-ietf-imapext-sort-19.txt".
	
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

--NextPart
Content-Type: Multipart/Alternative; Boundary="OtherAccess"

--OtherAccess
Content-Type: Message/External-body;
	access-type="mail-server";
	server="[email-masked]"

Content-Type: text/plain
Content-ID:	<[email-masked]>

ENCODING mime
FILE /internet-drafts/draft-ietf-imapext-sort-19.txt

--OtherAccess
Content-Type: Message/External-body;
	name="draft-ietf-imapext-sort-19.txt";
	site="ftp.ietf.org";
	access-type="anon-ftp";
	directory="internet-drafts"

Content-Type: text/plain
Content-ID:	<[email-masked]>

--OtherAccess--

--NextPart--

Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Date: Wed, 05 Sep 2007 15:30:57 +0100
From: Dave Cridland <[email-masked]>
To: Dan Karp <[email-masked]>
Cc: Pete Resnick <[email-masked]>, <[email-masked]>, Alexey Melnikov <[email-masked]>
Sender: [email-masked]

On Wed Sep  5 15:13:10 2007, Dan Karp wrote:
> You're right.  I was thinking only in one direction (SEARCH with an
> unknown charset, which leads to a [BADCHARSET]), but not the other.
> 
> 
Yes, but SEARCH with a CHARSET of UTF-8 and a single high-octet  
sequence will work fine, they're caught here, not in IMAP. Although  
arguably, section 4.2 of this draft might be read as preventing that  
sort of behaviour.

[it = "i;octet"]
> It's actually pretty bad for case-sensitive matching, as you don't  
> always know the target message's charset.  I'd vote for omitting it
> entirely.
> 
> 
Well, section 4.2 kills off unknown charsets before they hit the  
comparator - even "i;octet" doesn't see them. So those collate (and  
are matched) differently.

"i;octet" certainly isn't very good for "proper" case-sensitive  
matches, I just think we've basically got the support mandated by  
"i;unicode-casemap" anyway, so I don't see a compelling argument  
*not* to have it - at the same time, I'm not desperate to have it  
either.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Date: Wed, 05 Sep 2007 15:18:00 +0100
From: Dave Cridland <[email-masked]>
To: Dave Cridland <[email-masked]>
Cc: Dan Karp <[email-masked]>, Pete Resnick <[email-masked]>, <[email-masked]>, Alexey Melnikov <[email-masked]>, Timo Sirainen <[email-masked]>
Sender: [email-masked]

On Wed Sep  5 14:53:59 2007, Dave Cridland wrote:
>> This is for the message data, which isn't a problem. The problem  
>> is   i;octet would allow searching with any kind of search keys.  
>> For   example "ä" in UTF-8 is 0xC3 0xA4. With i;octet it would be   
>> possible  to search for only 0xA4 and it would have to match UTF-8  
>>  "ä" letters.
> 
> But you can search for that using "i;unicode-casemap" anyway. It's   
> the fallback position described in step 1(b). There is no invalid   
> input for "i;unicode-casemap".

Actually, I might be slightly off there.

"Ä" is 0xC3 0x84. You can certainly match an "Ä" by searching for  
0x84, but whether 0xA4 matches "ä" or not depends if the draft is  
saying that non-decodable input - ie, step 1(b) stuff - is still  
compared (with "i;octet") against prepared, decodable input.

Reading closely, that might be the more accurate interpretation, in  
which case 0x84 will match strings containing either "ä" or "Ä" - and  
presumably other characters - when using "i;unicode-casemap". (That's  
not to say that 0xA4 matches nothing, of course).

I don't see how this helps your indexing problem, actually it looks  
like it makes it even more unpleasant, to me. (Although I freely  
admit to not knowing much about indexing).

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Date: Wed, 5 Sep 2007 07:13:10 -0700 (PDT)
From: Dan Karp <[email-masked]>
To: Dave Cridland <[email-masked]>
Cc: Pete Resnick <[email-masked]>, [email-masked], Alexey Melnikov <[email-masked]>
Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Sender: [email-masked]

> I'm not sure how one could implement it without implementing all of  
> "i;octet", given such things as:
> 
>      (b) If the input string is in an unknown charset, or an
>          invalid sequence occurs in step (1)(a), conversion ceases.
>          No further preparation is performed, and any partial
>          preparation results are discarded.  The original string is
>          used unchanged with the i;octet comparator.

You're right.  I was thinking only in one direction (SEARCH with an
unknown charset, which leads to a [BADCHARSET]), but not the other.

> >   But, more to the point, is it OK to remove the MUST on i;octet?
> 
> I'd lean toward having it. It's useful for case-insensitive matching, 
> at least, which is rather useful on occasion. But I won't scream if  
> it's not mandated. I just think an implementation providing  
> "i;unicode-casemap" is extremely close to providing "i;octet" anyway.

It's actually pretty bad for case-sensitive matching, as you don't 
always know the target message's charset.  I'd vote for omitting it
entirely.

Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Date: Wed, 05 Sep 2007 14:53:59 +0100
From: Dave Cridland <[email-masked]>
To: Timo Sirainen <[email-masked]>
Cc: Dan Karp <[email-masked]>, Pete Resnick <[email-masked]>, <[email-masked]>, Alexey Melnikov <[email-masked]>
Sender: [email-masked]

On Wed Sep  5 14:44:11 2007, Timo Sirainen wrote:
> On 5.9.2007, at 16.23, Dave Cridland wrote:
> 
>> On Wed Sep  5 13:34:35 2007, Dan Karp wrote:
>>> > > > Well, given that you have to implement "i;octet" anyway for  
>>>   >  > > > "i;unicode-casemap", I'm not sure it makes any real   
>>> difference  >  > > > whether we mandate it or not.
>>> > > > > That's not the case.  It certainly isn't so in my case...
>>> > > Well, you don't have to expose it, but you do need to   
>>> implement all  > of its operations.
>>> Again, I'm pretty sure that's not so.
>> 
>> I'm not sure how one could implement it without implementing all  
>> of  "i;octet", given such things as:
>> 
>>    Following the above preparation process on each string, the   
>> equality,
>>    ordering and substring operations are as for i;octet.
>> 
>> And:
>> 
>>           (b) If the input string is in an unknown charset, or an
>>               invalid sequence occurs in step (1)(a), conversion   
>> ceases.
>>               No further preparation is performed, and any partial
>>               preparation results are discarded.  The original   
>> string is
>>               used unchanged with the i;octet comparator.
> 
> This is for the message data, which isn't a problem. The problem is  
>  i;octet would allow searching with any kind of search keys. For   
> example "ä" in UTF-8 is 0xC3 0xA4. With i;octet it would be  
> possible  to search for only 0xA4 and it would have to match UTF-8  
> "ä" letters.

But you can search for that using "i;unicode-casemap" anyway. It's  
the fallback position described in step 1(b). There is no invalid  
input for "i;unicode-casemap".

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Cc: Dan Karp <[email-masked]>, Pete Resnick <[email-masked]>, <[email-masked]>, Alexey Melnikov <[email-masked]>
From: Timo Sirainen <[email-masked]>
Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Date: Wed, 5 Sep 2007 16:44:11 +0300
To: Dave Cridland <[email-masked]>
Sender: [email-masked]

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--Apple-Mail-1-239254042
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset=ISO-8859-1; delsp=yes; format=flowed

On 5.9.2007, at 16.23, Dave Cridland wrote:

> On Wed Sep  5 13:34:35 2007, Dan Karp wrote:
>> > > > Well, given that you have to implement "i;octet" anyway for  =20=

>> >  > > > "i;unicode-casemap", I'm not sure it makes any real =20
>> difference  >  > > > whether we mandate it or not.
>> > > > > That's not the case.  It certainly isn't so in my case...
>> > > Well, you don't have to expose it, but you do need to =20
>> implement all  > of its operations.
>> Again, I'm pretty sure that's not so.
>
> I'm not sure how one could implement it without implementing all of =20=

> "i;octet", given such things as:
>
>    Following the above preparation process on each string, the =20
> equality,
>    ordering and substring operations are as for i;octet.
>
> And:
>
>           (b) If the input string is in an unknown charset, or an
>               invalid sequence occurs in step (1)(a), conversion =20
> ceases.
>               No further preparation is performed, and any partial
>               preparation results are discarded.  The original =20
> string is
>               used unchanged with the i;octet comparator.

This is for the message data, which isn't a problem. The problem is =20
i;octet would allow searching with any kind of search keys. For =20
example "=E4" in UTF-8 is 0xC3 0xA4. With i;octet it would be possible =20=

to search for only 0xA4 and it would have to match UTF-8 "=E4" letters. =20=

Full text search indexing for IMAP already has to deal with substring =20=

matching, requiring it to deal with octet matching also would make it =20=

even more bloaty for no real advantage.=

--Apple-Mail-1-239254042
content-type: application/pgp-signature; x-mac-type=70674453;
	name=PGP.sig
content-description: This is a digitally signed message part
content-disposition: inline; filename=PGP.sig
content-transfer-encoding: 7bit

--Apple-Mail-1-239254042--

Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Date: Wed, 05 Sep 2007 14:23:40 +0100
From: Dave Cridland <[email-masked]>
To: Dan Karp <[email-masked]>
Cc: Pete Resnick <[email-masked]>, <[email-masked]>, Alexey Melnikov <[email-masked]>
Sender: [email-masked]

On Wed Sep  5 13:34:35 2007, Dan Karp wrote:
> > > > Well, given that you have to implement "i;octet" anyway for   
> >  > > > "i;unicode-casemap", I'm not sure it makes any real  
> difference  >  > > > whether we mandate it or not.
> > > > > That's not the case.  It certainly isn't so in my case...
> > > Well, you don't have to expose it, but you do need to implement  
> all  > of its operations.
> 
> Again, I'm pretty sure that's not so.

I'm not sure how one could implement it without implementing all of  
"i;octet", given such things as:

    Following the above preparation process on each string, the  
equality,
    ordering and substring operations are as for i;octet.

And:

           (b) If the input string is in an unknown charset, or an
               invalid sequence occurs in step (1)(a), conversion  
ceases.
               No further preparation is performed, and any partial
               preparation results are discarded.  The original  
string is
               used unchanged with the i;octet comparator.

>   But, more to the point, is it OK to remove the MUST on i;octet?

I'd lean toward having it. It's useful for case-insensitive matching,  
at least, which is rather useful on occasion. But I won't scream if  
it's not mandated. I just think an implementation providing  
"i;unicode-casemap" is extremely close to providing "i;octet" anyway.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Date: Wed, 5 Sep 2007 05:34:35 -0700 (PDT)
From: Dan Karp <[email-masked]>
To: Dave Cridland <[email-masked]>
Cc: Pete Resnick <[email-masked]>, [email-masked], Alexey Melnikov <[email-masked]>
Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Sender: [email-masked]

> > > Well, given that you have to implement "i;octet" anyway for  >  
> > > "i;unicode-casemap", I'm not sure it makes any real difference  >  
> > > whether we mandate it or not.
> > 
> > That's not the case.  It certainly isn't so in my case...
> 
> Well, you don't have to expose it, but you do need to implement all  
> of its operations.

Again, I'm pretty sure that's not so.  But, more to the point, is it OK to remove the MUST on i;octet?

Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Date: Wed, 05 Sep 2007 13:29:14 +0100
From: Dave Cridland <[email-masked]>
To: Dan Karp <[email-masked]>
Cc: Pete Resnick <[email-masked]>, <[email-masked]>, Alexey Melnikov <[email-masked]>
Sender: [email-masked]

On Wed Sep  5 13:14:17 2007, Dan Karp wrote:
> 
> > Well, given that you have to implement "i;octet" anyway for  >  
> "i;unicode-casemap", I'm not sure it makes any real difference  >  
> whether we mandate it or not.
> 
> That's not the case.  It certainly isn't so in my case...
> 
> 
Well, you don't have to expose it, but you do need to implement all  
of its operations.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Date: Wed, 5 Sep 2007 05:14:17 -0700 (PDT)
From: Dan Karp <[email-masked]>
To: Dave Cridland <[email-masked]>
Cc: Pete Resnick <[email-masked]>, [email-masked], Alexey Melnikov <[email-masked]>
Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Sender: [email-masked]

> Well, given that you have to implement "i;octet" anyway for  
> "i;unicode-casemap", I'm not sure it makes any real difference  
> whether we mandate it or not.

That's not the case.  It certainly isn't so in my case...

Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Date: Wed, 05 Sep 2007 12:10:11 +0100
From: Dave Cridland <[email-masked]>
To: Alexey Melnikov <[email-masked]>
Cc: Pete Resnick <[email-masked]>, <[email-masked]>, Dan Karp <[email-masked]>
Sender: [email-masked]

On Wed Sep  5 11:23:22 2007, Alexey Melnikov wrote:
> 
> Dan Karp wrote:
> 
>>> This is a Working Group Last Call on  
>>> draft-ietf-imapext-i18n-12.txt. It has been reviewed quite a bit  
>>> already, so I think there is no need
>>> to go longer than **1 WEEK** for this. If there are no  
>>> significant issues, I will write this up and submit it for IETF  
>>> Last Call at COB next Tuesday. If there are issues to be  
>>> addressed, Arnt can fix them in a -13 and I will write that one  
>>> up.
>>> 
>>> Please submit comments NOW.
>>>    
>> Last I saw, the discussion was leaning towards (but hadn't  
>> finalized on) removing the MUST on the i;ascii-casemap and i;octet  
>> comparators and introducing a MUST on i;unicode-casemap.  Given  
>> that that's probably the largest COMPARATOR implementation issue  
>> for servers, we should decide it one way or the other before  
>> submitting.
>> 
>> Apologies if I've missed a resolution to this discussion.
>>  
> No, you haven't missed anything.
> 
> I think there is mailing list consensus to remove MUST on the  
> i;ascii-casemap and i;octet comparators and introducing a MUST on  
> i;unicode-casemap.
> 
> I don't think we have any consensus on whether i;octet should be  
> mentioned at all. I would be Ok with saying MAY i;octet, but on the  
> other hand I am also Ok if it is not mentioned at all.

Well, given that you have to implement "i;octet" anyway for  
"i;unicode-casemap", I'm not sure it makes any real difference  
whether we mandate it or not.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Date: Wed, 05 Sep 2007 11:23:22 +0100
From: Alexey Melnikov <[email-masked]>
To: Dan Karp <[email-masked]>
CC: Pete Resnick <[email-masked]>, IMAP Extensions <[email-masked]>
Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Sender: [email-masked]

Dan Karp wrote:

>>This is a Working Group Last Call on draft-ietf-imapext-i18n-12.txt. 
>>It has been reviewed quite a bit already, so I think there is no need
>>to go longer than **1 WEEK** for this. If there are no significant 
>>issues, I will write this up and submit it for IETF Last Call at COB 
>>next Tuesday. If there are issues to be addressed, Arnt can fix them 
>>in a -13 and I will write that one up.
>>
>>Please submit comments NOW.
>>    
>>
>Last I saw, the discussion was leaning towards (but hadn't finalized on) removing the MUST on the i;ascii-casemap and i;octet comparators and introducing a MUST on i;unicode-casemap.  Given that that's probably the largest COMPARATOR implementation issue for servers, we should decide it one way or the other before submitting.
>
>Apologies if I've missed a resolution to this discussion.
>  
>
No, you haven't missed anything.

I think there is mailing list consensus to remove MUST on the 
i;ascii-casemap and i;octet comparators and introducing a MUST on 
i;unicode-casemap.

I don't think we have any consensus on whether i;octet should be 
mentioned at all. I would be Ok with saying MAY i;octet, but on the 
other hand I am also Ok if it is not mentioned at all.

Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
From: Timo Sirainen <[email-masked]>
To: Dan Karp <[email-masked]>
Cc: Pete Resnick <[email-masked]>, IMAP Extensions <[email-masked]>
Date: Wed, 05 Sep 2007 00:48:52 +0300
Sender: [email-masked]

--=-qxWC3pqEM4gPpyfNtjrk
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

On Tue, 2007-09-04 at 14:32 -0700, Dan Karp wrote:
> > This is a Working Group Last Call on draft-ietf-imapext-i18n-12.txt.=20
> > It has been reviewed quite a bit already, so I think there is no need
> > to go longer than **1 WEEK** for this. If there are no significant=20
> > issues, I will write this up and submit it for IETF Last Call at COB=20
> > next Tuesday. If there are issues to be addressed, Arnt can fix them=20
> > in a -13 and I will write that one up.
> >=20
> > Please submit comments NOW.
>=20
> Last I saw, the discussion was leaning towards (but hadn't finalized on) =
removing the MUST on the i;ascii-casemap and i;octet comparators and introd=
ucing a MUST on i;unicode-casemap.  Given that that's probably the largest =
COMPARATOR implementation issue for servers, we should decide it one way or=
 the other before submitting.
>=20
> Apologies if I've missed a resolution to this discussion.

Same here. Especially about removing i;octet requirement.

--=-qxWC3pqEM4gPpyfNtjrk
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

--=-qxWC3pqEM4gPpyfNtjrk--

Date: Tue, 4 Sep 2007 14:32:22 -0700 (PDT)
From: Dan Karp <[email-masked]>
To: Pete Resnick <[email-masked]>
Cc: IMAP Extensions <[email-masked]>
Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Sender: [email-masked]

> This is a Working Group Last Call on draft-ietf-imapext-i18n-12.txt. 
> It has been reviewed quite a bit already, so I think there is no need
> to go longer than **1 WEEK** for this. If there are no significant 
> issues, I will write this up and submit it for IETF Last Call at COB 
> next Tuesday. If there are issues to be addressed, Arnt can fix them 
> in a -13 and I will write that one up.
> 
> Please submit comments NOW.

Last I saw, the discussion was leaning towards (but hadn't finalized on) removing the MUST on the i;ascii-casemap and i;octet comparators and introducing a MUST on i;unicode-casemap.  Given that that's probably the largest COMPARATOR implementation issue for servers, we should decide it one way or the other before submitting.

Apologies if I've missed a resolution to this discussion.

Date: Tue, 4 Sep 2007 15:59:32 -0500
To: IMAP Extensions <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Sender: [email-masked]

This fell through the cracks:

This is a Working Group Last Call on draft-ietf-imapext-i18n-12.txt. 
It has been reviewed quite a bit already, so I think there is no need 
to go longer than **1 WEEK** for this. If there are no significant 
issues, I will write this up and submit it for IETF Last Call at COB 
next Tuesday. If there are issues to be addressed, Arnt can fix them 
in a -13 and I will write that one up.

Please submit comments NOW.

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

Date: Fri, 21 Sep 2007 07:46:51 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: Alexey Melnikov <[email-masked]>
cc: Arnt Gulbrandsen <[email-masked]>, IMAP Extensions <[email-masked]>
Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Sender: [email-masked]

On Thu, 20 Sep 2007, Alexey Melnikov wrote:
>>>>    This allows i;unicode-casemap to be used for collation
>>>>    in IMAP servers that do not implement IMAP-I18N.
>>> I am not sure I understand why you are trying to say here. By IMAP-I18N I 
>>> assume you've meant the COMPARATOR extension?
>> No, I mean the entire IMAP-I18N document.
> IMAP-I18N effectively consist of 2 parts: the COMPARATOR extension and the 
> LANGUAGE extension. The LANGUAGE extension has nothing to do with searching.

Right, and what I am saying is that all IMAP servers should use (or be 
upgraded to use) i;unicode-casemap today, even if they do not implement 
COMPARATOR and/or LANGUAGE.

I consider both COMPARATOR and LANGUAGE to be frills, but upgrading 
searching to use i;unicode-casemap to be essential.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Date: Thu, 20 Sep 2007 23:43:05 +0100
From: Alexey Melnikov <[email-masked]>
To: Mark Crispin <[email-masked]>
CC: Arnt Gulbrandsen <[email-masked]>, IMAP Extensions <[email-masked]>
Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Sender: [email-masked]

Mark Crispin wrote:

> On Sat, 15 Sep 2007, Alexey Melnikov wrote:
>
>>>    This allows i;unicode-casemap to be used for collation
>>>    in IMAP servers that do not implement IMAP-I18N.
>>
>> I am not sure I understand why you are trying to say here. By 
>> IMAP-I18N I assume you've meant the COMPARATOR extension?
>
> No, I mean the entire IMAP-I18N document.

IMAP-I18N effectively consist of 2 parts: the COMPARATOR extension and 
the LANGUAGE extension. The LANGUAGE extension has nothing to do with 
searching.

>   There's a lot in IMAP-I18N, but the immediate pressing problem is 
> SEARCH comparison and SORT/THREAD collation.
>
> Every server should use i;unicode-casemap instead of something like 
> "convert 'a'-'z' to 'A'-'Z' in both strings, then use strstr() or 
> strcmp() as appropriate".  ALL servers that do the latter should be 
> upgraded to use i;unicode-casecmp *now*, even if full IMAP-I18N 
> support is not going to happen right away.
>
> I'd go so far as to say that this requirement belongs in the base 
> specification and should be in the next update to the base specification.

I agree with the last sentence.

>> And I don't like statements like this in general. This document can't 
>> put any requirements on implementations that don't support any 
>> extension from it.
>
> For now, it can be a should, or a recommendation pending an update in 
> the base specification.

Such statement has to be tied to an extension (such as UNICASEMAP and 
COMPARATOR).
In theory an implementation can say that it complies with RFC XXXX 
(future IMAP-I18N RFC number), but doesn't support any extensions from 
it, but that would be silly.

> Nonetheless, the day when a server could get away with ASCII casemap 
> and strstr()/strcmp() is at an end.
>
> The c-client library has an open source implementation that is easily 
> separable from the rest of c-client.
>
>>> Then, you need to decide if other collations used in IMAP MUST 
>>> follow the charset resolution process laid out in 
>>> i;unicode-casemap.  If so, perhaps you want to make 
>>> i;unicode-casemap normative (if it isn't already) and state that its 
>>> Step 1 is mandatory for other collations.
>>
>> IMAP I18N can't change requirements on collations without changing 
>> RFC 4790. So this is not an option, unless there is WG consensus that 
>> RFC 4790 is seriously broken and need to be replaced.
>
> I was unclear here; this is not a requirement on collations, but 
> rather a requirement on IMAP's use of collations.  If another 
> collation does not have this procedure (which is possible since RFC 
> 4790 does not require it), then IMAP requires that this procedure be 
> performed (as an IMAP-mandated "step 0") prior to the collation action 
> if that collation is used in IMAP.
>
> "You are allowed to use X in Z.  But if X does not do Y as part of its 
> steps, then you must first do Y yourself before doing X in Z."
>
>> However, IMAP I18N can state that the resolution process in IMAP is 
>> the same as Step 1 from i;unicode-casemap.
>
> We may be saying the same thing.

After you clarification above, I agree.

> The basic point here is that you don't have to do the same thing twice 
> if the collation also requires it, but you must do it at least once in 
> IMAP.
>
>>> Another alternative is that IMAP-I18N can specify a different 
>>> resolution procedure.  If i;unicode-casemap only sees well-formed 
>>> UTF-8 strings, then all of its Step 1 becomes a no-op.
>>
>> This is also an option.
>
> That is, however, an option that I hope is not chosen.

Date: Sat, 15 Sep 2007 08:16:36 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: Alexey Melnikov <[email-masked]>
cc: Arnt Gulbrandsen <[email-masked]>, IMAP Extensions <[email-masked]>
Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Sender: [email-masked]

On Sat, 15 Sep 2007, Alexey Melnikov wrote:
>>    This allows i;unicode-casemap to be used for collation
>>    in IMAP servers that do not implement IMAP-I18N.
> I am not sure I understand why you are trying to say here. By IMAP-I18N I 
> assume you've meant the COMPARATOR extension?

No, I mean the entire IMAP-I18N document.  There's a lot in IMAP-I18N, but 
the immediate pressing problem is SEARCH comparison and SORT/THREAD 
collation.

Every server should use i;unicode-casemap instead of something like 
"convert 'a'-'z' to 'A'-'Z' in both strings, then use strstr() or strcmp() 
as appropriate".  ALL servers that do the latter should be upgraded to use 
i;unicode-casecmp *now*, even if full IMAP-I18N support is not going to 
happen right away.

I'd go so far as to say that this requirement belongs in the base 
specification and should be in the next update to the base specification.

> And I don't like statements like this in general. This document can't put any 
> requirements on implementations that don't support any extension from it.

For now, it can be a should, or a recommendation pending an update in the 
base specification.  Nonetheless, the day when a server could get away 
with ASCII casemap and strstr()/strcmp() is at an end.

The c-client library has an open source implementation that is easily 
separable from the rest of c-client.

>> Then, you need to decide if other collations used in IMAP MUST follow the 
>> charset resolution process laid out in i;unicode-casemap.  If so, perhaps 
>> you want to make i;unicode-casemap normative (if it isn't already) and 
>> state that its Step 1 is mandatory for other collations.
> IMAP I18N can't change requirements on collations without changing RFC 4790. 
> So this is not an option, unless there is WG consensus that RFC 4790 is 
> seriously broken and need to be replaced.

I was unclear here; this is not a requirement on collations, but rather a 
requirement on IMAP's use of collations.  If another collation does not 
have this procedure (which is possible since RFC 4790 does not require 
it), then IMAP requires that this procedure be performed (as an 
IMAP-mandated "step 0") prior to the collation action if that collation is 
used in IMAP.

"You are allowed to use X in Z.  But if X does not do Y as part of its 
steps, then you must first do Y yourself before doing X in Z."

> However, IMAP I18N can state that the resolution process in IMAP is the same 
> as Step 1 from i;unicode-casemap.

We may be saying the same thing.  The basic point here is that you don't 
have to do the same thing twice if the collation also requires it, but you 
must do it at least once in IMAP.

>> Another alternative is that IMAP-I18N can specify a different resolution 
>> procedure.  If i;unicode-casemap only sees well-formed UTF-8 strings, then 
>> all of its Step 1 becomes a no-op.
> This is also an option.

That is, however, an option that I hope is not chosen.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Date: Sat, 15 Sep 2007 11:28:27 +0100
From: Alexey Melnikov <[email-masked]>
To: Mark Crispin <[email-masked]>
CC: Arnt Gulbrandsen <[email-masked]>, IMAP Extensions <[email-masked]>
Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Sender: [email-masked]

Mark Crispin wrote:

> On Sun, 9 Sep 2007, Alexey Melnikov wrote:
>
>> What does "convert character encodings compatibly" means? (Compatibly 
>> for what?) A collation defines which character set it operates on. I 
>> suggest we just say so. Or, considering that support for UTF-8 
>> charset is required by this document, we can just say that all 
>> strings are converted to Unicode before invoking the active 
>> comparator. This will match what Sieve spec is saying.
>
> Better would be to lay out the problem.  Something to the effect of:
>
>    IMAP strings may be in arbitrary charsets, which must be resolved
>    against the collation's input domain.

I like this part so far.

>    The i;unicode-casemap collation accepts input in arbitrary charsets,
>    and specifies a charset resolution procedure involving conversion to
>    Unicode and fallback to octet comparison for strings that can not be
>    converted.

This is just summarizing i;unicode-casemap behavior. I don't mind this 
text, as long as it remains informative.

>    This allows i;unicode-casemap to be used for collation
>    in IMAP servers that do not implement IMAP-I18N.

I am not sure I understand why you are trying to say here. By IMAP-I18N 
I assume you've meant the COMPARATOR extension?

And I don't like statements like this in general. This document can't 
put any requirements on implementations that don't support any extension 
from it.

> Then, you need to decide if other collations used in IMAP MUST follow 
> the charset resolution process laid out in i;unicode-casemap.  If so, 
> perhaps you want to make i;unicode-casemap normative (if it isn't 
> already) and state that its Step 1 is mandatory for other collations.

IMAP I18N can't change requirements on collations without changing RFC 
4790. So this is not an option, unless there is WG consensus that RFC 
4790 is seriously broken and need to be replaced.
However, IMAP I18N can state that the resolution process in IMAP is the 
same as Step 1 from i;unicode-casemap.

> Another alternative is that IMAP-I18N can specify a different 
> resolution procedure.  If i;unicode-casemap only sees well-formed 
> UTF-8 strings, then all of its Step 1 becomes a no-op.

This is also an option.

> This is why I fought so hard to keep Step 1; I felt that it was 
> necessary for i;unicode-casemap to be usable without an i18n document 
> for that protocol.  [Put another way: Sieve can override Step 1 if it 
> so desires by feeding i;unicode-casemap only well-formed UTF-8 strings.] 

Date: Wed, 12 Sep 2007 20:26:33 -0500
To: Arnt Gulbrandsen <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Cc: [email-masked]
Sender: [email-masked]

On 9/12/07 at 3:00 PM +0200, Arnt Gulbrandsen wrote:

>(FYI: I've been ill. Still not quite well. My mail backlog will take 
>many days to work off.)

Sorry to hear it. I'll hold off on any action until you're caught up 
and let me know that we're good to go for IETF Last Call.

Feel better soon.

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated

Date: Wed, 12 Sep 2007 15:09:39 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Cc: Alexey Melnikov <[email-masked]>, Pete Resnick <[email-masked]>, Dan Karp <[email-masked]>, Timo Sirainen <[email-masked]>, Dave Cridland <[email-masked]>
Sender: [email-masked]

Dave Cridland writes:
> But you can search for that using "i;unicode-casemap" anyway. It's  
> the fallback position described in step 1(b). There is no invalid  
> input for "i;unicode-casemap".

The SEARCH key doesn't have to not accept invalid input. AFAICT, for a 
command such as "a search charset utf-8 body x", the server is 
permitted to check the UTF-8-ness of x and say BAD if it hates what it 
sees. (Permitted or required, I don't think it matters in this case.)

Arnt

Date: Wed, 12 Sep 2007 15:00:35 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Cc: Alexey Melnikov <[email-masked]>, Pete Resnick <[email-masked]>, Dan Karp <[email-masked]>, Dave Cridland <[email-masked]>
Sender: [email-masked]

(FYI: I've been ill. Still not quite well. My mail backlog will take 
many days to work off.)

Dave Cridland writes:
> On Wed Sep  5 13:14:17 2007, Dan Karp wrote:
>
>> > Well, given that you have to implement "i;octet" anyway for  >  
>> "i;unicode-casemap", I'm not sure it makes any real difference  >  
>> whether we mandate it or not.
>>
>> That's not the case.  It certainly isn't so in my case...
>
> Well, you don't have to expose it, but you do need to implement all  
> of its operations.

Perhaps. I don't do it (or won't do it - the code isn't complete). I'm 
not sure whether my finished implementation should be said to comply 
with i;unicode-casemap, though.

Arnt

Date: Mon, 10 Sep 2007 23:02:33 +0100
From: Alexey Melnikov <[email-masked]>
To: Mark Crispin <[email-masked]>
CC: Pete Resnick <[email-masked]>, Arnt Gulbrandsen <[email-masked]>, IMAP Extensions <[email-masked]>
Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Sender: [email-masked]

Mark Crispin wrote:

>> Also, the document doesn't describe what should happen if removal of 
>> RFC 2047 or RFC 2045 encoding or conversion to the character set 
>> supported by collation fails.
>
> IMHO: a failure in removing RFC 2047 encoding can only happen as a 
> result of a syntax error.  Therefore, the string wasn't RFC 2047, it 
> just kinda looked like RFC 2047.

This is a good argument. Thanks.

 [...]

>>  Sieve treats a comparator result of "undefined" the same as a result
>>  of "no-match". That is, this specification
>>  does not provide any means to directly search for invalid comparator 
>> input.
>
> This effect is doable by the resolution method, without requiring the 
> comparator to have an "undefined result".

Undefined result can be detected by a comparator once resolution is 
already complete. So handling of undefined results has to be specified 
explicitly.

Note that none of the existing comparators ever return "undefined", 
however a future comparator might.

> That is, the resolution method generate strings that can not match.

Date: Sun, 9 Sep 2007 14:20:01 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: Alexey Melnikov <[email-masked]>
cc: Pete Resnick <[email-masked]>, Arnt Gulbrandsen <[email-masked]>, IMAP Extensions <[email-masked]>
Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Sender: [email-masked]

On Sun, 9 Sep 2007, Alexey Melnikov wrote:
> Please correct me if I am wrong, but I don't think that RFC 2047 encoding can 
> be specified in an argument to SEARCH command. So I suggest the text in () to 
> be removed.

You are correct.

> What does "convert character encodings compatibly" means? (Compatibly for 
> what?) A collation defines which character set it operates on. I suggest we 
> just say so. Or, considering that support for UTF-8 charset is required by 
> this document, we can just say that all strings are converted to Unicode 
> before invoking the active comparator. This will match what Sieve spec is 
> saying.

Better would be to lay out the problem.  Something to the effect of:

    IMAP strings may be in arbitrary charsets, which must be resolved
    against the collation's input domain.

    The i;unicode-casemap collation accepts input in arbitrary charsets,
    and specifies a charset resolution procedure involving conversion to
    Unicode and fallback to octet comparison for strings that can not be
    converted.  This allows i;unicode-casemap to be used for collation
    in IMAP servers that do not implement IMAP-I18N.

Then, you need to decide if other collations used in IMAP MUST follow the 
charset resolution process laid out in i;unicode-casemap.  If so, perhaps 
you want to make i;unicode-casemap normative (if it isn't already) and 
state that its Step 1 is mandatory for other collations.

Another alternative is that IMAP-I18N can specify a different resolution 
procedure.  If i;unicode-casemap only sees well-formed UTF-8 strings, then 
all of its Step 1 becomes a no-op.  This is why I fought so hard to keep 
Step 1; I felt that it was necessary for i;unicode-casemap to be usable 
without an i18n document for that protocol.  [Put another way: Sieve can 
override Step 1 if it so desires by feeding i;unicode-casemap only 
well-formed UTF-8 strings.]

> Also, the document doesn't describe what should happen if removal of RFC 2047 
> or RFC 2045 encoding or conversion to the character set supported by 
> collation fails.

IMHO: a failure in removing RFC 2047 encoding can only happen as a result 
of a syntax error.  Therefore, the string wasn't RFC 2047, it just kinda 
looked like RFC 2047.

Handling of charset conversion failure is as stated above.

> Sieve specifies the following:
>     Text that the implementation cannot convert to Unicode for any reason MAY be
>     treated as plain US-ASCII (including any [RFC 2047] syntax) or
>     processed according to local conventions.
> (FYI, the unicasemap collation conforms to the MAY)

Basically, it says "syntax error" => "random ASCII text, not RFC 2047".

>  Sieve treats a comparator result of "undefined" the same as a result
>  of "no-match". That is, this specification
>  does not provide any means to directly search for invalid comparator input.

This effect is doable by the resolution method, without requiring the 
comparator to have an "undefined result".  That is, the resolution method 
generate strings that can not match.

> I believe that was some support on the mailing list in favor of UNICASEMAP 
> extension, which requires the server to only implement i;unicode-casemap + 
> SEARCH/SORT behavior.

I agree.  Something in IMAP-I18N should state that SEARCH SHOULD use 
i;unicode-casemap even if IMAP-I18N is not implemented; and that new 
implementations of the base specification MUST do so.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Date: Sun, 09 Sep 2007 21:46:39 +0100
From: Alexey Melnikov <[email-masked]>
To: Pete Resnick <[email-masked]>, Arnt Gulbrandsen <[email-masked]>
CC: IMAP Extensions <[email-masked]>
Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Sender: [email-masked]

Pete Resnick wrote:

> This fell through the cracks:
>
> This is a Working Group Last Call on draft-ietf-imapext-i18n-12.txt. 
> It has been reviewed quite a bit already, so I think there is no need 
> to go longer than **1 WEEK** for this. If there are no significant 
> issues, I will write this up and submit it for IETF Last Call at COB 
> next Tuesday. If there are issues to be addressed, Arnt can fix them 
> in a -13 and I will write that one up.
>
> Please submit comments NOW.

Note that I am trying to digest information from several documents (RFC 
3501, RFC 4790, draft-ietf-imapext-i18n-12.txt, 
draft-crispin-collation-unicasemap-07.txt, 3028bis), so I am apologizing 
in advance if some of my suggestions are contradictory. So please review 
this carefully.

 >4.2 Comparators and Character Encodings
 >
 >    When SEARCH, SORT, THREAD or another command needs to perform
 >    collation operations on messages (or on the command's arguments),

Please correct me if I am wrong, but I don't think that RFC 2047 
encoding can be specified in an argument to SEARCH command. So I suggest 
the text in () to be removed.

 >    the server MUST remove MIME encoding (see [RFC2047] for headers and
 >    [RFC2045] for bodyparts)

Good so far.

 >    and convert character encodings compatibly
 >    before doing the collation operation.

What does "convert character encodings compatibly" means? (Compatibly 
for what?) A collation defines which character set it operates on. I 
suggest we just say so. Or, considering that support for UTF-8 charset 
is required by this document, we can just say that all strings are 
converted to Unicode before invoking the active comparator. This will 
match what Sieve spec is saying.

Also, the document doesn't describe what should happen if removal of RFC 
2047 or RFC 2045 encoding or conversion to the character set supported 
by collation fails. Sieve specifies the following:

      Text that the implementation cannot convert to Unicode for any 
reason MAY be
      treated as plain US-ASCII (including any [RFC 2047] syntax) or
      processed according to local conventions.

(FYI, the unicasemap collation conforms to the MAY)

 >
 >    Strings encoded using unknown character encodings should sort
 >    together with invalid input (as defined by the active collation) for
 >    the SORT and THREAD commands.

The document also needs to define how collation's "undefined" result 
(i.e. what is returned if at least one of the input strings is "invalid" 
according to the active collation) is to be handled. FYI, Sieve says the 
following:

   Sieve treats a comparator result of "undefined" the same as a result
   of "no-match". That is, this specification
   does not provide any means to directly search for invalid comparator 
input.

RFC 3501, section 6.4.4 says:

      In all search keys that use strings, a message matches the key if
      the string is a substring of the field.  The matching is
      case-insensitive.

This might be obvious, but can draft-ietf-imapext-i18n-12.txt be more 
explicit that it replaces the second sentence with a call to a comparator?

I believe that was some support on the mailing list in favor of 
UNICASEMAP extension, which requires the server to only implement 
i;unicode-casemap + SEARCH/SORT behavior.

Minor nit reported by ID nits:

  -- No information found for draft-daboo-imap- - is the name correct?

I think this is because the draft name was split into 2 lines:

    [METADATA] Daboo, C., "IMAP METADATA Extension", draft-daboo-imap-
               annotatemore-10 (work in progress), November 2006.

I am not sure it is worth fixing, but if you can do this easily, that 
would be great.

To: [email-masked]
Cc: [email-masked]
From: [email-masked]
Subject: I-D ACTION:draft-ietf-imapext-sort-19.txt 
Date: Thu, 06 Sep 2007 15:15:01 -0400
Sender: [email-masked]

--NextPart

A New Internet-Draft is available from the on-line Internet-Drafts 
directories.
This draft is a work item of the Internet Message Access Protocol Extension Working Group of the IETF.

	Title		: INTERNET MESSAGE ACCESS PROTOCOL - SORT AND THREAD EXTENSIONS
	Author(s)	: M. Crispin, K. Murchison
	Filename	: draft-ietf-imapext-sort-19.txt
	Pages		: 0
	Date		: 2007-9-6
	
This document describes the base-level server-based sorting and
    threading extensions to the [IMAP] protocol.  These extensions
    provide substantial performance improvements for IMAP clients which
    offer sorted and threaded views.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-imapext-sort-19.txt

To remove yourself from the I-D Announcement list, send a message to 
[email-masked] with the word unsubscribe in the body of 
the message. 
You can also visit https://www1.ietf.org/mailman/listinfo/I-D-announce 
to change your subscription settings.

Internet-Drafts are also available by anonymous FTP. Login with the 
username "anonymous" and a password of your e-mail address. After 
logging in, type "cd internet-drafts" and then 
"get draft-ietf-imapext-sort-19.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html 
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

Internet-Drafts can also be obtained by e-mail.

Send a message to:
	[email-masked].
In the body type:
	"FILE /internet-drafts/draft-ietf-imapext-sort-19.txt".
	
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

--NextPart
Content-Type: Multipart/Alternative; Boundary="OtherAccess"

--OtherAccess
Content-Type: Message/External-body;
	access-type="mail-server";
	server="[email-masked]"

Content-Type: text/plain
Content-ID:	<[email-masked]>

ENCODING mime
FILE /internet-drafts/draft-ietf-imapext-sort-19.txt

--OtherAccess
Content-Type: Message/External-body;
	name="draft-ietf-imapext-sort-19.txt";
	site="ftp.ietf.org";
	access-type="anon-ftp";
	directory="internet-drafts"

Content-Type: text/plain
Content-ID:	<[email-masked]>

--OtherAccess--

--NextPart--

Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Date: Wed, 05 Sep 2007 15:30:57 +0100
From: Dave Cridland <[email-masked]>
To: Dan Karp <[email-masked]>
Cc: Pete Resnick <[email-masked]>, <[email-masked]>, Alexey Melnikov <[email-masked]>
Sender: [email-masked]

On Wed Sep  5 15:13:10 2007, Dan Karp wrote:
> You're right.  I was thinking only in one direction (SEARCH with an
> unknown charset, which leads to a [BADCHARSET]), but not the other.
> 
> 
Yes, but SEARCH with a CHARSET of UTF-8 and a single high-octet  
sequence will work fine, they're caught here, not in IMAP. Although  
arguably, section 4.2 of this draft might be read as preventing that  
sort of behaviour.

[it = "i;octet"]
> It's actually pretty bad for case-sensitive matching, as you don't  
> always know the target message's charset.  I'd vote for omitting it
> entirely.
> 
> 
Well, section 4.2 kills off unknown charsets before they hit the  
comparator - even "i;octet" doesn't see them. So those collate (and  
are matched) differently.

"i;octet" certainly isn't very good for "proper" case-sensitive  
matches, I just think we've basically got the support mandated by  
"i;unicode-casemap" anyway, so I don't see a compelling argument  
*not* to have it - at the same time, I'm not desperate to have it  
either.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Date: Wed, 05 Sep 2007 15:18:00 +0100
From: Dave Cridland <[email-masked]>
To: Dave Cridland <[email-masked]>
Cc: Dan Karp <[email-masked]>, Pete Resnick <[email-masked]>, <[email-masked]>, Alexey Melnikov <[email-masked]>, Timo Sirainen <[email-masked]>
Sender: [email-masked]

On Wed Sep  5 14:53:59 2007, Dave Cridland wrote:
>> This is for the message data, which isn't a problem. The problem  
>> is   i;octet would allow searching with any kind of search keys.  
>> For   example "ä" in UTF-8 is 0xC3 0xA4. With i;octet it would be   
>> possible  to search for only 0xA4 and it would have to match UTF-8  
>>  "ä" letters.
> 
> But you can search for that using "i;unicode-casemap" anyway. It's   
> the fallback position described in step 1(b). There is no invalid   
> input for "i;unicode-casemap".

Actually, I might be slightly off there.

"Ä" is 0xC3 0x84. You can certainly match an "Ä" by searching for  
0x84, but whether 0xA4 matches "ä" or not depends if the draft is  
saying that non-decodable input - ie, step 1(b) stuff - is still  
compared (with "i;octet") against prepared, decodable input.

Reading closely, that might be the more accurate interpretation, in  
which case 0x84 will match strings containing either "ä" or "Ä" - and  
presumably other characters - when using "i;unicode-casemap". (That's  
not to say that 0xA4 matches nothing, of course).

I don't see how this helps your indexing problem, actually it looks  
like it makes it even more unpleasant, to me. (Although I freely  
admit to not knowing much about indexing).

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Date: Wed, 5 Sep 2007 07:13:10 -0700 (PDT)
From: Dan Karp <[email-masked]>
To: Dave Cridland <[email-masked]>
Cc: Pete Resnick <[email-masked]>, [email-masked], Alexey Melnikov <[email-masked]>
Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Sender: [email-masked]

> I'm not sure how one could implement it without implementing all of  
> "i;octet", given such things as:
> 
>      (b) If the input string is in an unknown charset, or an
>          invalid sequence occurs in step (1)(a), conversion ceases.
>          No further preparation is performed, and any partial
>          preparation results are discarded.  The original string is
>          used unchanged with the i;octet comparator.

You're right.  I was thinking only in one direction (SEARCH with an
unknown charset, which leads to a [BADCHARSET]), but not the other.

> >   But, more to the point, is it OK to remove the MUST on i;octet?
> 
> I'd lean toward having it. It's useful for case-insensitive matching, 
> at least, which is rather useful on occasion. But I won't scream if  
> it's not mandated. I just think an implementation providing  
> "i;unicode-casemap" is extremely close to providing "i;octet" anyway.

It's actually pretty bad for case-sensitive matching, as you don't 
always know the target message's charset.  I'd vote for omitting it
entirely.

Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Date: Wed, 05 Sep 2007 14:53:59 +0100
From: Dave Cridland <[email-masked]>
To: Timo Sirainen <[email-masked]>
Cc: Dan Karp <[email-masked]>, Pete Resnick <[email-masked]>, <[email-masked]>, Alexey Melnikov <[email-masked]>
Sender: [email-masked]

On Wed Sep  5 14:44:11 2007, Timo Sirainen wrote:
> On 5.9.2007, at 16.23, Dave Cridland wrote:
> 
>> On Wed Sep  5 13:34:35 2007, Dan Karp wrote:
>>> > > > Well, given that you have to implement "i;octet" anyway for  
>>>   >  > > > "i;unicode-casemap", I'm not sure it makes any real   
>>> difference  >  > > > whether we mandate it or not.
>>> > > > > That's not the case.  It certainly isn't so in my case...
>>> > > Well, you don't have to expose it, but you do need to   
>>> implement all  > of its operations.
>>> Again, I'm pretty sure that's not so.
>> 
>> I'm not sure how one could implement it without implementing all  
>> of  "i;octet", given such things as:
>> 
>>    Following the above preparation process on each string, the   
>> equality,
>>    ordering and substring operations are as for i;octet.
>> 
>> And:
>> 
>>           (b) If the input string is in an unknown charset, or an
>>               invalid sequence occurs in step (1)(a), conversion   
>> ceases.
>>               No further preparation is performed, and any partial
>>               preparation results are discarded.  The original   
>> string is
>>               used unchanged with the i;octet comparator.
> 
> This is for the message data, which isn't a problem. The problem is  
>  i;octet would allow searching with any kind of search keys. For   
> example "ä" in UTF-8 is 0xC3 0xA4. With i;octet it would be  
> possible  to search for only 0xA4 and it would have to match UTF-8  
> "ä" letters.

But you can search for that using "i;unicode-casemap" anyway. It's  
the fallback position described in step 1(b). There is no invalid  
input for "i;unicode-casemap".

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Cc: Dan Karp <[email-masked]>, Pete Resnick <[email-masked]>, <[email-masked]>, Alexey Melnikov <[email-masked]>
From: Timo Sirainen <[email-masked]>
Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Date: Wed, 5 Sep 2007 16:44:11 +0300
To: Dave Cridland <[email-masked]>
Sender: [email-masked]

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--Apple-Mail-1-239254042
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset=ISO-8859-1; delsp=yes; format=flowed

On 5.9.2007, at 16.23, Dave Cridland wrote:

> On Wed Sep  5 13:34:35 2007, Dan Karp wrote:
>> > > > Well, given that you have to implement "i;octet" anyway for  =20=

>> >  > > > "i;unicode-casemap", I'm not sure it makes any real =20
>> difference  >  > > > whether we mandate it or not.
>> > > > > That's not the case.  It certainly isn't so in my case...
>> > > Well, you don't have to expose it, but you do need to =20
>> implement all  > of its operations.
>> Again, I'm pretty sure that's not so.
>
> I'm not sure how one could implement it without implementing all of =20=

> "i;octet", given such things as:
>
>    Following the above preparation process on each string, the =20
> equality,
>    ordering and substring operations are as for i;octet.
>
> And:
>
>           (b) If the input string is in an unknown charset, or an
>               invalid sequence occurs in step (1)(a), conversion =20
> ceases.
>               No further preparation is performed, and any partial
>               preparation results are discarded.  The original =20
> string is
>               used unchanged with the i;octet comparator.

This is for the message data, which isn't a problem. The problem is =20
i;octet would allow searching with any kind of search keys. For =20
example "=E4" in UTF-8 is 0xC3 0xA4. With i;octet it would be possible =20=

to search for only 0xA4 and it would have to match UTF-8 "=E4" letters. =20=

Full text search indexing for IMAP already has to deal with substring =20=

matching, requiring it to deal with octet matching also would make it =20=

even more bloaty for no real advantage.=

--Apple-Mail-1-239254042
content-type: application/pgp-signature; x-mac-type=70674453;
	name=PGP.sig
content-description: This is a digitally signed message part
content-disposition: inline; filename=PGP.sig
content-transfer-encoding: 7bit

--Apple-Mail-1-239254042--

Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Date: Wed, 05 Sep 2007 14:23:40 +0100
From: Dave Cridland <[email-masked]>
To: Dan Karp <[email-masked]>
Cc: Pete Resnick <[email-masked]>, <[email-masked]>, Alexey Melnikov <[email-masked]>
Sender: [email-masked]

On Wed Sep  5 13:34:35 2007, Dan Karp wrote:
> > > > Well, given that you have to implement "i;octet" anyway for   
> >  > > > "i;unicode-casemap", I'm not sure it makes any real  
> difference  >  > > > whether we mandate it or not.
> > > > > That's not the case.  It certainly isn't so in my case...
> > > Well, you don't have to expose it, but you do need to implement  
> all  > of its operations.
> 
> Again, I'm pretty sure that's not so.

I'm not sure how one could implement it without implementing all of  
"i;octet", given such things as:

    Following the above preparation process on each string, the  
equality,
    ordering and substring operations are as for i;octet.

And:

           (b) If the input string is in an unknown charset, or an
               invalid sequence occurs in step (1)(a), conversion  
ceases.
               No further preparation is performed, and any partial
               preparation results are discarded.  The original  
string is
               used unchanged with the i;octet comparator.

>   But, more to the point, is it OK to remove the MUST on i;octet?

I'd lean toward having it. It's useful for case-insensitive matching,  
at least, which is rather useful on occasion. But I won't scream if  
it's not mandated. I just think an implementation providing  
"i;unicode-casemap" is extremely close to providing "i;octet" anyway.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Date: Wed, 5 Sep 2007 05:34:35 -0700 (PDT)
From: Dan Karp <[email-masked]>
To: Dave Cridland <[email-masked]>
Cc: Pete Resnick <[email-masked]>, [email-masked], Alexey Melnikov <[email-masked]>
Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Sender: [email-masked]

> > > Well, given that you have to implement "i;octet" anyway for  >  
> > > "i;unicode-casemap", I'm not sure it makes any real difference  >  
> > > whether we mandate it or not.
> > 
> > That's not the case.  It certainly isn't so in my case...
> 
> Well, you don't have to expose it, but you do need to implement all  
> of its operations.

Again, I'm pretty sure that's not so.  But, more to the point, is it OK to remove the MUST on i;octet?

Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Date: Wed, 05 Sep 2007 13:29:14 +0100
From: Dave Cridland <[email-masked]>
To: Dan Karp <[email-masked]>
Cc: Pete Resnick <[email-masked]>, <[email-masked]>, Alexey Melnikov <[email-masked]>
Sender: [email-masked]

On Wed Sep  5 13:14:17 2007, Dan Karp wrote:
> 
> > Well, given that you have to implement "i;octet" anyway for  >  
> "i;unicode-casemap", I'm not sure it makes any real difference  >  
> whether we mandate it or not.
> 
> That's not the case.  It certainly isn't so in my case...
> 
> 
Well, you don't have to expose it, but you do need to implement all  
of its operations.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Date: Wed, 5 Sep 2007 05:14:17 -0700 (PDT)
From: Dan Karp <[email-masked]>
To: Dave Cridland <[email-masked]>
Cc: Pete Resnick <[email-masked]>, [email-masked], Alexey Melnikov <[email-masked]>
Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Sender: [email-masked]

> Well, given that you have to implement "i;octet" anyway for  
> "i;unicode-casemap", I'm not sure it makes any real difference  
> whether we mandate it or not.

That's not the case.  It certainly isn't so in my case...

Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Date: Wed, 05 Sep 2007 12:10:11 +0100
From: Dave Cridland <[email-masked]>
To: Alexey Melnikov <[email-masked]>
Cc: Pete Resnick <[email-masked]>, <[email-masked]>, Dan Karp <[email-masked]>
Sender: [email-masked]

On Wed Sep  5 11:23:22 2007, Alexey Melnikov wrote:
> 
> Dan Karp wrote:
> 
>>> This is a Working Group Last Call on  
>>> draft-ietf-imapext-i18n-12.txt. It has been reviewed quite a bit  
>>> already, so I think there is no need
>>> to go longer than **1 WEEK** for this. If there are no  
>>> significant issues, I will write this up and submit it for IETF  
>>> Last Call at COB next Tuesday. If there are issues to be  
>>> addressed, Arnt can fix them in a -13 and I will write that one  
>>> up.
>>> 
>>> Please submit comments NOW.
>>>    
>> Last I saw, the discussion was leaning towards (but hadn't  
>> finalized on) removing the MUST on the i;ascii-casemap and i;octet  
>> comparators and introducing a MUST on i;unicode-casemap.  Given  
>> that that's probably the largest COMPARATOR implementation issue  
>> for servers, we should decide it one way or the other before  
>> submitting.
>> 
>> Apologies if I've missed a resolution to this discussion.
>>  
> No, you haven't missed anything.
> 
> I think there is mailing list consensus to remove MUST on the  
> i;ascii-casemap and i;octet comparators and introducing a MUST on  
> i;unicode-casemap.
> 
> I don't think we have any consensus on whether i;octet should be  
> mentioned at all. I would be Ok with saying MAY i;octet, but on the  
> other hand I am also Ok if it is not mentioned at all.

Well, given that you have to implement "i;octet" anyway for  
"i;unicode-casemap", I'm not sure it makes any real difference  
whether we mandate it or not.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Date: Wed, 05 Sep 2007 11:23:22 +0100
From: Alexey Melnikov <[email-masked]>
To: Dan Karp <[email-masked]>
CC: Pete Resnick <[email-masked]>, IMAP Extensions <[email-masked]>
Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Sender: [email-masked]

Dan Karp wrote:

>>This is a Working Group Last Call on draft-ietf-imapext-i18n-12.txt. 
>>It has been reviewed quite a bit already, so I think there is no need
>>to go longer than **1 WEEK** for this. If there are no significant 
>>issues, I will write this up and submit it for IETF Last Call at COB 
>>next Tuesday. If there are issues to be addressed, Arnt can fix them 
>>in a -13 and I will write that one up.
>>
>>Please submit comments NOW.
>>    
>>
>Last I saw, the discussion was leaning towards (but hadn't finalized on) removing the MUST on the i;ascii-casemap and i;octet comparators and introducing a MUST on i;unicode-casemap.  Given that that's probably the largest COMPARATOR implementation issue for servers, we should decide it one way or the other before submitting.
>
>Apologies if I've missed a resolution to this discussion.
>  
>
No, you haven't missed anything.

I think there is mailing list consensus to remove MUST on the 
i;ascii-casemap and i;octet comparators and introducing a MUST on 
i;unicode-casemap.

I don't think we have any consensus on whether i;octet should be 
mentioned at all. I would be Ok with saying MAY i;octet, but on the 
other hand I am also Ok if it is not mentioned at all.

Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
From: Timo Sirainen <[email-masked]>
To: Dan Karp <[email-masked]>
Cc: Pete Resnick <[email-masked]>, IMAP Extensions <[email-masked]>
Date: Wed, 05 Sep 2007 00:48:52 +0300
Sender: [email-masked]

--=-qxWC3pqEM4gPpyfNtjrk
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

On Tue, 2007-09-04 at 14:32 -0700, Dan Karp wrote:
> > This is a Working Group Last Call on draft-ietf-imapext-i18n-12.txt.=20
> > It has been reviewed quite a bit already, so I think there is no need
> > to go longer than **1 WEEK** for this. If there are no significant=20
> > issues, I will write this up and submit it for IETF Last Call at COB=20
> > next Tuesday. If there are issues to be addressed, Arnt can fix them=20
> > in a -13 and I will write that one up.
> >=20
> > Please submit comments NOW.
>=20
> Last I saw, the discussion was leaning towards (but hadn't finalized on) =
removing the MUST on the i;ascii-casemap and i;octet comparators and introd=
ucing a MUST on i;unicode-casemap.  Given that that's probably the largest =
COMPARATOR implementation issue for servers, we should decide it one way or=
 the other before submitting.
>=20
> Apologies if I've missed a resolution to this discussion.

Same here. Especially about removing i;octet requirement.

--=-qxWC3pqEM4gPpyfNtjrk
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

--=-qxWC3pqEM4gPpyfNtjrk--

Date: Tue, 4 Sep 2007 14:32:22 -0700 (PDT)
From: Dan Karp <[email-masked]>
To: Pete Resnick <[email-masked]>
Cc: IMAP Extensions <[email-masked]>
Subject: Re: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Sender: [email-masked]

> This is a Working Group Last Call on draft-ietf-imapext-i18n-12.txt. 
> It has been reviewed quite a bit already, so I think there is no need
> to go longer than **1 WEEK** for this. If there are no significant 
> issues, I will write this up and submit it for IETF Last Call at COB 
> next Tuesday. If there are issues to be addressed, Arnt can fix them 
> in a -13 and I will write that one up.
> 
> Please submit comments NOW.

Last I saw, the discussion was leaning towards (but hadn't finalized on) removing the MUST on the i;ascii-casemap and i;octet comparators and introducing a MUST on i;unicode-casemap.  Given that that's probably the largest COMPARATOR implementation issue for servers, we should decide it one way or the other before submitting.

Apologies if I've missed a resolution to this discussion.

Date: Tue, 4 Sep 2007 15:59:32 -0500
To: IMAP Extensions <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: **1 WEEK** WGLC on draft-ietf-imapext-i18n-12.txt
Sender: [email-masked]

This fell through the cracks:

This is a Working Group Last Call on draft-ietf-imapext-i18n-12.txt. 
It has been reviewed quite a bit already, so I think there is no need 
to go longer than **1 WEEK** for this. If there are no significant 
issues, I will write this up and submit it for IETF Last Call at COB 
next Tuesday. If there are issues to be addressed, Arnt can fix them 
in a -13 and I will write that one up.

Please submit comments NOW.

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102