Subject: Fwd: draft-resnick-2822upd-01 posted
From: Pete Resnick <[email-masked]>
To: IETF-POP3EXT:;, IETF-IMAPEXT:;, IETF-SMTP:;, LEMONADE:;, IMA:;
Date: Thu, 26 Apr 2007 23:59:37 -0500
List-Id: <ietf-imapext.imc.org>

Please follow the discussion on [email-masked].

--- begin forwarded text

Date: Thu, 26 Apr 2007 16:55:06 -0500
To: [email-masked]
From: Pete Resnick <[email-masked]>
Subject: draft-resnick-2822upd-01 posted

You may have noticed that 2822upd-01 was posted.

<http://www.ietf.org/internet-drafts/draft-resnick-2822upd-01.txt>
Also:
<http://www.qualcomm.com/~presnick/draft-resnick-2822upd-01.html>
<http://www.qualcomm.com/~presnick/draft-resnick-2822upd-01.xml>
<http://www.qualcomm.com/~presnick/draft-resnick-2822upd-01.txt>

To make rfcdiff work, I've put a copy of RFC 2822 on my web site with 
some of the later sections rearranged so that they're in the same 
order as the current draft. You can see the diffs between the two 
here:

<http://tools.ietf.org/rfcdiff?url1=http://www.qualcomm.com/~presnick/rfc2822-fixed.txt&url2=http://www.ietf.org/internet-drafts/draft-resnick-2822upd-01.txt>

Though overall very few, there are real changes from 2822 in this 
document that need review. See the changes section at the bottom. I 
did *not* go whole hog and switch to Bruce's syntax, simply because I 
wanted to limit the number of changes and see if we can't get this 
thing to Draft along with 2821bis. But I did incorporate (hopefully 
all of) the corrections that you all have pointed out over the last 
couple of years.

Please review with a fine toothed comb. If there are more than just a 
few problems, perhaps I'll ask Tony Hansen or Philip Guenther (who 
have at assorted time offered assistance) to start an issues list.

Thanks in advance for your assistance.

pr

--- end forwarded text

-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

Date: Wed, 18 Apr 2007 22:49:37 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: Dan Karp <[email-masked]>
cc: IMAP Extensions WG <[email-masked]>, Enhancements to Internet email to support diverse service enivronments <[email-masked]>, IMAP Protocol Interest List <[email-masked]>
Subject: Re: [lemonade] SURVEY: How does SEARCH match in your IMAP server?
Sender: [email-masked]

  This message is in MIME format.  The first part should be readable text,
  while the remaining parts are likely unreadable without MIME-aware tools.

--0-559819627-1176960403=:11458
Content-Type: TEXT/PLAIN; CHARSET=UTF-8; format=flowed
Content-Transfer-Encoding: QUOTED-PRINTABLE
Content-ID: <[email-masked]>

Your test examples aren't any good, because your message is in UTF-8 but=20
you have texts that are claimed to be ISO 8859-1.  I will try to answer=20
what I *think* you intended.

If my presumptions are wrong, then I reserve the right to change my=20
answers.

On Wed, 18 Apr 2007, Dan Karp wrote:
> server's name and version

UW imapd, version imap-2006 series unless noted otherwise.

> I love M=C3=B6tley Cr=C3=BCe !

Presumably, you mean "I love M" 0xf6 "tley Cr" 0xfc "e"

> A001 SEARCH BODY "love"

matches

> A002 SEARCH BODY "LOVE"

matches

> A003 SEARCH CHARSET "iso-8859-1" BODY {4}
> Cr=C3=BCe

Presumably, you mean "Cr" 0xfc "e"

matches in all versions since imap-4.1

error in imap-4 and earlier (10+ year old US-ASCII only version)

> A004 SEARCH CHARSET "utf-8" BODY {5}
> Cr=C3=83=C2=BCe

Presuably, you mean "Cr" 0xc3 0xbc "e"

matches in all versions since imap-4.1

error in imap-4 and earlier (10+ year old US-ASCII only version)

> A005 SEARCH CHARSET "iso-8859-1" BODY {4}
> CR=C3=9CE

Presumably, you mean "CR" 0xdc "E"

matches in imap-2006 series.

does not match in earlier versions (earlier versions did i;ascii-casemap).

error in imap-4 and earlier (10+ year old US-ASCII only version)

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.
--0-559819627-1176960403=:11458--

Date: Wed, 18 Apr 2007 21:23:29 -0700 (PDT)
From: Dan Karp <[email-masked]>
To: IMAP Extensions WG <[email-masked]>, Enhancements to Internet email to support diverse service enivronments <[email-masked]>, IMAP Protocol Interest List <[email-masked]>
Cc: Arnt Gulbrandsen <[email-masked]>
Subject: SURVEY: How does SEARCH match in your IMAP server?
Sender: [email-masked]

We're collecting hard data to help refine an upcoming IMAP extension
(http://www.ietf.org/internet-drafts/draft-ietf-imapext-i18n-10.txt).
If you're developing an IMAP server, it would be a *great* help if you
could take 3 minutes to answer a few questions about your server
implementation.  If you email responses back to me along with your
server's name and version, I'll compile the results.

Assume that you've just injected the following message into an empty
INBOX:

----------------------------------------------------------------------
Subject: Music
Date: Tue, 17 Apr 2007 16:53:02 -0700 (PDT)
From: [email-masked]
To: [email-masked]
Content-Type: text/plain; charset="iso-8859-1"

I love Mötley Crüe !
----------------------------------------------------------------------

For each of the following SEARCH requests, would your server match the
message, not match the message, return NO [BADCHARSET], or return some
other NO or BAD response?

A001 SEARCH BODY "love"

A002 SEARCH BODY "LOVE"

A003 SEARCH CHARSET "iso-8859-1" BODY {4}
Crüe

A004 SEARCH CHARSET "utf-8" BODY {5}
CrÃ¼e

A005 SEARCH CHARSET "iso-8859-1" BODY {4}
CRÜE

Date: Wed, 18 Apr 2007 21:19:04 -0700 (PDT)
From: Dan Karp <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: Eric Burger <[email-masked]>, Alexey Melnikov <[email-masked]>, Timo Sirainen <[email-masked]>, Cyrus Daboo <[email-masked]>, [email-masked], [email-masked]
Subject: Re: ANNOTATEMORE and COMPARATOR
Sender: [email-masked]

> > Is this true?  Do these servers really implement i;ascii-casemap as
> > defined in RFC 4790?  I'd expect any RFC 3501-compliant server that
> > knows about more charsets than just US-ASCII to be supporting 
> > i;unicode-casemap and *not* i;ascii-casemap because of 
> > case-insensitive "SEARCH CHARSET".
> 
> Build some consensus please.

I chatted with Arnt, and he thinks it'd be worthwhile to canvas the
troops and get some hard data on what various servers actually implement
for their default SEARCH comparator.  I've thrown together a little
5-question survey which I'll send out under separate cover.

Subject: Re: ANNOTATEMORE and COMPARATOR
From: Timo Sirainen <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: [email-masked], [email-masked], Eric Burger <[email-masked]>, Alexey Melnikov <[email-masked]>, Dan Karp <[email-masked]>, Cyrus Daboo <[email-masked]>
Date: Tue, 17 Apr 2007 21:54:36 +0300
Sender: [email-masked]

--=-baA9QaoKY2tQJvsfXhIn
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

On Tue, 2007-04-17 at 20:25 +0200, Arnt Gulbrandsen wrote:
> >>  > If your aim is to make baseline COMPARATOR support fairly
> >>  > lightweight for IMAP servers, you may be constrained to only
> >>  > requiring i;unicode-casemap.
> >>
> >>  Mumble. Given how many IMAP servers actually implement=20
> >>  i;ascii-casemap and nothing else, I'm not happy doing that. Make=20
> >>  some other people agree with you so Pete will say "WG consensus".
> >
> > Is this true?  Do these servers really implement i;ascii-casemap as=20
> > defined in RFC 4790?  I'd expect any RFC 3501-compliant server that=20
> > knows about more charsets than just US-ASCII to be supporting=20
> > i;unicode-casemap and *not* i;ascii-casemap because of=20
> > case-insensitive "SEARCH CHARSET".
>=20
> Build some consensus please.

My server implements only i;ascii-casemap currently, but that's just
because I've been busy getting it stable and no-one's complained. I'm
going to move to i;unicode-casemap soon, and I don't see a need for
i;ascii-casemap. I'd think anyone who is going to implement COMPARATOR
won't see i;unicode-casemap requirement as a problem.

--=-baA9QaoKY2tQJvsfXhIn
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

--=-baA9QaoKY2tQJvsfXhIn--

Date: Tue, 17 Apr 2007 20:51:24 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: [lemonade] Re: ANNOTATEMORE and COMPARATOR
Cc: Philip Guenther <[email-masked]>
Sender: [email-masked]

That shouldn't apply to someone like Dan, since he stores the data as UTF-16.

He may have a nasty problem converting some of his Subject fields to 
unicode, but it's not a collation problem in his case.

Arnt

Date: Tue, 17 Apr 2007 20:25:36 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: ANNOTATEMORE and COMPARATOR
Cc: Eric Burger <[email-masked]>, Alexey Melnikov <[email-masked]>, Dan Karp <[email-masked]>, Timo Sirainen <[email-masked]>, Cyrus Daboo <[email-masked]>
Sender: [email-masked]

Dan Karp writes:
>>  I wanted to permit implementations of i;ascii-casemap to do whatever 
>>  comes easiest outside US-ASCII. However, the final RFC doesn't 
>>  reflect my viewpoint in this respect. I don't remember why and 
>>  don't feel like searching my archive. I certainly wasn't aware of 
>>  the argument you bring up now.
>>
>>  Do you think this is a signficant issue? It doesn't affect searching 
>>  to a noticeable degree, and its impact on sorting is O(size of 
>>  result set), not O(size of mailbox).
>
> Actually, yeah, it does impact search. If you search for 
> "M<o-umlaut>tley Cr<u-umlaut>e", i;ascii-casemap doesn't allow that 
> to match "M<O-umlaut>TLEY CR<U-umlaut>E". If you've only stored an 
> upcased version of the text, you can't tell the case on the original 
> and thus you can't determine if the non-ASCII accented characters 
> match under i;octet.

By the time you get that far, you've eliminated almost all messages not 
in the actual result set, so opening and rechecking isn't so awfully 
expensive.

>>  > If your aim is to make baseline COMPARATOR support fairly
>>  > lightweight for IMAP servers, you may be constrained to only
>>  > requiring i;unicode-casemap.
>>
>>  Mumble. Given how many IMAP servers actually implement 
>>  i;ascii-casemap and nothing else, I'm not happy doing that. Make 
>>  some other people agree with you so Pete will say "WG consensus".
>
> Is this true?  Do these servers really implement i;ascii-casemap as 
> defined in RFC 4790?  I'd expect any RFC 3501-compliant server that 
> knows about more charsets than just US-ASCII to be supporting 
> i;unicode-casemap and *not* i;ascii-casemap because of 
> case-insensitive "SEARCH CHARSET".

Build some consensus please. I don't want to argue today - I'm in a mood 
to help little old ladies halfway across the street. Being civil is a 
strain.

Arnt

Date: Tue, 17 Apr 2007 10:06:04 -0700 (PDT)
From: Dan Karp <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: Eric Burger <[email-masked]>, Alexey Melnikov <[email-masked]>, Timo Sirainen <[email-masked]>, Cyrus Daboo <[email-masked]>, [email-masked], [email-masked]
Subject: Re: ANNOTATEMORE and COMPARATOR
Sender: [email-masked]

> I wanted to permit implementations of i;ascii-casemap to do whatever 
> comes easiest outside US-ASCII. However, the final RFC doesn't reflect
> my viewpoint in this respect. I don't remember why and don't feel like
> searching my archive. I certainly wasn't aware of the argument you 
> bring up now.
> 
> Do you think this is a signficant issue? It doesn't affect searching
> to a noticeable degree, and its impact on sorting is O(size of result 
> set), not O(size of mailbox).

Actually, yeah, it does impact search.  If you search for
"M<o-umlaut>tley Cr<u-umlaut>e", i;ascii-casemap doesn't allow that to
match "M<O-umlaut>TLEY CR<U-umlaut>E".  If you've only stored an
upcased version of the text, you can't tell the case on the original
and thus you can't determine if the non-ASCII accented characters
match under i;octet.

> > If your aim is to make baseline COMPARATOR support fairly
> > lightweight for IMAP servers, you may be constrained to only
> > requiring i;unicode-casemap.
> 
> Mumble. Given how many IMAP servers actually implement i;ascii-casemap
> and nothing else, I'm not happy doing that. Make some other people 
> agree with you so Pete will say "WG consensus".

Is this true?  Do these servers really implement i;ascii-casemap as
defined in RFC 4790?  I'd expect any RFC 3501-compliant server that
knows about more charsets than just US-ASCII to be supporting
i;unicode-casemap and *not* i;ascii-casemap because of case-insensitive
"SEARCH CHARSET".

Date: Tue, 17 Apr 2007 18:45:19 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: ANNOTATEMORE and COMPARATOR
Cc: Eric Burger <[email-masked]>, Alexey Melnikov <[email-masked]>, Dan Karp <[email-masked]>, Timo Sirainen <[email-masked]>, Cyrus Daboo <[email-masked]>
Sender: [email-masked]

Dan Karp writes:
> i;ascii-casemap is problematic as well, for roughly the same reasons:
>
>    Its equality, ordering, and substring operations are as for 
>    i;octet, except that at first, the lower-case letters (octet 
>    values 97-122) in each input string are changed to upper case 
>    (octet values 65-90).
>
> Servers wishing to optimize i;unicode-casemap may well store only a 
> searchable upper case version of the text, which will sort 
> incorrectly under i;ascii-casemap.

Good point.

I wanted to permit implementations of i;ascii-casemap to do whatever 
comes easiest outside US-ASCII. However, the final RFC doesn't reflect 
my viewpoint in this respect. I don't remember why and don't feel like 
searching my archive. I certainly wasn't aware of the argument you 
bring up now.

Do you think this is a signficant issue? It doesn't affect searching to 
a noticeable degree, and its impact on sorting is O(size of result 
set), not O(size of mailbox).

> If your aim is to make baseline COMPARATOR support fairly lightweight 
> for IMAP servers, you may be constrained to only requiring 
> i;unicode-casemap.

Mumble. Given how many IMAP servers actually implement i;ascii-casemap 
and nothing else, I'm not happy doing that. Make some other people 
agree with you so Pete will say "WG consensus".

Arnt

Date: Tue, 17 Apr 2007 08:52:18 -0700 (PDT)
From: Dan Karp <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: Eric Burger <[email-masked]>, Alexey Melnikov <[email-masked]>, Timo Sirainen <[email-masked]>, Cyrus Daboo <[email-masked]>, [email-masked], [email-masked]
Subject: Re: ANNOTATEMORE and COMPARATOR
Sender: [email-masked]

> > IMAP (via SEARCH) essentially requires that message content be 
> > searchable in some intermediate transcoded form (we chose UTF-16). 
> > COMPARATOR (via "i;octet") requires that message content be 
> > searchable in its original form pace content-encoding.
> 
> Only i;octet does that... we did a lot of work for RFC4790 to allow
> your kind of implementation, so I think it should be reasonable that 
> COMPARATOR allows it too.
> 
> As far as I can see, the mention of i;octet in 
> draft-crispin-collation-unicasemap is the only substantial requirement
> on i;octet, and it's weak. The one in draft-ietf-imapext-i18n is a 
> leftover - something I forgot to delete when its rationale
> disappeared.

i;ascii-casemap is problematic as well, for roughly the same reasons:

   Its equality, ordering, and substring operations are as for i;octet,
   except that at first, the lower-case letters (octet values 97-122) in
   each input string are changed to upper case (octet values 65-90).

Servers wishing to optimize i;unicode-casemap may well store only a
searchable upper case version of the text, which will sort incorrectly
under i;ascii-casemap.  COMPARATOR requires servers to implement both
i;ascii-casemap and i;octet (though i;octet sounds like it may be on the
way out).  If your aim is to make baseline COMPARATOR support fairly
lightweight for IMAP servers, you may be constrained to only requiring
i;unicode-casemap.

Subject: Re: [lemonade] Re: ANNOTATEMORE and COMPARATOR
Date: Tue, 17 Apr 2007 16:20:06 +0100
From: Dave Cridland <[email-masked]>
To: Mark Crispin <[email-masked]>
Cc: Enhancements to Internet email to support diverse service enivronments <[email-masked]>, <[email-masked]>
Sender: [email-masked]

On Tue Apr 17 14:30:14 2007, Mark Crispin wrote:
> 
> On Tue, 17 Apr 2007, Dave Cridland wrote:
>> I don't think it's quite right to state that "everything in IMAP 
>> is case-insensitive", since it seems intuitive to me that subject 
>> lines and other human-orientated text is demonstratively 
>> case-sensitive; it merely may be more useful to search it 
>> case-insensitively for much of the time.
> 
> Take a time machine back 21 years and tell me that I am wrong for 
> specifying that everything in IMAP is case-insensitive.  I would 
> have told you back then that you are wrong, and I see no good cause 
> to reverse that choice today.

NO, ON FURTHER CONSIDERATION YOU'RE ABSOLUTELY RIGHT. CASE IS NEVER 
INDICATIVE OF ANYTHING, THEREFORE IT MAKES NO SENSE TO SEARCH FOR IT 
IN ANY INSTANCE. ALSO, ALL EMAIL SHOULD BE PRESENTED ON PROPER 80 
COLUMN DISPLAYS, WITH NICE GREEN TEXT ON A PALE GREY SMEARED 
BACKGROUND YOU'RE ENCOURAGED TO CALL BLACK.

Seriously, though, while I'd readily agree that case should be 
irrelevant in IMAP protocol elements - we've already had that 
argument, in fact - I don't see that the same applies to 
human-created, human-directed text, such as subject fields, where the 
paragraph above does show that case conveys information.

It seems reasonable that we wouldn't want to mandate that clients 
should be prevented from searching case-sensitively. Similarly, many 
IMAP servers, by stripping away spaces and punctuation before the 
search test, can harm the usability of searches.

Not always. Just sometimes. I don't think that's a contentious 
statement, and I suspect I've worded this badly if you're disagreeing 
with me this strongly.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Date: Tue, 17 Apr 2007 06:30:14 -0700
From: Mark Crispin <[email-masked]>
To: Dave Cridland <[email-masked]>
cc: Enhancements to Internet email to support diverse service enivronments <[email-masked]>, IMAP Extensions WG <[email-masked]>
Subject: Re: [lemonade] Re: ANNOTATEMORE and COMPARATOR
Organization: Networks & Distributed Computing
Sender: [email-masked]

On Tue, 17 Apr 2007, Dave Cridland wrote:
> I don't think it's quite right to state that "everything in IMAP is 
> case-insensitive", since it seems intuitive to me that subject lines and 
> other human-orientated text is demonstratively case-sensitive; it merely may 
> be more useful to search it case-insensitively for much of the time.

Take a time machine back 21 years and tell me that I am wrong for 
specifying that everything in IMAP is case-insensitive.  I would have told 
you back then that you are wrong, and I see no good cause to reverse that 
choice today.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Subject: Re: [lemonade] Re: ANNOTATEMORE and COMPARATOR
Date: Tue, 17 Apr 2007 10:20:20 +0100
From: Dave Cridland <[email-masked]>
To: Mark Crispin <[email-masked]>
Cc: Arnt Gulbrandsen <[email-masked]>, Timo Sirainen <[email-masked]>, Enhancements to Internet email to support diverse service enivronments <[email-masked]>, <[email-masked]>
Sender: [email-masked]

On Tue Apr 17 00:04:24 2007, Mark Crispin wrote:
> 
> On Mon, 16 Apr 2007, Dave Cridland wrote:
>> ACAP deals in arbitrary binary data which *might* be treatable as 
>> text - the server has no way of knowing. "i;octet" is remarkably 
>> useful at doing exact matches, whether text or binary, and was the 
>> only comparator capable of doing case-sensitive matching on text.
>> In IMAP, this is the only use, and there are undoubtedly better 
>> ways of doing this.
> 
> I don't understand.  Everything in IMAP is case-insensitive by 
> definition. There is, AFAIK, no need or use for i;octet anywhere in 
> IMAP.

As I said, I was searching for OMA in subject lines rather a lot 
yesterday - if I were able to have specified a case-sensitive match, 
I would have done, since the sequence "oma" appears rather a lot in 
spurious results. (Such as "Domain", "Automaton", and rather 
interestingly, "to make" and "from a" on several implementations).

I don't think it's quite right to state that "everything in IMAP is 
case-insensitive", since it seems intuitive to me that subject lines 
and other human-orientated text is demonstratively case-sensitive; it 
merely may be more useful to search it case-insensitively for much of 
the time.

(This is not a demand for an "i;unicode-not-casemap" or anything, 
merely an observation.)

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Subject: Re: [lemonade] Re: ANNOTATEMORE and COMPARATOR
Date: Tue, 17 Apr 2007 10:11:21 +0100
From: Dave Cridland <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: Alexey Melnikov <[email-masked]>, Mark Crispin <[email-masked]>, Timo Sirainen <[email-masked]>, Cyrus Daboo <[email-masked]>, Enhancements to Internet email to support diverse service enivronments <[email-masked]>, <[email-masked]>
Sender: [email-masked]

On Tue Apr 17 09:38:31 2007, Arnt Gulbrandsen wrote:
> i;unicode-casemap contains a sentence which might be construed as 
> "and use i;octet from this point on". I'll try to suggest a 
> alternative wording there, something that reads naturally for an 
> UCS-4 implementation without i;octet.

Well, to do "i;octet" would mean encoding to UTF-8 first.

However, you get the same result if you perform a dictionary-like 
ordering over UCS-4, as far as I'm aware, so it's not a problem for 
collations, and equality tests work fine too.

It's things like substring matching which cause problems, since you 
can search for a substring matching partial UTF-8 sequences in 
principle, however, if you've already called those out as illegal 
input at a protocol level (or "super-comparator" level in this case), 
even that's blocked.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Date: Tue, 17 Apr 2007 10:43:54 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: ANNOTATEMORE and COMPARATOR
Cc: Eric Burger <[email-masked]>, Alexey Melnikov <[email-masked]>, Dan Karp <[email-masked]>, Timo Sirainen <[email-masked]>, Cyrus Daboo <[email-masked]>
Sender: [email-masked]

Dan Karp writes:
> IMAP (via SEARCH) essentially requires that message content be 
> searchable in some intermediate transcoded form (we chose UTF-16). 
> COMPARATOR (via "i;octet") requires that message content be 
> searchable in its original form pace content-encoding.

Only i;octet does that... we did a lot of work for RFC4790 to allow your 
kind of implementation, so I think it should be reasonable that 
COMPARATOR allows it too.

As far as I can see, the mention of i;octet in 
draft-crispin-collation-unicasemap is the only substantial requirement 
on i;octet, and it's weak. The one in draft-ietf-imapext-i18n is a 
leftover - something I forgot to delete when its rationale disappeared.

Arnt

Date: Tue, 17 Apr 2007 10:38:31 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: [lemonade] Re: ANNOTATEMORE and COMPARATOR
Cc: Alexey Melnikov <[email-masked]>, Mark Crispin <[email-masked]>, Timo Sirainen <[email-masked]>, Cyrus Daboo <[email-masked]>
Sender: [email-masked]

Mark Crispin writes:
> On Mon, 16 Apr 2007, Arnt Gulbrandsen wrote:
>>> The only place I see i;octet required is i18n draft, so I guess it's 
>>> still possible to change it?
>> That's the stated requirement. There may be an actual requirement 
>> elsewhere. SORT comes to mind.
>
> No. SORT would use i;unicode-casemap or (older servers) 
> i;ascii-casemap. I always assumed that i;octet was there as a 
> baseline for documentation purposes.  I never implemen ted it.

SORT requires a complete sort, doesn't it? Nothing other than i;octet 
provides that any more.

i;unicode-casemap contains a sentence which might be construed as "and 
use i;octet from this point on". I'll try to suggest a alternative 
wording there, something that reads naturally for an UCS-4 
implementation without i;octet.

>>> Is there a reason why a client should be able to force a server to 
>>> use i;ascii-casemap if i;unicode-casemap is also available?
>> Yes, but they're odd and weird.
>
> Examples, please?

Any test suite. A test client wishing to test everything required by 
lemonade would append some carefully constructed messages and use 
search on them with i;ascii-casemap. An IMAP client which connects to n 
IMAP servers and executes the same commands on each, noting any 
differences between the server responses.

I thought of something rather different last night, can't remember what 
it was. But odd and weird - nothing a MUA would do.

Arnt

Cc: [email-masked], Mark Crispin <[email-masked]>, Cyrus Daboo <[email-masked]>, Dave Cridland <[email-masked]>, Kjetil Torgrim Homme <[email-masked]>
From: Lisa Dusseault <[email-masked]>
Subject: Re: SORT on addresses
Date: Mon, 16 Apr 2007 16:58:14 -0700
To: Arnt Gulbrandsen <[email-masked]>
Sender: [email-masked]

On Apr 12, 2007, at 7:05 AM, Arnt Gulbrandsen wrote:

>
> Dave Cridland writes:
>> Yes, exactly - you've put it clearer than I did, but that's my  
>> intention.
>
> Fine phrasing.
>
>> I'd add a caveat that while I'll like to go all the way with the  
>> examination of what's needed, I wouldn't want to commit to  
>> producing capabilities to cover everything we find.
>
> Charter: Examine the types of sort, search and thread features  
> offered by modern email clients. Provide server-side extensions (to  
> IMAP, SMTP or other protocols) for that which it can be done faster  
> and/or better on the server.

Speaking personally, +1

Email clients like OSAF's Chandler and Opera M2 are ill-served by  
servers that assume that mail is divided into digestible chunks in  
child mailboxes/folders and that folders are the primary way users  
find emails.  Synch errors seem commoner (not assigning blame, merely  
noting what may be a spurious connection; I actually have no idea why  
this would be the case).

Faceted browsers are potentially a strong UI model for applications  
where the user wants to find a particular object among thousands in a  
library or repository (MP3s in a music library, email, files on disk,  
bookmarks, etc).  Many users are unwilling to sort all their data  
objects into folders and keywords or tags are not always filled in  
for individual objects even in a system that supports tagging.   Most  
users can't easily construct search queries with multiple terms from  
scratch, yet they can use faceted browsers which implicitly do  
multiple term queries.  Faceted browsing in some sense transcends  
both folder-sorted browsing and tagging/searching.  In order to work  
well however it needs well-indexed data and good search functions.

Lisa

Date: Mon, 16 Apr 2007 16:04:24 -0700 (Pacific Daylight Time)
From: Mark Crispin <[email-masked]>
To: Dave Cridland <[email-masked]>
cc: Arnt Gulbrandsen <[email-masked]>, Timo Sirainen <[email-masked]>, Enhancements to Internet email to support diverse service enivronments <[email-masked]>, [email-masked]
Subject: Re: [lemonade] Re: ANNOTATEMORE and COMPARATOR
Organization: Networks & Distributed Computing
Sender: [email-masked]

On Mon, 16 Apr 2007, Dave Cridland wrote:
> ACAP deals in arbitrary binary data which *might* be treatable as text - the 
> server has no way of knowing. "i;octet" is remarkably useful at doing exact 
> matches, whether text or binary, and was the only comparator capable of doing 
> case-sensitive matching on text.
> In IMAP, this is the only use, and there are undoubtedly better ways of doing 
> this.

I don't understand.  Everything in IMAP is case-insensitive by definition. 
There is, AFAIK, no need or use for i;octet anywhere in IMAP.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Subject: Re: [lemonade] Re: ANNOTATEMORE and COMPARATOR
Date: Mon, 16 Apr 2007 22:59:09 +0100
From: Dave Cridland <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: Mark Crispin <[email-masked]>, Timo Sirainen <[email-masked]>, Enhancements to Internet email to support diverse service enivronments <[email-masked]>, <[email-masked]>
Sender: [email-masked]

On Mon Apr 16 20:38:51 2007, Arnt Gulbrandsen wrote:
> Mark Crispin writes:
>> AFAIK, i;octet was documented for completeness.
> 
> FWIW: 2244 specified it, I don't know why, although I'm sure the 
> reason is in an archive somewhere. 4790 defined a registry and 
> inserted the preexisting collations.

ACAP deals in arbitrary binary data which *might* be treatable as 
text - the server has no way of knowing. "i;octet" is remarkably 
useful at doing exact matches, whether text or binary, and was the 
only comparator capable of doing case-sensitive matching on text.

In IMAP, this is the only use, and there are undoubtedly better ways 
of doing this.

(I do feel the need, sometimes, for case sensitivity - as today, when 
searching for messages with the subject line including "OMA").

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Cc: [email-masked], [email-masked], Alexey Melnikov <[email-masked]>, Cyrus Daboo <[email-masked]>
From: Timo Sirainen <[email-masked]>
Subject: Re: [lemonade] Re: ANNOTATEMORE and COMPARATOR
Date: Mon, 16 Apr 2007 22:54:09 +0300
To: Arnt Gulbrandsen <[email-masked]>
Sender: [email-masked]

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--Apple-Mail-1-877554154
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed

On 16.4.2007, at 22.35, Arnt Gulbrandsen wrote:

>> I would also want to see i;octet requirement gone. Actually I'd go  
>> even a bit further. Is there a reason why a client should be able  
>> to force a server to use i;ascii-casemap if i;unicode-casemap is  
>> also available?
>
> Yes, but they're odd and weird.
>
>> I can't think of any, but it would reduce optimization  
>> possibilities for server implementations.
>
> i;ascii-casemap is the same as i;unicode-casemap for the  
> intersection of their definition set, which would give you these  
> optimisations back. (IIRC, and it's late and...)

I'm just worried that some clients (or users) would select the  
i;ascii-casemap even when they could (or should) use i;unicode-casemap.

Non-ASCII characters with i;ascii-casemap are different, so if the  
server optimized i;unicode-casemap searches and a client did a non- 
ASCII search with i;ascii-casemap, the server would have to verify  
the unicode-optimized matches by searching through the messages to  
see if the non-ASCII casing matches exactly or not. Probably not that  
big of a deal, but it would be pretty pointless extra code.

--Apple-Mail-1-877554154
content-type: application/pgp-signature; x-mac-type=70674453;
	name=PGP.sig
content-description: This is a digitally signed message part
content-disposition: inline; filename=PGP.sig
content-transfer-encoding: 7bit

--Apple-Mail-1-877554154--

Date: Mon, 16 Apr 2007 12:44:33 -0700 (Pacific Daylight Time)
From: Mark Crispin <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
cc: [email-masked], [email-masked], Cyrus Daboo <[email-masked]>, Alexey Melnikov <[email-masked]>, Timo Sirainen <[email-masked]>
Subject: Re: [lemonade] Re: ANNOTATEMORE and COMPARATOR
Organization: Networks & Distributed Computing
Sender: [email-masked]

On Mon, 16 Apr 2007, Arnt Gulbrandsen wrote:
>> The only place I see i;octet required is i18n draft, so I guess it's still 
>> possible to change it?
> That's the stated requirement. There may be an actual requirement elsewhere. 
> SORT comes to mind.

No. SORT would use i;unicode-casemap or (older servers) i;ascii-casemap. 
I always assumed that i;octet was there as a baseline for documentation 
purposes.  I never implemen ted it.

>> Is there a reason why a client should be able to force a 
>> server to use i;ascii-casemap if i;unicode-casemap is also available?
> Yes, but they're odd and weird.

Examples, please?  My software did i;ascii-casemap, but only in old 
versions as an interim measure.  Newer versions are i;unicode-casemap 
only.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Date: Mon, 16 Apr 2007 21:38:51 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: [lemonade] Re: ANNOTATEMORE and COMPARATOR
Cc: Mark Crispin <[email-masked]>, Timo Sirainen <[email-masked]>
Sender: [email-masked]

Mark Crispin writes:
> AFAIK, i;octet was documented for completeness.

FWIW: 2244 specified it, I don't know why, although I'm sure the reason 
is in an archive somewhere. 4790 defined a registry and inserted the 
preexisting collations.

Arnt

Date: Mon, 16 Apr 2007 21:35:41 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: [lemonade] Re: ANNOTATEMORE and COMPARATOR
Cc: Alexey Melnikov <[email-masked]>, Timo Sirainen <[email-masked]>, Cyrus Daboo <[email-masked]>
Sender: [email-masked]

Timo Sirainen writes:
> On Mon, 2007-04-16 at 19:08 +0200, Arnt Gulbrandsen wrote:
>>  I'm wondering whether the i;octet requirement has outlived its 
>>  rationale. I know why it needed to be there in the early drafts 
>>  (other collations fell back to i;octet for some input), but I don't 
>>  know whether it needs to any more...?
>
> The only place I see i;octet required is i18n draft, so I guess it's 
> still possible to change it?

That's the stated requirement. There may be an actual requirement 
elsewhere. SORT comes to mind.

> I would also want to see i;octet requirement gone. Actually I'd go 
> even a bit further. Is there a reason why a client should be able to 
> force a server to use i;ascii-casemap if i;unicode-casemap is also 
> available?

Yes, but they're odd and weird.

> I can't think of any, but it would reduce optimization possibilities 
> for server implementations.

i;ascii-casemap is the same as i;unicode-casemap for the intersection of 
their definition set, which would give you these optimisations back. 
(IIRC, and it's late and...)

Arnt

Date: Mon, 16 Apr 2007 11:00:47 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: Timo Sirainen <[email-masked]>
cc: [email-masked], IMAP Extensions WG <[email-masked]>
Subject: Re: [lemonade] Re: ANNOTATEMORE and COMPARATOR
Sender: [email-masked]

IMHO:

i;ascii-casemap is important because there are legacy implementations that 
implement it.  Specifically, i;ascii-casemap *is* effectively the 
comparator of IMAP since its creation in 1986.  I believe that most modern 
IMAP server implementations do some version of i;unicode-casemap instead 
of i;ascii-casemap, and there is no way for the client to know with a 
server that does not otherwise support COMPARATOR.

I can't imagine that a client would choose i;ascii-casemap instead of 
i;unicode-casemap.

AFAIK, i;octet was documented for completeness.

I would be alright with i;unicode-casemap as being mandatory-to-implement 
for COMPARATOR, and optional-to-implement (albeit with a SHOULD) instead 
of i;ascii-casemap for non-COMPARATOR servers.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Date: Mon, 16 Apr 2007 18:51:40 +0100
From: Alexey Melnikov <[email-masked]>
To: Timo Sirainen <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: [lemonade] Re: ANNOTATEMORE and COMPARATOR
Sender: [email-masked]

Timo Sirainen wrote:

>Is the i;basic mentioned in i18n draft the same as i;octet?
>  
>
No. i;basic is defined in draft-gulbrandsen-collation-basic-01.txt.
Note that the mandatory-to-implement in i18n draft is going to be 
i;unicode-casemap.

Subject: Re: [lemonade] Re: ANNOTATEMORE and COMPARATOR
From: Timo Sirainen <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: [email-masked], [email-masked], Alexey Melnikov <[email-masked]>, Cyrus Daboo <[email-masked]>
Date: Mon, 16 Apr 2007 20:43:33 +0300
Sender: [email-masked]

--=-K+TEBP4YHYMuMTvRZ4hT
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

On Mon, 2007-04-16 at 19:08 +0200, Arnt Gulbrandsen wrote:
> I'm wondering whether the i;octet requirement has outlived its=20
> rationale. I know why it needed to be there in the early drafts (other=20
> collations fell back to i;octet for some input), but I don't know=20
> whether it needs to any more...?

The only place I see i;octet required is i18n draft, so I guess it's
still possible to change it?

I would also want to see i;octet requirement gone. Actually I'd go even
a bit further. Is there a reason why a client should be able to force a
server to use i;ascii-casemap if i;unicode-casemap is also available? I
can't think of any, but it would reduce optimization possibilities for
server implementations.

Is the i;basic mentioned in i18n draft the same as i;octet?

--=-K+TEBP4YHYMuMTvRZ4hT
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

--=-K+TEBP4YHYMuMTvRZ4hT--

Date: Mon, 16 Apr 2007 19:20:05 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: [lemonade] Re: ANNOTATEMORE and COMPARATOR
Cc: Mark Crispin <[email-masked]>, Timo Sirainen <[email-masked]>
Sender: [email-masked]

Mark Crispin writes:
> For collations in the IMAP context, UTF-16 is problematic for i;octet,

yes.

> i;ascii-casemap, and i;unicode-casemap.  I'm not sure about i;basic.

None of these - the problem was raised about a year ago by an RFC 4790 
reviewer and we solved it.

Using UTF-16 or UTF-8 instead of a sensible unicode encoding may make it 
more difficult to implement e.g. i;ascii-casemap, but not especially. 
Every kind of processing is more difficult in UTF-* than in UCS-*, IMO, 
and most collations are just specific examples of this general rules.

Arnt

Date: Mon, 16 Apr 2007 10:19:17 -0700 (PDT)
From: Dan Karp <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: Eric Burger <[email-masked]>, Alexey Melnikov <[email-masked]>, Timo Sirainen <[email-masked]>, Cyrus Daboo <[email-masked]>, [email-masked], [email-masked]
Subject: Re: ANNOTATEMORE and COMPARATOR
Sender: [email-masked]

> >   a) COMPARATOR is a reasonably heavy extension to implement
> 
> I don't think COMPARATOR ought to be that. Various other extensions 
> require it, and it seems wrong for me that COMPARATOR should be
> heavier than its users. Tail wagging dog.
> 
> Tell me what makes COMPARATOR heavy?

IMAP (via SEARCH) essentially requires that message content be
searchable in some intermediate transcoded form (we chose UTF-16).
COMPARATOR (via "i;octet") requires that message content be searchable
in its original form pace content-encoding.

Date: Mon, 16 Apr 2007 10:15:04 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: Timo Sirainen <[email-masked]>
cc: Dan Karp <[email-masked]>, [email-masked], IMAP Extensions WG <[email-masked]>
Subject: Re: ANNOTATEMORE and COMPARATOR
Sender: [email-masked]

On Mon, 16 Apr 2007, Timo Sirainen wrote:
> On Mon, 2007-04-16 at 09:41 -0700, Dan Karp wrote:
>> Searchable content is effectively transcoded to UTF-16.  The original 
>> content needed for "i;octet" is available only by serially opening, 
>> reading, and reparsing each message in the mailbox (stored 1 file per 
>> message), which is sufficiently nonperformant that we can't permit it.
> Do you support full substring searches as standard SEARCH command
> requires? In Zimbra's feature list I see "Ability to search using a
> prefix plus a wildcard", which seems like a "no" answer. In that case
> you're already noncompliant.

True.  I'm not sure what "search using a prefix plus a wildcard" is 
supposed to mean in IMAP searching, which is of substrings only.

For collations in the IMAP context, UTF-16 is problematic for i;octet, 
i;ascii-casemap, and i;unicode-casemap.  I'm not sure about i;basic.

I suggest that you use either UTF-8 or UCS-4.  The attraction of UTF-16 is 
to save memory space and a mostly-constant width if most characters are in 
the BMP (thus effectively the same as UCS-2 handling), but at the cost of 
MUCH more complex handling of non-BMP characters.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Date: Mon, 16 Apr 2007 19:12:25 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: ANNOTATEMORE and COMPARATOR
Cc: Eric Burger <[email-masked]>, Alexey Melnikov <[email-masked]>, Dan Karp <[email-masked]>, Timo Sirainen <[email-masked]>, Cyrus Daboo <[email-masked]>
Sender: [email-masked]

Dan Karp writes:
>   a) COMPARATOR is a reasonably heavy extension to implement

I don't think COMPARATOR ought to be that. Various other extensions 
require it, and it seems wrong for me that COMPARATOR should be heavier 
than its users. Tail wagging dog.

Tell me what makes COMPATOR heavy?

Arnt

Date: Mon, 16 Apr 2007 19:08:37 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: ANNOTATEMORE and COMPARATOR
Cc: Eric Burger <[email-masked]>, Alexey Melnikov <[email-masked]>, Dan Karp <[email-masked]>, Cyrus Daboo <[email-masked]>
Sender: [email-masked]

Dan Karp writes:
> Searchable content is effectively transcoded to UTF-16.  The original 
> content needed for "i;octet" is available only by serially opening, 
> reading, and reparsing each message in the mailbox (stored 1 file per 
> message), which is sufficiently nonperformant that we can't permit 
> it.

Makes sense. (We do something similar, although in our case it the 
problem is mitigated by other circumstances.)

I'm wondering whether the i;octet requirement has outlived its 
rationale. I know why it needed to be there in the early drafts (other 
collations fell back to i;octet for some input), but I don't know 
whether it needs to any more...?

Arnt

Date: Mon, 16 Apr 2007 10:03:51 -0700 (PDT)
From: Dan Karp <[email-masked]>
To: Timo Sirainen <[email-masked]>
Cc: Arnt Gulbrandsen <[email-masked]>, Eric Burger <[email-masked]>, Alexey Melnikov <[email-masked]>, Cyrus Daboo <[email-masked]>, [email-masked], [email-masked]
Subject: Re: ANNOTATEMORE and COMPARATOR
Sender: [email-masked]

> > Searchable content is effectively transcoded to UTF-16.  The
> > original content needed for "i;octet" is available only by serially
> > opening, reading, and reparsing each message in the mailbox (stored 1
> > file per message), which is sufficiently nonperformant that we can't
> > permit it.
> 
> Do you support full substring searches as standard SEARCH command
> requires? In Zimbra's feature list I see "Ability to search using a
> prefix plus a wildcard", which seems like a "no" answer. In that case
> you're already noncompliant.

*That* is a red herring, and I believe that you know it.  Can we please
redirect the discussion back to the core issue, which is that

  a) COMPARATOR is a reasonably heavy extension to implement
  b) METADATA is meant to be a reasonably light extension to implement
  c) METADATA is being positioned as a prerequisite for many new
extensions
  d) No METADATA entries appear able to make use of the COMPARATOR
functionality

Subject: Re: ANNOTATEMORE and COMPARATOR
From: Timo Sirainen <[email-masked]>
To: Dan Karp <[email-masked]>
Cc: Arnt Gulbrandsen <[email-masked]>, Eric Burger <[email-masked]>, Alexey Melnikov <[email-masked]>, Cyrus Daboo <[email-masked]>, [email-masked], [email-masked]
Date: Mon, 16 Apr 2007 19:53:10 +0300
Sender: [email-masked]

--=-fYPDKaFTsPDGb6/paBZs
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

On Mon, 2007-04-16 at 09:41 -0700, Dan Karp wrote:
> > > Unfortunately, my server implementation is not compatible with=20
> > > COMPARATOR, in particular the requirement to support i;octet in=20
> > > SEARCH.
> >=20
> > Why not?
>=20
> Searchable content is effectively transcoded to UTF-16.  The original con=
tent needed for "i;octet" is available only by serially opening, reading, a=
nd reparsing each message in the mailbox (stored 1 file per message), which=
 is sufficiently nonperformant that we can't permit it.

Do you support full substring searches as standard SEARCH command
requires? In Zimbra's feature list I see "Ability to search using a
prefix plus a wildcard", which seems like a "no" answer. In that case
you're already noncompliant.

--=-fYPDKaFTsPDGb6/paBZs
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

--=-fYPDKaFTsPDGb6/paBZs--

Date: Mon, 16 Apr 2007 09:41:33 -0700 (PDT)
From: Dan Karp <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: Eric Burger <[email-masked]>, Alexey Melnikov <[email-masked]>, Cyrus Daboo <[email-masked]>, [email-masked], [email-masked]
Subject: Re: ANNOTATEMORE and COMPARATOR
Sender: [email-masked]

> > Unfortunately, my server implementation is not compatible with 
> > COMPARATOR, in particular the requirement to support i;octet in 
> > SEARCH.
> 
> Why not?

Searchable content is effectively transcoded to UTF-16.  The original content needed for "i;octet" is available only by serially opening, reading, and reparsing each message in the mailbox (stored 1 file per message), which is sufficiently nonperformant that we can't permit it.

Date: Mon, 16 Apr 2007 18:34:46 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: ANNOTATEMORE and COMPARATOR
Cc: Eric Burger <[email-masked]>, Alexey Melnikov <[email-masked]>, Dan Karp <[email-masked]>, Cyrus Daboo <[email-masked]>
Sender: [email-masked]

Dan Karp writes:
> Unfortunately, my server implementation is not compatible with 
> COMPARATOR, in particular the requirement to support i;octet in 
> SEARCH.

Why not?

Arnt

Date: Mon, 16 Apr 2007 09:14:57 -0700 (PDT)
From: Dan Karp <[email-masked]>
To: Alexey Melnikov <[email-masked]>
Cc: Eric Burger <[email-masked]>, Enhancements to Internet email to support diverse service enivronments <[email-masked]>, IMAP Extensions WG <[email-masked]>, Cyrus Daboo <[email-masked]>
Subject: ANNOTATEMORE and COMPARATOR
Sender: [email-masked]

I'd really like to implement LEMONADE.  LEMONADE consists of several
IMAP extensions which require METADATA, and the current METADATA draft
in turn requires LIST-EXTENDED and COMPARATOR.  Unfortunately, my server
implementation is not compatible with COMPARATOR, in particular the
requirement to support i;octet in SEARCH.  I'm hoping that there's a way
to come to a compromise whereby METADATA can return to not requiring
COMPARATOR while still maintaining all its core functionality.  I'd hate
for such a small thing to be a roadblock for my and others' LEMONADE
implementations.

Right now, METADATA uses COMPARATOR to specify LIST-EXTENDED selection
constraints.  The GETMETADATA command doesn't appear to support these
constraints at present (though this may be an oversight):

    getmetadata       = "GETMETADATA" SP entries SP attribs
    entries           = entry-match /
                        "(" entry-match *(SP entry-match) ")"
    entry-match       = list-mailbox
    attribs           = attrib / "(" attrib *(SP attrib) ")"
    attrib            = astring

I've looked through all the entries defined by the base METADATA draft
as well as those defined by several of the LEMONADE drafts that metion
METADATA (/comment, /motd, /admin, /sort, /thread, /check, /checkperiod,
/convert/..., etc.).  None of these appear to benefit in a meaningful
way from the LIST-EXTENDED match/collation feature, so I'm gathering
that the inclusion of COMPARATOR is intended as more of a nice-to-have
with an eye towards accommodating possible future needs rather than a
real core METADATA feature.  If this is the case -- and towards the end
of making METADATA a workable foundation extension that every server can
implement -- I'd like to request that the LIST-EXTENDED METADATA
selection option type be removed from the METADATA draft.  If in the
future there is a new METADATA entry that would substantially benefit
from using COMPARATOR with LIST-EXTENDED, it should be possible to
reintroduce this functionality (or a subset thereof) in the same RFC
that registers the new entry.

Content-class: urn:content-classes:message
Subject: RE: [lemonade] Deprecate IMAP IDLE?
Date: Sun, 15 Apr 2007 13:08:52 +0300
From: <[email-masked]>
To: <[email-masked]>, <[email-masked]>
Cc: <[email-masked]>
Sender: [email-masked]

Dear Pete,
You said:

>Take a look at LEMONADE, 
>for example. There is really little going on in LEMONADE that 
>could not have been done in IMAPEXT. It's pretty much the same 
>people and it's mostly IMAP work. 

I think that this one might argue that deprecating IMAP IDLE is in fact
in the charter for LEMONADE.
Joke level: 10%.

Date: Sat, 14 Apr 2007 15:02:08 -0500
To: Arnt Gulbrandsen <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: Re: [lemonade] Deprecate IMAP IDLE?
Cc: IMAP Extensions WG <[email-masked]>
Sender: [email-masked]

On 4/14/07 at 9:16 PM +0200, Arnt Gulbrandsen wrote:

>Don't you think we should prolong the life of IMAPEXT by keeping it 
>alive until each of its standards-track products either has reached 
>full standard or certainly won't proceed further along the standards 
>track?
>
>(I'm only half joking. Well, only 60% joking.)

OK, so since we're only half joking, let me be half (well, maybe 
three-quarters) serious.

Working groups are supposed to be short-lived task-oriented groups, 
not long lived review bodies looking for things to do. Aside from the 
organizational reasons to do that, I think there is a good (albeit 
sociological/psychological) reason to shut down old working groups 
and start up new ones: People simply run out of energy to do good 
work in long ongoing efforts and get re-energized when there is a 
new, specific effort. Take a look at LEMONADE, for example. There is 
really little going on in LEMONADE that could not have been done in 
IMAPEXT. It's pretty much the same people and it's mostly IMAP work. 
But the fact that it was a new effort with a specific (well, mostly 
specific) goal in mind re-energized people and got them working 
toward that goal. The act of chartering (getting everyone to step 
back and decide what they're doing in the "bigger picture", coming up 
with a case for the IESG and the rest of the community, dividing up 
the tasks, etc.), writing -00 drafts, getting new chairs and document 
editors, along with the simple fact that the work *feels* new, makes 
the work go forward.

So, no, I don't think it's good to keep working groups around when 
they get really old. (This does leave open the question of how to 
progress things along the standards track without a WG, and I think 
that the IESG should--and is starting to--look at that issue.)

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

Date: Sat, 14 Apr 2007 21:16:20 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: Pete Resnick <[email-masked]>
Subject: Re: [lemonade] Deprecate IMAP IDLE?
Cc: IMAP Extensions WG <[email-masked]>
Sender: [email-masked]

Hi there Mr. Chair. Don't you think we should prolong the life of 
IMAPEXT by keeping it alive until each of its standards-track products 
either has reached full standard or certainly won't proceed further 
along the standards track?

(I'm only half joking. Well, only 60% joking.)

Arnt

Date: Sat, 14 Apr 2007 11:55:45 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: Pete Resnick <[email-masked]>
cc: IMAP Extensions WG <[email-masked]>
Subject: Re: SORT on addresses
Sender: [email-masked]

On Sat, 14 Apr 2007, Pete Resnick wrote:
> On 4/12/07 at 5:57 PM +0100, Alexey Melnikov wrote:
>> Mark Crispin wrote:
>>> And once again, IMAPEXT finds yet another way to prolong its life.
>> IMHO, for this work to happen as an official WG, either IMAPEXT needs to 
>> recharter, or a new WG should be formed.
>> I am slightly in favor of the latter.
> *If* this work happens, I am *overwhelmingly* in favor of the later.

+1 (in case it wasn't obvious)

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Date: Sat, 14 Apr 2007 13:52:49 -0500
To: Alexey Melnikov <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: Re: SORT on addresses
Cc: Mark Crispin <[email-masked]>, IMAP Extensions WG <[email-masked]>
Sender: [email-masked]

On 4/12/07 at 5:57 PM +0100, Alexey Melnikov wrote:

>Mark Crispin wrote:
>
>>And once again, IMAPEXT finds yet another way to prolong its life.
>
>IMHO, for this work to happen as an official WG, either IMAPEXT 
>needs to recharter, or a new WG should be formed.
>I am slightly in favor of the latter.

*If* this work happens, I am *overwhelmingly* in favor of the later.

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

Date: Thu, 12 Apr 2007 11:26:28 -0700
From: Peter Coates <[email-masked]>
Subject: RE: SORT on addresses
To: [email-masked]
Sender: [email-masked]

To: "Fredrick the Great" <fred@domain>, friends: bob@domain, sally@domain;
	Sort on Fredrick the Great

To: fred@domain, friends: bob@domain, sally@domain;
	Sort on fred@domain

To: friends: bob@domain, sally@domain;, fred@domain
	Sort friends

To: friends: "Robert the Bruce"<bob@domain>, sally@domain;, fred@domain
	Sort friends

Peter

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
On Behalf Of Philip Guenther
Sent: April 12, 2007 11:02
To: Peter Coates
Cc: [email-masked]
Subject: RE: SORT on addresses

On Thu, 12 Apr 2007, Peter Coates wrote:
> The display name of the group if there is one, failing which the 
> display name of the first address, failing which the first address. Is 
> the algorithm.

So this:

To: fred@domain, friends: bob@domain, sally@domain;

will sort on 'friends' and not 'fred'?  Interesting.

Philip Guenther

Date: Thu, 12 Apr 2007 11:05:12 -0700
From: Peter Coates <[email-masked]>
Subject: RE: SORT on addresses
To: "'Philip Guenther'" <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

No.  The algorithm uses the first address it finds.  If that is a group, OK, but
in this case the first thing it finds is fred@domain, so that is what it uses

Peter 

-----Original Message-----
From: Philip Guenther [mailto:[email-masked]] 
Sent: April 12, 2007 11:02
To: Peter Coates
Cc: [email-masked]
Subject: RE: SORT on addresses

On Thu, 12 Apr 2007, Peter Coates wrote:
> The display name of the group if there is one, failing which the 
> display name of the first address, failing which the first address. Is 
> the algorithm.

So this:

To: fred@domain, friends: bob@domain, sally@domain;

will sort on 'friends' and not 'fred'?  Interesting.

Philip Guenther

Date: Thu, 12 Apr 2007 12:02:01 -0600
From: Philip Guenther <[email-masked]>
To: Peter Coates <[email-masked]>
cc: [email-masked]
Subject: RE: SORT on addresses
Sender: [email-masked]

On Thu, 12 Apr 2007, Peter Coates wrote:
> The display name of the group if there is one, failing which the display 
> name of the first address, failing which the first address. Is the 
> algorithm.

So this:

To: fred@domain, friends: bob@domain, sally@domain;

will sort on 'friends' and not 'fred'?  Interesting.

Philip Guenther

Date: Thu, 12 Apr 2007 10:54:30 -0700
From: Peter Coates <[email-masked]>
Subject: RE: SORT on addresses
To: "'Philip Guenther'" <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

The display name of the group if there is one, failing which the display name of
the first address, failing which the first address. Is the algorithm.  

-----Original Message-----
From: Philip Guenther [mailto:[email-masked]] 
Sent: April 12, 2007 10:33
To: Peter Coates
Cc: [email-masked]
Subject: RE: SORT on addresses

On Thu, 12 Apr 2007, Peter Coates wrote:
> Being incompetent, I replied, rather than replied all when I sent a 
> note saying that the Sun imap server uses sort keys X-SUN-TO, 
> X-SUN-CC, X-SUN-FROM to sort by display-name or if display name is not 
> present the email address.  For obvious reasons I like that algorithm.

The display-name of the first actual address or will it use the display-name of
a group?

Hmm, the current SORT address keys use the first addr-mailbox, so they don't
skip groups but will sort something like

To: friends: bob@domain, sally@domain;

with the key "friends"...which will be annoying if the display-name contains an
RFC 2047 word, as there's no mention of decoding them like there is for the
SUBJECT key.

Philip Guenther

Date: Thu, 12 Apr 2007 11:33:21 -0600
From: Philip Guenther <[email-masked]>
To: Peter Coates <[email-masked]>
cc: [email-masked]
Subject: RE: SORT on addresses
Sender: [email-masked]

On Thu, 12 Apr 2007, Peter Coates wrote:
> Being incompetent, I replied, rather than replied all when I sent a note 
> saying that the Sun imap server uses sort keys X-SUN-TO, X-SUN-CC, 
> X-SUN-FROM to sort by display-name or if display name is not present the 
> email address.  For obvious reasons I like that algorithm.

The display-name of the first actual address or will it use the 
display-name of a group?

Hmm, the current SORT address keys use the first addr-mailbox, so they 
don't skip groups but will sort something like

To: friends: bob@domain, sally@domain;

with the key "friends"...which will be annoying if the display-name 
contains an RFC 2047 word, as there's no mention of decoding them like 
there is for the SUBJECT key.

Philip Guenther

Date: Thu, 12 Apr 2007 10:16:58 -0700
From: Mark Crispin <[email-masked]>
To: Alexey Melnikov <[email-masked]>
cc: IMAP Extensions WG <[email-masked]>
Subject: Re: SORT on addresses
Organization: Networks & Distributed Computing
Sender: [email-masked]

On Thu, 12 Apr 2007, Alexey Melnikov wrote:
> IMHO, for this work to happen as an official WG, either IMAPEXT needs to 
> recharter, or a new WG should be formed.
> I am slightly in favor of the latter.

Personally, I feel that it is long past time for IMAPEXT to die.  Nor 
should there be a replacement any time in the short-term future.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Date: Thu, 12 Apr 2007 17:57:28 +0100
From: Alexey Melnikov <[email-masked]>
To: Mark Crispin <[email-masked]>
CC: IMAP Extensions WG <[email-masked]>
Subject: Re: SORT on addresses
Sender: [email-masked]

Mark Crispin wrote:

> And once again, IMAPEXT finds yet another way to prolong its life.

IMHO, for this work to happen as an official WG, either IMAPEXT needs to 
recharter, or a new WG should be formed.
I am slightly in favor of the latter.

Date: Thu, 12 Apr 2007 09:30:47 -0700
From: Peter Coates <[email-masked]>
Subject: RE: SORT on addresses
To: "'Cyrus Daboo'" <[email-masked]>, "'Dave Cridland'" <[email-masked]>, "'Mark Crispin'" <[email-masked]>
Cc: [email-masked], "'Kjetil Torgrim Homme'" <[email-masked]>
Sender: [email-masked]

There are a class of clients that DO use server side sort and search, and those
are web clients.  These are probably used more than desktop imap clients.  And
lemonade is premised on the assumption that mobile devices will use imap and are
too strapped for processing power to do more than the necessary.

Being incompetent, I replied, rather than replied all when I sent a note saying
that the Sun imap server uses sort keys X-SUN-TO, X-SUN-CC, X-SUN-FROM to sort
by display-name or if display name is not present the email address.  For
obvious reasons I like that algorithm.

Peter

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
On Behalf Of Cyrus Daboo
Sent: April 12, 2007 06:27
To: Dave Cridland; Mark Crispin
Cc: [email-masked]; Kjetil Torgrim Homme
Subject: Re: SORT on addresses

Hi Dave,

--On April 12, 2007 10:03:46 AM +0100 Dave Cridland <[email-masked]>
wrote:

> I think that there would be serious advantage to taking a step back at 
> this point, and examining how users and UAs are currently searching 
> through mail, and see how we could make this both more efficient and 
> closer to the end-users' requirements. Right now, I think that IMAP 
> provides only rudimentary "organizational" abilities, and I think that 
> they're showing up as a weak point in IMAP.

Lets face it, very few clients really use server-side sort/thread/search. 
Most download and cache and do local operations because they can offer much,
much more in the way of features for that.

If we are going to make changes to the server-side behavior we should go all the
way. Examine the types of features offered by desktop clients doing local
operations and match those with server-side capabilities. Anything less is a
poor half-measure.

--
Cyrus Daboo

Subject: Re: SORT on addresses
Date: Thu, 12 Apr 2007 16:29:12 +0100
From: Dave Cridland <[email-masked]>
To: Mark Crispin <[email-masked]>
Cc: <[email-masked]>
Sender: [email-masked]

On Thu Apr 12 15:29:27 2007, Mark Crispin wrote:
> 
> And once again, IMAPEXT finds yet another way to prolong its life.

No, we should come up with a much more contrived acronym than that. 
:-)

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Date: Thu, 12 Apr 2007 11:27:33 -0400
From: Tony Hansen <[email-masked]>
CC: IMAP Extensions WG <[email-masked]>
Subject: Re: SORT on addresses
Sender: [email-masked]

what, you expected a rest? Silly Pooh Bear, in the words of the immortal
Christopher Robin. :-)

Mark Crispin wrote:
> 
> And once again, IMAPEXT finds yet another way to prolong its life.

Date: Thu, 12 Apr 2007 16:45:38 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: SORT on addresses
Cc: Mark Crispin <[email-masked]>
Sender: [email-masked]

Mark Crispin writes:
> And once again, IMAPEXT finds yet another way to prolong its life.

IMAP is the protocol that just won't lie down and die.

Arnt

Subject: Re: SORT on addresses
From: Kjetil Torgrim Homme <[email-masked]>
To: Cyrus Daboo <[email-masked]>
Cc: [email-masked]
Date: Thu, 12 Apr 2007 16:42:16 +0200
Sender: [email-masked]

On Thu, 2007-04-12 at 09:27 -0400, Cyrus Daboo wrote:
> Lets face it, very few clients really use server-side sort/thread/search. 
> Most download and cache and do local operations because they can offer 
> much, much more in the way of features for that.

ironically(?), webmail applications are probably the most common user of
"stateless" IMAP.  this is a space where improvements in server and
client are relatively easily deployed, so it's not a bad space to work
in.

> If we are going to make changes to the server-side behavior we should go 
> all the way. Examine the types of features offered by desktop clients doing 
> local operations and match those with server-side capabilities. Anything 
> less is a poor half-measure.

I wouldn't word it that strongly, but sure -- sounds like you have the
work cut out for us.  one thing is to finalise complicated stuff like
"context", there are also simple things like standardise on flags for
"Spam" and "Ham" which could do wonders for client interoperability.

-- 
Kjetil T.

Date: Thu, 12 Apr 2007 07:29:27 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: IMAP Extensions WG <[email-masked]>
Subject: Re: SORT on addresses
Sender: [email-masked]

And once again, IMAPEXT finds yet another way to prolong its life.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Date: Thu, 12 Apr 2007 16:05:32 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: SORT on addresses
Cc: Mark Crispin <[email-masked]>, Cyrus Daboo <[email-masked]>, Dave Cridland <[email-masked]>, Kjetil Torgrim Homme <[email-masked]>
Sender: [email-masked]

Dave Cridland writes:
> Yes, exactly - you've put it clearer than I did, but that's my intention.

Fine phrasing.

> I'd add a caveat that while I'll like to go all the way with the 
> examination of what's needed, I wouldn't want to commit to producing 
> capabilities to cover everything we find.

Charter: Examine the types of sort, search and thread features offered 
by modern email clients. Provide server-side extensions (to IMAP, SMTP 
or other protocols) for that which it can be done faster and/or better 
on the server.

Arnt

Subject: Re: SORT on addresses
Date: Thu, 12 Apr 2007 14:46:07 +0100
From: Dave Cridland <[email-masked]>
To: Cyrus Daboo <[email-masked]>
Cc: <[email-masked]>, Kjetil Torgrim Homme <[email-masked]>, Mark Crispin <[email-masked]>
Sender: [email-masked]

On Thu Apr 12 14:27:19 2007, Cyrus Daboo wrote:
> 
> Hi Dave,
> 
> --On April 12, 2007 10:03:46 AM +0100 Dave Cridland 
> <[email-masked]> wrote:
> 
>> I think that there would be serious advantage to taking a step 
>> back at
>> this point, and examining how users and UAs are currently searching
>> through mail, and see how we could make this both more efficient 
>> and
>> closer to the end-users' requirements. Right now, I think that IMAP
>> provides only rudimentary "organizational" abilities, and I think 
>> that
>> they're showing up as a weak point in IMAP.
> 
> Lets face it, very few clients really use server-side 
> sort/thread/search. Most download and cache and do local operations 
> because they can offer much, much more in the way of features for 
> that.
> 
> If we are going to make changes to the server-side behavior we 
> should go all the way. Examine the types of features offered by 
> desktop clients doing local operations and match those with 
> server-side capabilities. Anything less is a poor half-measure.

Yes, exactly - you've put it clearer than I did, but that's my 
intention.

I'd add a caveat that while I'll like to go all the way with the 
examination of what's needed, I wouldn't want to commit to producing 
capabilities to cover everything we find.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Subject: Re: SORT on addresses
From: Kjetil Torgrim Homme <[email-masked]>
To: [email-masked]
Date: Thu, 12 Apr 2007 15:45:02 +0200
Sender: [email-masked]

first, I'd like to thank everyone for their reasoned and constructive
responses.

On Thu, 2007-04-12 at 10:03 +0100, Dave Cridland wrote:
> On Thu Apr 12 02:38:18 2007, Mark Crispin wrote:
> > Now, with that said, I would not oppose a SORTEXT proposal that 
> > adds new keys (as opposed to changing the current keys) that work 
> > as you propose.
> 
> I would oppose a proposal *now*. I don't think we know nearly enough 
> to base a decision on concerning which keys to add.

fair enough.  Peter Coates told me in a private reply that Sun has an
extension which sorts on display-name, which very well may be more
aligned with what users want/expect anyway.  there are probably other
options.

> So, here's a counter-proposal:
> 
> We setup a mailing list, or at least agree to use an existing one, 
> invite a few UA developers and search-agent people [since we're heavy 
> on the server-side as usual], and start to figure out what's needed, 
> what's practical, and get to work on it. If we seem to be making 
> progress, we turn it into a WG if it seems to suit.

sounds good to me.
-- 
Kjetil T.

Date: Thu, 12 Apr 2007 09:27:19 -0400
From: Cyrus Daboo <[email-masked]>
To: Dave Cridland <[email-masked]>, Mark Crispin <[email-masked]>
cc: [email-masked], Kjetil Torgrim Homme <[email-masked]>
Subject: Re: SORT on addresses
Sender: [email-masked]

Hi Dave,

--On April 12, 2007 10:03:46 AM +0100 Dave Cridland <[email-masked]> 
wrote:

> I think that there would be serious advantage to taking a step back at
> this point, and examining how users and UAs are currently searching
> through mail, and see how we could make this both more efficient and
> closer to the end-users' requirements. Right now, I think that IMAP
> provides only rudimentary "organizational" abilities, and I think that
> they're showing up as a weak point in IMAP.

Lets face it, very few clients really use server-side sort/thread/search. 
Most download and cache and do local operations because they can offer 
much, much more in the way of features for that.

If we are going to make changes to the server-side behavior we should go 
all the way. Examine the types of features offered by desktop clients doing 
local operations and match those with server-side capabilities. Anything 
less is a poor half-measure.

-- 
Cyrus Daboo

Date: Thu, 12 Apr 2007 09:16:23 -0400
From: Ken Murchison <[email-masked]>
Organization: Carnegie Mellon University
To: [email-masked]
Subject: Re: SORT on addresses
Sender: [email-masked]

Dave Cridland wrote:

> So, here's a counter-proposal:
> 
> We setup a mailing list, or at least agree to use an existing one, 
> invite a few UA developers and search-agent people [since we're heavy on 
> the server-side as usual], and start to figure out what's needed, what's 
> practical, and get to work on it. If we seem to be making progress, we 
> turn it into a WG if it seems to suit.
> 
> The aim would be to get, or look into:
> - a set of useful SEARCH keys, SORT keys, etc.
> - multimailbox SEARCHing.
> - Inexact searching, such as SCAN provides.
> - Sorting by relevance.
> - probably other search and organizational things. (Perhaps we could 
> stretch to a re-examination of ANNOTATE, or resurrect Alexey's keywords 
> draft - not saying we should do either, but looking into them might be 
> useful).
> 
> I have no clear idea if this would be considered IETF or IRTF territory 
> - I suspect it might start out informal, and become formalized in one or 
> other or both.
> 
> Who's willing to participate in something like this?

I would be (speaking for CMU).

-- 
Kenneth Murchison
Systems Programmer
Project Cyrus Developer/Maintainer
Carnegie Mellon University

Date: Thu, 12 Apr 2007 12:30:17 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: SORT on addresses
Cc: Mark Crispin <[email-masked]>, Dave Cridland <[email-masked]>, Kjetil Torgrim Homme <[email-masked]>
Sender: [email-masked]

Dave Cridland writes:
> I spoke to a few people in Prague about IMAP and email, and the 
> general area of searching and managing large quantities of email was 
> raised by several of them, so perhaps the time is right.

Indeed.

> So, here's a counter-proposal:
>
> We setup a mailing list, or at least agree to use an existing one, 
> invite a few UA developers and search-agent people [since we're heavy 
> on the server-side as usual], and start to figure out what's needed, 
> what's practical, and get to work on it. If we seem to be making 
> progress, we turn it into a WG if it seems to suit.
>
> The aim would be to get, or look into:
> - a set of useful SEARCH keys, SORT keys, etc.
> - multimailbox SEARCHing.
> - Inexact searching, such as SCAN provides.
> - Sorting by relevance.
> - probably other search and organizational things. (Perhaps we could 
> stretch to a re-examination of ANNOTATE, or resurrect Alexey's 
> keywords draft - not saying we should do either, but looking into 
> them might be useful).
>
> I have no clear idea if this would be considered IETF or IRTF 
> territory - I suspect it might start out informal, and become 
> formalized in one or other or both.
>
> Who's willing to participate in something like this?

Arnt is. No surprise I'm sure.

Arnt

Subject: Re: SORT on addresses
Date: Thu, 12 Apr 2007 10:03:46 +0100
From: Dave Cridland <[email-masked]>
To: Mark Crispin <[email-masked]>
Cc: <[email-masked]>, Kjetil Torgrim Homme <[email-masked]>
Sender: [email-masked]

On Thu Apr 12 02:38:18 2007, Mark Crispin wrote:
> 
> On Wed, 11 Apr 2007, Kjetil Torgrim Homme wrote:
>> it seems to me that this behaviour is mostly useful in "old-skool"
>> environments, where username@host was common.  e.g., I could be
>> [email-masked], [email-masked] or [email-masked]
>> depending on which workstation I was logged in that day.  I 
>> sincerely
>> doubt this is widespread today.
> 
> Please be assured that it is still quite common and still very much 
> a fact of life for many people.
> 
> 
It is for you. Not for me, and I suspect that sorting by phrase would 
provide a much more useful method than looking at the address at all.

(For you: mrc@ is almost uniformly Mark Crispin, and "SORT (FROM) 
UTF-8 ..." sorts all your mails the "right" way. For me: I'm dave@ 
for home, and dave.cridland@ for work. I'm pretty sure I'm not the 
only dave@, too. SORT (FROM) is useless for my mails as a result.)

>>      * will end-users think that changing the sort key to be the
>>        complete RFC2822 addr-spec (or RFC2822 group) is an 
>> improvement?
>> my answer is, to the extent they notice it, an emphatic yes.
> 
> What surety is there that such is the case?  Your opinion that this 
> is an "improvement" has been noted.  That does not constitute 
> research.
> 
> 
I certainly agree that *a* sort key needs to exist to allow for 
collating the results by "the user". I'm even convinced that blindly 
changing FROM/TO/CC would benefit the majority of users, but ...

>>      * is it problematic that current deployments of the draft 
>> break
>>        wrt compliance?
>> my answer is, not really.  this is not a behavioural change which
>> affects clients, it only affects the humans who use the clients.
> 
> Ahem.
> 
> A change in behavior that has been implemented and documented for 
> over a decade is quite problematic.
> 
> 
... the above point unnerves me, and moreover I cannot be certain 
that the assertion that the sort results are blindly passed through 
to the user is correct.

> Now, with that said, I would not oppose a SORTEXT proposal that 
> adds new keys (as opposed to changing the current keys) that work 
> as you propose.

I would oppose a proposal *now*. I don't think we know nearly enough 
to base a decision on concerning which keys to add.

I think that there would be serious advantage to taking a step back 
at this point, and examining how users and UAs are currently 
searching through mail, and see how we could make this both more 
efficient and closer to the end-users' requirements. Right now, I 
think that IMAP provides only rudimentary "organizational" abilities, 
and I think that they're showing up as a weak point in IMAP.

I spoke to a few people in Prague about IMAP and email, and the 
general area of searching and managing large quantities of email was 
raised by several of them, so perhaps the time is right.

So, here's a counter-proposal:

We setup a mailing list, or at least agree to use an existing one, 
invite a few UA developers and search-agent people [since we're heavy 
on the server-side as usual], and start to figure out what's needed, 
what's practical, and get to work on it. If we seem to be making 
progress, we turn it into a WG if it seems to suit.

The aim would be to get, or look into:
- a set of useful SEARCH keys, SORT keys, etc.
- multimailbox SEARCHing.
- Inexact searching, such as SCAN provides.
- Sorting by relevance.
- probably other search and organizational things. (Perhaps we could 
stretch to a re-examination of ANNOTATE, or resurrect Alexey's 
keywords draft - not saying we should do either, but looking into 
them might be useful).

I have no clear idea if this would be considered IETF or IRTF 
territory - I suspect it might start out informal, and become 
formalized in one or other or both.

Who's willing to participate in something like this?

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Subject: Re: SORT on addresses
From: Timo Sirainen <[email-masked]>
To: Mark Crispin <[email-masked]>
Cc: Kjetil Torgrim Homme <[email-masked]>, [email-masked]
Date: Thu, 12 Apr 2007 11:14:24 +0300
Sender: [email-masked]

--=-cH0f2lcWAd7jk9WgZyt8
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

On Wed, 2007-04-11 at 18:38 -0700, Mark Crispin wrote:
> Now, with that said, I would not oppose a SORTEXT proposal that adds new=20
> keys (as opposed to changing the current keys) that work as you propose.

If someone starts that, perhaps it could also include REFERENCES2?
http://mailman1.u.washington.edu/pipermail/imap-protocol/2006-December/0003=
22.html

Or maybe I'll just need to write a draft myself some day.

--=-cH0f2lcWAd7jk9WgZyt8
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

--=-cH0f2lcWAd7jk9WgZyt8--

Date: Wed, 11 Apr 2007 18:38:18 -0700 (Pacific Daylight Time)
From: Mark Crispin <[email-masked]>
To: Kjetil Torgrim Homme <[email-masked]>
cc: [email-masked]
Subject: Re: SORT on addresses
Organization: Networks & Distributed Computing
Sender: [email-masked]

On Wed, 11 Apr 2007, Kjetil Torgrim Homme wrote:
> it seems to me that this behaviour is mostly useful in "old-skool"
> environments, where username@host was common.  e.g., I could be
> [email-masked], [email-masked] or [email-masked]
> depending on which workstation I was logged in that day.  I sincerely
> doubt this is widespread today.

Please be assured that it is still quite common and still very much a fact 
of life for many people.

>      * will end-users think that changing the sort key to be the
>        complete RFC2822 addr-spec (or RFC2822 group) is an improvement?
> my answer is, to the extent they notice it, an emphatic yes.

What surety is there that such is the case?  Your opinion that this is an 
"improvement" has been noted.  That does not constitute research.

>      * is it problematic that current deployments of the draft break
>        wrt compliance?
> my answer is, not really.  this is not a behavioural change which
> affects clients, it only affects the humans who use the clients.

Ahem.

A change in behavior that has been implemented and documented for over a 
decade is quite problematic.

A declaration that software which has been distributed for more than a 
decade is suddenly "non-compliant" is quite problematic.

The only reason why SORT is "still a draft" is that it got hung up on i18n 
collation.  SORT has been a cast-in-stone specification for production 
software for many years.  It can not be arbitrarily changed.

Now, with that said, I would not oppose a SORTEXT proposal that adds new 
keys (as opposed to changing the current keys) that work as you propose.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Subject: SORT on addresses
From: Kjetil Torgrim Homme <[email-masked]>
To: [email-masked]
Date: Wed, 11 Apr 2007 07:27:54 +0200
Sender: [email-masked]

I came across some grumblings on the info-cyrus mailing list about the
behaviour of SORT with respect to FROM and TO.

[Rob Mueller]:
> It's defined by the IMAP sort extension.
> 
> http://tools.ietf.org/html/draft-ietf-imapext-sort-19
> 
>          FROM
>             [IMAP] addr-mailbox of the first "From" address.
> 
>          TO
>             [IMAP] addr-mailbox of the first "To" address.
> 
> So it really only sorts by the localpart of the email address, not
> even the domain is included, something that's annoyed our users as
> well since you basically end up with UID sorting for the localparts
> that are the same but different domains. That's the "standard"
> though... *sigh*

I was surprised to see this explanation of the behaviour, so I had to
look it up (addr-mailbox being local-part wasn't obvious to me), and
tried to find the justification for it in the archives.

way back in 2004-11-09, Dave Cridland wrote about the mismatch between
SEARCH and SORT (SEARCH looks at the headers verbatim, SORT does
decoding first), and went on to say:

[Dave Cridland]:
> I confess to being totally bewildered by the FROM, TO and CC sort
> keys, as they all appear to only sort on the local-part of the email
> address - that *is* the "bit before the @", right? So different
> domains collate equally? I can't understand why this should be the
> case.

[Mark Crispin] responded:
> In practice, not all that many people use those keys.
> 
> The idea is to tend to collect messages from a single person together,
> based upon the tendency of people to have the same local-part for his
> email address even though they may use many different sites.  Note
> that word "tend".
> 
> It's useful, in some very limited cases.

it seems to me that this behaviour is mostly useful in "old-skool"
environments, where username@host was common.  e.g., I could be
[email-masked], [email-masked] or [email-masked]
depending on which workstation I was logged in that day.  I sincerely
doubt this is widespread today.

a given localpart will typically belong to a single entity within a
given mailbox, e.g., in my personal INBOX, "christb" is unique, and
there are "only" three distinct people with localpart "chris" (in my
INBOX.lists there are 12 "chris").  role addresses increase the chance
of mixups, though: "info", "noreply", "orders", "post", "service",
"support", "webmaster" are all examples from my INBOX which belong to
numerous domains and companies, and it's not at all intuitive to me that
all addresses sharing one of these localparts are sorted as the same
address.

to give an example of when this is annoying, consider when I want to
file messages from certain people into separate folders.  most clients
allow selecting a block by specifying the start and end in the message
list, so I can operate on all of them.  when the FROM criteria is equal,
it falls back on sequence number, which means I have to pick messages
individually since the senders are interleaved.  in this scenario, even
if I wanted to move all messages from "chris" regardless of domain, it's
not a problem, since I can still select them as a single block.

the change in behaviour is only observable in edge cases, which is why I
dare bring it up at all so late in the game.  I'd like to pose two
questions:

      * will end-users think that changing the sort key to be the
        complete RFC2822 addr-spec (or RFC2822 group) is an improvement?

my answer is, to the extent they notice it, an emphatic yes.

      * is it problematic that current deployments of the draft break
        wrt compliance?

my answer is, not really.  this is not a behavioural change which
affects clients, it only affects the humans who use the clients.

-- 
thank you for reading,
Kjetil T.

Date: Mon, 09 Apr 2007 20:07:40 +0100
From: Alexey Melnikov <[email-masked]>
To: Dan Karp <[email-masked]>, Cyrus Daboo <[email-masked]>
CC: Eric Burger <[email-masked]>, Enhancements to Internet email to support  diverse service enivronments <[email-masked]>, IMAP Extensions WG <[email-masked]>
Subject: Re: [lemonade] Expert Review: annotatemore (like WGLC)
Sender: [email-masked]

Dan Karp wrote:

>>http://www.ietf.org/internet-drafts/draft-daboo-imap-annotatemore-11.txt
>>    
>>
>Comments interspersed btween snippets of the draft:
>  
>
Hi Dan,

>   Changes from -10 to -11:
>   7.   Added match type and collation identifier to the LIST-EXTENDED
>        selection option.
>   8.   Made support for IMAP-I18N a requirement.
>
>   Text comparisons may be done as part of the GETMETADATA or LIST-
>   EXTENDED commands.  As a result, support for the COMPARATOR
>   [I-D.ietf-imapext-i18n] extension is REQUIRED.
>
>GETMETADATA doesn't actually do text comparisons, right?  The BNF
>doesn't seem to tie in the "matchtype" or "collation" productions.
>  
>
Check the <list-select-metadata>, it references both.

>But more to the point, I believe that the intention was that METADATA
>should be a simple, baseline extenstion that will be required by a wide
>range of other extensions.  If that's the case, *PLEASE* reconsider the
>wisdom of adding the LT, LE, GT, and GE matchers (and thus the
>COMPARATOR requirement) to this draft.  It doesn't apear to be core
>functionality for METADATA, and since COMPARATOR appears nontrivial to
>implement
>
Can you please elaborate on this?
(Note that the COMPARATOR extension is about to switch to 
i;unicode-casemap collation (as mandatory to implement), which is much 
easier to implement than i;basic)

> this may not only prevent server authors from implementing
>METADATA but also keep us from impelenting any future extension that
>requires METADATA.
>
>
>   A user can only set and retrieve private or shared annotations on a
>   mailbox which exists and is returned to them via a LIST or LSUB
>   command, irrespective of whether they have read or write access to
>   the actual message content of the mailbox.  If the client attempts to
>   set or retrieve annotations on other mailboxes, the server MUST
>   respond with a NO response.
>
>I can understand the logic behind requiring LIST/LSUB access for reading
>public mailbox annotations and for setting private annotations.  But
>that seems to be a very low bar for setting public mailbox annotations.
>Adding a new ACL permission for setting annotations would make this
>extension more complex and would also introduce a dependency on the ACL
>extension,
>
Dependency in what sense?

> but perhaps requiring [READ-WRITE] access to the mailbox
>might be reasonable?
>
>
>   The new SETMETADATA command and the METADATA response each have a
>   mailbox name argument, indicating that the annotations being referred
>   to are attached to the specified mailbox.  An empty string can be
>   used for the mailbox name to signify server annotations.
>
>What rights are required to set public and private annotations on the
>server?
>  
>
None. But maybe this should be stated explicitly.

>   This command sets the specified list of entries by adding or
>   replacing the specified attributes with the values provided, on the
>   specified existing mailboxes.  Clients can use NIL for values of
>   attributes it wants to remove from entries.  The server MAY return a
>   METADATA response containing the updated annotation data.
>
>In this case, the server may return a METADATA response that doesn't
>contain the updated annotation data or a LIST response that does,
>correct?
>
>
>   This extension adds the GETMETADATA command.  This allows clients to
>   retrieve server annotations.  Mailbox annotations are retrieved via
>   the extended LIST command described in the next section.
>
>   This extension adds the SETMETADATA command.  This allows clients to
>   set annotations.
>
>It's a little incongruous that GETMETADATA is used only to retrieve
>server annotations, but SETMETADATA can be used to set both server and
>mailbox annotations.  I understand that the mailbox-level function of
>GETMETADATA has been passed to the LIST-EXTENDED feature, but generally
>it's clearer when a GETFOO and SETFOO pair cover the same target space.
>  
>
Yes, but the alternative is to have 2 ways of getting annotations, which 
is worse.

>   If the METADATA extension is present, support for shared annotations
>   is REQUIRED, whilst support for private annotations is OPTIONAL.
>   This recognizes the fact that support for private annotations may
>   introduce significantly more complexity to a server in terms of
>   tracking ownership of the annotations, how quota is determined for
>   users based on their own annotations etc.  Clients that support the
>   METADATA extension MUST handle both shared and private annotations.
>
>Is there a mechanism for informing clients that a server does not
>support private annotations?  Or is a client expected to determine this
>from METADATA NOPRIVATE response codes on a SETMETADATA NO response?
>  
>
Currently the latter.

>If the latter is the case, won't client implementations end up allowing
>users one attempt to set private annotations and only then inform them
>that the functionality is not supported by the server?
>
Clients can try to workaround this by trying to set an annotation to a 
fake value.
Yes, this doesn't look very nice, but works.

Defining a new capability wouldn't work, because in general case 
NOPRIVATE can be namespace or even mailbox-specific.

We can add a new \NoPrivate per mailbox attribute to return this in LIST?

>  (This would probably occur once per IMAP session...)
>
>
>   If the server is unable to set an annotation because the size of its
>   value is too large, the server MUST return a tagged NO response with
>   a "[METADATA TOOBIG]" response code.
>
>Likewise, is there a mechanism for a server to tell its clients the
>maximum annotation length it supports?  Or is it up to the client to
>determine this from METADATA TOOBIG error code responses?
>  
>
The latter.
Also note, that the server might return TOOBIG because it simply ran out 
of storage space, so advertising the maximum might not be that helpful 
to the client.

>   Entry and attribute names are case-sensitive.
>
>   Entry and attribute names MUST NOT contain asterisk ("*") or percent
>   ("%") characters and MUST NOT contain non-ASCII characters or the NUL
>   octet.  Invalid entry or attribute names result in a BAD response in
>   any IMAP commands where they are used.
>
>   Use of control or punctuation characters in entry and attribute names
>   is strongly discouraged.
>
>It's probably cleaner in the long run to use the BNF to explicitly
>specify a limited set of permitted characters for entry and attribute
>names.  Constraining them to something like "atom" plus the delimiters
>is probably not a bad idea.
>
>Given that these names are already constrained to ASCII,
>
I don't think this is the case! Annotation names can contain UTF-8.

> I'd favor eliminating the case-sensitivity requirement.  It differs from other
>IMAP commands and extensions' general consistent overall case-
>insensitivity (e.g. RFC 3501 keywords).
>
>
>   value
>      A string or binary data representing the value of the annotation.
>      To delete an annotation, the client can store "NIL" into the
>      value.  The content represented by the string is determined by the
>      Content-Type used to register the entry (see Section 6.1 for entry
>      registration templates).  Where applicable, the registered
>      Content-Type MUST include a charset parameter.  Text values SHOULD
>      use the utf-8 [RFC3629] character set.
>
>Is there any reason not to mandate UTF-8 for text values?
>
Values can also be binary and not every binary value is a valid UTF-8.

>  We seem to be
>moving towards standardizing on UTF-8 in IMAP, and this seems a good
>opportunity to draw a line.
>  
>
"Values for annotations published on Standard track MUST use charset=UTF-8"?
This still allows vendors to do whatever they want with their private 
annotations.

>   /sort
>      Defines the default sort criteria [I-D.ietf-imapext-sort] to use
>      when first displaying the mailbox contents to the user, or NIL if
>      sorting is not required.  Note that the presence of this attribute
>      does not imply that the server supports the IMAP SORT
>      [I-D.ietf-imapext-sort] extension - servers can choose to support
>      or not support SORT independently of this extension.  In the
>      absence of the SORT extension, clients can choose to interpret
>      this entry as suggesting a client-side sort ordering of the
>      corresponding mailbox.
>
>   /thread
>      Defines the default thread criteria [I-D.ietf-imapext-sort] to use
>      when first displaying the mailbox contents to the user, or NIL if
>      threading is not required.  If both sort and thread are not NIL,
>      then threading should take precedence over sorting.  Note that the
>      presence of this attribute does not imply that the server supports
>      the IMAP THREAD [I-D.ietf-imapext-sort] extension - servers can
>      choose to support or not support THREAD independently of this
>      extension.  In the absence of the THREAD extension, clients can
>      choose to interpret this entry as a client-side thread ordering of
>      the corresponding mailbox.
>
>These registered annotations are odd ducks.  They seem most appropriate
>as private annotations on a mailbox, but in practice they may be
>confusing to the end user because sorting and threading preferences are
>often client-specific, not user-specific.
>
>For instance, a user may prefer to read their mail nonthreaded on the
>desktop, but on a mobile device they may want to make use of the reduced
>visual footprint of a threaded display.  Likewise, more likely than not
>a user won't expect their sort preferences to automatically be
>transferred between their devices and clients.
>  
>
I would let other people comment on this.

>   Mailboxes or the server as a whole may have zero or more annotations
>   associated with them.  An annotation contains a uniquely named entry
>   with a set of uniquely named attributes, each of which has a value.
>   Annotations can be added to mailboxes when a mailbox name is provided
>   as the first argument to the new command, or to the server as a whole
>   when the empty string is provided as the first argument to the new
>   command.
>
>Just a nitpick here, but you should probably refer to SETMETADATA
>command by name since (a) it hasn't been introduced at this point in the
>document and (b) there is more than one new command added by the draft.
>  
>
Agreed.

>   Each annotation is an entry that has a hierarchical name, with each
>   component of the name separated by a slash ("/").  An entry name MUST
>   NOT contain two consecutive "/" characters and MUST NOT end with a
>   "/" character.
>
>   Each entry is made up of a set of attributes.  Each attribute has a
>   hierarchical name, with each component of the name separated by a
>   period (".").  An attribute name MUST NOT contain two consecutive "."
>   characters and MUST NOT end with a "." character.
>
>My last comment is personal preference: I fear that this may be a bit of
>overkill for folder annotation.  I understand the need for this type of
>flexible syntax for ACAP's purposes, but the use cases for IMAP appear
>significantly more limited in scope.  In accordance with the KISS
>philosophy, is there any chance I might be able to convince everyone to
>consider shedding a layer of hierarchy here?
>  
>
This is not going to be a problem in practice. IMHO, new attributes are 
unlikely to be standardized.
IMHO, the current model of value.priv + value.shared per annotation 
should work for 99% cases.

Date: Mon, 9 Apr 2007 08:50:10 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: [lemonade] Deprecate IMAP IDLE?
Cc: Eric Burger <[email-masked]>
Sender: [email-masked]

Eric Burger writes:
> Because of this, it was proposed that IMAP NOTIFY would supercede IMAP 
> IDLE, leading to its deprecation.
>
> What do we think of this? As chair, my only care is whether we say it 
> obsoletes RFC 2177 or not.

I think obsoleting 2177 is out of scope for Lemonade. All Lemonade 
can/should do is say things in profile-bis.

Arnt

Date: Fri, 06 Apr 2007 14:38:35 -0400
From: Barry Leiba <[email-masked]>
To: [email-masked]
Subject: Re: friends of imap dinner tonight
Sender: [email-masked]

A little late here, but...

> It was a pleasant dinner.

I'm sure it was, and I'm sorry to've missed it.  Perhaps in future, we 
should make sure that the friends of IMAP and the friends of DKIM 
coordinate, so those of us who have lots of friends can do both.

:-)

Barry

Date: Mon, 2 Apr 2007 21:31:14 -0700 (PDT)
From: Dan Karp <[email-masked]>
To: Eric Burger <[email-masked]>
Cc: Enhancements to Internet email to support diverse service enivronments <[email-masked]>, IMAP Extensions WG <[email-masked]>
Subject: Re: [lemonade] Expert Review: annotatemore (like WGLC)
Sender: [email-masked]

> http://www.ietf.org/internet-drafts/draft-daboo-imap-annotatemore-11.txt

Comments interspersed btween snippets of the draft:

   Changes from -10 to -11:
   7.   Added match type and collation identifier to the LIST-EXTENDED
        selection option.
   8.   Made support for IMAP-I18N a requirement.

   Text comparisons may be done as part of the GETMETADATA or LIST-
   EXTENDED commands.  As a result, support for the COMPARATOR
   [I-D.ietf-imapext-i18n] extension is REQUIRED.

GETMETADATA doesn't actually do text comparisons, right?  The BNF
doesn't seem to tie in the "matchtype" or "collation" productions.

But more to the point, I believe that the intention was that METADATA
should be a simple, baseline extenstion that will be required by a wide
range of other extensions.  If that's the case, *PLEASE* reconsider the
wisdom of adding the LT, LE, GT, and GE matchers (and thus the
COMPARATOR requirement) to this draft.  It doesn't apear to be core
functionality for METADATA, and since COMPARATOR appears nontrivial to
implement this may not only prevent server authors from implementing
METADATA but also keep us from impelenting any future extension that
requires METADATA.

   A user can only set and retrieve private or shared annotations on a
   mailbox which exists and is returned to them via a LIST or LSUB
   command, irrespective of whether they have read or write access to
   the actual message content of the mailbox.  If the client attempts to
   set or retrieve annotations on other mailboxes, the server MUST
   respond with a NO response.

I can understand the logic behind requiring LIST/LSUB access for reading
public mailbox annotations and for setting private annotations.  But
that seems to be a very low bar for setting public mailbox annotations.
Adding a new ACL permission for setting annotations would make this
extension more complex and would also introduce a dependency on the ACL
extension, but perhaps requiring [READ-WRITE] access to the mailbox
might be reasonable?

   The new SETMETADATA command and the METADATA response each have a
   mailbox name argument, indicating that the annotations being referred
   to are attached to the specified mailbox.  An empty string can be
   used for the mailbox name to signify server annotations.

What rights are required to set public and private annotations on the
server?

   This command sets the specified list of entries by adding or
   replacing the specified attributes with the values provided, on the
   specified existing mailboxes.  Clients can use NIL for values of
   attributes it wants to remove from entries.  The server MAY return a
   METADATA response containing the updated annotation data.

In this case, the server may return a METADATA response that doesn't
contain the updated annotation data or a LIST response that does,
correct?

   This extension adds the GETMETADATA command.  This allows clients to
   retrieve server annotations.  Mailbox annotations are retrieved via
   the extended LIST command described in the next section.

   This extension adds the SETMETADATA command.  This allows clients to
   set annotations.

It's a little incongruous that GETMETADATA is used only to retrieve
server annotations, but SETMETADATA can be used to set both server and
mailbox annotations.  I understand that the mailbox-level function of
GETMETADATA has been passed to the LIST-EXTENDED feature, but generally
it's clearer when a GETFOO and SETFOO pair cover the same target space.

   If the METADATA extension is present, support for shared annotations
   is REQUIRED, whilst support for private annotations is OPTIONAL.
   This recognizes the fact that support for private annotations may
   introduce significantly more complexity to a server in terms of
   tracking ownership of the annotations, how quota is determined for
   users based on their own annotations etc.  Clients that support the
   METADATA extension MUST handle both shared and private annotations.

Is there a mechanism for informing clients that a server does not
support private annotations?  Or is a client expected to determine this
from METADATA NOPRIVATE response codes on a SETMETADATA NO response?

If the latter is the case, won't client implementations end up allowing
users one attempt to set private annotations and only then inform them
that the functionality is not supported by the server?  (This would
probably occur once per IMAP session...)

   If the server is unable to set an annotation because the size of its
   value is too large, the server MUST return a tagged NO response with
   a "[METADATA TOOBIG]" response code.

Likewise, is there a mechanism for a server to tell its clients the
maximum annotation length it supports?  Or is it up to the client to
determine this from METADATA TOOBIG error code responses?

   Entry and attribute names are case-sensitive.

   Entry and attribute names MUST NOT contain asterisk ("*") or percent
   ("%") characters and MUST NOT contain non-ASCII characters or the NUL
   octet.  Invalid entry or attribute names result in a BAD response in
   any IMAP commands where they are used.

   Use of control or punctuation characters in entry and attribute names
   is strongly discouraged.

It's probably cleaner in the long run to use the BNF to explicitly
specify a limited set of permitted characters for entry and attribute
names.  Constraining them to something like "atom" plus the delimiters
is probably not a bad idea.

Given that these names are already constrained to ASCII, I'd favor
eliminating the case-sensitivity requirement.  It differs from other
IMAP commands and extensions' general consistent overall case-
insensitivity (e.g. RFC 3501 keywords).

   value
      A string or binary data representing the value of the annotation.
      To delete an annotation, the client can store "NIL" into the
      value.  The content represented by the string is determined by the
      Content-Type used to register the entry (see Section 6.1 for entry
      registration templates).  Where applicable, the registered
      Content-Type MUST include a charset parameter.  Text values SHOULD
      use the utf-8 [RFC3629] character set.

Is there any reason not to mandate UTF-8 for text values?  We seem to be
moving towards standardizing on UTF-8 in IMAP, and this seems a good
opportunity to draw a line.

   /sort
      Defines the default sort criteria [I-D.ietf-imapext-sort] to use
      when first displaying the mailbox contents to the user, or NIL if
      sorting is not required.  Note that the presence of this attribute
      does not imply that the server supports the IMAP SORT
      [I-D.ietf-imapext-sort] extension - servers can choose to support
      or not support SORT independently of this extension.  In the
      absence of the SORT extension, clients can choose to interpret
      this entry as suggesting a client-side sort ordering of the
      corresponding mailbox.

   /thread
      Defines the default thread criteria [I-D.ietf-imapext-sort] to use
      when first displaying the mailbox contents to the user, or NIL if
      threading is not required.  If both sort and thread are not NIL,
      then threading should take precedence over sorting.  Note that the
      presence of this attribute does not imply that the server supports
      the IMAP THREAD [I-D.ietf-imapext-sort] extension - servers can
      choose to support or not support THREAD independently of this
      extension.  In the absence of the THREAD extension, clients can
      choose to interpret this entry as a client-side thread ordering of
      the corresponding mailbox.

These registered annotations are odd ducks.  They seem most appropriate
as private annotations on a mailbox, but in practice they may be
confusing to the end user because sorting and threading preferences are
often client-specific, not user-specific.

For instance, a user may prefer to read their mail nonthreaded on the
desktop, but on a mobile device they may want to make use of the reduced
visual footprint of a threaded display.  Likewise, more likely than not
a user won't expect their sort preferences to automatically be
transferred between their devices and clients.

   Mailboxes or the server as a whole may have zero or more annotations
   associated with them.  An annotation contains a uniquely named entry
   with a set of uniquely named attributes, each of which has a value.
   Annotations can be added to mailboxes when a mailbox name is provided
   as the first argument to the new command, or to the server as a whole
   when the empty string is provided as the first argument to the new
   command.

Just a nitpick here, but you should probably refer to SETMETADATA
command by name since (a) it hasn't been introduced at this point in the
document and (b) there is more than one new command added by the draft.

   Each annotation is an entry that has a hierarchical name, with each
   component of the name separated by a slash ("/").  An entry name MUST
   NOT contain two consecutive "/" characters and MUST NOT end with a
   "/" character.

   Each entry is made up of a set of attributes.  Each attribute has a
   hierarchical name, with each component of the name separated by a
   period (".").  An attribute name MUST NOT contain two consecutive "."
   characters and MUST NOT end with a "." character.

My last comment is personal preference: I fear that this may be a bit of
overkill for folder annotation.  I understand the need for this type of
flexible syntax for ACAP's purposes, but the use cases for IMAP appear
significantly more limited in scope.  In accordance with the KISS
philosophy, is there any chance I might be able to convince everyone to
consider shedding a layer of hierarchy here?

Date: Thu, 26 Apr 2007 23:59:37 -0500
To: IETF-POP3EXT:;, IETF-IMAPEXT:;, IETF-SMTP:;, LEMONADE:;, IMA:;
From: Pete Resnick <[email-masked]>
Subject: Fwd: draft-resnick-2822upd-01 posted
Sender: [email-masked]

Please follow the discussion on [email-masked].

--- begin forwarded text

Date: Thu, 26 Apr 2007 16:55:06 -0500
To: [email-masked]
From: Pete Resnick <[email-masked]>
Subject: draft-resnick-2822upd-01 posted

You may have noticed that 2822upd-01 was posted.

<http://www.ietf.org/internet-drafts/draft-resnick-2822upd-01.txt>
Also:
<http://www.qualcomm.com/~presnick/draft-resnick-2822upd-01.html>
<http://www.qualcomm.com/~presnick/draft-resnick-2822upd-01.xml>
<http://www.qualcomm.com/~presnick/draft-resnick-2822upd-01.txt>

To make rfcdiff work, I've put a copy of RFC 2822 on my web site with 
some of the later sections rearranged so that they're in the same 
order as the current draft. You can see the diffs between the two 
here:

<http://tools.ietf.org/rfcdiff?url1=http://www.qualcomm.com/~presnick/rfc2822-fixed.txt&url2=http://www.ietf.org/internet-drafts/draft-resnick-2822upd-01.txt>

Though overall very few, there are real changes from 2822 in this 
document that need review. See the changes section at the bottom. I 
did *not* go whole hog and switch to Bruce's syntax, simply because I 
wanted to limit the number of changes and see if we can't get this 
thing to Draft along with 2821bis. But I did incorporate (hopefully 
all of) the corrections that you all have pointed out over the last 
couple of years.

Please review with a fine toothed comb. If there are more than just a 
few problems, perhaps I'll ask Tony Hansen or Philip Guenther (who 
have at assorted time offered assistance) to start an issues list.

Thanks in advance for your assistance.

pr

--- end forwarded text

-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

Date: Wed, 18 Apr 2007 22:49:37 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: Dan Karp <[email-masked]>
cc: IMAP Extensions WG <[email-masked]>, Enhancements to Internet email to support diverse service enivronments <[email-masked]>, IMAP Protocol Interest List <[email-masked]>
Subject: Re: [lemonade] SURVEY: How does SEARCH match in your IMAP server?
Sender: [email-masked]

  This message is in MIME format.  The first part should be readable text,
  while the remaining parts are likely unreadable without MIME-aware tools.

--0-559819627-1176960403=:11458
Content-Type: TEXT/PLAIN; CHARSET=UTF-8; format=flowed
Content-Transfer-Encoding: QUOTED-PRINTABLE
Content-ID: <[email-masked]>

Your test examples aren't any good, because your message is in UTF-8 but=20
you have texts that are claimed to be ISO 8859-1.  I will try to answer=20
what I *think* you intended.

If my presumptions are wrong, then I reserve the right to change my=20
answers.

On Wed, 18 Apr 2007, Dan Karp wrote:
> server's name and version

UW imapd, version imap-2006 series unless noted otherwise.

> I love M=C3=B6tley Cr=C3=BCe !

Presumably, you mean "I love M" 0xf6 "tley Cr" 0xfc "e"

> A001 SEARCH BODY "love"

matches

> A002 SEARCH BODY "LOVE"

matches

> A003 SEARCH CHARSET "iso-8859-1" BODY {4}
> Cr=C3=BCe

Presumably, you mean "Cr" 0xfc "e"

matches in all versions since imap-4.1

error in imap-4 and earlier (10+ year old US-ASCII only version)

> A004 SEARCH CHARSET "utf-8" BODY {5}
> Cr=C3=83=C2=BCe

Presuably, you mean "Cr" 0xc3 0xbc "e"

matches in all versions since imap-4.1

error in imap-4 and earlier (10+ year old US-ASCII only version)

> A005 SEARCH CHARSET "iso-8859-1" BODY {4}
> CR=C3=9CE

Presumably, you mean "CR" 0xdc "E"

matches in imap-2006 series.

does not match in earlier versions (earlier versions did i;ascii-casemap).

error in imap-4 and earlier (10+ year old US-ASCII only version)

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.
--0-559819627-1176960403=:11458--

Date: Wed, 18 Apr 2007 21:23:29 -0700 (PDT)
From: Dan Karp <[email-masked]>
To: IMAP Extensions WG <[email-masked]>, Enhancements to Internet email to support diverse service enivronments <[email-masked]>, IMAP Protocol Interest List <[email-masked]>
Cc: Arnt Gulbrandsen <[email-masked]>
Subject: SURVEY: How does SEARCH match in your IMAP server?
Sender: [email-masked]

We're collecting hard data to help refine an upcoming IMAP extension
(http://www.ietf.org/internet-drafts/draft-ietf-imapext-i18n-10.txt).
If you're developing an IMAP server, it would be a *great* help if you
could take 3 minutes to answer a few questions about your server
implementation.  If you email responses back to me along with your
server's name and version, I'll compile the results.

Assume that you've just injected the following message into an empty
INBOX:

----------------------------------------------------------------------
Subject: Music
Date: Tue, 17 Apr 2007 16:53:02 -0700 (PDT)
From: [email-masked]
To: [email-masked]
Content-Type: text/plain; charset="iso-8859-1"

I love Mötley Crüe !
----------------------------------------------------------------------

For each of the following SEARCH requests, would your server match the
message, not match the message, return NO [BADCHARSET], or return some
other NO or BAD response?

A001 SEARCH BODY "love"

A002 SEARCH BODY "LOVE"

A003 SEARCH CHARSET "iso-8859-1" BODY {4}
Crüe

A004 SEARCH CHARSET "utf-8" BODY {5}
CrÃ¼e

A005 SEARCH CHARSET "iso-8859-1" BODY {4}
CRÜE

Date: Wed, 18 Apr 2007 21:19:04 -0700 (PDT)
From: Dan Karp <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: Eric Burger <[email-masked]>, Alexey Melnikov <[email-masked]>, Timo Sirainen <[email-masked]>, Cyrus Daboo <[email-masked]>, [email-masked], [email-masked]
Subject: Re: ANNOTATEMORE and COMPARATOR
Sender: [email-masked]

> > Is this true?  Do these servers really implement i;ascii-casemap as
> > defined in RFC 4790?  I'd expect any RFC 3501-compliant server that
> > knows about more charsets than just US-ASCII to be supporting 
> > i;unicode-casemap and *not* i;ascii-casemap because of 
> > case-insensitive "SEARCH CHARSET".
> 
> Build some consensus please.

I chatted with Arnt, and he thinks it'd be worthwhile to canvas the
troops and get some hard data on what various servers actually implement
for their default SEARCH comparator.  I've thrown together a little
5-question survey which I'll send out under separate cover.

Subject: Re: ANNOTATEMORE and COMPARATOR
From: Timo Sirainen <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: [email-masked], [email-masked], Eric Burger <[email-masked]>, Alexey Melnikov <[email-masked]>, Dan Karp <[email-masked]>, Cyrus Daboo <[email-masked]>
Date: Tue, 17 Apr 2007 21:54:36 +0300
Sender: [email-masked]

--=-baA9QaoKY2tQJvsfXhIn
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

On Tue, 2007-04-17 at 20:25 +0200, Arnt Gulbrandsen wrote:
> >>  > If your aim is to make baseline COMPARATOR support fairly
> >>  > lightweight for IMAP servers, you may be constrained to only
> >>  > requiring i;unicode-casemap.
> >>
> >>  Mumble. Given how many IMAP servers actually implement=20
> >>  i;ascii-casemap and nothing else, I'm not happy doing that. Make=20
> >>  some other people agree with you so Pete will say "WG consensus".
> >
> > Is this true?  Do these servers really implement i;ascii-casemap as=20
> > defined in RFC 4790?  I'd expect any RFC 3501-compliant server that=20
> > knows about more charsets than just US-ASCII to be supporting=20
> > i;unicode-casemap and *not* i;ascii-casemap because of=20
> > case-insensitive "SEARCH CHARSET".
>=20
> Build some consensus please.

My server implements only i;ascii-casemap currently, but that's just
because I've been busy getting it stable and no-one's complained. I'm
going to move to i;unicode-casemap soon, and I don't see a need for
i;ascii-casemap. I'd think anyone who is going to implement COMPARATOR
won't see i;unicode-casemap requirement as a problem.

--=-baA9QaoKY2tQJvsfXhIn
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

--=-baA9QaoKY2tQJvsfXhIn--

Date: Tue, 17 Apr 2007 20:51:24 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: [lemonade] Re: ANNOTATEMORE and COMPARATOR
Cc: Philip Guenther <[email-masked]>
Sender: [email-masked]

That shouldn't apply to someone like Dan, since he stores the data as UTF-16.

He may have a nasty problem converting some of his Subject fields to 
unicode, but it's not a collation problem in his case.

Arnt

Date: Tue, 17 Apr 2007 20:25:36 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: ANNOTATEMORE and COMPARATOR
Cc: Eric Burger <[email-masked]>, Alexey Melnikov <[email-masked]>, Dan Karp <[email-masked]>, Timo Sirainen <[email-masked]>, Cyrus Daboo <[email-masked]>
Sender: [email-masked]

Dan Karp writes:
>>  I wanted to permit implementations of i;ascii-casemap to do whatever 
>>  comes easiest outside US-ASCII. However, the final RFC doesn't 
>>  reflect my viewpoint in this respect. I don't remember why and 
>>  don't feel like searching my archive. I certainly wasn't aware of 
>>  the argument you bring up now.
>>
>>  Do you think this is a signficant issue? It doesn't affect searching 
>>  to a noticeable degree, and its impact on sorting is O(size of 
>>  result set), not O(size of mailbox).
>
> Actually, yeah, it does impact search. If you search for 
> "M<o-umlaut>tley Cr<u-umlaut>e", i;ascii-casemap doesn't allow that 
> to match "M<O-umlaut>TLEY CR<U-umlaut>E". If you've only stored an 
> upcased version of the text, you can't tell the case on the original 
> and thus you can't determine if the non-ASCII accented characters 
> match under i;octet.

By the time you get that far, you've eliminated almost all messages not 
in the actual result set, so opening and rechecking isn't so awfully 
expensive.

>>  > If your aim is to make baseline COMPARATOR support fairly
>>  > lightweight for IMAP servers, you may be constrained to only
>>  > requiring i;unicode-casemap.
>>
>>  Mumble. Given how many IMAP servers actually implement 
>>  i;ascii-casemap and nothing else, I'm not happy doing that. Make 
>>  some other people agree with you so Pete will say "WG consensus".
>
> Is this true?  Do these servers really implement i;ascii-casemap as 
> defined in RFC 4790?  I'd expect any RFC 3501-compliant server that 
> knows about more charsets than just US-ASCII to be supporting 
> i;unicode-casemap and *not* i;ascii-casemap because of 
> case-insensitive "SEARCH CHARSET".

Build some consensus please. I don't want to argue today - I'm in a mood 
to help little old ladies halfway across the street. Being civil is a 
strain.

Arnt

Date: Tue, 17 Apr 2007 10:06:04 -0700 (PDT)
From: Dan Karp <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: Eric Burger <[email-masked]>, Alexey Melnikov <[email-masked]>, Timo Sirainen <[email-masked]>, Cyrus Daboo <[email-masked]>, [email-masked], [email-masked]
Subject: Re: ANNOTATEMORE and COMPARATOR
Sender: [email-masked]

> I wanted to permit implementations of i;ascii-casemap to do whatever 
> comes easiest outside US-ASCII. However, the final RFC doesn't reflect
> my viewpoint in this respect. I don't remember why and don't feel like
> searching my archive. I certainly wasn't aware of the argument you 
> bring up now.
> 
> Do you think this is a signficant issue? It doesn't affect searching
> to a noticeable degree, and its impact on sorting is O(size of result 
> set), not O(size of mailbox).

Actually, yeah, it does impact search.  If you search for
"M<o-umlaut>tley Cr<u-umlaut>e", i;ascii-casemap doesn't allow that to
match "M<O-umlaut>TLEY CR<U-umlaut>E".  If you've only stored an
upcased version of the text, you can't tell the case on the original
and thus you can't determine if the non-ASCII accented characters
match under i;octet.

> > If your aim is to make baseline COMPARATOR support fairly
> > lightweight for IMAP servers, you may be constrained to only
> > requiring i;unicode-casemap.
> 
> Mumble. Given how many IMAP servers actually implement i;ascii-casemap
> and nothing else, I'm not happy doing that. Make some other people 
> agree with you so Pete will say "WG consensus".

Is this true?  Do these servers really implement i;ascii-casemap as
defined in RFC 4790?  I'd expect any RFC 3501-compliant server that
knows about more charsets than just US-ASCII to be supporting
i;unicode-casemap and *not* i;ascii-casemap because of case-insensitive
"SEARCH CHARSET".

Date: Tue, 17 Apr 2007 18:45:19 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: ANNOTATEMORE and COMPARATOR
Cc: Eric Burger <[email-masked]>, Alexey Melnikov <[email-masked]>, Dan Karp <[email-masked]>, Timo Sirainen <[email-masked]>, Cyrus Daboo <[email-masked]>
Sender: [email-masked]

Dan Karp writes:
> i;ascii-casemap is problematic as well, for roughly the same reasons:
>
>    Its equality, ordering, and substring operations are as for 
>    i;octet, except that at first, the lower-case letters (octet 
>    values 97-122) in each input string are changed to upper case 
>    (octet values 65-90).
>
> Servers wishing to optimize i;unicode-casemap may well store only a 
> searchable upper case version of the text, which will sort 
> incorrectly under i;ascii-casemap.

Good point.

I wanted to permit implementations of i;ascii-casemap to do whatever 
comes easiest outside US-ASCII. However, the final RFC doesn't reflect 
my viewpoint in this respect. I don't remember why and don't feel like 
searching my archive. I certainly wasn't aware of the argument you 
bring up now.

Do you think this is a signficant issue? It doesn't affect searching to 
a noticeable degree, and its impact on sorting is O(size of result 
set), not O(size of mailbox).

> If your aim is to make baseline COMPARATOR support fairly lightweight 
> for IMAP servers, you may be constrained to only requiring 
> i;unicode-casemap.

Mumble. Given how many IMAP servers actually implement i;ascii-casemap 
and nothing else, I'm not happy doing that. Make some other people 
agree with you so Pete will say "WG consensus".

Arnt

Date: Tue, 17 Apr 2007 08:52:18 -0700 (PDT)
From: Dan Karp <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: Eric Burger <[email-masked]>, Alexey Melnikov <[email-masked]>, Timo Sirainen <[email-masked]>, Cyrus Daboo <[email-masked]>, [email-masked], [email-masked]
Subject: Re: ANNOTATEMORE and COMPARATOR
Sender: [email-masked]

> > IMAP (via SEARCH) essentially requires that message content be 
> > searchable in some intermediate transcoded form (we chose UTF-16). 
> > COMPARATOR (via "i;octet") requires that message content be 
> > searchable in its original form pace content-encoding.
> 
> Only i;octet does that... we did a lot of work for RFC4790 to allow
> your kind of implementation, so I think it should be reasonable that 
> COMPARATOR allows it too.
> 
> As far as I can see, the mention of i;octet in 
> draft-crispin-collation-unicasemap is the only substantial requirement
> on i;octet, and it's weak. The one in draft-ietf-imapext-i18n is a 
> leftover - something I forgot to delete when its rationale
> disappeared.

i;ascii-casemap is problematic as well, for roughly the same reasons:

   Its equality, ordering, and substring operations are as for i;octet,
   except that at first, the lower-case letters (octet values 97-122) in
   each input string are changed to upper case (octet values 65-90).

Servers wishing to optimize i;unicode-casemap may well store only a
searchable upper case version of the text, which will sort incorrectly
under i;ascii-casemap.  COMPARATOR requires servers to implement both
i;ascii-casemap and i;octet (though i;octet sounds like it may be on the
way out).  If your aim is to make baseline COMPARATOR support fairly
lightweight for IMAP servers, you may be constrained to only requiring
i;unicode-casemap.

Subject: Re: [lemonade] Re: ANNOTATEMORE and COMPARATOR
Date: Tue, 17 Apr 2007 16:20:06 +0100
From: Dave Cridland <[email-masked]>
To: Mark Crispin <[email-masked]>
Cc: Enhancements to Internet email to support diverse service enivronments <[email-masked]>, <[email-masked]>
Sender: [email-masked]

On Tue Apr 17 14:30:14 2007, Mark Crispin wrote:
> 
> On Tue, 17 Apr 2007, Dave Cridland wrote:
>> I don't think it's quite right to state that "everything in IMAP 
>> is case-insensitive", since it seems intuitive to me that subject 
>> lines and other human-orientated text is demonstratively 
>> case-sensitive; it merely may be more useful to search it 
>> case-insensitively for much of the time.
> 
> Take a time machine back 21 years and tell me that I am wrong for 
> specifying that everything in IMAP is case-insensitive.  I would 
> have told you back then that you are wrong, and I see no good cause 
> to reverse that choice today.

NO, ON FURTHER CONSIDERATION YOU'RE ABSOLUTELY RIGHT. CASE IS NEVER 
INDICATIVE OF ANYTHING, THEREFORE IT MAKES NO SENSE TO SEARCH FOR IT 
IN ANY INSTANCE. ALSO, ALL EMAIL SHOULD BE PRESENTED ON PROPER 80 
COLUMN DISPLAYS, WITH NICE GREEN TEXT ON A PALE GREY SMEARED 
BACKGROUND YOU'RE ENCOURAGED TO CALL BLACK.

Seriously, though, while I'd readily agree that case should be 
irrelevant in IMAP protocol elements - we've already had that 
argument, in fact - I don't see that the same applies to 
human-created, human-directed text, such as subject fields, where the 
paragraph above does show that case conveys information.

It seems reasonable that we wouldn't want to mandate that clients 
should be prevented from searching case-sensitively. Similarly, many 
IMAP servers, by stripping away spaces and punctuation before the 
search test, can harm the usability of searches.

Not always. Just sometimes. I don't think that's a contentious 
statement, and I suspect I've worded this badly if you're disagreeing 
with me this strongly.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Date: Tue, 17 Apr 2007 06:30:14 -0700
From: Mark Crispin <[email-masked]>
To: Dave Cridland <[email-masked]>
cc: Enhancements to Internet email to support diverse service enivronments <[email-masked]>, IMAP Extensions WG <[email-masked]>
Subject: Re: [lemonade] Re: ANNOTATEMORE and COMPARATOR
Organization: Networks & Distributed Computing
Sender: [email-masked]

On Tue, 17 Apr 2007, Dave Cridland wrote:
> I don't think it's quite right to state that "everything in IMAP is 
> case-insensitive", since it seems intuitive to me that subject lines and 
> other human-orientated text is demonstratively case-sensitive; it merely may 
> be more useful to search it case-insensitively for much of the time.

Take a time machine back 21 years and tell me that I am wrong for 
specifying that everything in IMAP is case-insensitive.  I would have told 
you back then that you are wrong, and I see no good cause to reverse that 
choice today.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Subject: Re: [lemonade] Re: ANNOTATEMORE and COMPARATOR
Date: Tue, 17 Apr 2007 10:20:20 +0100
From: Dave Cridland <[email-masked]>
To: Mark Crispin <[email-masked]>
Cc: Arnt Gulbrandsen <[email-masked]>, Timo Sirainen <[email-masked]>, Enhancements to Internet email to support diverse service enivronments <[email-masked]>, <[email-masked]>
Sender: [email-masked]

On Tue Apr 17 00:04:24 2007, Mark Crispin wrote:
> 
> On Mon, 16 Apr 2007, Dave Cridland wrote:
>> ACAP deals in arbitrary binary data which *might* be treatable as 
>> text - the server has no way of knowing. "i;octet" is remarkably 
>> useful at doing exact matches, whether text or binary, and was the 
>> only comparator capable of doing case-sensitive matching on text.
>> In IMAP, this is the only use, and there are undoubtedly better 
>> ways of doing this.
> 
> I don't understand.  Everything in IMAP is case-insensitive by 
> definition. There is, AFAIK, no need or use for i;octet anywhere in 
> IMAP.

As I said, I was searching for OMA in subject lines rather a lot 
yesterday - if I were able to have specified a case-sensitive match, 
I would have done, since the sequence "oma" appears rather a lot in 
spurious results. (Such as "Domain", "Automaton", and rather 
interestingly, "to make" and "from a" on several implementations).

I don't think it's quite right to state that "everything in IMAP is 
case-insensitive", since it seems intuitive to me that subject lines 
and other human-orientated text is demonstratively case-sensitive; it 
merely may be more useful to search it case-insensitively for much of 
the time.

(This is not a demand for an "i;unicode-not-casemap" or anything, 
merely an observation.)

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Subject: Re: [lemonade] Re: ANNOTATEMORE and COMPARATOR
Date: Tue, 17 Apr 2007 10:11:21 +0100
From: Dave Cridland <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: Alexey Melnikov <[email-masked]>, Mark Crispin <[email-masked]>, Timo Sirainen <[email-masked]>, Cyrus Daboo <[email-masked]>, Enhancements to Internet email to support diverse service enivronments <[email-masked]>, <[email-masked]>
Sender: [email-masked]

On Tue Apr 17 09:38:31 2007, Arnt Gulbrandsen wrote:
> i;unicode-casemap contains a sentence which might be construed as 
> "and use i;octet from this point on". I'll try to suggest a 
> alternative wording there, something that reads naturally for an 
> UCS-4 implementation without i;octet.

Well, to do "i;octet" would mean encoding to UTF-8 first.

However, you get the same result if you perform a dictionary-like 
ordering over UCS-4, as far as I'm aware, so it's not a problem for 
collations, and equality tests work fine too.

It's things like substring matching which cause problems, since you 
can search for a substring matching partial UTF-8 sequences in 
principle, however, if you've already called those out as illegal 
input at a protocol level (or "super-comparator" level in this case), 
even that's blocked.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Date: Tue, 17 Apr 2007 10:43:54 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: ANNOTATEMORE and COMPARATOR
Cc: Eric Burger <[email-masked]>, Alexey Melnikov <[email-masked]>, Dan Karp <[email-masked]>, Timo Sirainen <[email-masked]>, Cyrus Daboo <[email-masked]>
Sender: [email-masked]

Dan Karp writes:
> IMAP (via SEARCH) essentially requires that message content be 
> searchable in some intermediate transcoded form (we chose UTF-16). 
> COMPARATOR (via "i;octet") requires that message content be 
> searchable in its original form pace content-encoding.

Only i;octet does that... we did a lot of work for RFC4790 to allow your 
kind of implementation, so I think it should be reasonable that 
COMPARATOR allows it too.

As far as I can see, the mention of i;octet in 
draft-crispin-collation-unicasemap is the only substantial requirement 
on i;octet, and it's weak. The one in draft-ietf-imapext-i18n is a 
leftover - something I forgot to delete when its rationale disappeared.

Arnt

Date: Tue, 17 Apr 2007 10:38:31 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: [lemonade] Re: ANNOTATEMORE and COMPARATOR
Cc: Alexey Melnikov <[email-masked]>, Mark Crispin <[email-masked]>, Timo Sirainen <[email-masked]>, Cyrus Daboo <[email-masked]>
Sender: [email-masked]

Mark Crispin writes:
> On Mon, 16 Apr 2007, Arnt Gulbrandsen wrote:
>>> The only place I see i;octet required is i18n draft, so I guess it's 
>>> still possible to change it?
>> That's the stated requirement. There may be an actual requirement 
>> elsewhere. SORT comes to mind.
>
> No. SORT would use i;unicode-casemap or (older servers) 
> i;ascii-casemap. I always assumed that i;octet was there as a 
> baseline for documentation purposes.  I never implemen ted it.

SORT requires a complete sort, doesn't it? Nothing other than i;octet 
provides that any more.

i;unicode-casemap contains a sentence which might be construed as "and 
use i;octet from this point on". I'll try to suggest a alternative 
wording there, something that reads naturally for an UCS-4 
implementation without i;octet.

>>> Is there a reason why a client should be able to force a server to 
>>> use i;ascii-casemap if i;unicode-casemap is also available?
>> Yes, but they're odd and weird.
>
> Examples, please?

Any test suite. A test client wishing to test everything required by 
lemonade would append some carefully constructed messages and use 
search on them with i;ascii-casemap. An IMAP client which connects to n 
IMAP servers and executes the same commands on each, noting any 
differences between the server responses.

I thought of something rather different last night, can't remember what 
it was. But odd and weird - nothing a MUA would do.

Arnt

Cc: [email-masked], Mark Crispin <[email-masked]>, Cyrus Daboo <[email-masked]>, Dave Cridland <[email-masked]>, Kjetil Torgrim Homme <[email-masked]>
From: Lisa Dusseault <[email-masked]>
Subject: Re: SORT on addresses
Date: Mon, 16 Apr 2007 16:58:14 -0700
To: Arnt Gulbrandsen <[email-masked]>
Sender: [email-masked]

On Apr 12, 2007, at 7:05 AM, Arnt Gulbrandsen wrote:

>
> Dave Cridland writes:
>> Yes, exactly - you've put it clearer than I did, but that's my  
>> intention.
>
> Fine phrasing.
>
>> I'd add a caveat that while I'll like to go all the way with the  
>> examination of what's needed, I wouldn't want to commit to  
>> producing capabilities to cover everything we find.
>
> Charter: Examine the types of sort, search and thread features  
> offered by modern email clients. Provide server-side extensions (to  
> IMAP, SMTP or other protocols) for that which it can be done faster  
> and/or better on the server.

Speaking personally, +1

Email clients like OSAF's Chandler and Opera M2 are ill-served by  
servers that assume that mail is divided into digestible chunks in  
child mailboxes/folders and that folders are the primary way users  
find emails.  Synch errors seem commoner (not assigning blame, merely  
noting what may be a spurious connection; I actually have no idea why  
this would be the case).

Faceted browsers are potentially a strong UI model for applications  
where the user wants to find a particular object among thousands in a  
library or repository (MP3s in a music library, email, files on disk,  
bookmarks, etc).  Many users are unwilling to sort all their data  
objects into folders and keywords or tags are not always filled in  
for individual objects even in a system that supports tagging.   Most  
users can't easily construct search queries with multiple terms from  
scratch, yet they can use faceted browsers which implicitly do  
multiple term queries.  Faceted browsing in some sense transcends  
both folder-sorted browsing and tagging/searching.  In order to work  
well however it needs well-indexed data and good search functions.

Lisa

Date: Mon, 16 Apr 2007 16:04:24 -0700 (Pacific Daylight Time)
From: Mark Crispin <[email-masked]>
To: Dave Cridland <[email-masked]>
cc: Arnt Gulbrandsen <[email-masked]>, Timo Sirainen <[email-masked]>, Enhancements to Internet email to support diverse service enivronments <[email-masked]>, [email-masked]
Subject: Re: [lemonade] Re: ANNOTATEMORE and COMPARATOR
Organization: Networks & Distributed Computing
Sender: [email-masked]

On Mon, 16 Apr 2007, Dave Cridland wrote:
> ACAP deals in arbitrary binary data which *might* be treatable as text - the 
> server has no way of knowing. "i;octet" is remarkably useful at doing exact 
> matches, whether text or binary, and was the only comparator capable of doing 
> case-sensitive matching on text.
> In IMAP, this is the only use, and there are undoubtedly better ways of doing 
> this.

I don't understand.  Everything in IMAP is case-insensitive by definition. 
There is, AFAIK, no need or use for i;octet anywhere in IMAP.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Subject: Re: [lemonade] Re: ANNOTATEMORE and COMPARATOR
Date: Mon, 16 Apr 2007 22:59:09 +0100
From: Dave Cridland <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: Mark Crispin <[email-masked]>, Timo Sirainen <[email-masked]>, Enhancements to Internet email to support diverse service enivronments <[email-masked]>, <[email-masked]>
Sender: [email-masked]

On Mon Apr 16 20:38:51 2007, Arnt Gulbrandsen wrote:
> Mark Crispin writes:
>> AFAIK, i;octet was documented for completeness.
> 
> FWIW: 2244 specified it, I don't know why, although I'm sure the 
> reason is in an archive somewhere. 4790 defined a registry and 
> inserted the preexisting collations.

ACAP deals in arbitrary binary data which *might* be treatable as 
text - the server has no way of knowing. "i;octet" is remarkably 
useful at doing exact matches, whether text or binary, and was the 
only comparator capable of doing case-sensitive matching on text.

In IMAP, this is the only use, and there are undoubtedly better ways 
of doing this.

(I do feel the need, sometimes, for case sensitivity - as today, when 
searching for messages with the subject line including "OMA").

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Cc: [email-masked], [email-masked], Alexey Melnikov <[email-masked]>, Cyrus Daboo <[email-masked]>
From: Timo Sirainen <[email-masked]>
Subject: Re: [lemonade] Re: ANNOTATEMORE and COMPARATOR
Date: Mon, 16 Apr 2007 22:54:09 +0300
To: Arnt Gulbrandsen <[email-masked]>
Sender: [email-masked]

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--Apple-Mail-1-877554154
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed

On 16.4.2007, at 22.35, Arnt Gulbrandsen wrote:

>> I would also want to see i;octet requirement gone. Actually I'd go  
>> even a bit further. Is there a reason why a client should be able  
>> to force a server to use i;ascii-casemap if i;unicode-casemap is  
>> also available?
>
> Yes, but they're odd and weird.
>
>> I can't think of any, but it would reduce optimization  
>> possibilities for server implementations.
>
> i;ascii-casemap is the same as i;unicode-casemap for the  
> intersection of their definition set, which would give you these  
> optimisations back. (IIRC, and it's late and...)

I'm just worried that some clients (or users) would select the  
i;ascii-casemap even when they could (or should) use i;unicode-casemap.

Non-ASCII characters with i;ascii-casemap are different, so if the  
server optimized i;unicode-casemap searches and a client did a non- 
ASCII search with i;ascii-casemap, the server would have to verify  
the unicode-optimized matches by searching through the messages to  
see if the non-ASCII casing matches exactly or not. Probably not that  
big of a deal, but it would be pretty pointless extra code.

--Apple-Mail-1-877554154
content-type: application/pgp-signature; x-mac-type=70674453;
	name=PGP.sig
content-description: This is a digitally signed message part
content-disposition: inline; filename=PGP.sig
content-transfer-encoding: 7bit

--Apple-Mail-1-877554154--

Date: Mon, 16 Apr 2007 12:44:33 -0700 (Pacific Daylight Time)
From: Mark Crispin <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
cc: [email-masked], [email-masked], Cyrus Daboo <[email-masked]>, Alexey Melnikov <[email-masked]>, Timo Sirainen <[email-masked]>
Subject: Re: [lemonade] Re: ANNOTATEMORE and COMPARATOR
Organization: Networks & Distributed Computing
Sender: [email-masked]

On Mon, 16 Apr 2007, Arnt Gulbrandsen wrote:
>> The only place I see i;octet required is i18n draft, so I guess it's still 
>> possible to change it?
> That's the stated requirement. There may be an actual requirement elsewhere. 
> SORT comes to mind.

No. SORT would use i;unicode-casemap or (older servers) i;ascii-casemap. 
I always assumed that i;octet was there as a baseline for documentation 
purposes.  I never implemen ted it.

>> Is there a reason why a client should be able to force a 
>> server to use i;ascii-casemap if i;unicode-casemap is also available?
> Yes, but they're odd and weird.

Examples, please?  My software did i;ascii-casemap, but only in old 
versions as an interim measure.  Newer versions are i;unicode-casemap 
only.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Date: Mon, 16 Apr 2007 21:38:51 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: [lemonade] Re: ANNOTATEMORE and COMPARATOR
Cc: Mark Crispin <[email-masked]>, Timo Sirainen <[email-masked]>
Sender: [email-masked]

Mark Crispin writes:
> AFAIK, i;octet was documented for completeness.

FWIW: 2244 specified it, I don't know why, although I'm sure the reason 
is in an archive somewhere. 4790 defined a registry and inserted the 
preexisting collations.

Arnt

Date: Mon, 16 Apr 2007 21:35:41 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: [lemonade] Re: ANNOTATEMORE and COMPARATOR
Cc: Alexey Melnikov <[email-masked]>, Timo Sirainen <[email-masked]>, Cyrus Daboo <[email-masked]>
Sender: [email-masked]

Timo Sirainen writes:
> On Mon, 2007-04-16 at 19:08 +0200, Arnt Gulbrandsen wrote:
>>  I'm wondering whether the i;octet requirement has outlived its 
>>  rationale. I know why it needed to be there in the early drafts 
>>  (other collations fell back to i;octet for some input), but I don't 
>>  know whether it needs to any more...?
>
> The only place I see i;octet required is i18n draft, so I guess it's 
> still possible to change it?

That's the stated requirement. There may be an actual requirement 
elsewhere. SORT comes to mind.

> I would also want to see i;octet requirement gone. Actually I'd go 
> even a bit further. Is there a reason why a client should be able to 
> force a server to use i;ascii-casemap if i;unicode-casemap is also 
> available?

Yes, but they're odd and weird.

> I can't think of any, but it would reduce optimization possibilities 
> for server implementations.

i;ascii-casemap is the same as i;unicode-casemap for the intersection of 
their definition set, which would give you these optimisations back. 
(IIRC, and it's late and...)

Arnt

Date: Mon, 16 Apr 2007 11:00:47 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: Timo Sirainen <[email-masked]>
cc: [email-masked], IMAP Extensions WG <[email-masked]>
Subject: Re: [lemonade] Re: ANNOTATEMORE and COMPARATOR
Sender: [email-masked]

IMHO:

i;ascii-casemap is important because there are legacy implementations that 
implement it.  Specifically, i;ascii-casemap *is* effectively the 
comparator of IMAP since its creation in 1986.  I believe that most modern 
IMAP server implementations do some version of i;unicode-casemap instead 
of i;ascii-casemap, and there is no way for the client to know with a 
server that does not otherwise support COMPARATOR.

I can't imagine that a client would choose i;ascii-casemap instead of 
i;unicode-casemap.

AFAIK, i;octet was documented for completeness.

I would be alright with i;unicode-casemap as being mandatory-to-implement 
for COMPARATOR, and optional-to-implement (albeit with a SHOULD) instead 
of i;ascii-casemap for non-COMPARATOR servers.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Date: Mon, 16 Apr 2007 18:51:40 +0100
From: Alexey Melnikov <[email-masked]>
To: Timo Sirainen <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: [lemonade] Re: ANNOTATEMORE and COMPARATOR
Sender: [email-masked]

Timo Sirainen wrote:

>Is the i;basic mentioned in i18n draft the same as i;octet?
>  
>
No. i;basic is defined in draft-gulbrandsen-collation-basic-01.txt.
Note that the mandatory-to-implement in i18n draft is going to be 
i;unicode-casemap.

Subject: Re: [lemonade] Re: ANNOTATEMORE and COMPARATOR
From: Timo Sirainen <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: [email-masked], [email-masked], Alexey Melnikov <[email-masked]>, Cyrus Daboo <[email-masked]>
Date: Mon, 16 Apr 2007 20:43:33 +0300
Sender: [email-masked]

--=-K+TEBP4YHYMuMTvRZ4hT
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

On Mon, 2007-04-16 at 19:08 +0200, Arnt Gulbrandsen wrote:
> I'm wondering whether the i;octet requirement has outlived its=20
> rationale. I know why it needed to be there in the early drafts (other=20
> collations fell back to i;octet for some input), but I don't know=20
> whether it needs to any more...?

The only place I see i;octet required is i18n draft, so I guess it's
still possible to change it?

I would also want to see i;octet requirement gone. Actually I'd go even
a bit further. Is there a reason why a client should be able to force a
server to use i;ascii-casemap if i;unicode-casemap is also available? I
can't think of any, but it would reduce optimization possibilities for
server implementations.

Is the i;basic mentioned in i18n draft the same as i;octet?

--=-K+TEBP4YHYMuMTvRZ4hT
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

--=-K+TEBP4YHYMuMTvRZ4hT--

Date: Mon, 16 Apr 2007 19:20:05 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: [lemonade] Re: ANNOTATEMORE and COMPARATOR
Cc: Mark Crispin <[email-masked]>, Timo Sirainen <[email-masked]>
Sender: [email-masked]

Mark Crispin writes:
> For collations in the IMAP context, UTF-16 is problematic for i;octet,

yes.

> i;ascii-casemap, and i;unicode-casemap.  I'm not sure about i;basic.

None of these - the problem was raised about a year ago by an RFC 4790 
reviewer and we solved it.

Using UTF-16 or UTF-8 instead of a sensible unicode encoding may make it 
more difficult to implement e.g. i;ascii-casemap, but not especially. 
Every kind of processing is more difficult in UTF-* than in UCS-*, IMO, 
and most collations are just specific examples of this general rules.

Arnt

Date: Mon, 16 Apr 2007 10:19:17 -0700 (PDT)
From: Dan Karp <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: Eric Burger <[email-masked]>, Alexey Melnikov <[email-masked]>, Timo Sirainen <[email-masked]>, Cyrus Daboo <[email-masked]>, [email-masked], [email-masked]
Subject: Re: ANNOTATEMORE and COMPARATOR
Sender: [email-masked]

> >   a) COMPARATOR is a reasonably heavy extension to implement
> 
> I don't think COMPARATOR ought to be that. Various other extensions 
> require it, and it seems wrong for me that COMPARATOR should be
> heavier than its users. Tail wagging dog.
> 
> Tell me what makes COMPARATOR heavy?

IMAP (via SEARCH) essentially requires that message content be
searchable in some intermediate transcoded form (we chose UTF-16).
COMPARATOR (via "i;octet") requires that message content be searchable
in its original form pace content-encoding.

Date: Mon, 16 Apr 2007 10:15:04 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: Timo Sirainen <[email-masked]>
cc: Dan Karp <[email-masked]>, [email-masked], IMAP Extensions WG <[email-masked]>
Subject: Re: ANNOTATEMORE and COMPARATOR
Sender: [email-masked]

On Mon, 16 Apr 2007, Timo Sirainen wrote:
> On Mon, 2007-04-16 at 09:41 -0700, Dan Karp wrote:
>> Searchable content is effectively transcoded to UTF-16.  The original 
>> content needed for "i;octet" is available only by serially opening, 
>> reading, and reparsing each message in the mailbox (stored 1 file per 
>> message), which is sufficiently nonperformant that we can't permit it.
> Do you support full substring searches as standard SEARCH command
> requires? In Zimbra's feature list I see "Ability to search using a
> prefix plus a wildcard", which seems like a "no" answer. In that case
> you're already noncompliant.

True.  I'm not sure what "search using a prefix plus a wildcard" is 
supposed to mean in IMAP searching, which is of substrings only.

For collations in the IMAP context, UTF-16 is problematic for i;octet, 
i;ascii-casemap, and i;unicode-casemap.  I'm not sure about i;basic.

I suggest that you use either UTF-8 or UCS-4.  The attraction of UTF-16 is 
to save memory space and a mostly-constant width if most characters are in 
the BMP (thus effectively the same as UCS-2 handling), but at the cost of 
MUCH more complex handling of non-BMP characters.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Date: Mon, 16 Apr 2007 19:12:25 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: ANNOTATEMORE and COMPARATOR
Cc: Eric Burger <[email-masked]>, Alexey Melnikov <[email-masked]>, Dan Karp <[email-masked]>, Timo Sirainen <[email-masked]>, Cyrus Daboo <[email-masked]>
Sender: [email-masked]

Dan Karp writes:
>   a) COMPARATOR is a reasonably heavy extension to implement

I don't think COMPARATOR ought to be that. Various other extensions 
require it, and it seems wrong for me that COMPARATOR should be heavier 
than its users. Tail wagging dog.

Tell me what makes COMPATOR heavy?

Arnt

Date: Mon, 16 Apr 2007 19:08:37 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: ANNOTATEMORE and COMPARATOR
Cc: Eric Burger <[email-masked]>, Alexey Melnikov <[email-masked]>, Dan Karp <[email-masked]>, Cyrus Daboo <[email-masked]>
Sender: [email-masked]

Dan Karp writes:
> Searchable content is effectively transcoded to UTF-16.  The original 
> content needed for "i;octet" is available only by serially opening, 
> reading, and reparsing each message in the mailbox (stored 1 file per 
> message), which is sufficiently nonperformant that we can't permit 
> it.

Makes sense. (We do something similar, although in our case it the 
problem is mitigated by other circumstances.)

I'm wondering whether the i;octet requirement has outlived its 
rationale. I know why it needed to be there in the early drafts (other 
collations fell back to i;octet for some input), but I don't know 
whether it needs to any more...?

Arnt

Date: Mon, 16 Apr 2007 10:03:51 -0700 (PDT)
From: Dan Karp <[email-masked]>
To: Timo Sirainen <[email-masked]>
Cc: Arnt Gulbrandsen <[email-masked]>, Eric Burger <[email-masked]>, Alexey Melnikov <[email-masked]>, Cyrus Daboo <[email-masked]>, [email-masked], [email-masked]
Subject: Re: ANNOTATEMORE and COMPARATOR
Sender: [email-masked]

> > Searchable content is effectively transcoded to UTF-16.  The
> > original content needed for "i;octet" is available only by serially
> > opening, reading, and reparsing each message in the mailbox (stored 1
> > file per message), which is sufficiently nonperformant that we can't
> > permit it.
> 
> Do you support full substring searches as standard SEARCH command
> requires? In Zimbra's feature list I see "Ability to search using a
> prefix plus a wildcard", which seems like a "no" answer. In that case
> you're already noncompliant.

*That* is a red herring, and I believe that you know it.  Can we please
redirect the discussion back to the core issue, which is that

  a) COMPARATOR is a reasonably heavy extension to implement
  b) METADATA is meant to be a reasonably light extension to implement
  c) METADATA is being positioned as a prerequisite for many new
extensions
  d) No METADATA entries appear able to make use of the COMPARATOR
functionality

Subject: Re: ANNOTATEMORE and COMPARATOR
From: Timo Sirainen <[email-masked]>
To: Dan Karp <[email-masked]>
Cc: Arnt Gulbrandsen <[email-masked]>, Eric Burger <[email-masked]>, Alexey Melnikov <[email-masked]>, Cyrus Daboo <[email-masked]>, [email-masked], [email-masked]
Date: Mon, 16 Apr 2007 19:53:10 +0300
Sender: [email-masked]

--=-fYPDKaFTsPDGb6/paBZs
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

On Mon, 2007-04-16 at 09:41 -0700, Dan Karp wrote:
> > > Unfortunately, my server implementation is not compatible with=20
> > > COMPARATOR, in particular the requirement to support i;octet in=20
> > > SEARCH.
> >=20
> > Why not?
>=20
> Searchable content is effectively transcoded to UTF-16.  The original con=
tent needed for "i;octet" is available only by serially opening, reading, a=
nd reparsing each message in the mailbox (stored 1 file per message), which=
 is sufficiently nonperformant that we can't permit it.

Do you support full substring searches as standard SEARCH command
requires? In Zimbra's feature list I see "Ability to search using a
prefix plus a wildcard", which seems like a "no" answer. In that case
you're already noncompliant.

--=-fYPDKaFTsPDGb6/paBZs
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

--=-fYPDKaFTsPDGb6/paBZs--

Date: Mon, 16 Apr 2007 09:41:33 -0700 (PDT)
From: Dan Karp <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: Eric Burger <[email-masked]>, Alexey Melnikov <[email-masked]>, Cyrus Daboo <[email-masked]>, [email-masked], [email-masked]
Subject: Re: ANNOTATEMORE and COMPARATOR
Sender: [email-masked]

> > Unfortunately, my server implementation is not compatible with 
> > COMPARATOR, in particular the requirement to support i;octet in 
> > SEARCH.
> 
> Why not?

Searchable content is effectively transcoded to UTF-16.  The original content needed for "i;octet" is available only by serially opening, reading, and reparsing each message in the mailbox (stored 1 file per message), which is sufficiently nonperformant that we can't permit it.

Date: Mon, 16 Apr 2007 18:34:46 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: ANNOTATEMORE and COMPARATOR
Cc: Eric Burger <[email-masked]>, Alexey Melnikov <[email-masked]>, Dan Karp <[email-masked]>, Cyrus Daboo <[email-masked]>
Sender: [email-masked]

Dan Karp writes:
> Unfortunately, my server implementation is not compatible with 
> COMPARATOR, in particular the requirement to support i;octet in 
> SEARCH.

Why not?

Arnt

Date: Mon, 16 Apr 2007 09:14:57 -0700 (PDT)
From: Dan Karp <[email-masked]>
To: Alexey Melnikov <[email-masked]>
Cc: Eric Burger <[email-masked]>, Enhancements to Internet email to support diverse service enivronments <[email-masked]>, IMAP Extensions WG <[email-masked]>, Cyrus Daboo <[email-masked]>
Subject: ANNOTATEMORE and COMPARATOR
Sender: [email-masked]

I'd really like to implement LEMONADE.  LEMONADE consists of several
IMAP extensions which require METADATA, and the current METADATA draft
in turn requires LIST-EXTENDED and COMPARATOR.  Unfortunately, my server
implementation is not compatible with COMPARATOR, in particular the
requirement to support i;octet in SEARCH.  I'm hoping that there's a way
to come to a compromise whereby METADATA can return to not requiring
COMPARATOR while still maintaining all its core functionality.  I'd hate
for such a small thing to be a roadblock for my and others' LEMONADE
implementations.

Right now, METADATA uses COMPARATOR to specify LIST-EXTENDED selection
constraints.  The GETMETADATA command doesn't appear to support these
constraints at present (though this may be an oversight):

    getmetadata       = "GETMETADATA" SP entries SP attribs
    entries           = entry-match /
                        "(" entry-match *(SP entry-match) ")"
    entry-match       = list-mailbox
    attribs           = attrib / "(" attrib *(SP attrib) ")"
    attrib            = astring

I've looked through all the entries defined by the base METADATA draft
as well as those defined by several of the LEMONADE drafts that metion
METADATA (/comment, /motd, /admin, /sort, /thread, /check, /checkperiod,
/convert/..., etc.).  None of these appear to benefit in a meaningful
way from the LIST-EXTENDED match/collation feature, so I'm gathering
that the inclusion of COMPARATOR is intended as more of a nice-to-have
with an eye towards accommodating possible future needs rather than a
real core METADATA feature.  If this is the case -- and towards the end
of making METADATA a workable foundation extension that every server can
implement -- I'd like to request that the LIST-EXTENDED METADATA
selection option type be removed from the METADATA draft.  If in the
future there is a new METADATA entry that would substantially benefit
from using COMPARATOR with LIST-EXTENDED, it should be possible to
reintroduce this functionality (or a subset thereof) in the same RFC
that registers the new entry.

Content-class: urn:content-classes:message
Subject: RE: [lemonade] Deprecate IMAP IDLE?
Date: Sun, 15 Apr 2007 13:08:52 +0300
From: <[email-masked]>
To: <[email-masked]>, <[email-masked]>
Cc: <[email-masked]>
Sender: [email-masked]

Dear Pete,
You said:

>Take a look at LEMONADE, 
>for example. There is really little going on in LEMONADE that 
>could not have been done in IMAPEXT. It's pretty much the same 
>people and it's mostly IMAP work. 

I think that this one might argue that deprecating IMAP IDLE is in fact
in the charter for LEMONADE.
Joke level: 10%.

Date: Sat, 14 Apr 2007 15:02:08 -0500
To: Arnt Gulbrandsen <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: Re: [lemonade] Deprecate IMAP IDLE?
Cc: IMAP Extensions WG <[email-masked]>
Sender: [email-masked]

On 4/14/07 at 9:16 PM +0200, Arnt Gulbrandsen wrote:

>Don't you think we should prolong the life of IMAPEXT by keeping it 
>alive until each of its standards-track products either has reached 
>full standard or certainly won't proceed further along the standards 
>track?
>
>(I'm only half joking. Well, only 60% joking.)

OK, so since we're only half joking, let me be half (well, maybe 
three-quarters) serious.

Working groups are supposed to be short-lived task-oriented groups, 
not long lived review bodies looking for things to do. Aside from the 
organizational reasons to do that, I think there is a good (albeit 
sociological/psychological) reason to shut down old working groups 
and start up new ones: People simply run out of energy to do good 
work in long ongoing efforts and get re-energized when there is a 
new, specific effort. Take a look at LEMONADE, for example. There is 
really little going on in LEMONADE that could not have been done in 
IMAPEXT. It's pretty much the same people and it's mostly IMAP work. 
But the fact that it was a new effort with a specific (well, mostly 
specific) goal in mind re-energized people and got them working 
toward that goal. The act of chartering (getting everyone to step 
back and decide what they're doing in the "bigger picture", coming up 
with a case for the IESG and the rest of the community, dividing up 
the tasks, etc.), writing -00 drafts, getting new chairs and document 
editors, along with the simple fact that the work *feels* new, makes 
the work go forward.

So, no, I don't think it's good to keep working groups around when 
they get really old. (This does leave open the question of how to 
progress things along the standards track without a WG, and I think 
that the IESG should--and is starting to--look at that issue.)

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

Date: Sat, 14 Apr 2007 21:16:20 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: Pete Resnick <[email-masked]>
Subject: Re: [lemonade] Deprecate IMAP IDLE?
Cc: IMAP Extensions WG <[email-masked]>
Sender: [email-masked]

Hi there Mr. Chair. Don't you think we should prolong the life of 
IMAPEXT by keeping it alive until each of its standards-track products 
either has reached full standard or certainly won't proceed further 
along the standards track?

(I'm only half joking. Well, only 60% joking.)

Arnt

Date: Sat, 14 Apr 2007 11:55:45 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: Pete Resnick <[email-masked]>
cc: IMAP Extensions WG <[email-masked]>
Subject: Re: SORT on addresses
Sender: [email-masked]

On Sat, 14 Apr 2007, Pete Resnick wrote:
> On 4/12/07 at 5:57 PM +0100, Alexey Melnikov wrote:
>> Mark Crispin wrote:
>>> And once again, IMAPEXT finds yet another way to prolong its life.
>> IMHO, for this work to happen as an official WG, either IMAPEXT needs to 
>> recharter, or a new WG should be formed.
>> I am slightly in favor of the latter.
> *If* this work happens, I am *overwhelmingly* in favor of the later.

+1 (in case it wasn't obvious)

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Date: Sat, 14 Apr 2007 13:52:49 -0500
To: Alexey Melnikov <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: Re: SORT on addresses
Cc: Mark Crispin <[email-masked]>, IMAP Extensions WG <[email-masked]>
Sender: [email-masked]

On 4/12/07 at 5:57 PM +0100, Alexey Melnikov wrote:

>Mark Crispin wrote:
>
>>And once again, IMAPEXT finds yet another way to prolong its life.
>
>IMHO, for this work to happen as an official WG, either IMAPEXT 
>needs to recharter, or a new WG should be formed.
>I am slightly in favor of the latter.

*If* this work happens, I am *overwhelmingly* in favor of the later.

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

Date: Thu, 12 Apr 2007 11:26:28 -0700
From: Peter Coates <[email-masked]>
Subject: RE: SORT on addresses
To: [email-masked]
Sender: [email-masked]

To: "Fredrick the Great" <fred@domain>, friends: bob@domain, sally@domain;
	Sort on Fredrick the Great

To: fred@domain, friends: bob@domain, sally@domain;
	Sort on fred@domain

To: friends: bob@domain, sally@domain;, fred@domain
	Sort friends

To: friends: "Robert the Bruce"<bob@domain>, sally@domain;, fred@domain
	Sort friends

Peter

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
On Behalf Of Philip Guenther
Sent: April 12, 2007 11:02
To: Peter Coates
Cc: [email-masked]
Subject: RE: SORT on addresses

On Thu, 12 Apr 2007, Peter Coates wrote:
> The display name of the group if there is one, failing which the 
> display name of the first address, failing which the first address. Is 
> the algorithm.

So this:

To: fred@domain, friends: bob@domain, sally@domain;

will sort on 'friends' and not 'fred'?  Interesting.

Philip Guenther

Date: Thu, 12 Apr 2007 11:05:12 -0700
From: Peter Coates <[email-masked]>
Subject: RE: SORT on addresses
To: "'Philip Guenther'" <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

No.  The algorithm uses the first address it finds.  If that is a group, OK, but
in this case the first thing it finds is fred@domain, so that is what it uses

Peter 

-----Original Message-----
From: Philip Guenther [mailto:[email-masked]] 
Sent: April 12, 2007 11:02
To: Peter Coates
Cc: [email-masked]
Subject: RE: SORT on addresses

On Thu, 12 Apr 2007, Peter Coates wrote:
> The display name of the group if there is one, failing which the 
> display name of the first address, failing which the first address. Is 
> the algorithm.

So this:

To: fred@domain, friends: bob@domain, sally@domain;

will sort on 'friends' and not 'fred'?  Interesting.

Philip Guenther

Date: Thu, 12 Apr 2007 12:02:01 -0600
From: Philip Guenther <[email-masked]>
To: Peter Coates <[email-masked]>
cc: [email-masked]
Subject: RE: SORT on addresses
Sender: [email-masked]

On Thu, 12 Apr 2007, Peter Coates wrote:
> The display name of the group if there is one, failing which the display 
> name of the first address, failing which the first address. Is the 
> algorithm.

So this:

To: fred@domain, friends: bob@domain, sally@domain;

will sort on 'friends' and not 'fred'?  Interesting.

Philip Guenther

Date: Thu, 12 Apr 2007 10:54:30 -0700
From: Peter Coates <[email-masked]>
Subject: RE: SORT on addresses
To: "'Philip Guenther'" <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

The display name of the group if there is one, failing which the display name of
the first address, failing which the first address. Is the algorithm.  

-----Original Message-----
From: Philip Guenther [mailto:[email-masked]] 
Sent: April 12, 2007 10:33
To: Peter Coates
Cc: [email-masked]
Subject: RE: SORT on addresses

On Thu, 12 Apr 2007, Peter Coates wrote:
> Being incompetent, I replied, rather than replied all when I sent a 
> note saying that the Sun imap server uses sort keys X-SUN-TO, 
> X-SUN-CC, X-SUN-FROM to sort by display-name or if display name is not 
> present the email address.  For obvious reasons I like that algorithm.

The display-name of the first actual address or will it use the display-name of
a group?

Hmm, the current SORT address keys use the first addr-mailbox, so they don't
skip groups but will sort something like

To: friends: bob@domain, sally@domain;

with the key "friends"...which will be annoying if the display-name contains an
RFC 2047 word, as there's no mention of decoding them like there is for the
SUBJECT key.

Philip Guenther

Date: Thu, 12 Apr 2007 11:33:21 -0600
From: Philip Guenther <[email-masked]>
To: Peter Coates <[email-masked]>
cc: [email-masked]
Subject: RE: SORT on addresses
Sender: [email-masked]

On Thu, 12 Apr 2007, Peter Coates wrote:
> Being incompetent, I replied, rather than replied all when I sent a note 
> saying that the Sun imap server uses sort keys X-SUN-TO, X-SUN-CC, 
> X-SUN-FROM to sort by display-name or if display name is not present the 
> email address.  For obvious reasons I like that algorithm.

The display-name of the first actual address or will it use the 
display-name of a group?

Hmm, the current SORT address keys use the first addr-mailbox, so they 
don't skip groups but will sort something like

To: friends: bob@domain, sally@domain;

with the key "friends"...which will be annoying if the display-name 
contains an RFC 2047 word, as there's no mention of decoding them like 
there is for the SUBJECT key.

Philip Guenther

Date: Thu, 12 Apr 2007 10:16:58 -0700
From: Mark Crispin <[email-masked]>
To: Alexey Melnikov <[email-masked]>
cc: IMAP Extensions WG <[email-masked]>
Subject: Re: SORT on addresses
Organization: Networks & Distributed Computing
Sender: [email-masked]

On Thu, 12 Apr 2007, Alexey Melnikov wrote:
> IMHO, for this work to happen as an official WG, either IMAPEXT needs to 
> recharter, or a new WG should be formed.
> I am slightly in favor of the latter.

Personally, I feel that it is long past time for IMAPEXT to die.  Nor 
should there be a replacement any time in the short-term future.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Date: Thu, 12 Apr 2007 17:57:28 +0100
From: Alexey Melnikov <[email-masked]>
To: Mark Crispin <[email-masked]>
CC: IMAP Extensions WG <[email-masked]>
Subject: Re: SORT on addresses
Sender: [email-masked]

Mark Crispin wrote:

> And once again, IMAPEXT finds yet another way to prolong its life.

IMHO, for this work to happen as an official WG, either IMAPEXT needs to 
recharter, or a new WG should be formed.
I am slightly in favor of the latter.

Date: Thu, 12 Apr 2007 09:30:47 -0700
From: Peter Coates <[email-masked]>
Subject: RE: SORT on addresses
To: "'Cyrus Daboo'" <[email-masked]>, "'Dave Cridland'" <[email-masked]>, "'Mark Crispin'" <[email-masked]>
Cc: [email-masked], "'Kjetil Torgrim Homme'" <[email-masked]>
Sender: [email-masked]

There are a class of clients that DO use server side sort and search, and those
are web clients.  These are probably used more than desktop imap clients.  And
lemonade is premised on the assumption that mobile devices will use imap and are
too strapped for processing power to do more than the necessary.

Being incompetent, I replied, rather than replied all when I sent a note saying
that the Sun imap server uses sort keys X-SUN-TO, X-SUN-CC, X-SUN-FROM to sort
by display-name or if display name is not present the email address.  For
obvious reasons I like that algorithm.

Peter

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
On Behalf Of Cyrus Daboo
Sent: April 12, 2007 06:27
To: Dave Cridland; Mark Crispin
Cc: [email-masked]; Kjetil Torgrim Homme
Subject: Re: SORT on addresses

Hi Dave,

--On April 12, 2007 10:03:46 AM +0100 Dave Cridland <[email-masked]>
wrote:

> I think that there would be serious advantage to taking a step back at 
> this point, and examining how users and UAs are currently searching 
> through mail, and see how we could make this both more efficient and 
> closer to the end-users' requirements. Right now, I think that IMAP 
> provides only rudimentary "organizational" abilities, and I think that 
> they're showing up as a weak point in IMAP.

Lets face it, very few clients really use server-side sort/thread/search. 
Most download and cache and do local operations because they can offer much,
much more in the way of features for that.

If we are going to make changes to the server-side behavior we should go all the
way. Examine the types of features offered by desktop clients doing local
operations and match those with server-side capabilities. Anything less is a
poor half-measure.

--
Cyrus Daboo

Subject: Re: SORT on addresses
Date: Thu, 12 Apr 2007 16:29:12 +0100
From: Dave Cridland <[email-masked]>
To: Mark Crispin <[email-masked]>
Cc: <[email-masked]>
Sender: [email-masked]

On Thu Apr 12 15:29:27 2007, Mark Crispin wrote:
> 
> And once again, IMAPEXT finds yet another way to prolong its life.

No, we should come up with a much more contrived acronym than that. 
:-)

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Date: Thu, 12 Apr 2007 11:27:33 -0400
From: Tony Hansen <[email-masked]>
CC: IMAP Extensions WG <[email-masked]>
Subject: Re: SORT on addresses
Sender: [email-masked]

what, you expected a rest? Silly Pooh Bear, in the words of the immortal
Christopher Robin. :-)

Mark Crispin wrote:
> 
> And once again, IMAPEXT finds yet another way to prolong its life.

Date: Thu, 12 Apr 2007 16:45:38 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: SORT on addresses
Cc: Mark Crispin <[email-masked]>
Sender: [email-masked]

Mark Crispin writes:
> And once again, IMAPEXT finds yet another way to prolong its life.

IMAP is the protocol that just won't lie down and die.

Arnt

Subject: Re: SORT on addresses
From: Kjetil Torgrim Homme <[email-masked]>
To: Cyrus Daboo <[email-masked]>
Cc: [email-masked]
Date: Thu, 12 Apr 2007 16:42:16 +0200
Sender: [email-masked]

On Thu, 2007-04-12 at 09:27 -0400, Cyrus Daboo wrote:
> Lets face it, very few clients really use server-side sort/thread/search. 
> Most download and cache and do local operations because they can offer 
> much, much more in the way of features for that.

ironically(?), webmail applications are probably the most common user of
"stateless" IMAP.  this is a space where improvements in server and
client are relatively easily deployed, so it's not a bad space to work
in.

> If we are going to make changes to the server-side behavior we should go 
> all the way. Examine the types of features offered by desktop clients doing 
> local operations and match those with server-side capabilities. Anything 
> less is a poor half-measure.

I wouldn't word it that strongly, but sure -- sounds like you have the
work cut out for us.  one thing is to finalise complicated stuff like
"context", there are also simple things like standardise on flags for
"Spam" and "Ham" which could do wonders for client interoperability.

-- 
Kjetil T.

Date: Thu, 12 Apr 2007 07:29:27 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: IMAP Extensions WG <[email-masked]>
Subject: Re: SORT on addresses
Sender: [email-masked]

And once again, IMAPEXT finds yet another way to prolong its life.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Date: Thu, 12 Apr 2007 16:05:32 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: SORT on addresses
Cc: Mark Crispin <[email-masked]>, Cyrus Daboo <[email-masked]>, Dave Cridland <[email-masked]>, Kjetil Torgrim Homme <[email-masked]>
Sender: [email-masked]

Dave Cridland writes:
> Yes, exactly - you've put it clearer than I did, but that's my intention.

Fine phrasing.

> I'd add a caveat that while I'll like to go all the way with the 
> examination of what's needed, I wouldn't want to commit to producing 
> capabilities to cover everything we find.

Charter: Examine the types of sort, search and thread features offered 
by modern email clients. Provide server-side extensions (to IMAP, SMTP 
or other protocols) for that which it can be done faster and/or better 
on the server.

Arnt

Subject: Re: SORT on addresses
Date: Thu, 12 Apr 2007 14:46:07 +0100
From: Dave Cridland <[email-masked]>
To: Cyrus Daboo <[email-masked]>
Cc: <[email-masked]>, Kjetil Torgrim Homme <[email-masked]>, Mark Crispin <[email-masked]>
Sender: [email-masked]

On Thu Apr 12 14:27:19 2007, Cyrus Daboo wrote:
> 
> Hi Dave,
> 
> --On April 12, 2007 10:03:46 AM +0100 Dave Cridland 
> <[email-masked]> wrote:
> 
>> I think that there would be serious advantage to taking a step 
>> back at
>> this point, and examining how users and UAs are currently searching
>> through mail, and see how we could make this both more efficient 
>> and
>> closer to the end-users' requirements. Right now, I think that IMAP
>> provides only rudimentary "organizational" abilities, and I think 
>> that
>> they're showing up as a weak point in IMAP.
> 
> Lets face it, very few clients really use server-side 
> sort/thread/search. Most download and cache and do local operations 
> because they can offer much, much more in the way of features for 
> that.
> 
> If we are going to make changes to the server-side behavior we 
> should go all the way. Examine the types of features offered by 
> desktop clients doing local operations and match those with 
> server-side capabilities. Anything less is a poor half-measure.

Yes, exactly - you've put it clearer than I did, but that's my 
intention.

I'd add a caveat that while I'll like to go all the way with the 
examination of what's needed, I wouldn't want to commit to producing 
capabilities to cover everything we find.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Subject: Re: SORT on addresses
From: Kjetil Torgrim Homme <[email-masked]>
To: [email-masked]
Date: Thu, 12 Apr 2007 15:45:02 +0200
Sender: [email-masked]

first, I'd like to thank everyone for their reasoned and constructive
responses.

On Thu, 2007-04-12 at 10:03 +0100, Dave Cridland wrote:
> On Thu Apr 12 02:38:18 2007, Mark Crispin wrote:
> > Now, with that said, I would not oppose a SORTEXT proposal that 
> > adds new keys (as opposed to changing the current keys) that work 
> > as you propose.
> 
> I would oppose a proposal *now*. I don't think we know nearly enough 
> to base a decision on concerning which keys to add.

fair enough.  Peter Coates told me in a private reply that Sun has an
extension which sorts on display-name, which very well may be more
aligned with what users want/expect anyway.  there are probably other
options.

> So, here's a counter-proposal:
> 
> We setup a mailing list, or at least agree to use an existing one, 
> invite a few UA developers and search-agent people [since we're heavy 
> on the server-side as usual], and start to figure out what's needed, 
> what's practical, and get to work on it. If we seem to be making 
> progress, we turn it into a WG if it seems to suit.

sounds good to me.
-- 
Kjetil T.

Date: Thu, 12 Apr 2007 09:27:19 -0400
From: Cyrus Daboo <[email-masked]>
To: Dave Cridland <[email-masked]>, Mark Crispin <[email-masked]>
cc: [email-masked], Kjetil Torgrim Homme <[email-masked]>
Subject: Re: SORT on addresses
Sender: [email-masked]

Hi Dave,

--On April 12, 2007 10:03:46 AM +0100 Dave Cridland <[email-masked]> 
wrote:

> I think that there would be serious advantage to taking a step back at
> this point, and examining how users and UAs are currently searching
> through mail, and see how we could make this both more efficient and
> closer to the end-users' requirements. Right now, I think that IMAP
> provides only rudimentary "organizational" abilities, and I think that
> they're showing up as a weak point in IMAP.

Lets face it, very few clients really use server-side sort/thread/search. 
Most download and cache and do local operations because they can offer 
much, much more in the way of features for that.

If we are going to make changes to the server-side behavior we should go 
all the way. Examine the types of features offered by desktop clients doing 
local operations and match those with server-side capabilities. Anything 
less is a poor half-measure.

-- 
Cyrus Daboo

Date: Thu, 12 Apr 2007 09:16:23 -0400
From: Ken Murchison <[email-masked]>
Organization: Carnegie Mellon University
To: [email-masked]
Subject: Re: SORT on addresses
Sender: [email-masked]

Dave Cridland wrote:

> So, here's a counter-proposal:
> 
> We setup a mailing list, or at least agree to use an existing one, 
> invite a few UA developers and search-agent people [since we're heavy on 
> the server-side as usual], and start to figure out what's needed, what's 
> practical, and get to work on it. If we seem to be making progress, we 
> turn it into a WG if it seems to suit.
> 
> The aim would be to get, or look into:
> - a set of useful SEARCH keys, SORT keys, etc.
> - multimailbox SEARCHing.
> - Inexact searching, such as SCAN provides.
> - Sorting by relevance.
> - probably other search and organizational things. (Perhaps we could 
> stretch to a re-examination of ANNOTATE, or resurrect Alexey's keywords 
> draft - not saying we should do either, but looking into them might be 
> useful).
> 
> I have no clear idea if this would be considered IETF or IRTF territory 
> - I suspect it might start out informal, and become formalized in one or 
> other or both.
> 
> Who's willing to participate in something like this?

I would be (speaking for CMU).

-- 
Kenneth Murchison
Systems Programmer
Project Cyrus Developer/Maintainer
Carnegie Mellon University

Date: Thu, 12 Apr 2007 12:30:17 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: SORT on addresses
Cc: Mark Crispin <[email-masked]>, Dave Cridland <[email-masked]>, Kjetil Torgrim Homme <[email-masked]>
Sender: [email-masked]

Dave Cridland writes:
> I spoke to a few people in Prague about IMAP and email, and the 
> general area of searching and managing large quantities of email was 
> raised by several of them, so perhaps the time is right.

Indeed.

> So, here's a counter-proposal:
>
> We setup a mailing list, or at least agree to use an existing one, 
> invite a few UA developers and search-agent people [since we're heavy 
> on the server-side as usual], and start to figure out what's needed, 
> what's practical, and get to work on it. If we seem to be making 
> progress, we turn it into a WG if it seems to suit.
>
> The aim would be to get, or look into:
> - a set of useful SEARCH keys, SORT keys, etc.
> - multimailbox SEARCHing.
> - Inexact searching, such as SCAN provides.
> - Sorting by relevance.
> - probably other search and organizational things. (Perhaps we could 
> stretch to a re-examination of ANNOTATE, or resurrect Alexey's 
> keywords draft - not saying we should do either, but looking into 
> them might be useful).
>
> I have no clear idea if this would be considered IETF or IRTF 
> territory - I suspect it might start out informal, and become 
> formalized in one or other or both.
>
> Who's willing to participate in something like this?

Arnt is. No surprise I'm sure.

Arnt

Subject: Re: SORT on addresses
Date: Thu, 12 Apr 2007 10:03:46 +0100
From: Dave Cridland <[email-masked]>
To: Mark Crispin <[email-masked]>
Cc: <[email-masked]>, Kjetil Torgrim Homme <[email-masked]>
Sender: [email-masked]

On Thu Apr 12 02:38:18 2007, Mark Crispin wrote:
> 
> On Wed, 11 Apr 2007, Kjetil Torgrim Homme wrote:
>> it seems to me that this behaviour is mostly useful in "old-skool"
>> environments, where username@host was common.  e.g., I could be
>> [email-masked], [email-masked] or [email-masked]
>> depending on which workstation I was logged in that day.  I 
>> sincerely
>> doubt this is widespread today.
> 
> Please be assured that it is still quite common and still very much 
> a fact of life for many people.
> 
> 
It is for you. Not for me, and I suspect that sorting by phrase would 
provide a much more useful method than looking at the address at all.

(For you: mrc@ is almost uniformly Mark Crispin, and "SORT (FROM) 
UTF-8 ..." sorts all your mails the "right" way. For me: I'm dave@ 
for home, and dave.cridland@ for work. I'm pretty sure I'm not the 
only dave@, too. SORT (FROM) is useless for my mails as a result.)

>>      * will end-users think that changing the sort key to be the
>>        complete RFC2822 addr-spec (or RFC2822 group) is an 
>> improvement?
>> my answer is, to the extent they notice it, an emphatic yes.
> 
> What surety is there that such is the case?  Your opinion that this 
> is an "improvement" has been noted.  That does not constitute 
> research.
> 
> 
I certainly agree that *a* sort key needs to exist to allow for 
collating the results by "the user". I'm even convinced that blindly 
changing FROM/TO/CC would benefit the majority of users, but ...

>>      * is it problematic that current deployments of the draft 
>> break
>>        wrt compliance?
>> my answer is, not really.  this is not a behavioural change which
>> affects clients, it only affects the humans who use the clients.
> 
> Ahem.
> 
> A change in behavior that has been implemented and documented for 
> over a decade is quite problematic.
> 
> 
... the above point unnerves me, and moreover I cannot be certain 
that the assertion that the sort results are blindly passed through 
to the user is correct.

> Now, with that said, I would not oppose a SORTEXT proposal that 
> adds new keys (as opposed to changing the current keys) that work 
> as you propose.

I would oppose a proposal *now*. I don't think we know nearly enough 
to base a decision on concerning which keys to add.

I think that there would be serious advantage to taking a step back 
at this point, and examining how users and UAs are currently 
searching through mail, and see how we could make this both more 
efficient and closer to the end-users' requirements. Right now, I 
think that IMAP provides only rudimentary "organizational" abilities, 
and I think that they're showing up as a weak point in IMAP.

I spoke to a few people in Prague about IMAP and email, and the 
general area of searching and managing large quantities of email was 
raised by several of them, so perhaps the time is right.

So, here's a counter-proposal:

We setup a mailing list, or at least agree to use an existing one, 
invite a few UA developers and search-agent people [since we're heavy 
on the server-side as usual], and start to figure out what's needed, 
what's practical, and get to work on it. If we seem to be making 
progress, we turn it into a WG if it seems to suit.

The aim would be to get, or look into:
- a set of useful SEARCH keys, SORT keys, etc.
- multimailbox SEARCHing.
- Inexact searching, such as SCAN provides.
- Sorting by relevance.
- probably other search and organizational things. (Perhaps we could 
stretch to a re-examination of ANNOTATE, or resurrect Alexey's 
keywords draft - not saying we should do either, but looking into 
them might be useful).

I have no clear idea if this would be considered IETF or IRTF 
territory - I suspect it might start out informal, and become 
formalized in one or other or both.

Who's willing to participate in something like this?

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Subject: Re: SORT on addresses
From: Timo Sirainen <[email-masked]>
To: Mark Crispin <[email-masked]>
Cc: Kjetil Torgrim Homme <[email-masked]>, [email-masked]
Date: Thu, 12 Apr 2007 11:14:24 +0300
Sender: [email-masked]

--=-cH0f2lcWAd7jk9WgZyt8
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

On Wed, 2007-04-11 at 18:38 -0700, Mark Crispin wrote:
> Now, with that said, I would not oppose a SORTEXT proposal that adds new=20
> keys (as opposed to changing the current keys) that work as you propose.

If someone starts that, perhaps it could also include REFERENCES2?
http://mailman1.u.washington.edu/pipermail/imap-protocol/2006-December/0003=
22.html

Or maybe I'll just need to write a draft myself some day.

--=-cH0f2lcWAd7jk9WgZyt8
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

--=-cH0f2lcWAd7jk9WgZyt8--

Date: Wed, 11 Apr 2007 18:38:18 -0700 (Pacific Daylight Time)
From: Mark Crispin <[email-masked]>
To: Kjetil Torgrim Homme <[email-masked]>
cc: [email-masked]
Subject: Re: SORT on addresses
Organization: Networks & Distributed Computing
Sender: [email-masked]

On Wed, 11 Apr 2007, Kjetil Torgrim Homme wrote:
> it seems to me that this behaviour is mostly useful in "old-skool"
> environments, where username@host was common.  e.g., I could be
> [email-masked], [email-masked] or [email-masked]
> depending on which workstation I was logged in that day.  I sincerely
> doubt this is widespread today.

Please be assured that it is still quite common and still very much a fact 
of life for many people.

>      * will end-users think that changing the sort key to be the
>        complete RFC2822 addr-spec (or RFC2822 group) is an improvement?
> my answer is, to the extent they notice it, an emphatic yes.

What surety is there that such is the case?  Your opinion that this is an 
"improvement" has been noted.  That does not constitute research.

>      * is it problematic that current deployments of the draft break
>        wrt compliance?
> my answer is, not really.  this is not a behavioural change which
> affects clients, it only affects the humans who use the clients.

Ahem.

A change in behavior that has been implemented and documented for over a 
decade is quite problematic.

A declaration that software which has been distributed for more than a 
decade is suddenly "non-compliant" is quite problematic.

The only reason why SORT is "still a draft" is that it got hung up on i18n 
collation.  SORT has been a cast-in-stone specification for production 
software for many years.  It can not be arbitrarily changed.

Now, with that said, I would not oppose a SORTEXT proposal that adds new 
keys (as opposed to changing the current keys) that work as you propose.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Subject: SORT on addresses
From: Kjetil Torgrim Homme <[email-masked]>
To: [email-masked]
Date: Wed, 11 Apr 2007 07:27:54 +0200
Sender: [email-masked]

I came across some grumblings on the info-cyrus mailing list about the
behaviour of SORT with respect to FROM and TO.

[Rob Mueller]:
> It's defined by the IMAP sort extension.
> 
> http://tools.ietf.org/html/draft-ietf-imapext-sort-19
> 
>          FROM
>             [IMAP] addr-mailbox of the first "From" address.
> 
>          TO
>             [IMAP] addr-mailbox of the first "To" address.
> 
> So it really only sorts by the localpart of the email address, not
> even the domain is included, something that's annoyed our users as
> well since you basically end up with UID sorting for the localparts
> that are the same but different domains. That's the "standard"
> though... *sigh*

I was surprised to see this explanation of the behaviour, so I had to
look it up (addr-mailbox being local-part wasn't obvious to me), and
tried to find the justification for it in the archives.

way back in 2004-11-09, Dave Cridland wrote about the mismatch between
SEARCH and SORT (SEARCH looks at the headers verbatim, SORT does
decoding first), and went on to say:

[Dave Cridland]:
> I confess to being totally bewildered by the FROM, TO and CC sort
> keys, as they all appear to only sort on the local-part of the email
> address - that *is* the "bit before the @", right? So different
> domains collate equally? I can't understand why this should be the
> case.

[Mark Crispin] responded:
> In practice, not all that many people use those keys.
> 
> The idea is to tend to collect messages from a single person together,
> based upon the tendency of people to have the same local-part for his
> email address even though they may use many different sites.  Note
> that word "tend".
> 
> It's useful, in some very limited cases.

it seems to me that this behaviour is mostly useful in "old-skool"
environments, where username@host was common.  e.g., I could be
[email-masked], [email-masked] or [email-masked]
depending on which workstation I was logged in that day.  I sincerely
doubt this is widespread today.

a given localpart will typically belong to a single entity within a
given mailbox, e.g., in my personal INBOX, "christb" is unique, and
there are "only" three distinct people with localpart "chris" (in my
INBOX.lists there are 12 "chris").  role addresses increase the chance
of mixups, though: "info", "noreply", "orders", "post", "service",
"support", "webmaster" are all examples from my INBOX which belong to
numerous domains and companies, and it's not at all intuitive to me that
all addresses sharing one of these localparts are sorted as the same
address.

to give an example of when this is annoying, consider when I want to
file messages from certain people into separate folders.  most clients
allow selecting a block by specifying the start and end in the message
list, so I can operate on all of them.  when the FROM criteria is equal,
it falls back on sequence number, which means I have to pick messages
individually since the senders are interleaved.  in this scenario, even
if I wanted to move all messages from "chris" regardless of domain, it's
not a problem, since I can still select them as a single block.

the change in behaviour is only observable in edge cases, which is why I
dare bring it up at all so late in the game.  I'd like to pose two
questions:

      * will end-users think that changing the sort key to be the
        complete RFC2822 addr-spec (or RFC2822 group) is an improvement?

my answer is, to the extent they notice it, an emphatic yes.

      * is it problematic that current deployments of the draft break
        wrt compliance?

my answer is, not really.  this is not a behavioural change which
affects clients, it only affects the humans who use the clients.

-- 
thank you for reading,
Kjetil T.

Date: Mon, 09 Apr 2007 20:07:40 +0100
From: Alexey Melnikov <[email-masked]>
To: Dan Karp <[email-masked]>, Cyrus Daboo <[email-masked]>
CC: Eric Burger <[email-masked]>, Enhancements to Internet email to support  diverse service enivronments <[email-masked]>, IMAP Extensions WG <[email-masked]>
Subject: Re: [lemonade] Expert Review: annotatemore (like WGLC)
Sender: [email-masked]

Dan Karp wrote:

>>http://www.ietf.org/internet-drafts/draft-daboo-imap-annotatemore-11.txt
>>    
>>
>Comments interspersed btween snippets of the draft:
>  
>
Hi Dan,

>   Changes from -10 to -11:
>   7.   Added match type and collation identifier to the LIST-EXTENDED
>        selection option.
>   8.   Made support for IMAP-I18N a requirement.
>
>   Text comparisons may be done as part of the GETMETADATA or LIST-
>   EXTENDED commands.  As a result, support for the COMPARATOR
>   [I-D.ietf-imapext-i18n] extension is REQUIRED.
>
>GETMETADATA doesn't actually do text comparisons, right?  The BNF
>doesn't seem to tie in the "matchtype" or "collation" productions.
>  
>
Check the <list-select-metadata>, it references both.

>But more to the point, I believe that the intention was that METADATA
>should be a simple, baseline extenstion that will be required by a wide
>range of other extensions.  If that's the case, *PLEASE* reconsider the
>wisdom of adding the LT, LE, GT, and GE matchers (and thus the
>COMPARATOR requirement) to this draft.  It doesn't apear to be core
>functionality for METADATA, and since COMPARATOR appears nontrivial to
>implement
>
Can you please elaborate on this?
(Note that the COMPARATOR extension is about to switch to 
i;unicode-casemap collation (as mandatory to implement), which is much 
easier to implement than i;basic)

> this may not only prevent server authors from implementing
>METADATA but also keep us from impelenting any future extension that
>requires METADATA.
>
>
>   A user can only set and retrieve private or shared annotations on a
>   mailbox which exists and is returned to them via a LIST or LSUB
>   command, irrespective of whether they have read or write access to
>   the actual message content of the mailbox.  If the client attempts to
>   set or retrieve annotations on other mailboxes, the server MUST
>   respond with a NO response.
>
>I can understand the logic behind requiring LIST/LSUB access for reading
>public mailbox annotations and for setting private annotations.  But
>that seems to be a very low bar for setting public mailbox annotations.
>Adding a new ACL permission for setting annotations would make this
>extension more complex and would also introduce a dependency on the ACL
>extension,
>
Dependency in what sense?

> but perhaps requiring [READ-WRITE] access to the mailbox
>might be reasonable?
>
>
>   The new SETMETADATA command and the METADATA response each have a
>   mailbox name argument, indicating that the annotations being referred
>   to are attached to the specified mailbox.  An empty string can be
>   used for the mailbox name to signify server annotations.
>
>What rights are required to set public and private annotations on the
>server?
>  
>
None. But maybe this should be stated explicitly.

>   This command sets the specified list of entries by adding or
>   replacing the specified attributes with the values provided, on the
>   specified existing mailboxes.  Clients can use NIL for values of
>   attributes it wants to remove from entries.  The server MAY return a
>   METADATA response containing the updated annotation data.
>
>In this case, the server may return a METADATA response that doesn't
>contain the updated annotation data or a LIST response that does,
>correct?
>
>
>   This extension adds the GETMETADATA command.  This allows clients to
>   retrieve server annotations.  Mailbox annotations are retrieved via
>   the extended LIST command described in the next section.
>
>   This extension adds the SETMETADATA command.  This allows clients to
>   set annotations.
>
>It's a little incongruous that GETMETADATA is used only to retrieve
>server annotations, but SETMETADATA can be used to set both server and
>mailbox annotations.  I understand that the mailbox-level function of
>GETMETADATA has been passed to the LIST-EXTENDED feature, but generally
>it's clearer when a GETFOO and SETFOO pair cover the same target space.
>  
>
Yes, but the alternative is to have 2 ways of getting annotations, which 
is worse.

>   If the METADATA extension is present, support for shared annotations
>   is REQUIRED, whilst support for private annotations is OPTIONAL.
>   This recognizes the fact that support for private annotations may
>   introduce significantly more complexity to a server in terms of
>   tracking ownership of the annotations, how quota is determined for
>   users based on their own annotations etc.  Clients that support the
>   METADATA extension MUST handle both shared and private annotations.
>
>Is there a mechanism for informing clients that a server does not
>support private annotations?  Or is a client expected to determine this
>from METADATA NOPRIVATE response codes on a SETMETADATA NO response?
>  
>
Currently the latter.

>If the latter is the case, won't client implementations end up allowing
>users one attempt to set private annotations and only then inform them
>that the functionality is not supported by the server?
>
Clients can try to workaround this by trying to set an annotation to a 
fake value.
Yes, this doesn't look very nice, but works.

Defining a new capability wouldn't work, because in general case 
NOPRIVATE can be namespace or even mailbox-specific.

We can add a new \NoPrivate per mailbox attribute to return this in LIST?

>  (This would probably occur once per IMAP session...)
>
>
>   If the server is unable to set an annotation because the size of its
>   value is too large, the server MUST return a tagged NO response with
>   a "[METADATA TOOBIG]" response code.
>
>Likewise, is there a mechanism for a server to tell its clients the
>maximum annotation length it supports?  Or is it up to the client to
>determine this from METADATA TOOBIG error code responses?
>  
>
The latter.
Also note, that the server might return TOOBIG because it simply ran out 
of storage space, so advertising the maximum might not be that helpful 
to the client.

>   Entry and attribute names are case-sensitive.
>
>   Entry and attribute names MUST NOT contain asterisk ("*") or percent
>   ("%") characters and MUST NOT contain non-ASCII characters or the NUL
>   octet.  Invalid entry or attribute names result in a BAD response in
>   any IMAP commands where they are used.
>
>   Use of control or punctuation characters in entry and attribute names
>   is strongly discouraged.
>
>It's probably cleaner in the long run to use the BNF to explicitly
>specify a limited set of permitted characters for entry and attribute
>names.  Constraining them to something like "atom" plus the delimiters
>is probably not a bad idea.
>
>Given that these names are already constrained to ASCII,
>
I don't think this is the case! Annotation names can contain UTF-8.

> I'd favor eliminating the case-sensitivity requirement.  It differs from other
>IMAP commands and extensions' general consistent overall case-
>insensitivity (e.g. RFC 3501 keywords).
>
>
>   value
>      A string or binary data representing the value of the annotation.
>      To delete an annotation, the client can store "NIL" into the
>      value.  The content represented by the string is determined by the
>      Content-Type used to register the entry (see Section 6.1 for entry
>      registration templates).  Where applicable, the registered
>      Content-Type MUST include a charset parameter.  Text values SHOULD
>      use the utf-8 [RFC3629] character set.
>
>Is there any reason not to mandate UTF-8 for text values?
>
Values can also be binary and not every binary value is a valid UTF-8.

>  We seem to be
>moving towards standardizing on UTF-8 in IMAP, and this seems a good
>opportunity to draw a line.
>  
>
"Values for annotations published on Standard track MUST use charset=UTF-8"?
This still allows vendors to do whatever they want with their private 
annotations.

>   /sort
>      Defines the default sort criteria [I-D.ietf-imapext-sort] to use
>      when first displaying the mailbox contents to the user, or NIL if
>      sorting is not required.  Note that the presence of this attribute
>      does not imply that the server supports the IMAP SORT
>      [I-D.ietf-imapext-sort] extension - servers can choose to support
>      or not support SORT independently of this extension.  In the
>      absence of the SORT extension, clients can choose to interpret
>      this entry as suggesting a client-side sort ordering of the
>      corresponding mailbox.
>
>   /thread
>      Defines the default thread criteria [I-D.ietf-imapext-sort] to use
>      when first displaying the mailbox contents to the user, or NIL if
>      threading is not required.  If both sort and thread are not NIL,
>      then threading should take precedence over sorting.  Note that the
>      presence of this attribute does not imply that the server supports
>      the IMAP THREAD [I-D.ietf-imapext-sort] extension - servers can
>      choose to support or not support THREAD independently of this
>      extension.  In the absence of the THREAD extension, clients can
>      choose to interpret this entry as a client-side thread ordering of
>      the corresponding mailbox.
>
>These registered annotations are odd ducks.  They seem most appropriate
>as private annotations on a mailbox, but in practice they may be
>confusing to the end user because sorting and threading preferences are
>often client-specific, not user-specific.
>
>For instance, a user may prefer to read their mail nonthreaded on the
>desktop, but on a mobile device they may want to make use of the reduced
>visual footprint of a threaded display.  Likewise, more likely than not
>a user won't expect their sort preferences to automatically be
>transferred between their devices and clients.
>  
>
I would let other people comment on this.

>   Mailboxes or the server as a whole may have zero or more annotations
>   associated with them.  An annotation contains a uniquely named entry
>   with a set of uniquely named attributes, each of which has a value.
>   Annotations can be added to mailboxes when a mailbox name is provided
>   as the first argument to the new command, or to the server as a whole
>   when the empty string is provided as the first argument to the new
>   command.
>
>Just a nitpick here, but you should probably refer to SETMETADATA
>command by name since (a) it hasn't been introduced at this point in the
>document and (b) there is more than one new command added by the draft.
>  
>
Agreed.

>   Each annotation is an entry that has a hierarchical name, with each
>   component of the name separated by a slash ("/").  An entry name MUST
>   NOT contain two consecutive "/" characters and MUST NOT end with a
>   "/" character.
>
>   Each entry is made up of a set of attributes.  Each attribute has a
>   hierarchical name, with each component of the name separated by a
>   period (".").  An attribute name MUST NOT contain two consecutive "."
>   characters and MUST NOT end with a "." character.
>
>My last comment is personal preference: I fear that this may be a bit of
>overkill for folder annotation.  I understand the need for this type of
>flexible syntax for ACAP's purposes, but the use cases for IMAP appear
>significantly more limited in scope.  In accordance with the KISS
>philosophy, is there any chance I might be able to convince everyone to
>consider shedding a layer of hierarchy here?
>  
>
This is not going to be a problem in practice. IMHO, new attributes are 
unlikely to be standardized.
IMHO, the current model of value.priv + value.shared per annotation 
should work for 99% cases.

Date: Mon, 9 Apr 2007 08:50:10 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: [lemonade] Deprecate IMAP IDLE?
Cc: Eric Burger <[email-masked]>
Sender: [email-masked]

Eric Burger writes:
> Because of this, it was proposed that IMAP NOTIFY would supercede IMAP 
> IDLE, leading to its deprecation.
>
> What do we think of this? As chair, my only care is whether we say it 
> obsoletes RFC 2177 or not.

I think obsoleting 2177 is out of scope for Lemonade. All Lemonade 
can/should do is say things in profile-bis.

Arnt

Date: Fri, 06 Apr 2007 14:38:35 -0400
From: Barry Leiba <[email-masked]>
To: [email-masked]
Subject: Re: friends of imap dinner tonight
Sender: [email-masked]

A little late here, but...

> It was a pleasant dinner.

I'm sure it was, and I'm sorry to've missed it.  Perhaps in future, we 
should make sure that the friends of IMAP and the friends of DKIM 
coordinate, so those of us who have lots of friends can do both.

:-)

Barry

Date: Mon, 2 Apr 2007 21:31:14 -0700 (PDT)
From: Dan Karp <[email-masked]>
To: Eric Burger <[email-masked]>
Cc: Enhancements to Internet email to support diverse service enivronments <[email-masked]>, IMAP Extensions WG <[email-masked]>
Subject: Re: [lemonade] Expert Review: annotatemore (like WGLC)
Sender: [email-masked]

> http://www.ietf.org/internet-drafts/draft-daboo-imap-annotatemore-11.txt

Comments interspersed btween snippets of the draft:

   Changes from -10 to -11:
   7.   Added match type and collation identifier to the LIST-EXTENDED
        selection option.
   8.   Made support for IMAP-I18N a requirement.

   Text comparisons may be done as part of the GETMETADATA or LIST-
   EXTENDED commands.  As a result, support for the COMPARATOR
   [I-D.ietf-imapext-i18n] extension is REQUIRED.

GETMETADATA doesn't actually do text comparisons, right?  The BNF
doesn't seem to tie in the "matchtype" or "collation" productions.

But more to the point, I believe that the intention was that METADATA
should be a simple, baseline extenstion that will be required by a wide
range of other extensions.  If that's the case, *PLEASE* reconsider the
wisdom of adding the LT, LE, GT, and GE matchers (and thus the
COMPARATOR requirement) to this draft.  It doesn't apear to be core
functionality for METADATA, and since COMPARATOR appears nontrivial to
implement this may not only prevent server authors from implementing
METADATA but also keep us from impelenting any future extension that
requires METADATA.

   A user can only set and retrieve private or shared annotations on a
   mailbox which exists and is returned to them via a LIST or LSUB
   command, irrespective of whether they have read or write access to
   the actual message content of the mailbox.  If the client attempts to
   set or retrieve annotations on other mailboxes, the server MUST
   respond with a NO response.

I can understand the logic behind requiring LIST/LSUB access for reading
public mailbox annotations and for setting private annotations.  But
that seems to be a very low bar for setting public mailbox annotations.
Adding a new ACL permission for setting annotations would make this
extension more complex and would also introduce a dependency on the ACL
extension, but perhaps requiring [READ-WRITE] access to the mailbox
might be reasonable?

   The new SETMETADATA command and the METADATA response each have a
   mailbox name argument, indicating that the annotations being referred
   to are attached to the specified mailbox.  An empty string can be
   used for the mailbox name to signify server annotations.

What rights are required to set public and private annotations on the
server?

   This command sets the specified list of entries by adding or
   replacing the specified attributes with the values provided, on the
   specified existing mailboxes.  Clients can use NIL for values of
   attributes it wants to remove from entries.  The server MAY return a
   METADATA response containing the updated annotation data.

In this case, the server may return a METADATA response that doesn't
contain the updated annotation data or a LIST response that does,
correct?

   This extension adds the GETMETADATA command.  This allows clients to
   retrieve server annotations.  Mailbox annotations are retrieved via
   the extended LIST command described in the next section.

   This extension adds the SETMETADATA command.  This allows clients to
   set annotations.

It's a little incongruous that GETMETADATA is used only to retrieve
server annotations, but SETMETADATA can be used to set both server and
mailbox annotations.  I understand that the mailbox-level function of
GETMETADATA has been passed to the LIST-EXTENDED feature, but generally
it's clearer when a GETFOO and SETFOO pair cover the same target space.

   If the METADATA extension is present, support for shared annotations
   is REQUIRED, whilst support for private annotations is OPTIONAL.
   This recognizes the fact that support for private annotations may
   introduce significantly more complexity to a server in terms of
   tracking ownership of the annotations, how quota is determined for
   users based on their own annotations etc.  Clients that support the
   METADATA extension MUST handle both shared and private annotations.

Is there a mechanism for informing clients that a server does not
support private annotations?  Or is a client expected to determine this
from METADATA NOPRIVATE response codes on a SETMETADATA NO response?

If the latter is the case, won't client implementations end up allowing
users one attempt to set private annotations and only then inform them
that the functionality is not supported by the server?  (This would
probably occur once per IMAP session...)

   If the server is unable to set an annotation because the size of its
   value is too large, the server MUST return a tagged NO response with
   a "[METADATA TOOBIG]" response code.

Likewise, is there a mechanism for a server to tell its clients the
maximum annotation length it supports?  Or is it up to the client to
determine this from METADATA TOOBIG error code responses?

   Entry and attribute names are case-sensitive.

   Entry and attribute names MUST NOT contain asterisk ("*") or percent
   ("%") characters and MUST NOT contain non-ASCII characters or the NUL
   octet.  Invalid entry or attribute names result in a BAD response in
   any IMAP commands where they are used.

   Use of control or punctuation characters in entry and attribute names
   is strongly discouraged.

It's probably cleaner in the long run to use the BNF to explicitly
specify a limited set of permitted characters for entry and attribute
names.  Constraining them to something like "atom" plus the delimiters
is probably not a bad idea.

Given that these names are already constrained to ASCII, I'd favor
eliminating the case-sensitivity requirement.  It differs from other
IMAP commands and extensions' general consistent overall case-
insensitivity (e.g. RFC 3501 keywords).

   value
      A string or binary data representing the value of the annotation.
      To delete an annotation, the client can store "NIL" into the
      value.  The content represented by the string is determined by the
      Content-Type used to register the entry (see Section 6.1 for entry
      registration templates).  Where applicable, the registered
      Content-Type MUST include a charset parameter.  Text values SHOULD
      use the utf-8 [RFC3629] character set.

Is there any reason not to mandate UTF-8 for text values?  We seem to be
moving towards standardizing on UTF-8 in IMAP, and this seems a good
opportunity to draw a line.

   /sort
      Defines the default sort criteria [I-D.ietf-imapext-sort] to use
      when first displaying the mailbox contents to the user, or NIL if
      sorting is not required.  Note that the presence of this attribute
      does not imply that the server supports the IMAP SORT
      [I-D.ietf-imapext-sort] extension - servers can choose to support
      or not support SORT independently of this extension.  In the
      absence of the SORT extension, clients can choose to interpret
      this entry as suggesting a client-side sort ordering of the
      corresponding mailbox.

   /thread
      Defines the default thread criteria [I-D.ietf-imapext-sort] to use
      when first displaying the mailbox contents to the user, or NIL if
      threading is not required.  If both sort and thread are not NIL,
      then threading should take precedence over sorting.  Note that the
      presence of this attribute does not imply that the server supports
      the IMAP THREAD [I-D.ietf-imapext-sort] extension - servers can
      choose to support or not support THREAD independently of this
      extension.  In the absence of the THREAD extension, clients can
      choose to interpret this entry as a client-side thread ordering of
      the corresponding mailbox.

These registered annotations are odd ducks.  They seem most appropriate
as private annotations on a mailbox, but in practice they may be
confusing to the end user because sorting and threading preferences are
often client-specific, not user-specific.

For instance, a user may prefer to read their mail nonthreaded on the
desktop, but on a mobile device they may want to make use of the reduced
visual footprint of a threaded display.  Likewise, more likely than not
a user won't expect their sort preferences to automatically be
transferred between their devices and clients.

   Mailboxes or the server as a whole may have zero or more annotations
   associated with them.  An annotation contains a uniquely named entry
   with a set of uniquely named attributes, each of which has a value.
   Annotations can be added to mailboxes when a mailbox name is provided
   as the first argument to the new command, or to the server as a whole
   when the empty string is provided as the first argument to the new
   command.

Just a nitpick here, but you should probably refer to SETMETADATA
command by name since (a) it hasn't been introduced at this point in the
document and (b) there is more than one new command added by the draft.

   Each annotation is an entry that has a hierarchical name, with each
   component of the name separated by a slash ("/").  An entry name MUST
   NOT contain two consecutive "/" characters and MUST NOT end with a
   "/" character.

   Each entry is made up of a set of attributes.  Each attribute has a
   hierarchical name, with each component of the name separated by a
   period (".").  An attribute name MUST NOT contain two consecutive "."
   characters and MUST NOT end with a "." character.

My last comment is personal preference: I fear that this may be a bit of
overkill for folder annotation.  I understand the need for this type of
flexible syntax for ACAP's purposes, but the use cases for IMAP appear
significantly more limited in scope.  In accordance with the KISS
philosophy, is there any chance I might be able to convince everyone to
consider shedding a layer of hierarchy here?