Subject: Re: [Imap-protocol] Private FETCH items
From: Tony Finch <[email-masked]>
To: [email-masked], IMAP Protocol Interest List <[email-masked]>
Date: Mon, 27 Aug 2007 11:31:20 +0100
List-Id: <ietf-imapext.imc.org>

On Thu, 23 Aug 2007, Ken Murchison wrote:
>
> I probably wasn't clear in the use case.  Cyrus now has the ability to
> replicate a mailstore in near realtime.  Several sites (including CMU in the
> near future) are using this to have redundant mailstores in case of a hardware
> failure.  Rather than create a new tool or protocol to verify the consistency
> of the mailstore pairs, extending IMAP with a couple of new FETCH items seems
> like a reasonable thing to do.

I should note that the MD5 replica consistency check has been very useful
in finding lurking bugs in the code, both the original version and across
versions as the code has evolved. It's also good for discovering bit-
smashing hardware failures that haven't (yet) affected lower levels of the
os/driver/raid stack. You can only be sure of data integrity if you test
it end-to-end.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
IRISH SEA: SOUTHERLY, BACKING NORTHEASTERLY FOR A TIME, 3 OR 4. SLIGHT OR
MODERATE. SHOWERS. MODERATE OR GOOD, OCCASIONALLY POOR.

Date: Sun, 26 Aug 2007 16:12:41 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: UID SEARCH responses
Cc: Alexey Melnikov <[email-masked]>, Pawel Salek <[email-masked]>
Sender: [email-masked]

Alexey Melnikov writes:
> I can't think of any difference for standard commands.

I thought of two differences, both insignificant.

1. If the client has a cache and doesn't know which cached UIDs 
represent extant messages, it cannot use an ESEARCH response to 
retrieve messages from its cache. It has to clean its cache of expunged 
messages (as it should do anyway).

2. SEARCH implicitly returns a count of results, ESEARCH ALL does not, 
so clients who want to display a count of results have to use both 
COUNT and ALL.

Someday I'm going to check what my server does. I suspect it may return 
the shorter response where UID ranges may include nonexistsent 
messages, but I don't know. Or care. IMO this is a nonissue.

Arnt

Date: Fri, 24 Aug 2007 16:42:18 -0400
From: Ken Murchison <[email-masked]>
Organization: Carnegie Mellon University
To: Arnt Gulbrandsen <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: Private FETCH items
Sender: [email-masked]

Arnt Gulbrandsen wrote:
> Ken Murchison writes:
>> Arnt Gulbrandsen wrote:
>>> What does the GUID do?
>>
>> Uniquely identifies a message so that if the same message is present 
>> in more than one mailbox, its contents only gets replicated once.
> 
> I'm afraid I phrased the question badly. What does the code do when 
> asked for a GUID?

Pulls the message GUID out of the index file for the mailbox.

-- 
Kenneth Murchison
Systems Programmer
Project Cyrus Developer/Maintainer
Carnegie Mellon University

To: IMAP Extensions WG <[email-masked]>
From: Lisa Dusseault <[email-masked]>
Subject: Fwd: Last Call: draft-saintandre-jabberid (The Jabber-ID Header  Field) to Proposed Standard 
Date: Fri, 24 Aug 2007 09:56:20 -0700
Sender: [email-masked]

--Apple-Mail-10--786017334
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
	charset=US-ASCII;
	delsp=yes;
	format=flowed

FYI on this last call document relating to email header fields.   
Please note that some review on the Apps-review team list has already  
suggested this go to Experimental instead  (http://www1.ietf.org/mail- 
archive/web/apps-review/current/msg00062.html ).

Thanks,
Lisa

Begin forwarded message:
From: The IESG <[email-masked]>
Date: August 24, 2007 6:51:05 AM PDT
To: IETF-Announce <[email-masked]>
Subject: Last Call: draft-saintandre-jabberid (The Jabber-ID Header   
Field) to Proposed Standard
Reply-To: [email-masked]

The IESG has received a request from an individual submitter to consider
the following document:

- 'The Jabber-ID Header Field '
    <draft-saintandre-jabberid-05.txt> as a Proposed Standard

The IESG plans to make a decision in the next few weeks, and solicits
final comments on this action.  Please send substantive comments to the
[email-masked] mailing lists by 2007-09-21. Exceptionally,
comments may be sent to [email-masked] instead. In either case, please
retain the beginning of the Subject line to allow automated sorting.

The file can be obtained via
http://www.ietf.org/internet-drafts/draft-saintandre-jabberid-05.txt

IESG discussion can be tracked via
https://datatracker.ietf.org/public/pidtracker.cgi? 
command=view_id&dTag=14260&rfc_flag=0

_______________________________________________
IETF-Announce mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf-announce

--Apple-Mail-10--786017334
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=ISO-8859-1

<HTML><BODY style=3D"word-wrap: break-word; -khtml-nbsp-mode: space; =
-khtml-line-break: after-white-space; "><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">FYI on this =
last call document relating to email header fields.=A0 Please note that =
some review on the Apps-review team list has already suggested this go =
to Experimental instead=A0 (<A =
href=3D"http://www1.ietf.org/mail-archive/web/apps-review/current/msg00062=
.html"><FONT class=3D"Apple-style-span" =
color=3D"#0020E2">http://www1.ietf.org/mail-archive/web/apps-review/curren=
t/msg00062.html</FONT></A> ).</DIV><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal =
normal normal 12px/normal Helvetica; min-height: 14px; "><BR></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; ">Thanks,</DIV><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
">Lisa</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
margin-bottom: 0px; margin-left: 0px; font: normal normal normal =
12px/normal Helvetica; min-height: 14px; "><BR></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; ">Begin forwarded message:</DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; "><B>From: </B><FONT class=3D"Apple-style-span" =
color=3D"#001ED2">The IESG &lt;</FONT><A =
href=3D"mailto:[email-masked]"><FONT class=3D"Apple-style-span" =
color=3D"#0020E2">[email-masked]</FONT></A><FONT =
class=3D"Apple-style-span" color=3D"#001ED2">&gt;</FONT></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; "><B>Date: </B><FONT class=3D"Apple-style-span" =
color=3D"#001ED2">August 24, 2007 6:51:05 AM PDT</FONT></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; "><B>To: </B><FONT class=3D"Apple-style-span" =
color=3D"#001ED2">IETF-Announce &lt;</FONT><A =
href=3D"mailto:[email-masked]"><FONT class=3D"Apple-style-span" =
color=3D"#0020E2">[email-masked]</FONT></A><FONT =
class=3D"Apple-style-span" color=3D"#001ED2">&gt;</FONT></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; "><B>Subject: </B><FONT class=3D"Apple-style-span" =
color=3D"#001ED2"><B>Last Call: draft-saintandre-jabberid (The Jabber-ID =
Header</B></FONT><FONT class=3D"Apple-style-span" color=3D"#001ED2"><B>=A0=
 </B></FONT><FONT class=3D"Apple-style-span" color=3D"#001ED2"><B>Field) =
to Proposed Standard</B></FONT><FONT class=3D"Apple-style-span" =
color=3D"#001ED2"><B>=A0</B></FONT></DIV><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><B>Reply-To: =
</B><A href=3D"mailto:[email-masked]"><FONT class=3D"Apple-style-span" =
color=3D"#0020E2">[email-masked]</FONT></A></DIV><DIV style=3D"margin-top: =
0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: =
normal normal normal 12px/normal Helvetica; min-height: 14px; "><FONT =
class=3D"Apple-style-span" color=3D"#001ED2"><BR></FONT></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; "><FONT class=3D"Apple-style-span" color=3D"#001ED2">The=
 IESG has received a request from an individual submitter to =
consider</FONT><FONT class=3D"Apple-style-span" =
color=3D"#001ED2">=A0</FONT></DIV><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT =
class=3D"Apple-style-span" color=3D"#001ED2">the following =
document:</FONT></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
margin-bottom: 0px; margin-left: 0px; font: normal normal normal =
12px/normal Helvetica; min-height: 14px; "><FONT =
class=3D"Apple-style-span" color=3D"#001ED2"><BR></FONT></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; "><FONT class=3D"Apple-style-span" color=3D"#001ED2">- =
'The Jabber-ID Header Field '</FONT></DIV><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT =
class=3D"Apple-style-span" color=3D"#001ED2">=A0=A0 </FONT><FONT =
class=3D"Apple-style-span" =
color=3D"#001ED2">&lt;draft-saintandre-jabberid-05.txt&gt; as a Proposed =
Standard</FONT></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
margin-bottom: 0px; margin-left: 0px; font: normal normal normal =
12px/normal Helvetica; min-height: 14px; "><FONT =
class=3D"Apple-style-span" color=3D"#001ED2"><BR></FONT></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; "><FONT class=3D"Apple-style-span" color=3D"#001ED2">The=
 IESG plans to make a decision in the next few weeks, and =
solicits</FONT></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
margin-bottom: 0px; margin-left: 0px; "><FONT class=3D"Apple-style-span" =
color=3D"#001ED2">final comments on this action.</FONT><FONT =
class=3D"Apple-style-span" color=3D"#001ED2">=A0 </FONT><FONT =
class=3D"Apple-style-span" color=3D"#001ED2">Please send substantive =
comments to the</FONT></DIV><DIV style=3D"margin-top: 0px; margin-right: =
0px; margin-bottom: 0px; margin-left: 0px; "><A =
href=3D"mailto:[email-masked]">[email-masked]</A><FONT =
class=3D"Apple-style-span" color=3D"#001ED2"> mailing lists by =
2007-09-21. Exceptionally,</FONT><FONT class=3D"Apple-style-span" =
color=3D"#001ED2">=A0</FONT></DIV><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT =
class=3D"Apple-style-span" color=3D"#001ED2">comments may be sent to =
</FONT><A href=3D"mailto:[email-masked]"><FONT class=3D"Apple-style-span" =
color=3D"#0020E2">[email-masked]</FONT></A><FONT class=3D"Apple-style-span"=
 color=3D"#001ED2"> instead. In either case, please</FONT><FONT =
class=3D"Apple-style-span" color=3D"#001ED2">=A0</FONT></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; "><FONT class=3D"Apple-style-span" =
color=3D"#001ED2">retain the beginning of the Subject line to allow =
automated sorting.</FONT></DIV><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal =
normal normal 12px/normal Helvetica; min-height: 14px; "><FONT =
class=3D"Apple-style-span" color=3D"#001ED2"><BR></FONT></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; "><FONT class=3D"Apple-style-span" color=3D"#001ED2">The=
 file can be obtained via</FONT></DIV><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><A =
href=3D"http://www.ietf.org/internet-drafts/draft-saintandre-jabberid-05.t=
xt">http://www.ietf.org/internet-drafts/draft-saintandre-jabberid-05.txt</=
A></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: =
0px; margin-left: 0px; font: normal normal normal 12px/normal Helvetica; =
min-height: 14px; "><FONT class=3D"Apple-style-span" =
color=3D"#001ED2"><BR></FONT></DIV><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal =
normal normal 12px/normal Helvetica; min-height: 14px; "><FONT =
class=3D"Apple-style-span" color=3D"#001ED2"><BR></FONT></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; "><FONT class=3D"Apple-style-span" =
color=3D"#001ED2">IESG discussion can be tracked via</FONT></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; "><A =
href=3D"https://datatracker.ietf.org/public/pidtracker.cgi?command=3Dview_=
id&dTag=3D14260&rfc_flag=3D0">https://datatracker.ietf.org/public/pidtrack=
er.cgi?command=3Dview_id&amp;dTag=3D14260&amp;rfc_flag=3D0</A></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; font: normal normal normal 12px/normal Helvetica; =
min-height: 14px; "><FONT class=3D"Apple-style-span" =
color=3D"#001ED2"><BR></FONT></DIV><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal =
normal normal 12px/normal Helvetica; min-height: 14px; "><FONT =
class=3D"Apple-style-span" color=3D"#001ED2"><BR></FONT></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; "><FONT class=3D"Apple-style-span" =
color=3D"#001ED2">_______________________________________________</FONT></=
DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: =
0px; margin-left: 0px; "><FONT class=3D"Apple-style-span" =
color=3D"#001ED2">IETF-Announce mailing list</FONT></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; "><A =
href=3D"mailto:[email-masked]">[email-masked]</A></DIV><DI=
V style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; "><A =
href=3D"https://www1.ietf.org/mailman/listinfo/ietf-announce">https://www1=
.ietf.org/mailman/listinfo/ietf-announce</A></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; "><BR =
class=3D"khtml-block-placeholder"></DIV></BODY></HTML>=

--Apple-Mail-10--786017334--

Date: Fri, 24 Aug 2007 11:41:18 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: Private FETCH items
Cc: Ken Murchison <[email-masked]>, [email-masked]
Sender: [email-masked]

Ken Murchison writes:
> Arnt Gulbrandsen wrote:
>> What does the GUID do?
>
> Uniquely identifies a message so that if the same message is present 
> in more than one mailbox, its contents only gets replicated once.

I'm afraid I phrased the question badly. What does the code do when 
asked for a GUID?

Arnt

Date: Thu, 23 Aug 2007 10:34:06 -0400
From: Ken Murchison <[email-masked]>
Organization: Carnegie Mellon University
To: Arnt Gulbrandsen <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: Private FETCH items
Sender: [email-masked]

Arnt Gulbrandsen wrote:
> Sounds to me as the patch author really wants a FETCH COOKIE instead of 
> MD5/FILESIZE, returning an opaque cookie which should be the same if the 
> same message is present in two mailboxes/servers and different otherwise.

Correct.

> What does the GUID do?

Uniquely identifies a message so that if the same message is present in 
more than one mailbox, its contents only gets replicated once.

-- 
Kenneth Murchison
Systems Programmer
Project Cyrus Developer/Maintainer
Carnegie Mellon University

Date: Thu, 23 Aug 2007 10:32:53 -0400
From: Ken Murchison <[email-masked]>
Organization: Carnegie Mellon University
To: Lyndon Nerenberg <[email-masked]>
CC: IMAP Protocol Interest List <[email-masked]>, [email-masked]
Subject: Re: [Imap-protocol] Private FETCH items
Sender: [email-masked]

Lyndon Nerenberg wrote:
> 
> On Aug 22, 2007, at 12:05 PM, Ken Murchison wrote:
> 
>> FYI, the difference between RFC822.SIZE and RFC822.FILESIZE is that 
>> the former uses the cached size in the message index file, and the 
>> latter stat()s for the file size at the time of the request.
> 
> .FILESIZE is just  asking for trouble.  Nobody cares what the on-disk 
> representation is, and comparing file sizes is NOT the same as comparing 
> message content.

True.  The author of the patch may be using this for something other 
than what I was thinking.  I'll have to ask.

> .MD5 is useful in the general sense as a fast way to compare messages 
> for equality, although I think it would be better served in that context 
> if you could MD5 the body content without the 822 headers to be able to 
> handle Received: differences for the same message delivered to two or 
> more folders via different [SL]MTP transactions.  It would probably make 
> more sense to be able to generically MD5 by MIME body part.

I probably wasn't clear in the use case.  Cyrus now has the ability to 
replicate a mailstore in near realtime.  Several sites (including CMU in 
the near future) are using this to have redundant mailstores in case of 
a hardware failure.  Rather than create a new tool or protocol to verify 
the consistency of the mailstore pairs, extending IMAP with a couple of 
new FETCH items seems like a reasonable thing to do.

> What's a GUID?

As Mark deduced, its Global Unique ID

-- 
Kenneth Murchison
Systems Programmer
Project Cyrus Developer/Maintainer
Carnegie Mellon University

Date: Thu, 23 Aug 2007 09:03:30 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: Private FETCH items
Cc: Ken Murchison <[email-masked]>, [email-masked]
Sender: [email-masked]

Sounds to me as the patch author really wants a FETCH COOKIE instead of 
MD5/FILESIZE, returning an opaque cookie which should be the same if 
the same message is present in two mailboxes/servers and different 
otherwise.

What does the GUID do?

Arnt

Date: Wed, 22 Aug 2007 14:14:14 -0700
From: Mark Crispin <[email-masked]>
To: Lyndon Nerenberg <[email-masked]>
cc: Ken Murchison <[email-masked]>, [email-masked], IMAP Protocol Interest List <[email-masked]>
Subject: Re: [Imap-protocol] Private FETCH items
Organization: Networks & Distributed Computing
Sender: [email-masked]

On Wed, 22 Aug 2007, Lyndon Nerenberg wrote:
> .FILESIZE is just  asking for trouble.  Nobody cares what the on-disk 
> representation is, and comparing file sizes is NOT the same as comparing 
> message content.

I agree.  The c-client library (the beating heart of UW imapd, Pine, etc.) 
has a concept of an internal size, but that depends upon the mail store 
and thus is only used by mail store drivers.  It never gets as far as 
imapd (or other application), much less exported in the protocol.

Let us be clear here; any .FILESIZE value in any server which supports 
multiple mail store formats could (and WOULD) vary depending upon the 
mailbox format with no obvious indication to the user as to why.  This, by 
itself, should be enough cause to reject it.

> .MD5 is useful in the general sense as a fast way to compare messages for 
> equality, although I think it would be better served in that context if you 
> could MD5 the body content without the 822 headers to be able to handle 
> Received: differences for the same message delivered to two or more folders 
> via different [SL]MTP transactions.  It would probably make more sense to be 
> able to generically MD5 by MIME body part.

IMHO, this functionality belongs as part of CONVERT.  An MD5 checksum is, 
in effect, a conversion.

> What's a GUID?

Presumably this is a global UID which uniquely and permanently identifies 
the message on the server.

I think that IMAP should eventually have a GUID, but it needs careful 
design and review.  I don't think that we should take the first patch that 
comes along.

> But overall I guess I have to question why it's necessary to burden the 
> protocol with this just because somebody wrote some broken code.  I have done 
> an awful lot of server migrations over the years and I have never yet come up 
> against the problem this is trying to detect. I realize you intend this as 
> private data, but once it's in the wild, it will get used, for better and 
> worse.

I agree with this sentiment.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Subject: Re: [Imap-protocol] Private FETCH items
Date: Wed, 22 Aug 2007 22:13:42 +0100
From: Dave Cridland <[email-masked]>
To: Lyndon Nerenberg <[email-masked]>
Cc: <[email-masked]>, <[email-masked]>, Ken Murchison <[email-masked]>
Sender: [email-masked]

On Wed Aug 22 20:27:15 2007, Lyndon Nerenberg wrote:
> What's a GUID?
> 
> 
Past participle of the verb "to gooey".

> But overall I guess I have to question why it's necessary to burden 
>  the protocol with this just because somebody wrote some broken 
> code.   I have done an awful lot of server migrations over the 
> years and I  have never yet come up against the problem this is 
> trying to detect.  I realize you intend this as private data, but 
> once it's in the wild,  it will get used, for better and worse.

I can understand why adding private fetchable data would be in some 
way useful, though, irrespective of this particular case.

I'd personally opt for an X- prefix, at the very least. X-CMU- might 
be better. Then you have X-CMU-UID, X-CMU-CRC, X-CMU-STATSIZE, 
perhaps.

Note that I've proposed "CRC" instead of explicitly naming MD5, 
partly in case people are tempted to use it in clients, and partly to 
stop the flurry of emails you'll otherwise get telling you that MD5 
is insecure.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Cc: IMAP Protocol Interest List <[email-masked]>, [email-masked]
From: Lyndon Nerenberg <[email-masked]>
Subject: Re: [Imap-protocol] Private FETCH items
Date: Wed, 22 Aug 2007 12:27:15 -0700
To: Ken Murchison <[email-masked]>
Sender: [email-masked]

On Aug 22, 2007, at 12:05 PM, Ken Murchison wrote:

> FYI, the difference between RFC822.SIZE and RFC822.FILESIZE is that  
> the former uses the cached size in the message index file, and the  
> latter stat()s for the file size at the time of the request.

.FILESIZE is just  asking for trouble.  Nobody cares what the on-disk  
representation is, and comparing file sizes is NOT the same as  
comparing message content.

.MD5 is useful in the general sense as a fast way to compare messages  
for equality, although I think it would be better served in that  
context if you could MD5 the body content without the 822 headers to  
be able to handle Received: differences for the same message  
delivered to two or more folders via different [SL]MTP transactions.   
It would probably make more sense to be able to generically MD5 by  
MIME body part.

What's a GUID?

But overall I guess I have to question why it's necessary to burden  
the protocol with this just because somebody wrote some broken code.   
I have done an awful lot of server migrations over the years and I  
have never yet come up against the problem this is trying to detect.  
I realize you intend this as private data, but once it's in the wild,  
it will get used, for better and worse.

--lyndon

Date: Wed, 22 Aug 2007 15:05:18 -0400
From: Ken Murchison <[email-masked]>
Organization: Carnegie Mellon University
To: [email-masked], IMAP Extensions WG <[email-masked]>
Subject: Private FETCH items
Sender: [email-masked]

Folks,

I was recently given a patch to Cyrus IMAP which adds 3 new FETCH items 
(GUID, RFC822.MD5, and RFC822.FILESIZE) for a message which helps with 
consistency checking when a message store is replicated.

I'm considering applying this patch, since it will be useful for CMU and 
others, but I'd like to solicit opinions regarding adding new FETCH 
items vs. making these items new ANNOTATE entries (although Cyrus 
doesn't currently implement ANNOTATE).

FYI, the difference between RFC822.SIZE and RFC822.FILESIZE is that the 
former uses the cached size in the message index file, and the latter 
stat()s for the file size at the time of the request.  I'm not thrilled 
with the name of this item, so I'd be open to suggestions.

-- 
Kenneth Murchison
Systems Programmer
Project Cyrus Developer/Maintainer
Carnegie Mellon University

Date: Wed, 15 Aug 2007 00:12:18 -0400
From: Tony Hansen <[email-masked]>
To: ietf-822 mailing list <[email-masked]>
CC: SMTP Interest Group <[email-masked]>, LEMONADE WG <[email-masked]>, IMAP extensions mailing list <[email-masked]>, EAI WG <[email-masked]>, POP3 extensions mailing list <[email-masked]>, USEFOR WG <[email-masked]>, SIMPLE WG <[email-masked]>
Subject: Re: Mailing List Last Call for 2822 update internet-draft
Sender: [email-masked]

The mailing list last call for the 2822 update is now over.

I believe we have mailing list consensus on the issues that have been
raised during the last call. After an updated draft is posted by Pete,
we'll be moving the draft forward to IETF last call.

	Tony

Tony Hansen wrote:
> There have been some comments on draft-resnick-2822upd-* but they have
> dwindled down to none.
> 
> This is a "formal" Mailing List Last Call on
> draft-resnick-2822upd-02.txt. The last call will last for two weeks
> time, ending on August 10, 2007.
> 
> The document will be discussed on the 822 mailing list,
> <[email-masked]>. Please send your comments there.

Date: Mon, 13 Aug 2007 16:20:40 -0700 (Pacific Daylight Time)
From: Mark Crispin <[email-masked]>
To: Alexey Melnikov <[email-masked]>
cc: IMAP Extensions WG <[email-masked]>
Subject: Re: UID SEARCH responses
Organization: Networks & Distributed Computing
Sender: [email-masked]

Is there a problem with this plan as outlined below?  If not, then let's 
just do it and be done with the discussion.  It is undoubtably getting 
dreary to most people.

You need to do this work anyway, due to what I coyly allued-to as the 
"second reason" why UIDPLUS could not use the sequence-set rule. 
Currently, the following response is valid:
 	* ESEARCH (TAG "A283") UID ALL 2,10:*
Oops.

On Mon, 13 Aug 2007, Mark Crispin wrote:
> On Mon, 13 Aug 2007, Mark Crispin wrote:
>>> 2). Clarify which of the two choices should be used by ALL and add a new 
>>> return option for the other one.
>
> On second thought, this could work if you proceed as follows:
>
> Update RFC 4466 to include uid-set from RFC 4315.  Be sure to highlight that 
> uid-set MUST NOT include non-existent UIDs, whereas sequence-set does.
>
> Consider a possible sytax change for uid-set, e.g., using hyphens, to reduce 
> confusion.  That orphans UIDPLUS syntax, but the benefit may be worth not 
> adding another subtlety in IMAP.
>
> Define the new return option (and capability indicating its presence) to 
> return uid-set.
>
> As a quality of implementation note, a server which supports the new return 
> option maximally compresses in ALL.
>
> This maintains compatibility for all existing servers.  As a gesture of good 
> faith, I will pledge to implement the new return value as soon as it is 
> published as an RFC.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Date: Mon, 13 Aug 2007 16:05:08 -0700 (Pacific Daylight Time)
From: Mark Crispin <[email-masked]>
To: Dave Cridland <[email-masked]>
cc: Pawel Salek <[email-masked]>, [email-masked], Peter Coates <[email-masked]>
Subject: RE: UID SEARCH responses
Organization: Networks & Distributed Computing
Sender: [email-masked]

On Mon, 13 Aug 2007, Dave Cridland wrote:
> On Mon Aug 13 18:29:48 2007, Mark Crispin wrote:
>> 
>> On Mon, 13 Aug 2007, Dave Cridland wrote:
>>> Non-existent messages do not satisfy the SEARCH criteria
>> 
>> Why not in the case of UIDs?  They satisfy the criteria when they are 
>> arguments to the SEARCH commands.  Why don't they satisfy the criteria when 
>> the server returns it as a value?
>> 
>> 
> Eh? No, they don't. No message will satisfy a UID criteria containing as 
> argument a non-existent UID.

However, what we are talking about is not a single UID, but rather a 
range.  By definition in RFC 3501, messages can match a UID range, even if 
neither UID in the range currently exists.

> "UID SEARCH UID 1:*" gives you back all extant UIDs in the mailbox, not all 
> possible UIDs in the range. (Which'd be 1 through the highest UID existent at 
> the time of the execution of the command, of course).

No.  It would be the first assigned UID (not 1) through the highest UID 
existent at the time of the execution of the command.

> No, there's no inconsistency. Non-existent UIDs don't affect SEARCH's UID 
> criteria, and are explicitly ignored by the UID family of commands. 
> Otherwise, they mean what they say.

Where, in any of the IMAP specifications, is this "otherwise" stated? 
There is no such meaning in RFC 3501.  RFC 4731 is the first, and as yet 
only, document in which anyone claims that UID ranges work any differently 
in the SEARCH and UID commands in RFC 3501.

>> The ranges in question do not contain any messages which do not satisfy the 
>> search criteria.
> Sure. But two wrongs don't make a right, and in this case, the ranges do 
> contain UIDs which do not relate to messages which satisfy the search 
> criteria.

They may contain UIDs which do not relate to messages, just as the usage 
of UID sequence-set in RFC 3501.  Nonetheless, all UIDs which relate to 
messages satisfy the search criteria.

You're trying to have the same protocol element do two entirely different 
things.

>> RFC 3501 section 6.4.4 describes a command that takes a UID sequence-set 
>> argument with the same properties as the sequence-set in the UID commands 
>> (UID FETCH, UID STORE, UID SEARCH).  It explicitly states that ranges are 
>> permitted.
> No, sorry. It actually says:
>[snip]

The text that you quoted is from section 6.4.8, not 6.4.4.

RFC 3501 section 6.4.4 describes a command other than UID that takes UID 
ranges, but it does not separately state the sematics that are in 6.4.8. 
Either 6.4.4 does not have those semantics (and I think that a lot of 
people will complain should that suddenly happen), or the semantics of 
6.4.8 apply to more than just the UID command.

Your next argument will probably be that 6.4.4 inherits the semantics from 
6.4.8 because it is a command, and that server responses do not inherit 
semantics from commands.  If so, that raises quite a few interesting 
issues.  I don't think that you want to go there.

>> That may be just rationale.  Nonetheless, the ranges that my server returns 
>> are in complete compliance with that text, and there is NO text that 
>> prohibits those ranges.
> No, there's nothing that says that non-existent UIDs MAY be included, either.

Correct.  But, as we have seen many many times, the lack of specific 
permission to do something does not constitute prohibition.

> And if they aren't included, as I maintain the specification hints closest to 
> and intends, then they still can be used in this way.

I saw no such hints in RFC 4731.  I can't read minds; without an explicit 
statement of intentions I have to use my judgement.  Thus:

I read nothing in RFC 4731 about the ability to get the map.  The fact 
that the non-ESEARCH form of UID SEARCH is an accident; FETCH 1:* UID was 
the intended way to accomplish that.  It seemed pointless to have a third 
way to get the map.

All that was clear in RFC 4731 is that it attempts to compress the search 
results to be less chatty.  This is a real concern with mobile devices and 
users who have INBOXes with tens of thousands of messages.

It consider it absurd that you would intentionally make the mechanism 
*more* chatty in order to provide a triplicate of functionality.  Not only 
that, but with typical INBOXes (and *all* mailboxes on a server which does 
not assign consecutive UID values to a mailbox), ESEARCH ends up being 
more chatty than the non-ESEARCH!

>> I
>> see nothing in RFC 4315, other than the general principle of "don't do 
>> silly/harmful behavior", that precludes
>> 	[COPYUID 38201 1:5 4483:9857]
>> That needs to be fixed in that document.
> No, it doesn't - you called that one out explicitly, by stating that no 
> extraneous UIDs could be included.

That isn't what RFC 4315 prohibition on extraneous UIDs actually says, 
although clearly that was what it intended to mean.  This needs to be 
remedied.

Fortunately, there is a clear interpretation in RFC 4315 of what is 
correct.  I don't think that anyone can argue the benefits of any other 
interpretation.

Nonetheless, it is ill-specified/ambiguous and should be fixed.

> RFC2359 has similar language. That said, 
> it also states (in both cases) "[...] a UID set containing the UIDs [...]", 
> so it really needs that language, since otherwise you could include other 
> existent UIDs.

Again, there is no dispute as to what is correct; but the current wording 
in RFC 4315 is ill-specified/ambiguous.

>> However, in spite of RFC 4315's unfortunate use of the same syntax as 
>> sequence-set, it does NOT use the sequence-set rule.  It uses its own 
>> uid-set and uid-range rules.  There is a reason for that: it divorces 
>> UIDPLUS's return values from the semantics of sequence-set.
> And I still contend that there are no such semantics. The semantics you're 
> talking about are related to the behaviour of the UID command, and not the 
> sequence-set syntactic construct.

What, then, is the point for RFC 4315 to use uid-set?

Think carefully about this before you answer.  There are actually two 
reasons why RFC 4315 has its own uid-set rule.  I have stated one, but 
there is another.

That other reason also applies to ESEARCH.  That reason, if nothing else, 
should indicate to you why sequence-set is the wrong rule.

> That leaves the extended UID SEARCH as the tool for performing searches 
> (or obtaining the UID mapping) for areas where the UID mapping is not known, 
> but that's only useful if and only if the ALL result data does not contain 
> non-existent UIDs.

I agree with the former, but not the latter.  I can envision a UID only 
client doing ESEARCH and taking the returned sequence-set and feeding it 
straight to a UID FETCH command.

> SEARCH's UID criteria matches messages which have a UID which falls within 
> the set. I don't see why this mandates that clients must include any 
> non-existent UIDs in the set, although quite obviously doing so will not 
> effect the result.

Clients don't process the UID criterion in SEARCH; servers do.

Are you saying that SEARCH UID 18:23 must return an error if UID 21 does 
not exist?  That seems like a very bad idea to me.

If not, then the UID criterion in SEARCH handles sequence-set exactly as 
the UID command.  Why, now, is ESEARCH different?  What other sequence-set 
in IMAP behaves the same way.

> The UID command (or commands, depending on your thinking) provides explicit 
> behaviour in the presence of non-existent UIDs. It does not provide any 
> semantics on the sequence-set syntactic construct itself.

If that were the case, then it would not be possible for a invalid 
sequence-set with an invalid msgno to return a BAD response (and RFC 3501 
would be internally contradictory).  BAD indicates a protocol syntax 
error.  Command execution errors return a NO.

>>> It happens that non-existent UIDs are ignored by the server in commands, 
>>> but that's explicitly mentioned in the definitions of those commands.
>>> Server-provided sequence-sets never contain *, or non-existent UIDs.
>> Nowhere is this stated.  4466/4731 is the first occurrence.
>> UIDPLUS is not a precedent, since it returns uid-set instead of 
>> sequence-set.
> But RFC2359 did not, and managed just fine.

RFC 2359 returned RFC 2060's set (which is now sequence-set in RFC 3501), 
which was clearly a bug in RFC 2359 since a UIDPLUS did not, and could 
not, return a RFC 2060 set.

I made a point about there being two reasons why sequence-set (set) was 
wrong for UIDPLUS, and why it is wrong for ESEARCH.  Think about what may 
be the second reason.

Now, it is possible to claim that this second reason is not a bug in 
ESEARCH.  If so, then ESEARCH can't be used as a map-obtaining operation.

> But assuming I'm wrong, you're arguing that somehow, this search criterion 
> would cause only messages which satisfy the search criteria to be included in 
> results. Right?

Basically, this would have to work by explicitly defining(!!) that it 
switches on magic behavior to prevent collapsing of contiguous messages 
with non-contiguous UID values into ranges.  As this magic is explicitly 
defined, it can do as it wishes.

But on subsequent reflection, it's probably just to do Alexey's idea and 
have a return value that has this lesser form of collapse.  Magic is icky.

> Well, I'm not saying that the future is fully known to me, but I think a GUID 
> space is likely to be distinct from a UID space, too.

Why?  Is it to retain UID/UIDVALIDITY as a handle on a message that is 
shorter than a GUID?  IMHO, three handles on a message is at least one too 
many.

POP3 seems to have no problem with UIDs which are much longer than any 
IMAP UIDs and lack the strictly-ascending property of IMAP UIDs.  POP3's 
only UID operation is to fetch the map.  POP3 is far from dead (sigh!) and 
remains far more ubiquitous than IMAP (double sigh).

> either way, non-existent messages, 
> or references to messages which don't exist, are being included in the result 
> as passed to the client.

This is not an unheard-of occurance in real life.  "The university 
district encompasses all the streets from 37th to 55th" does not in any 
way imply that there is a 46th Street (there isn't, at least not in the 
University District in Seattle...46th Street does exist elsewhere).

"All meetings from July 2 through July 14 are cancelled" doesn't imply 
that there was a meeting scheduled on July 4 that is now cancelled.

Unless it is explicitly stated, I don't see how the usage of a range in 
sparse data in any way implies that all possible values within that range 
exist within the sparse data.

> There's a handful of cases where ESEARCH comes out bigger by a small number 
> of octets (the size of the correlator, possibly UID, and an extra E). In the 
> majority of cases, though, it's smaller, in many cases I've experienced, it's 
> drastically smaller, especially with the larger mailboxes we seem to be 
> dealing with these days.

It would certainly do so with non-INBOX mailboxes; or more specifically 
mailboxes that are mostly the targets of COPY/APPEND operations and are 
generally never pruned.

However, INBOX stands out as the 800 pound gorilla, especially for those 
users who don't use non-INBOX mailboxes.

> Oh, well I've only got 51069 messages right now, and my UID range isn't all 
> that sparse. Maybe my spam filtering is working.

How tolerant are you of false positives, and how much spam hits you?

With filtering, I see have about a 20-to-1 spam/non-spam ratio in my 
INBOX, and it's getting worse all the time.  I shudder to think of how 
much gets blocked by the filters.

Consequently, consecutive non-spam messages are a rarity in my mailbox. 
When they occur they are usually hot-topic threads that I end up deleting 
after reading.

That's the problem with dealing with anecdotal evidence.  You and I each 
have anecdotes, but that doesn't necessarily mean that either one is valid 
for the general case.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Date: Mon, 13 Aug 2007 13:46:23 -0700 (Pacific Daylight Time)
From: Mark Crispin <[email-masked]>
To: Dave Cridland <[email-masked]>
cc: [email-masked]
Subject: Re: 1:4294967295 (was Re: UID SEARCH responses)
Organization: Networks & Distributed Computing
Sender: [email-masked]

On Mon, 13 Aug 2007, Dave Cridland wrote:
>> On further study:
> No, please don't over analyze my reductios ad absurda (if you'll pardon my 
> butchering of Latin) - isn't that as bad as answering my rhetorical 
> questions?

Actually, it wasn't an absurd argument at all, and it raises some good 
points.

4294967295 is in many, but NOT all, ways equivalent to *.  Similarly, 1 is 
in many, but NOT all, ways equivalent to the UID associated with msgno 1.

The ways that they are not equivalent are interesting and unique; and the 
author of an IMAP client should be aware of these properties.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Subject: Re: 1:4294967295 (was Re: UID SEARCH responses)
Date: Mon, 13 Aug 2007 21:38:38 +0100
From: Dave Cridland <[email-masked]>
To: Mark Crispin <[email-masked]>
Cc: <[email-masked]>
Sender: [email-masked]

On Mon Aug 13 19:49:05 2007, Mark Crispin wrote:
> 
> On further study:

No, please don't over analyze my reductios ad absurda (if you'll 
pardon my butchering of Latin) - isn't that as bad as answering my 
rhetorical questions?

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Subject: RE: UID SEARCH responses
Date: Mon, 13 Aug 2007 21:38:15 +0100
From: Dave Cridland <[email-masked]>
To: Mark Crispin <[email-masked]>
Cc: Pawel Salek <[email-masked]>, <[email-masked]>, Peter Coates <[email-masked]>
Sender: [email-masked]

On Mon Aug 13 18:29:48 2007, Mark Crispin wrote:
> 
> On Mon, 13 Aug 2007, Dave Cridland wrote:
>> Non-existent messages do not satisfy the SEARCH criteria
> 
> Why not in the case of UIDs?  They satisfy the criteria when they 
> are arguments to the SEARCH commands.  Why don't they satisfy the 
> criteria when the server returns it as a value?
> 
> 
Eh? No, they don't. No message will satisfy a UID criteria containing 
as argument a non-existent UID.

"UID SEARCH UID 1:*" gives you back all extant UIDs in the mailbox, 
not all possible UIDs in the range. (Which'd be 1 through the highest 
UID existent at the time of the execution of the command, of course).

> That is an inconsistency between client-end and server-end.  Such 
> inconsistencies MUST be documented, carefully and explicitly.
> 
> 
No, there's no inconsistency. Non-existent UIDs don't affect SEARCH's 
UID criteria, and are explicitly ignored by the UID family of 
commands. Otherwise, they mean what they say.

> The ranges in question do not contain any messages which do not 
> satisfy the search criteria.
> 
> 
Sure. But two wrongs don't make a right, and in this case, the ranges 
do contain UIDs which do not relate to messages which satisfy the 
search criteria. The ALL result data isn't specified as "Does not 
return any messages which do not satisfy the SEARCH criteria", 
otherwise you'd be right on the money.

> You are being hung up on the assumption that there is an implicit 
> search criteria of UIDEXISTS.  But the documents do not state that 
> anywhere.
> 
> A search criteria of UIDEXISTS may be a way out of the problem.  I 
> would not object to adding that search criteria within the existing 
> framework.
> 
> 
Hang on. This is where you lost me, but more on this below.

>> First off, RFC3501 clearly states that it's the UID command which 
>> ignores any non-existent messages - that is, it's a behaviour of 
>> the command, and not a property of the syntax. In fact, page 61 of 
>> RFC3501 merely states that a UID sequence-set is identical to a 
>> message number sequence-set except it uses UIDs not message 
>> numbers. If you disagree, please quote the relevant parts to 
>> enlighten me - I'm always happy to be proven wrong.
> 
> RFC 3501 section 6.4.4 describes a command that takes a UID 
> sequence-set argument with the same properties as the sequence-set 
> in the UID commands (UID FETCH, UID STORE, UID SEARCH).  It 
> explicitly states that ranges are permitted.
> 
> Is that evidence enough?
> 
> 
No, sorry. It actually says:

      appropriate for the associated command.  However, the numbers in
      the sequence set argument are unique identifiers instead of
      message sequence numbers.  Sequence set ranges are permitted, 
but
      there is no guarantee that unique identifiers will be 
contiguous.

      A non-existent unique identifier is ignored without any error
      message generated.  Thus, it is possible for a UID FETCH command
      to return an OK without any data or a UID COPY or UID STORE to
      return an OK without performing any operations.

Note that the commands ignore them, it's not a product of the syntax 
that they exist. As it happens, this can be taken advantage of (and 
my own client does this) - a client may delibrately extend ranges 
over non-existent UIDs in order to gain a more compact 
representation. But it can do that because the command will ignore 
them without error, not becausethe syntactic construct is in some way 
magical.

>> Secondly, the ALL result mentions usage in subsequent commands, 
>> but it's not given as the reason for its existence. It actually 
>> says:
>>        criteria.  Unlike regular (unextended) SEARCH, the messages 
>> are
>>        always returned using the sequence-set syntax.  A 
>> sequence-set
>>        representation may be more compact and can be used as is in 
>> a
>>        subsequent command that accepts sequence-set.  Note, the 
>> client
>> Taking this reference to "a subsequent command" to be the purpose 
>> for its existence is stretching a point beyond its elastic limit. 
>> The phrase used merely provides rationale behind the use of a 
>> server-provided sequence-set
> 
> That may be just rationale.  Nonetheless, the ranges that my server 
> returns are in complete compliance with that text, and there is NO 
> text that prohibits those ranges.
> 
> 
No, there's nothing that says that non-existent UIDs MAY be included, 
either. And if they aren't included, as I maintain the specification 
hints closest to and intends, then they still can be used in this way.

>> (which had, incidentally, only been used by UIDPLUS prior to this, 
>> which itself obviously does not include non-existent UIDs).
> 
> That's a bug in RFC 4315.  It does not contain the word 
> "monotonically". I see nothing in RFC 4315, other than the general 
> principle of "don't do silly/harmful behavior", that precludes
> 	[COPYUID 38201 1:5 4483:9857]
> That needs to be fixed in that document.
> 
> 
No, it doesn't - you called that one out explicitly, by stating that 
no extraneous UIDs could be included. RFC2359 has similar language. 
That said, it also states (in both cases) "[...] a UID set containing 
the UIDs [...]", so it really needs that language, since otherwise 
you could include other existent UIDs.

> However, in spite of RFC 4315's unfortunate use of the same syntax 
> as sequence-set, it does NOT use the sequence-set rule.  It uses 
> its own uid-set and uid-range rules.  There is a reason for that: 
> it divorces UIDPLUS's return values from the semantics of 
> sequence-set.
> 
> 
And I still contend that there are no such semantics. The semantics 
you're talking about are related to the behaviour of the UID command, 
and not the sequence-set syntactic construct.

> The fact that there is no such divorce in RFC 4466/4731 is another 
> nail in the coffin.  How many more nails do I need to find?
> 
> 
No nails, but a coffin would be handy. Oh, and it's traditional to 
wait until the victim is actually dead.

> You can't use the "don't do silly/harmful behavior" principle in 
> the case of ESEARCH, since the behavior in question is neither 
> silly (it has demonstrably greater compression) nor harmful (there 
> are two other ways to get the map).  In fact, a mobile client that 
> has the map and only wants to do bandwidth-effective searches will 
> benefit from this behavior.
> 
> 
It'll benefit more from using SEARCH, instead of UID SEARCH. Timo 
pointed this out already, though, and I agree - it's what I've been 
using for some time. That leaves the extended UID SEARCH as the tool 
for performing searches (or obtaining the UID mapping) for areas 
where the UID mapping is not known, but that's only useful if and 
only if the ALL result data does not contain non-existent UIDs.

(Timo didn't mention, I think, that message sequence numbers are, by 
definition, always smaller than or equal to UIDs, and therefore can, 
sometimes, also have a smaller ASCII representation. Gotta get those 
octet counts down, y'know.)

>> Nope, it uses the same sequence-sets that every other IMAP usage 
>> does.
> 
> There are only two other usages of sequence-set: SEARCH and UID.
> 
> 
SEARCH's UID criteria matches messages which have a UID which falls 
within the set. I don't see why this mandates that clients must 
include any non-existent UIDs in the set, although quite obviously 
doing so will not effect the result.

The UID command (or commands, depending on your thinking) provides 
explicit behaviour in the presence of non-existent UIDs. It does not 
provide any semantics on the sequence-set syntactic construct itself.

>> It happens that non-existent UIDs are ignored by the server in 
>> commands, but that's explicitly mentioned in the definitions of 
>> those commands.
>> Server-provided sequence-sets never contain *, or non-existent 
>> UIDs.
> 
> Nowhere is this stated.  4466/4731 is the first occurrence.
> 
> UIDPLUS is not a precedent, since it returns uid-set instead of 
> sequence-set.
> 
> 
But RFC2359 did not, and managed just fine.

>>>  . forces all ESEARCH results to have longer, more 
>>> bandwidth-consuming,
>>>    results even if the client has the UID map and has no need for 
>>> such
>>>    longer results
>> I'd be happy to entertain a proposal for an alternate to ALL which 
>> had this behaviour.
> 
> The right way to do this is probably a new SEARCH criterion called 
> UIDEXISTS.  That way, you don't have to change the definition of 
> sequence-set and you don't have to change 4466/4731.
> 
> 
I'm still lost after much thought.

I'm assuming that this search criterion will only be satisfied by 
messages that have a UID that exists. By inference, this means that 
all messages will match, and, therefore, that this is equivalent to 
ALL.

But assuming I'm wrong, you're arguing that somehow, this search 
criterion would cause only messages which satisfy the search criteria 
to be included in results. Right?

But, you also argue that non-existent UIDs can also be included in 
results, anyway. So surely we're right back to square one? Or is 
there something I'm missing?

>> I certainly feel that servers that do not allocate UIDs 
>> contiguously are fairly rare, and you're absolutely correct in 
>> your assertion that ESEARCH performs much worse in those, and this 
>> will mean it's a handful of octets larger than an unextended 
>> SEARCH.
> 
> I am aware of multiple servers which do not allocate UIDs 
> contiguously. I don't know what I can say about these servers due 
> to NDA considerations. I will say that my comments about global 
> UIDs and 32-bits being doomed were not based upon flights of fancy.
> 
> 
Well, I'm not saying that the future is fully known to me, but I 
think a GUID space is likely to be distinct from a UID space, too.

>> Your reading of the specification implies that you think all 
>> non-existent messages match all possible SEARCH criteria.
> 
> If that were the case, then it would be possible to get the 
> UID/msgno map by doing UID SEARCH RETURN () NOT ALL!
> 
> 
Or NOT UIDEXISTS?

> You're misstating my position.  I claim that non-existent UIDs (not 
> "non-existent messages"; there is no such thing) do not figure into 
> the calculation at all.
> 
> 
Well, I'll readily agree that non-existent messages don't exist. But 
the SEARCH command conceptually passes back a list of messages, the 
client's just getting to pick the reference used to indicate which 
ones. So while you can split hairs and otherwise wax philosophical 
about whether your implementation is including non-existent UIDs or 
non-existent messages in the results, I don't think it's making much 
difference - either way, non-existent messages, or references to 
messages which don't exist, are being included in the result as 
passed to the client.

>> ESEARCH works with servers that do not allocate UIDs sequentially 
>> and contiguously. Just not as well.
> 
> "not as well" -> "worse than SEARCH"
> 
> That's not a good design.
> 
> 
There's a handful of cases where ESEARCH comes out bigger by a small 
number of octets (the size of the correlator, possibly UID, and an 
extra E). In the majority of cases, though, it's smaller, in many 
cases I've experienced, it's drastically smaller, especially with the 
larger mailboxes we seem to be dealing with these days.

If you can find something which contains the same amount of 
information, yet is consistently smaller, I'll be very impressed - 
I'd long since decided this was probably similar to finding a 
compression algorithm that reduced the size of all possible inputs. 
(Which is tricky, unless you just use smaller bits, of course.)

>> Similarly, it works with mailboxes than have had every second or 
>> third message expunged - again, this is a worst-case. Thankfully 
>> equally rare.
> 
> Rare?  RARE?!?!?!?!?!!
> 
> 
Rare! RARE!!!!!! R A R E ! ! ! !

Yes, I did say rare. Although not as dramatically, which I hope to 
have rectified. :-)

> "A mailbox which has every second or third message expunged" 
> describes INBOX for the great majority of users.  They have NO 
> other mailbox other than INBOX, they delete their spam and they 
> keep other messages.
> 
> 
That's me.

> Some of these guys have thousands, even tens of thousands, of 
> messages in their INBOX, all with sparse UIDs due to spam.
> 
> 
Oh, well I've only got 51069 messages right now, and my UID range 
isn't all that sparse. Maybe my spam filtering is working. Then 
again, I've found that the bigger a mailbox, the less sparse the UID 
mapping, as a general rule. A couple of thousand seems to be the 
worst case, for no reason I can readily make out.

But it's not a matter of saying "it's pretty sparse", or even "almost 
every range is only two long", you need to be thinking in terms of 
"no range exists which is longer than two", and that is rare in my 
experience, at least on servers which allocate their UIDs 
contiguously. You don't need a lot of ranges of three or more before 
ESEARCH covers its overhead. Also note that the higher the UID, the 
faster the overhead is covered.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Date: Mon, 13 Aug 2007 14:39:48 -0500
To: Mark Crispin <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: Re: UID SEARCH responses
Cc: Alexey Melnikov <[email-masked]>, Dave Cridland <[email-masked]>, [email-masked]
Sender: [email-masked]

So, I'm happy to have technical discussion of IMAP that are not WG 
related items (i.e., ESEARCH) continue here. (At some point today, 
I'll get the last call for i18n in among all of these messages, so 
please keep an eye out for it.) As ESEARCH is not a WG product, I 
don't have to be involved in any of the decisions here, except 
insofar as someone might ask my opinion about what the IMAPEXT WG 
thinks of it.

That said, I want to comment on one item, wearing only my "process geek" hat:

On 8/13/07 at 12:01 PM -0700, Mark Crispin wrote:

>On Mon, 13 Aug 2007, Alexey Melnikov wrote:
>>>I will vigorously oppose such a "clarification", including IESG 
>>>appeals if necessary.
>>
>>You must be kidding.
>
>I'm not kidding.  Such a clarification constitutes an ex post facto 
>declaration of deployed server non-compliance.  You can't do that in 
>a clarification.

You will be hard pressed to find the word "compliance" in RFC 2026. 
(It occurs in the places, none of which are terribly interesting for 
the discussion at hand.) What you will find is section 4.1.1, which 
says in part:

    A Proposed Standard specification is generally stable, has resolved
    known design choices, is believed to be well-understood, has received
    significant community review, and appears to enjoy enough community
    interest to be considered valuable.  However, further experience
    might result in a change or even retraction of the specification
    before it advances.

and:

    Implementors should treat Proposed Standards as immature
    specifications.  It is desirable to implement them in order to gain
    experience and to validate, test, and clarify the specification.
    However, since the content of Proposed Standards may be changed if
    problems are found or better solutions are identified, deploying
    implementations of such standards into a disruption-sensitive
    environment is not recommended.

Proposed Standards might have more done to them than "clarification". 
They might "change". And the fact that a particular implementation of 
an old version of a specification is not "compliant" with a new 
version of a specification is not a basis of an appeal according to 
section 6.5 of that document (as far as I read it).

The idea of our standards process has always been to get people to 
come to (sometimes rough) consensus so that implementations can 
interoperate, not to anoint implemenations with "compliance" or 
"non-compliance". I can think of several times in the past where my 
implementation did not agree with what a new version of a 
specification said to do, even if what I was doing was arguably a 
valid interpretation of the old specification. The specification was 
clarified/changed to meet the consensus of what most folks thought 
the old specification meant, and I fixed my code (sometimes making my 
old code non-conformant to the new specification). If the community 
(as judged by the IESG) decides that the best way to achieve 
interoperability is to change a specification such that one (or a 
few) implementations have to change their code to get into line, I 
think one would be hard-pressed to convince the IESG that this is a 
process violation.

Now back to your regularly scheduled technical discussion.

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

Date: Mon, 13 Aug 2007 12:37:59 -0700 (Pacific Daylight Time)
From: Mark Crispin <[email-masked]>
To: Alexey Melnikov <[email-masked]>
cc: IMAP Extensions WG <[email-masked]>
Subject: Re: UID SEARCH responses
Organization: Networks & Distributed Computing
Sender: [email-masked]

On Mon, 13 Aug 2007, Mark Crispin wrote:
>> 2). Clarify which of the two choices should be used by ALL and add a new 
>> return option for the other one.

On second thought, this could work if you proceed as follows:

Update RFC 4466 to include uid-set from RFC 4315.  Be sure to highlight 
that uid-set MUST NOT include non-existent UIDs, whereas sequence-set 
does.

Consider a possible sytax change for uid-set, e.g., using hyphens, to 
reduce confusion.  That orphans UIDPLUS syntax, but the benefit may be 
worth not adding another subtlety in IMAP.

Define the new return option (and capability indicating its presence) to 
return uid-set.

As a quality of implementation note, a server which supports the new 
return option maximally compresses in ALL.

This maintains compatibility for all existing servers.  As a gesture of 
good faith, I will pledge to implement the new return value as soon as it 
is published as an RFC.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Date: Mon, 13 Aug 2007 12:01:11 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: Alexey Melnikov <[email-masked]>
cc: Dave Cridland <[email-masked]>, [email-masked]
Subject: Re: UID SEARCH responses
Sender: [email-masked]

On Mon, 13 Aug 2007, Alexey Melnikov wrote:
>> I will vigorously oppose such a "clarification", including IESG appeals if 
>> necessary.
> You must be kidding.

I'm not kidding.  Such a clarification constitutes an ex post facto 
declaration of deployed server non-compliance.  You can't do that in a 
clarification.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Date: Mon, 13 Aug 2007 11:59:20 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: Alexey Melnikov <[email-masked]>
cc: IMAP Extensions WG <[email-masked]>
Subject: Re: UID SEARCH responses
Sender: [email-masked]

On Mon, 13 Aug 2007, Alexey Melnikov wrote:
> "deprecate and replace" is a non-starter, as far as I am concerned. There are 
> several other extensions reusing the ESEARCH response.

OK.

> But there are other choices which you've ignored:
> 1). Get rough consensus among implementors and update the spec to mandate 
> that servers implement one of the two choices.
> 2). Clarify which of the two choices should be used by ALL and add a new 
> return option for the other one.

I consider both of these to be non-starters, as it will result in a 
deployed compliant implementation being broken.

> 3). Deprecate ALL and replace it with 2 other return options

You would also have to deprecate (), which I think would be a good thing.

There are other choices:

4) Leave everything as-is; and take it as a learning experience.
5) Clarify in the document that either way is possible, and note that
     either FETCH or non-ESEARCH UID SEARCH is the appropriate command
     to get the UID map.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Date: Mon, 13 Aug 2007 19:51:18 +0100
From: Alexey Melnikov <[email-masked]>
To: Mark Crispin <[email-masked]>
CC: Dave Cridland <[email-masked]>, [email-masked]
Subject: Re: UID SEARCH responses
Sender: [email-masked]

Mark Crispin wrote:

> On Mon, 13 Aug 2007, Dave Cridland wrote:
>
>> RFC4731 defines MIN, MAX, and COUNT - none of which are part of this 
>> discussion - and ALL - which you are contending is broken. ALL can be 
>> deprecated and replaced by something else quite easily without 
>> affecting MIN, MAX, COUNT, or the ESEARCH response as a whole.
>
> You'll still have to create a new capability.  Otherwise the client 
> won't know the difference between RFC 4731 compliant servers and RFC 
> xxxx compliant servers.

Yes, if there is consensus to do that.

>> In addition, I still do not believe that there is consensus to do 
>> this - as far as I can tell, there is only one voice in support of 
>> such an action. That voice happens to have substantial weight as the 
>> author of RFC3501, however neither author of RFC4731, nor any other 
>> implementor, has leant any support to it, whereas there appears to me 
>> to be wide support in favour of retaining the intended behaviour, and 
>> clarifying it in errata and a subsequent revision.
>
> I will vigorously oppose such a "clarification", including IESG 
> appeals if necessary.

You must be kidding.

Date: Mon, 13 Aug 2007 11:49:05 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: IMAP Extensions WG <[email-masked]>
Subject: 1:4294967295 (was Re: UID SEARCH responses)
Sender: [email-masked]

On further study:

A server can return 1:4294967295 in the ESEARCH form of UID SEARCH ALL, 
but it can NOT generally return 4294967295 as the upper end of a range in 
other searches.

There are two types of non-existent UIDs:
  . those which do not now, and will never exist
  . those which may exist in the future

Any UID value greater than or equal to UIDNEXT is in the latter category. 
Such UID values are permitted in sequence-set by RFC 3501 (the prohibition 
is on message numbers only).  However, since these UIDs reflect potential 
messages, a server can not say that a sequence-set that includes such UIDs 
will still match the SEARCH criteria when that sequence-set is applied to 
a subsequent command.

Specifically, if a message is instantiated between the UID SEARCH and the 
next command, the result of 1:4294967295 would be a false positive which 
breaks IMAP's precedent of a false negative in such circumstances.

With that said, I agree that a server should NOT send :4294967295 in any 
case.  Instead, it should send the UID of the last message in the mailbox 
if it matches the mailbox.  I am less concerned about the 1: case.

There is a similar timing race in IMAP with flags, as these can change.

Perhaps what should have been done instead of ESEARCH was to create an 
operation which did FETCH/STORE/COPY with the argument being SEARCH 
criteria instead of a sequence-set, and defining the resulting operation 
as being atomic.  Maybe that would have reduced the need for CONDSTORE.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Date: Mon, 13 Aug 2007 19:47:53 +0100
From: Alexey Melnikov <[email-masked]>
To: Mark Crispin <[email-masked]>
CC: IMAP Extensions WG <[email-masked]>
Subject: Re: UID SEARCH responses
Sender: [email-masked]

Mark Crispin wrote:

> If Pawel is correct on the intent, then the ESEARCH RFC is broken 
> since it is ambiguous on that point; either live with the ambiguity, 
> or deprecate ESEARCH and replace it with some other syntax.

"deprecate and replace" is a non-starter, as far as I am concerned. 
There are several other extensions reusing the ESEARCH response.

But there are other choices which you've ignored:
1). Get rough consensus among implementors and update the spec to 
mandate that servers implement one of the two choices.
2). Clarify which of the two choices should be used by ALL and add a new 
return option for the other one.
3). Deprecate ALL and replace it with 2 other return options

Date: Mon, 13 Aug 2007 11:31:30 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: Timo Sirainen <[email-masked]>
cc: [email-masked]
Subject: Re: UID SEARCH responses
Sender: [email-masked]

On Mon, 13 Aug 2007, Timo Sirainen wrote:
> Uncompressed UID SEARCH ranges are useful if the (full) sequence <-> UID
> map isn't known yet for the client for this session. After the map is
> known, there should be no problems with compressed UID SEARCH ranges.

Correct.

> Then again, if the map is known client can just as well use SEARCH and
> get even more compressed ranges.

Correct.

> So if UID SEARCH ranges can include expunged messages, I don't see why a
> client would ever want to do UID SEARCHes instead of SEARCHes, because
> it's required to internally check the UID <-> sequence mapping anyway to
> get a usable result.

Consider a client that wants to download all messages that match a certain 
set of search criteria and then exit.  This client does not need the map; 
it just needs a sequence-set string to feed to UID FETCH.  It does the UID 
form instead of the non-UID form so as to allow expunging.

With that said, I agree with the overall sentiment that it is more useful 
to get the UID/msgno map for synchronization and subsequently use msgno 
commands.  Almost all instances of "UID-only" clients that I have studied 
in real-world usage end up doing unnecessary work.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Date: Mon, 13 Aug 2007 11:01:34 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: Dave Cridland <[email-masked]>
cc: [email-masked], Alexey Melnikov <[email-masked]>
Subject: Re: UID SEARCH responses
Sender: [email-masked]

On Mon, 13 Aug 2007, Dave Cridland wrote:
> RFC4731 defines MIN, MAX, and COUNT - none of which are part of this 
> discussion - and ALL - which you are contending is broken. ALL can be 
> deprecated and replaced by something else quite easily without affecting MIN, 
> MAX, COUNT, or the ESEARCH response as a whole.

You'll still have to create a new capability.  Otherwise the client won't 
know the difference between RFC 4731 compliant servers and RFC xxxx 
compliant servers.

> In addition, I still do not believe that there is consensus to do this - as 
> far as I can tell, there is only one voice in support of such an action. That 
> voice happens to have substantial weight as the author of RFC3501, however 
> neither author of RFC4731, nor any other implementor, has leant any support 
> to it, whereas there appears to me to be wide support in favour of retaining 
> the intended behaviour, and clarifying it in errata and a subsequent 
> revision.

I will vigorously oppose such a "clarification", including IESG appeals if 
necessary.

I believe that I have strong arguments:
  . although RFC 4315 has a syntax for UID sequence ranges that seem to
     have the same semantics as the authors of RFC 4731 claim to have
     intended, RFC 4315 uses its own rules to describe those ranges and
     does not use sequence-set in order to indicate differing semantics.
  . unlike RFC 4315, RFC 4466/4731 uses RFC 3501's sequence-set rule.
  . there are two other ways in the base specification to accomplish
     the functionality of acquiring the UID/msgno map.
  . RFC 4731 says nothing about acquiring the UID/msgno map.
  . RFC 4731 mentions use of the returned UID sequence-set in the
     IMAP commands which have the RFC 3501 sequence-set semantics.  My
     interpretation is fully in compliance with this use.
  . RFC 4731 gives, as a motivation, the desire to return a "more
     compact" representation.  My interpretation is fully in compliance
     with this motivation, and in fact it is demonstrably more compact
     than the contrary interpretation.
  . The operation of obtaining the UID/msgno map is typically a one-time
     operation in a session.  Once the map is known to the client,
     removal from the map can be mathematically calculated from an EXPUNGE
     response, and additional to the map can be obtained with a short
     map fetch.
  . Searches typically are done many times in a session, and as noted
     above the re-fetch of the map data is unnecessary.
  . INBOX tends to have frequent expunges, and consequently the UID space
     in an INBOX can become very sparse.  Anecdotally (I can collect the
     data if you insist), some people have very large INBOXes.  In one
     interpretation, ESEARCH optimizes search results; in the other, it
     does not.
  . In one interpretation, ESEARCH optimizes search results even if the
     server does not allocate contiguous UID values in a mailbox.  In
     the other, it does not.
  . I contend that the optimizing interpretion in the two previous points
     is superior.
  . No evidence has been offered to debunk the assertation that RFC 4731
     is ambiguous.
  . Assuming that RFC 4731 is ambiguous, server implementations with
     either interpretation are fully compliant.  The only issue is that a
     client must be aware of the issue.
  . Assuming that RFC 4731 is ambiguous on this point, there is an
     existing deployed implementation that is fully compliant with RFC
     4731 and would be broken by the proposed clarification.  This is
     therefore an incompatible change, not a clarification, and must be
     designed and published as such.

I will also oppose, but less vigorously, a new specification that 
deprecates ESEARCH entirely in favor of a new capability.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Subject: Re: UID SEARCH responses
From: Timo Sirainen <[email-masked]>
To: [email-masked]
Date: Mon, 13 Aug 2007 20:45:09 +0300
Sender: [email-masked]

--=-sSux3v7X/FfMQqRyZkC8
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

On Mon, 2007-08-13 at 17:54 +0100, Dave Cridland wrote:
> In addition, I still do not believe that there is consensus to do=20
> this - as far as I can tell, there is only one voice in support of=20
> such an action.

My thoughts from usefulness point of view:

Uncompressed UID SEARCH ranges are useful if the (full) sequence <-> UID
map isn't known yet for the client for this session. After the map is
known, there should be no problems with compressed UID SEARCH ranges.
Then again, if the map is known client can just as well use SEARCH and
get even more compressed ranges.

So if UID SEARCH ranges can include expunged messages, I don't see why a
client would ever want to do UID SEARCHes instead of SEARCHes, because
it's required to internally check the UID <-> sequence mapping anyway to
get a usable result.

--=-sSux3v7X/FfMQqRyZkC8
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

--=-sSux3v7X/FfMQqRyZkC8--

Date: Mon, 13 Aug 2007 10:29:48 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: Dave Cridland <[email-masked]>
cc: Pawel Salek <[email-masked]>, [email-masked], Peter Coates <[email-masked]>
Subject: RE: UID SEARCH responses
Sender: [email-masked]

On Mon, 13 Aug 2007, Dave Cridland wrote:
> Non-existent messages do not satisfy the SEARCH criteria

Why not in the case of UIDs?  They satisfy the criteria when they are 
arguments to the SEARCH commands.  Why don't they satisfy the criteria 
when the server returns it as a value?

That is an inconsistency between client-end and server-end.  Such 
inconsistencies MUST be documented, carefully and explicitly.

>> By "stand alone", do you mean "mathematically calculate the UID/msgno map"? 
>> If so, where in RFC 4731 is this property stated?
> The property of the ALL result that interests me is that it contains only 
> messages which satisfy the search criteria.

The ranges in question do not contain any messages which do not satisfy 
the search criteria.

You are being hung up on the assumption that there is an implicit search 
criteria of UIDEXISTS.  But the documents do not state that anywhere.

A search criteria of UIDEXISTS may be a way out of the problem.  I would 
not object to adding that search criteria within the existing framework.

> First off, RFC3501 clearly states that it's the UID command which ignores any 
> non-existent messages - that is, it's a behaviour of the command, and not a 
> property of the syntax. In fact, page 61 of RFC3501 merely states that a UID 
> sequence-set is identical to a message number sequence-set except it uses 
> UIDs not message numbers. If you disagree, please quote the relevant parts to 
> enlighten me - I'm always happy to be proven wrong.

RFC 3501 section 6.4.4 describes a command that takes a UID sequence-set 
argument with the same properties as the sequence-set in the UID commands 
(UID FETCH, UID STORE, UID SEARCH).  It explicitly states that ranges are 
permitted.

Is that evidence enough?

> Secondly, the ALL result mentions usage in subsequent commands, but it's not 
> given as the reason for its existence. It actually says:
>        criteria.  Unlike regular (unextended) SEARCH, the messages are
>        always returned using the sequence-set syntax.  A sequence-set
>        representation may be more compact and can be used as is in a
>        subsequent command that accepts sequence-set.  Note, the client
> Taking this reference to "a subsequent command" to be the purpose for its 
> existence is stretching a point beyond its elastic limit. The phrase used 
> merely provides rationale behind the use of a server-provided sequence-set

That may be just rationale.  Nonetheless, the ranges that my server 
returns are in complete compliance with that text, and there is NO text 
that prohibits those ranges.

> (which had, incidentally, only been used by UIDPLUS prior to this, which 
> itself obviously does not include non-existent UIDs).

That's a bug in RFC 4315.  It does not contain the word "monotonically". 
I see nothing in RFC 4315, other than the general principle of "don't do 
silly/harmful behavior", that precludes
 	[COPYUID 38201 1:5 4483:9857]
That needs to be fixed in that document.

However, in spite of RFC 4315's unfortunate use of the same syntax as 
sequence-set, it does NOT use the sequence-set rule.  It uses its own 
uid-set and uid-range rules.  There is a reason for that: it divorces 
UIDPLUS's return values from the semantics of sequence-set.

The fact that there is no such divorce in RFC 4466/4731 is another nail in 
the coffin.  How many more nails do I need to find?

Perhaps what you wanted to do was have uid-set and uid-range defined in 
RFC 4466, and then use those rules in RFC 4731.

You can't use the "don't do silly/harmful behavior" principle in the case 
of ESEARCH, since the behavior in question is neither silly (it has 
demonstrably greater compression) nor harmful (there are two other ways to 
get the map).  In fact, a mobile client that has the map and only wants to 
do bandwidth-effective searches will benefit from this behavior.

> Nope, it uses the same sequence-sets that every other IMAP usage does.

There are only two other usages of sequence-set: SEARCH and UID.

> It 
> happens that non-existent UIDs are ignored by the server in commands, but 
> that's explicitly mentioned in the definitions of those commands.
> Server-provided sequence-sets never contain *, or non-existent UIDs.

Nowhere is this stated.  4466/4731 is the first occurrence.

UIDPLUS is not a precedent, since it returns uid-set instead of 
sequence-set.

>>  . forces all ESEARCH results to have longer, more bandwidth-consuming,
>>    results even if the client has the UID map and has no need for such
>>    longer results
> I'd be happy to entertain a proposal for an alternate to ALL which had this 
> behaviour.

The right way to do this is probably a new SEARCH criterion called 
UIDEXISTS.  That way, you don't have to change the definition of 
sequence-set and you don't have to change 4466/4731.

> I certainly feel that servers that do not allocate UIDs contiguously are 
> fairly rare, and you're absolutely correct in your assertion that ESEARCH 
> performs much worse in those, and this will mean it's a handful of octets 
> larger than an unextended SEARCH.

I am aware of multiple servers which do not allocate UIDs contiguously. 
I don't know what I can say about these servers due to NDA considerations. 
I will say that my comments about global UIDs and 32-bits being doomed 
were not based upon flights of fancy.

> Your reading of the specification implies that you think all 
> non-existent messages match all possible SEARCH criteria.

If that were the case, then it would be possible to get the UID/msgno map 
by doing UID SEARCH RETURN () NOT ALL!

You're misstating my position.  I claim that non-existent UIDs (not 
"non-existent messages"; there is no such thing) do not figure into the 
calculation at all.

> ESEARCH works with servers that do not allocate UIDs sequentially and 
> contiguously. Just not as well.

"not as well" -> "worse than SEARCH"

That's not a good design.

> Similarly, it works with mailboxes than have 
> had every second or third message expunged - again, this is a worst-case. 
> Thankfully equally rare.

Rare?  RARE?!?!?!?!?!!

"A mailbox which has every second or third message expunged" describes 
INBOX for the great majority of users.  They have NO other mailbox other 
than INBOX, they delete their spam and they keep other messages.

Some of these guys have thousands, even tens of thousands, of messages in 
their INBOX, all with sparse UIDs due to spam.

>> It is a fortunate accident that UID SEARCH returns the map as well.  But 
>> how many applications that use UID SEARCH to obtain the UID/msgno map can 
>> handle:
>> 	tag UID SEARCH ALL
>> 	* SEARCH 4929 3012
>> 	* SEARCH 12 9993
>> 	tag OK done
>> Such a scenario is completely valid in RFC 3501!
> The fact that a specification does not flatly prohibit doing something 
> idiotic does not mean that it is a good thing for servers (or clients) to do 
> the idiotic thing.

It is not idiotic.  It is an intentional part of the base specification 
from inception.

There are circumstances in which a server may need to split SEARCH results 
into multiple untagged SEARCH.  There are also circumstances (typically 
defined by a mail store designed for non-IMAP) in which a server may need 
to issue out-of-order results in an untagged SEARCH.

The example above is a worst-case, and highly unlikely, but it shows up 
what would happen if a client engine does not implement the untagged data 
model of IMAP properly.

RFC 4731 specifically requires a single ESEARCH response, but leaves open 
the out-of-order possibility.  It also contains some weasel-wording about 
how ESEARCH may be extended for multiple ESEARCH responses.

> Hey, why stop there? If the client really wanted to know the beginning and 
> end of the range, it could use MIN/MAX. No, instead:
> * ESEARCH (TAG x02) UID ALL 1:4294967295
> According to your reading, that's valid.

That is correct.  With the exception of the non-ESEARCH form of UID 
SEARCH, ALL has never been a particularly useful criterion.  The only 
reason why ALL was in the original IMAP was to aid in server debugging.

There's no NONE, but NOT ALL serves that purpose.  It's even less useful 
than ALL, but it's there.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Subject: Re: UID SEARCH responses
Date: Mon, 13 Aug 2007 17:54:53 +0100
From: Dave Cridland <[email-masked]>
To: Mark Crispin <[email-masked]>
Cc: <[email-masked]>, Alexey Melnikov <[email-masked]>
Sender: [email-masked]

On Mon Aug 13 16:46:30 2007, Mark Crispin wrote:
> 
> On Mon, 13 Aug 2007, Alexey Melnikov wrote:
>>> You have to do more than that.  You have to deprecate all of 
>>> ESEARCH, and replace it with different syntax.
>> (Irrespectively to whether you are right or wrong on the issue.)
>> I am sorry Mark, but this is nonsense. RFC 4731 does NOT define 
>> ESEARCH response or extended SEARCH, they are defined in RFC 4466.
>> Both extended SEARCH and ESEARCH response were designed to be 
>> extensible.
> 
> That just means that, in order to have this new type of UID range, 
> RFC 4466 will also have be deprecated and replaced with a new 
> document that uses a rule other than sequence-set to define these 
> new magic blurdybloops that have UID ranges that violate RFC 3501's 
> definition of UID ranges.

Again, irrespective of whether RFC3501 actually does define UID 
ranges in that manner, this is simply wrong.

RFC4466 defines the extensible SEARCH framework, in particular 
defining the ESEARCH response itself. (It also defines other 
extensible syntactic frameworks).

RFC4731 defines MIN, MAX, and COUNT - none of which are part of this 
discussion - and ALL - which you are contending is broken. ALL can be 
deprecated and replaced by something else quite easily without 
affecting MIN, MAX, COUNT, or the ESEARCH response as a whole.

In addition, I still do not believe that there is consensus to do 
this - as far as I can tell, there is only one voice in support of 
such an action. That voice happens to have substantial weight as the 
author of RFC3501, however neither author of RFC4731, nor any other 
implementor, has leant any support to it, whereas there appears to me 
to be wide support in favour of retaining the intended behaviour, and 
clarifying it in errata and a subsequent revision.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Date: Mon, 13 Aug 2007 08:46:30 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: Alexey Melnikov <[email-masked]>
cc: [email-masked]
Subject: Re: UID SEARCH responses
Sender: [email-masked]

On Mon, 13 Aug 2007, Alexey Melnikov wrote:
>> You have to do more than that.  You have to deprecate all of ESEARCH, and 
>> replace it with different syntax.
> (Irrespectively to whether you are right or wrong on the issue.)
> I am sorry Mark, but this is nonsense. RFC 4731 does NOT define ESEARCH 
> response or extended SEARCH, they are defined in RFC 4466.
> Both extended SEARCH and ESEARCH response were designed to be extensible.

That just means that, in order to have this new type of UID range, RFC 
4466 will also have be deprecated and replaced with a new document that 
uses a rule other than sequence-set to define these new magic blurdybloops 
that have UID ranges that violate RFC 3501's definition of UID ranges.

The alternative is to cut losses and leave the specification as-is. 
There *are* two other ways to get the UID/msgno map.

Don't think for a minute that I'm not sympathetic.  I am.  RFC 4466/4731 
were published as RFCs before there was thorough interoperability testing 
(particularly with implementations written by implementors that were NOT 
part of the specification design process), and got bit by the Law of 
Unintended Consequences.  This has happened to me many times; the worst 
and most embarassing being the RFC 1730/2060 fiasco.

Protocol design is a tough game.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Date: Mon, 13 Aug 2007 12:05:30 +0100
From: Alexey Melnikov <[email-masked]>
To: [email-masked]
Subject: Re: UID SEARCH responses
Sender: [email-masked]

Dave Cridland wrote:

> On Fri Aug 10 06:47:40 2007, Mark Crispin wrote:
>
>> For what it's worth, UW imapd issues the
>>     * ESEARCH (TAG ".") UID ALL 38782:41424
>> response instead of
>>     * ESEARCH (TAG ".") UID ALL 
>> 38782,38789,40004,40301,40421,40814,41424
>> because there are no other UIDs in that range.
>
> Ah. That's definitely not what I intended, and it never crossed my 
> mind that this was a possible interpretation. The second is correct.

BTW, I intended the latter, as this is what I've coded into Isode's 
implementation.

Subject: RE: UID SEARCH responses
Date: Mon, 13 Aug 2007 11:45:45 +0100
From: Dave Cridland <[email-masked]>
To: Mark Crispin <[email-masked]>
Cc: Pawel Salek <[email-masked]>, <[email-masked]>, Peter Coates <[email-masked]>
Sender: [email-masked]

On Mon Aug 13 04:21:16 2007, Mark Crispin wrote:
> 
> On Sun, 12 Aug 2007, Peter Coates wrote:
>> Personally I think that having a search result that does not stand 
>> alone (as
>> in sends a range when some messages in the range do not exist) is 
>> bogus.
> 
> Why do you think that it does not "stand alone"?  The returned 
> sequence-set is explicitly and unambiguously valid as a 
> sequence-set to any other IMAP command and will absolutely return 
> to the matching messages and no others.
> 
> 
Yes, but that's not the purpose. That's merely given as an example of 
potential use, and reasoning behind the sequence-set syntax used. It 
does not, however, satisfy the definition of ALL, which is:

         Return all message numbers/UIDs that satisfy the SEARCH
         criteria.

Non-existent messages do not satisfy the SEARCH criteria, and if they 
are included, then they mean the client cannot detirmine the results 
of the search, rendering the results only useful if the client then 
issues a further command to expand them by some means. To my mind, 
that means the results do not stand on their own.

> By "stand alone", do you mean "mathematically calculate the 
> UID/msgno map"?  If so, where in RFC 4731 is this property stated?
> 
> 
The property of the ALL result that interests me is that it contains 
only messages which satisfy the search criteria.

> The definition of sequence-set in RFC 3501 explicitly allows UID 
> ranges to be used even when there are holes in the UID space.  
> Nothing in RFC 4731 states that its definition of sequence-set is 
> different from RFC 3501. In fact, there is text in RFC 4731 that 
> clearly states that the sequence-set returned from ESEARCH is 
> intended to be used as an RFC 3501 sequence-set.
> 
> 
No, I disagree, on pretty much all counts.

First off, RFC3501 clearly states that it's the UID command which 
ignores any non-existent messages - that is, it's a behaviour of the 
command, and not a property of the syntax. In fact, page 61 of 
RFC3501 merely states that a UID sequence-set is identical to a 
message number sequence-set except it uses UIDs not message numbers. 
If you disagree, please quote the relevant parts to enlighten me - 
I'm always happy to be proven wrong.

Secondly, the ALL result mentions usage in subsequent commands, but 
it's not given as the reason for its existence. It actually says:

         criteria.  Unlike regular (unextended) SEARCH, the messages 
are
         always returned using the sequence-set syntax.  A 
sequence-set
         representation may be more compact and can be used as is in a
         subsequent command that accepts sequence-set.  Note, the 
client

Taking this reference to "a subsequent command" to be the purpose for 
its existence is stretching a point beyond its elastic limit. The 
phrase used merely provides rationale behind the use of a 
server-provided sequence-set (which had, incidentally, only been used 
by UIDPLUS prior to this, which itself obviously does not include 
non-existent UIDs).

> Nowhere in RFC 4731 does it say that an RFC 4731 sequence-set has 
> new semantics that prohibit the server from returning ranges that 
> encompass holes in monotonically assigned UID space.
> 
> 
New semantics? Show me the old ones.

> If you assert that ESEARCH sequence-set has such semantics, and 
> MUST NOT send ranges that have holes, then in effect ESEARCH:
>  . creates a new type of sequence-set that is different from all 
> other
>    IMAP use of sequence-set

Nope, it uses the same sequence-sets that every other IMAP usage 
does. It happens that non-existent UIDs are ignored by the server in 
commands, but that's explicitly mentioned in the definitions of those 
commands. Server-provided sequence-sets never contain *, or 
non-existent UIDs.

>  . forces all ESEARCH results to have longer, more 
> bandwidth-consuming,
>    results even if the client has the UID map and has no need for 
> such
>    longer results

I'd be happy to entertain a proposal for an alternate to ALL which 
had this behaviour.

>  . forces ESEARCH results to ALWAYS be longer than SEARCH results 
> in any
>    server that does not monotonically assign UID values.

By "monotonic" you mean "contiguous"? ("Monotonically" has a bunch of 
different meanings, between fields and locations.)

I certainly feel that servers that do not allocate UIDs contiguously 
are fairly rare, and you're absolutely correct in your assertion that 
ESEARCH performs much worse in those, and this will mean it's a 
handful of octets larger than an unextended SEARCH.

>  . as a consequence of the previous statement creates an implicit
>    requirement, stated nowhere in any specification, that UIDs must 
> be
>    monotonically assigned in a mailbox.
> 
> 
Nope. It does imply a recommendation to server implementors that 
allocating contiguous UIDs is a good idea. (As a brief aside, IMAP 
UIDs most certainly are strictly monotonically increasing. Or merely 
monotonically increasing, depending on where you live.)

> I assert that ESEARCH sequence-set has no such semantics, and is:
>  . compatible with RFC 3501

Yes, I think it is, too.

>  . minimizes the results so that only non-matching messages cause 
> there
>    to be multiple entries in the sequence-set

That depends on whether you believe that a non-existent message 
matches or not. Your reading of the specification implies that you 
think all non-existent messages match all possible SEARCH criteria.

>  . does so even with servers that do not monotonically assign UID 
> values

ESEARCH works with servers that do not allocate UIDs sequentially and 
contiguously. Just not as well. Similarly, it works with mailboxes 
than have had every second or third message expunged - again, this is 
a worst-case. Thankfully equally rare.

>  . as a consequence of the previous states create no new implicit
>    requirements
> 
> 
But ESEARCH absolutely does not have any other requirements.

> This does mean that ESEARCH can't be used to get the UID/msgno map 
> as Pawel complains.  But there are two other commands in RFC 3501 
> that can accomplish that: FETCH and UID SEARCH.  FETCH has always 
> been the official way.
> 
> It is a fortunate accident that UID SEARCH returns the map as well. 
>  But how many applications that use UID SEARCH to obtain the 
> UID/msgno map can handle:
> 	tag UID SEARCH ALL
> 	* SEARCH 4929 3012
> 	* SEARCH 12 9993
> 	tag OK done
> Such a scenario is completely valid in RFC 3501!

The fact that a specification does not flatly prohibit doing 
something idiotic does not mean that it is a good thing for servers 
(or clients) to do the idiotic thing.

>   If your client can't handle that, it's already broken.  Hence 
> that strange warning in RFC 4731:
>          [...]  Note, the client
>          MUST NOT assume that messages/UIDs will be listed in any
>          particular order.
> 
> 
Indeed, that's the reason. We also wanted to make it explicit that 
results such as 10,3,47 where completely valid.

> Think of the effect of what you are saying.  For the benefit of 
> clients that are not satisfied with having TWO ways of getting the 
> UID/msgno map, IMAP must have THREE ways of getting the UID/msgno 
> map; and to provide that third way, an advertised protocol 
> optimization MUST be reduced and in some cases turned into a 
> chattiness pessimization!  Given:
> 	x01 UID SEARCH ALL
> 	* SEARCH 12 3012 4929 5948 6666 7739 8199 8839 9017 9993
> 	x01 OK done
> Compare:
> 	x02 UID SEARCH RETURN () ALL
> 	* ESEARCH (TAG x02) ALL 12 3012 4929 5948 6666 7739 8199 8839 9017 
> 9993
> 	x02 OK done

[Not that this affects your point, but it'd be comma-seperated, and 
have UID present]

> with:
> 	x02 UID SEARCH RETURN () ALL
> 	* ESEARCH (TAG x02) ALL 12:9993
> 	x02 OK done
> 
> 
Hey, why stop there? If the client really wanted to know the 
beginning and end of the range, it could use MIN/MAX. No, instead:

* ESEARCH (TAG x02) UID ALL 1:4294967295

According to your reading, that's valid.

> Furthermore, this pessimization only happens with UIDs.  Message 
> numbers won't get pessimized:
> 	x03 SEARCH RETURN () ALL
> 	* ESEARCH (TAG x02) ALL 1:10
> 	x03 OK done
> 
> 
Indeed - message numbers are always contiguously allocated by 
definition, so clients which use them can gain bandwidth gains. In 
fact, I do this in my own client in some cases, for precisely this 
reason.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Date: Mon, 13 Aug 2007 11:32:35 +0100
From: Alexey Melnikov <[email-masked]>
To: Mark Crispin <[email-masked]>
CC: [email-masked]
Subject: Re: UID SEARCH responses
Sender: [email-masked]

Mark Crispin wrote:

>> I disagree, I think you've misinterpreted the document. If enough 
>> people disagree, and think that UID sets are somehow defined to 
>> include missing UIDs - which I dispute - then I suppose we could 
>> issue a revision that deprecates ALL, replacing it with (perhaps) 
>> RANGE and MAP.
>
> You have to do more than that.  You have to deprecate all of ESEARCH, 
> and replace it with different syntax.

(Irrespectively to whether you are right or wrong on the issue.)
I am sorry Mark, but this is nonsense. RFC 4731 does NOT define ESEARCH 
response or extended SEARCH, they are defined in RFC 4466.
Both extended SEARCH and ESEARCH response were designed to be extensible.

Date: Mon, 13 Aug 2007 09:02:23 +0200
From: Pawel Salek <[email-masked]>
Subject: Re: UID SEARCH responses
To: Martin Konold <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

On 08/13/2007 05:32:56 AM, Martin Konold wrote:
> 
> Am Montag 13 August 2007 schrieben Sie:
> 
> Hi Pieter,
> 
> > Personally I think that having a search result that does not stand  
> alone
> > (as in sends a range when some messages in the range do not exist) is
> > bogus.
> 
> I am wondering if this can be kept at all times. Basically I consider  
> IMAP tobe incoherent anyway.
> In other words: When returning an explicit list of UIDs like 1,3,4  
> instead of1:4 not much is gained simply because at _any_ time it is  
> allowed within theprotocol that any message might have got  
> deleted/expunged in the meantime.

Not quite so - think of the example I gave: a server returning 1,3,4  
allows the client to drop the cached information about message with uid  
2. This information is absent in the 1:4 response. Expunges arriving at  
arbitrary times do not change this picture.

Pawel

From: Martin Konold <[email-masked]>
Organization: erfrakon
To: [email-masked]
Subject: Re: UID SEARCH responses
Date: Mon, 13 Aug 2007 05:32:56 +0200
Sender: [email-masked]

Am Montag 13 August 2007 schrieben Sie:

Hi Pieter,

> Personally I think that having a search result that does not stand alone
> (as in sends a range when some messages in the range do not exist) is
> bogus.

I am wondering if this can be kept at all times. Basically I consider IMAP to 
be incoherent anyway. 

In other words: When returning an explicit list of UIDs like 1,3,4 instead of 
1:4 not much is gained simply because at _any_ time it is allowed within the 
protocol that any message might have got deleted/expunged in the meantime.

Regards,
-- martin konold

-- 
e r f r a k o n
Erlewein, Frank, Konold & Partner - Beratende Ingenieure und Physiker
Sitz: Adolfstraße 23 Stuttgart - Partnerschaftsregister Stuttgart PR 126
http://www.erfrakon.com/

Date: Sun, 12 Aug 2007 20:21:16 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: Peter Coates <[email-masked]>
cc: "'Pawel Salek'" <[email-masked]>, [email-masked]
Subject: RE: UID SEARCH responses
Sender: [email-masked]

On Sun, 12 Aug 2007, Peter Coates wrote:
> Personally I think that having a search result that does not stand alone (as
> in sends a range when some messages in the range do not exist) is bogus.

Why do you think that it does not "stand alone"?  The returned 
sequence-set is explicitly and unambiguously valid as a sequence-set to 
any other IMAP command and will absolutely return to the matching messages 
and no others.

By "stand alone", do you mean "mathematically calculate the UID/msgno 
map"?  If so, where in RFC 4731 is this property stated?

> The results that are returned as a result of a UID SEARCH RETURN (UPDATE)
> command explicitly preclude sending a range (eg 1:4) to mean 1,4 even if 2
> and 3 do not exist.

Where does it "explicitly" state that in RFC 4731?

The syntax for SEARCH results is zero or more space delimited message 
numbers or UIDs.  Note that this is a different syntax from sequence-set. 
Not only does it not have ranges, it also uses space instead of comma for 
the delimiter.

The syntax for ESEARCH results is the RFC 3501 sequence-set rule, which 
has comma-delimited message numbers, UIDs, or ranges (a range being a pair 
of colon delimited message numbers or UIDs).  It is not unreasonable to 
believe that the use of the same syntax indicates the same semantics.

The definition of sequence-set in RFC 3501 explicitly allows UID ranges to 
be used even when there are holes in the UID space.  Nothing in RFC 4731 
states that its definition of sequence-set is different from RFC 3501. 
In fact, there is text in RFC 4731 that clearly states that the 
sequence-set returned from ESEARCH is intended to be used as an RFC 3501 
sequence-set.

Nowhere in RFC 4731 does it say that an RFC 4731 sequence-set has new 
semantics that prohibit the server from returning ranges that encompass 
holes in monotonically assigned UID space.

If you assert that ESEARCH sequence-set has such semantics, and MUST NOT 
send ranges that have holes, then in effect ESEARCH:
  . creates a new type of sequence-set that is different from all other
    IMAP use of sequence-set
  . forces all ESEARCH results to have longer, more bandwidth-consuming,
    results even if the client has the UID map and has no need for such
    longer results
  . forces ESEARCH results to ALWAYS be longer than SEARCH results in any
    server that does not monotonically assign UID values.
  . as a consequence of the previous statement creates an implicit
    requirement, stated nowhere in any specification, that UIDs must be
    monotonically assigned in a mailbox.

I assert that ESEARCH sequence-set has no such semantics, and is:
  . compatible with RFC 3501
  . minimizes the results so that only non-matching messages cause there
    to be multiple entries in the sequence-set
  . does so even with servers that do not monotonically assign UID values
  . as a consequence of the previous states create no new implicit
    requirements

This does mean that ESEARCH can't be used to get the UID/msgno map as 
Pawel complains.  But there are two other commands in RFC 3501 that can 
accomplish that: FETCH and UID SEARCH.  FETCH has always been the official 
way.

It is a fortunate accident that UID SEARCH returns the map as well.  But 
how many applications that use UID SEARCH to obtain the UID/msgno map can 
handle:
 	tag UID SEARCH ALL
 	* SEARCH 4929 3012
 	* SEARCH 12 9993
 	tag OK done
Such a scenario is completely valid in RFC 3501!  If your client can't 
handle that, it's already broken.  Hence that strange warning in RFC 4731:
          [...]  Note, the client
          MUST NOT assume that messages/UIDs will be listed in any
          particular order.

> I think it is unwise for a server knowingly to emit responses that are
> likely to cause clients confusion no matter what the legalistic reading of
> the RFC might be.

I will remember that statement the next time someone claims that 
implementations must MIME-process message headers that lack a 
MIME-Version: and accept overlong MIME encoded-word tokens in a header.

Think of the effect of what you are saying.  For the benefit of clients 
that are not satisfied with having TWO ways of getting the UID/msgno map, 
IMAP must have THREE ways of getting the UID/msgno map; and to provide 
that third way, an advertised protocol optimization MUST be reduced and in 
some cases turned into a chattiness pessimization!  Given:
 	x01 UID SEARCH ALL
 	* SEARCH 12 3012 4929 5948 6666 7739 8199 8839 9017 9993
 	x01 OK done
Compare:
 	x02 UID SEARCH RETURN () ALL
 	* ESEARCH (TAG x02) ALL 12 3012 4929 5948 6666 7739 8199 8839 9017 9993
 	x02 OK done
with:
 	x02 UID SEARCH RETURN () ALL
 	* ESEARCH (TAG x02) ALL 12:9993
 	x02 OK done

Furthermore, this pessimization only happens with UIDs.  Message numbers 
won't get pessimized:
 	x03 SEARCH RETURN () ALL
 	* ESEARCH (TAG x02) ALL 1:10
 	x03 OK done

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Date: Sun, 12 Aug 2007 19:11:31 -0700
From: Peter Coates <[email-masked]>
Subject: RE: UID SEARCH responses
To: "'Mark Crispin'" <[email-masked]>, "'Pawel Salek'" <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

I am surprised at the heat of this conversation.  I would hope for
conversations that generated light, not heat.

I happen not to agree with Mark's position, but recognize that it has some
validity.  However his point about 64bit UID is overstating the case.
Having UIDs not increment without gaps is likely either to break things
(shouldn't) or at the very least invalidate assumptions built into
optimization in the protocol or people's implementations.  I would hope that
any GUID mechanism introduced to IMAP would be separate from UIDs.

Personally I think that having a search result that does not stand alone (as
in sends a range when some messages in the range do not exist) is bogus.
The results that are returned as a result of a UID SEARCH RETURN (UPDATE)
command explicitly preclude sending a range (eg 1:4) to mean 1,4 even if 2
and 3 do not exist.  I have only limited skin in this argument as I am
mostly interested in the server side.
I think it is unwise for a server knowingly to emit responses that are
likely to cause clients confusion no matter what the legalistic reading of
the RFC might be.

Peter

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of Mark Crispin
Sent: August 12, 2007 15:02
To: Pawel Salek
Cc: [email-masked]
Subject: Re: UID SEARCH responses

On Sun, 12 Aug 2007, Pawel Salek wrote:
> I wonder whether it is relevant or not but the copy of RFC3501 that I use
[1] 
> does not mention UIDs on page 61... Can you please be more specific?

Page 61 of RFC 3501 is the latter part of section 6.4.8.  That is the 
section that describes the UID command!

>> Unlike message numbers, the UID sets
>> 	38782:41424
>> and
>> 	38782,38789,40004,40301,40421,40814,41424
>> are exactly equivalent if no other UIDs exist within that range.
> "...if no other UIDs exists...". The client may not know whether they do
or 
> do not exist.

That doesn't matter.  The server knows, and that exact sequence will 
refer to the messages in question, and only the messages in question.

> The information conveyed to client is  different in these two cases. I see

> that the basically only thing the client can do with the response
generated 
> by your implementation is to send the sequence-set back to the server.

That is what RFC 4731 says as being the purpose of those ESEARCH results.

> It cannot use it to eg. purge local cache for message with UID 40000 
> because it may exist.

Hence there are two other mechanisms to get the UID map which will 
accomplish that purpose

> I cannot help the feeling you are way too dramatic here. I have no idea
how 
> IETF deals with imprecise specifications but my impression is it is not
the 
> first time there is some dispute how to interpret RFCs.

The IETF generally does not make incompatible changes to a published RFC 
(I-Ds are a different matter) that will have the effect of rendering 
previously compliant implementations non-compliant.  There have been 
lawsuits, and threats of lawsuits, over such matters in the past.

> The usefulness of the 
> response generated by current version of UW-IMAP is strongly limited and 
> considered by me, a client implementer, a bug.

That is your opinion.  I disagree.  I see no point to having the 
complexity of range syntax in returned UID results if it is limited to use 
for those cases where there are three or more monotonically consecutive 
matching UIDs.  I will vigorously fight any ex post facto incompatible 
change to the published semantics of ESEARCH.

> Please observe it does not say "usable ONLY in other IMAP commands. Users 
> must be cautious that the UID sequence sets may be collapsed by the server
if 
> separated only by non-existing UIDs". I do not think your argument is
valid.

The absence of text in an RFC can not be construed to indicate that the 
RFC takes the opposite position.

The absence of text in an RFC indicates just one thing: that the RFC is 
silent on that matter.

However (and this is important!!) if an RFC does not forbid something that 
is syntactically and semantically valid in the RFC, then that something is 
permitted EVEN IF the author of the RFC thinks that it is bad or wrong or 
stupid behavior.

It is true that RFCs can not forbid every possible bad or wrong or stupid 
behavior.  But if the RFC does not prohibit the behavior in question, and 
that behavior can be reasonably inferred from the specification, then the 
only way to deal with a conflict of interpretation is either to convince 
the other side of the rightness of ones interpretation, or to accept that 
both interpretations are valid.

I have encountered that problem innumerable times.  The majority of IMAP 
client implementations things which I consider to bad or wrong or stupid. 
But, because RFC 3501 (and its predecessors) did not specifically forbid 
the behavior in question, nothing can be done.

In this case, I consider my interpretation to be reasonably inferred from 
the specification AND reasonable in the overall context of IMAP.

>>  . my interpretation returns a valid sequence-set usable in other IMAP
>>     commands
> But your interpretation cannot be used for the same client-side tasks as 
> ordinary UID SEARCH response can.

You can use ordinary UID SEARCH if you need a UID map, and use ESEARCH to 
do traffic-efficient searches once you have the map.

>>  . your interpretation prevents optimization unless there are at least
>>     three contiguous extant UIDs matching the result; without such the
>>     ESEARCH response is larger than the SEARCH result.
> And that's quite common case, I think.

That assumes that UIDs are assigned monotonically by the server, and that 
messages are rarely expunged.

In fact, the implicit requirement in your interpretation that UIDs be 
assigned monotonically by the server is the final nail in the coffin. 
IMAP has NO such requirement.

This is on top of earlier nails, such as the assumption that sequence-set 
means one thing in RFC 3501 and something completely different in RFC 
4731.  Nothing confuses matters more than to have protocol elements that 
change their meaning depending upon context or defining document.  People 
criticize IMAP (justly, in my opinion) for being weighed down with a mass 
of complexity.

> Why not try fixing implementation in question and add a clarification in
RFC 
> 4731 errata?

That only works if the implementor agrees that it is a bug in the 
implementation.  He does not.

He thinks that including ranges in ESEARCH is pointless otherwise.

He objects strongly to the idea of IMAP having two types of UID ranges 
with a subtle semantic difference between the two, not to mention adding a 
new hidden requirement to IMAP that UIDs be assigned monotonically.  IMAP 
already has entirely too many subtle semantic differences and hidden 
requirements; this is not a case where "more is better".

> I do not see any client 
> author being happy with UID ESEARCH responses generated by current UW-IMAP
- 
> what could it be used for?

It returns shorter search results if you do not need a UID/msgno map. 
You only need to get a UID/msgno map once in a session; after that 
additions to the map are a simple operation and removals from the map are 
easily calculated.  What's more, two other ways exist in IMAP to get the 
UID/msgno map.

Searches, on the other hand, are done many times in a session.  Why should 
those results be weighed down with extra traffic in the assumption that
  (1) the client wants to get the map
  (2) the client does not want to use the two ways which exist in the base
      IMAP specification
  (3) UIDs are assigned monotonically increasing.

> I will be happy, too, with only adding a clarification to RFC 4731 that
the 
> UID ESEARCH responses may include non-existing UIDs for the benefit of 
> compression. Such interpetation makes the extension less useful for me but
is 
> not a deal-breaker.

Either that, or let it be.

I understand that you would like to have a UID map operation that 
compresses monotonically increasing UIDs into ranges, but does not 
compress UID ranges that are not monotonically increasing.  However, some 
servers do not assign monotonically increasing UIDs; and the operation is 
pointless on those servers compared to the existing UID map operations.

Furthermore, we need to recognize that IMAP's lock into a 32-bit world 
will not be forever.  We are already encountering implementations that 
incorrectly treat IMAP numbers as being signed (and thus limited to 
2GB-1).  The first domino that is likely to fall is the QUOTA extension, 
followed by the base specifications's limit of message size to 4GB-1 
octets.

32-bit UIDs are also in trouble.  Although 4GB messages in a mailbox seems 
unplausible, lots of people have expressed unhappiness with UIDs not being 
globally unique and that creates pressure on a 32-bit space.  Should the 
floodgates of 64-bit be opened, globally unique UIDs will rear their head 
and pretty much preclude monotonically increasing UIDs in a single 
mailbox.

I don't think that any IMAP extension should be designed with implicit 
assumptions that are not in the base specification, but are doomed.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Date: Sun, 12 Aug 2007 15:01:51 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: Pawel Salek <[email-masked]>
cc: [email-masked]
Subject: Re: UID SEARCH responses
Sender: [email-masked]

On Sun, 12 Aug 2007, Pawel Salek wrote:
> I wonder whether it is relevant or not but the copy of RFC3501 that I use [1] 
> does not mention UIDs on page 61... Can you please be more specific?

Page 61 of RFC 3501 is the latter part of section 6.4.8.  That is the 
section that describes the UID command!

>> Unlike message numbers, the UID sets
>> 	38782:41424
>> and
>> 	38782,38789,40004,40301,40421,40814,41424
>> are exactly equivalent if no other UIDs exist within that range.
> "...if no other UIDs exists...". The client may not know whether they do or 
> do not exist.

That doesn't matter.  The server knows, and that exact sequence will 
refer to the messages in question, and only the messages in question.

> The information conveyed to client is  different in these two cases. I see 
> that the basically only thing the client can do with the response generated 
> by your implementation is to send the sequence-set back to the server.

That is what RFC 4731 says as being the purpose of those ESEARCH results.

> It cannot use it to eg. purge local cache for message with UID 40000 
> because it may exist.

Hence there are two other mechanisms to get the UID map which will 
accomplish that purpose

> I cannot help the feeling you are way too dramatic here. I have no idea how 
> IETF deals with imprecise specifications but my impression is it is not the 
> first time there is some dispute how to interpret RFCs.

The IETF generally does not make incompatible changes to a published RFC 
(I-Ds are a different matter) that will have the effect of rendering 
previously compliant implementations non-compliant.  There have been 
lawsuits, and threats of lawsuits, over such matters in the past.

> The usefulness of the 
> response generated by current version of UW-IMAP is strongly limited and 
> considered by me, a client implementer, a bug.

That is your opinion.  I disagree.  I see no point to having the 
complexity of range syntax in returned UID results if it is limited to use 
for those cases where there are three or more monotonically consecutive 
matching UIDs.  I will vigorously fight any ex post facto incompatible 
change to the published semantics of ESEARCH.

> Please observe it does not say "usable ONLY in other IMAP commands. Users 
> must be cautious that the UID sequence sets may be collapsed by the server if 
> separated only by non-existing UIDs". I do not think your argument is valid.

The absence of text in an RFC can not be construed to indicate that the 
RFC takes the opposite position.

The absence of text in an RFC indicates just one thing: that the RFC is 
silent on that matter.

However (and this is important!!) if an RFC does not forbid something that 
is syntactically and semantically valid in the RFC, then that something is 
permitted EVEN IF the author of the RFC thinks that it is bad or wrong or 
stupid behavior.

It is true that RFCs can not forbid every possible bad or wrong or stupid 
behavior.  But if the RFC does not prohibit the behavior in question, and 
that behavior can be reasonably inferred from the specification, then the 
only way to deal with a conflict of interpretation is either to convince 
the other side of the rightness of ones interpretation, or to accept that 
both interpretations are valid.

I have encountered that problem innumerable times.  The majority of IMAP 
client implementations things which I consider to bad or wrong or stupid. 
But, because RFC 3501 (and its predecessors) did not specifically forbid 
the behavior in question, nothing can be done.

In this case, I consider my interpretation to be reasonably inferred from 
the specification AND reasonable in the overall context of IMAP.

>>  . my interpretation returns a valid sequence-set usable in other IMAP
>>     commands
> But your interpretation cannot be used for the same client-side tasks as 
> ordinary UID SEARCH response can.

You can use ordinary UID SEARCH if you need a UID map, and use ESEARCH to 
do traffic-efficient searches once you have the map.

>>  . your interpretation prevents optimization unless there are at least
>>     three contiguous extant UIDs matching the result; without such the
>>     ESEARCH response is larger than the SEARCH result.
> And that's quite common case, I think.

That assumes that UIDs are assigned monotonically by the server, and that 
messages are rarely expunged.

In fact, the implicit requirement in your interpretation that UIDs be 
assigned monotonically by the server is the final nail in the coffin. 
IMAP has NO such requirement.

This is on top of earlier nails, such as the assumption that sequence-set 
means one thing in RFC 3501 and something completely different in RFC 
4731.  Nothing confuses matters more than to have protocol elements that 
change their meaning depending upon context or defining document.  People 
criticize IMAP (justly, in my opinion) for being weighed down with a mass 
of complexity.

> Why not try fixing implementation in question and add a clarification in RFC 
> 4731 errata?

That only works if the implementor agrees that it is a bug in the 
implementation.  He does not.

He thinks that including ranges in ESEARCH is pointless otherwise.

He objects strongly to the idea of IMAP having two types of UID ranges 
with a subtle semantic difference between the two, not to mention adding a 
new hidden requirement to IMAP that UIDs be assigned monotonically.  IMAP 
already has entirely too many subtle semantic differences and hidden 
requirements; this is not a case where "more is better".

> I do not see any client 
> author being happy with UID ESEARCH responses generated by current UW-IMAP - 
> what could it be used for?

It returns shorter search results if you do not need a UID/msgno map. 
You only need to get a UID/msgno map once in a session; after that 
additions to the map are a simple operation and removals from the map are 
easily calculated.  What's more, two other ways exist in IMAP to get the 
UID/msgno map.

Searches, on the other hand, are done many times in a session.  Why should 
those results be weighed down with extra traffic in the assumption that
  (1) the client wants to get the map
  (2) the client does not want to use the two ways which exist in the base
      IMAP specification
  (3) UIDs are assigned monotonically increasing.

> I will be happy, too, with only adding a clarification to RFC 4731 that the 
> UID ESEARCH responses may include non-existing UIDs for the benefit of 
> compression. Such interpetation makes the extension less useful for me but is 
> not a deal-breaker.

Either that, or let it be.

I understand that you would like to have a UID map operation that 
compresses monotonically increasing UIDs into ranges, but does not 
compress UID ranges that are not monotonically increasing.  However, some 
servers do not assign monotonically increasing UIDs; and the operation is 
pointless on those servers compared to the existing UID map operations.

Furthermore, we need to recognize that IMAP's lock into a 32-bit world 
will not be forever.  We are already encountering implementations that 
incorrectly treat IMAP numbers as being signed (and thus limited to 
2GB-1).  The first domino that is likely to fall is the QUOTA extension, 
followed by the base specifications's limit of message size to 4GB-1 
octets.

32-bit UIDs are also in trouble.  Although 4GB messages in a mailbox seems 
unplausible, lots of people have expressed unhappiness with UIDs not being 
globally unique and that creates pressure on a 32-bit space.  Should the 
floodgates of 64-bit be opened, globally unique UIDs will rear their head 
and pretty much preclude monotonically increasing UIDs in a single 
mailbox.

I don't think that any IMAP extension should be designed with implicit 
assumptions that are not in the base specification, but are doomed.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Date: Sun, 12 Aug 2007 22:37:57 +0200
From: Pawel Salek <[email-masked]>
Subject: Re: UID SEARCH responses
To: Mark Crispin <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

On 08/12/2007 03:01:26 AM, Mark Crispin wrote:
> 
> On Sat, 11 Aug 2007, Dave Cridland wrote:
> >> For what it's worth, UW imapd issues the
> >> 	* ESEARCH (TAG ".") UID ALL 38782:41424
> >> response instead of
> >> 	* ESEARCH (TAG ".") UID ALL  
> 38782,38789,40004,40301,40421,40814,41424
> >> because there are no other UIDs in that range.
> > Ah. That's definitely not what I intended, and it never crossed my  
> mind that> this was a possible interpretation. The second is correct.
> 
> Please quote me anyplace in RFC 4731 where it says that.

> > Ouch. I'd disagree, ALL is defined as "Return all message  
> numbers/UIDs that> satisfy the SEARCH criteria", not "Return all  
> message numbers/UIDs that> either satisfy the SEARCH criteria, or  
> don't exist, if that's easier".
> 
> You didn't pay adequate attention to the definition of UIDs in RFC  
> 3501.Pay particular attention to the text on page 61.

I wonder whether it is relevant or not but the copy of RFC3501 that I  
use [1] does not mention UIDs on page 61... Can you please be more  
specific?

> Unlike message numbers, the UID sets
>  	38782:41424
> and
>  	38782,38789,40004,40301,40421,40814,41424
> are exactly equivalent if no other UIDs exist within that range.

"...if no other UIDs exists...". The client may not know whether they  
do or do not exist.
  The information conveyed to client is  different in these two cases.  
I see that the basically only thing the client can do with the response  
generated by your implementation is to send the sequence-set back to  
the server. It cannot use it to eg. purge local cache for message with  
UID 40000 because it may exist.

> Furthermore, RFC 4731 specifically refers to the ESEARCH result  
> beinginterpreted as in the base specification:
>  	 A sequence-set
>           representation may be more compact and can be used as is in  
> a
>           subsequent command that accepts sequence-set.
> 
> If it was your intention to require that ESEARCH returns the map with  
> UIDsthen RFC 4731 failed to state so.
> 
> > I disagree, I think you've misinterpreted the document. If enough  
> people> disagree, and think that UID sets are somehow defined to  
> include missing UIDs> - which I dispute - then I suppose we could  
> issue a revision that deprecates> ALL, replacing it with (perhaps)  
> RANGE and MAP.
> 
> You have to do more than that.  You have to deprecate all of ESEARCH,  
> andreplace it with different syntax.
> 
> The horse is out of the barn.  RFC 4731 does not forbid my  
> interpretation,and my interpretation is out in the field.

I cannot help the feeling you are way too dramatic here. I have no idea  
how IETF deals with imprecise specifications but my impression is it is  
not the first time there is some dispute how to interpret RFCs. The  
usefulness of the response generated by current version of UW-IMAP is  
strongly limited and considered by me, a client implementer, a bug.   
And you and I know it is not the first bug in the ESEARCH  
implementation of UW-IMAP in spite of short history of its  
implementation in this code base (ok, they are not exactly comparable:  
the first one was an obvious one!) so why not change this response as  
well and be done with it?

> > No. It's very unusual in practise for a UID map sent by UID SEARCH  
> RETURN ()> ALL to equal the size of a UID SEARCH ALL. Normally, it's  
> significantly> smaller. You'd need to have a maximum of two  
> contiguous UIDs anywhere in the> mailbox to equal the size, and  
> that's vanishingly rare in practise.
> 
> That statement makes no sense at all!
> 
> In your interpretation (that a UID range must encompass contiguous  
> extantUIDs) ESEARCH responses will always be larger than SEARCH  
> responses (dueto the additional stuff in SEARCH responses) UNLESS  
> there are at leastthree contiguous extant UIDs.  It is LESS likely  
> that there be threecontiguous extant UIDs than there be two  
> contiguous extant UIDs.
> 
> I'm taking a hard line, because:
>   . UID SEARCH ALL exists as a command to get the map

And? One of the points with ESEARCH is provide a compressed response.

>   . RFC 4731 specifically refers to the ESEARCH result being a  
> sequence-set
>      usable in other IMAP commands

Please observe it does not say "usable ONLY in other IMAP commands.  
Users must be cautious that the UID sequence sets may be collapsed by  
the server if separated only by non-existing UIDs". I do not think your  
argument is valid.

>   . my interpretation returns a valid sequence-set usable in other  
> IMAP
>      commands

But your interpretation cannot be used for the same client-side tasks  
as ordinary UID SEARCH response can.

>   . my interpretation optimizes ESEARCH return results

Your interpretation does it at the expense of usefulness.

>   . your interpretation prevents optimization unless there are at  
> least
>      three contiguous extant UIDs matching the result; without such  
> the
>      ESEARCH response is larger than the SEARCH result.

And that's quite common case, I think.

>   . the sole benefit of your interpretion is to be able to use  
> ESEARCH as
>      an alternative to get the map

It appears to me that disconnected, bandwidth-limited clients may see  
the ability to receive the compressed map as a big advantage.

>   . RFC 4731 says nothing about UID ranges in ESEARCH being different  
> from
>      IMAP ranges.  IMAP does not require that UID ranges refer to
>      contiguous extant UIDs; in fact RFC 3501 says quite the opposite.
>   . RFC 4731 uses the same syntax rule as RFC 3501 for UID ranges.

Syntax does not imply semantics. One can argue that the extension  
provided by RFC 4731 is called E-SEARCH and it would be fair to expect  
that the provided capability is not inferior to ordinary SEARCH.

> That isn't to say that I am not sympathetic.  I am.  Nonetheless, the  
> shoehas been on the other foot many times in the past, and I've had  
> to dealwith the resulting fiasco.
> 
> I recommend living with the unexpected consequence.  Trying to fix  
> ESEARCH(which I argue isn't broken) will just diminish what little  
> credibilityIMAP extensions still have.

Why not try fixing implementation in question and add a clarification  
in RFC 4731 errata? It's not like people have already coded hundreds of  
applications that rely on this specific behavior of UW-IMAP. I do not  
see any client author being happy with UID ESEARCH responses generated  
by current UW-IMAP - what could it be used for?

I will be happy, too, with only adding a clarification to RFC 4731 that  
the UID ESEARCH responses may include non-existing UIDs for the benefit  
of compression. Such interpetation makes the extension less useful for  
me but is not a deal-breaker.

Pawel

[1] http://tools.ietf.org/html/rfc3501

Date: Sat, 11 Aug 2007 18:01:26 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: Dave Cridland <[email-masked]>
cc: [email-masked]
Subject: Re: UID SEARCH responses
Sender: [email-masked]

On Sat, 11 Aug 2007, Dave Cridland wrote:
>> For what it's worth, UW imapd issues the
>> 	* ESEARCH (TAG ".") UID ALL 38782:41424
>> response instead of
>> 	* ESEARCH (TAG ".") UID ALL 38782,38789,40004,40301,40421,40814,41424
>> because there are no other UIDs in that range.
> Ah. That's definitely not what I intended, and it never crossed my mind that 
> this was a possible interpretation. The second is correct.

Please quote me anyplace in RFC 4731 where it says that.

> Ouch. I'd disagree, ALL is defined as "Return all message numbers/UIDs that 
> satisfy the SEARCH criteria", not "Return all message numbers/UIDs that 
> either satisfy the SEARCH criteria, or don't exist, if that's easier".

You didn't pay adequate attention to the definition of UIDs in RFC 3501. 
Pay particular attention to the text on page 61.

Unlike message numbers, the UID sets
 	38782:41424
and
 	38782,38789,40004,40301,40421,40814,41424
are exactly equivalent if no other UIDs exist within that range.

Furthermore, RFC 4731 specifically refers to the ESEARCH result being 
interpreted as in the base specification:
 	 A sequence-set
          representation may be more compact and can be used as is in a
          subsequent command that accepts sequence-set.

If it was your intention to require that ESEARCH returns the map with UIDs 
then RFC 4731 failed to state so.

> I disagree, I think you've misinterpreted the document. If enough people 
> disagree, and think that UID sets are somehow defined to include missing UIDs 
> - which I dispute - then I suppose we could issue a revision that deprecates 
> ALL, replacing it with (perhaps) RANGE and MAP.

You have to do more than that.  You have to deprecate all of ESEARCH, and 
replace it with different syntax.

The horse is out of the barn.  RFC 4731 does not forbid my interpretation, 
and my interpretation is out in the field.

> No. It's very unusual in practise for a UID map sent by UID SEARCH RETURN () 
> ALL to equal the size of a UID SEARCH ALL. Normally, it's significantly 
> smaller. You'd need to have a maximum of two contiguous UIDs anywhere in the 
> mailbox to equal the size, and that's vanishingly rare in practise.

That statement makes no sense at all!

In your interpretation (that a UID range must encompass contiguous extant 
UIDs) ESEARCH responses will always be larger than SEARCH responses (due 
to the additional stuff in SEARCH responses) UNLESS there are at least 
three contiguous extant UIDs.  It is LESS likely that there be three 
contiguous extant UIDs than there be two contiguous extant UIDs.

I'm taking a hard line, because:
  . UID SEARCH ALL exists as a command to get the map
  . RFC 4731 specifically refers to the ESEARCH result being a sequence-set
     usable in other IMAP commands
  . my interpretation returns a valid sequence-set usable in other IMAP
     commands
  . my interpretation optimizes ESEARCH return results
  . your interpretation prevents optimization unless there are at least
     three contiguous extant UIDs matching the result; without such the
     ESEARCH response is larger than the SEARCH result
  . the sole benefit of your interpretion is to be able to use ESEARCH as
     an alternative to get the map
  . RFC 4731 says nothing about UID ranges in ESEARCH being different from
     IMAP ranges.  IMAP does not require that UID ranges refer to
     contiguous extant UIDs; in fact RFC 3501 says quite the opposite.
  . RFC 4731 uses the same syntax rule as RFC 3501 for UID ranges.

That isn't to say that I am not sympathetic.  I am.  Nonetheless, the shoe 
has been on the other foot many times in the past, and I've had to deal 
with the resulting fiasco.

I recommend living with the unexpected consequence.  Trying to fix ESEARCH 
(which I argue isn't broken) will just diminish what little credibility 
IMAP extensions still have.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Subject: Re: UID SEARCH responses
Date: Sat, 11 Aug 2007 22:59:46 +0100
From: Dave Cridland <[email-masked]>
To: Mark Crispin <[email-masked]>
Cc: <[email-masked]>
Sender: [email-masked]

On Fri Aug 10 06:47:40 2007, Mark Crispin wrote:
> 
> For what it's worth, UW imapd issues the
> 	* ESEARCH (TAG ".") UID ALL 38782:41424
> response instead of
> 	* ESEARCH (TAG ".") UID ALL 
> 38782,38789,40004,40301,40421,40814,41424
> because there are no other UIDs in that range.
> 
> 
Ah. That's definitely not what I intended, and it never crossed my 
mind that this was a possible interpretation. The second is correct.

> Pawel thought this is wrong, and that a UID range of 38782:41424 
> must mean all the possible values are in the map.
> 
> 
It does. Or, more accurately, it must mean that all those UIDs match 
the SEARCH criteria.

> I suggested that he bring the matter up on this list.  I reviewed 
> the ESEARCH RFC, and can find nothing to indicate that ESEARCH must 
> deliver a UID map (as SEARCH does) rather than a UID sequence set.
> 
> 
Ouch. I'd disagree, ALL is defined as "Return all message 
numbers/UIDs that satisfy the SEARCH criteria", not "Return all 
message numbers/UIDs that either satisfy the SEARCH criteria, or 
don't exist, if that's easier".

UID set syntax expands to all UIDs. The fact that UID FETCH etc 
happen to ignore UIDs means that clients MAY include non-existent 
UIDs as a method of reducing the bandwidth (or command length), and 
has nothing whatsoever to do with the set syntax itself. I can't find 
anything in RFC3501 which implies otherwise.

RFC4315 happens to insist on not including "extraneous UIDs", but 
that's belt and braces - I'd be happy to include such a phrase in a 
revision if it'd make people happier.

> If Pawel is correct on the intent, then the ESEARCH RFC is broken 
> since it is ambiguous on that point; either live with the 
> ambiguity, or deprecate ESEARCH and replace it with some other 
> syntax.  The horse is already out of the barn.
> 
> 
I disagree, I think you've misinterpreted the document. If enough 
people disagree, and think that UID sets are somehow defined to 
include missing UIDs - which I dispute - then I suppose we could 
issue a revision that deprecates ALL, replacing it with (perhaps) 
RANGE and MAP.

> I contend that if a client wants the map, then "UID SEARCH ALL" is 
> the correct command.  The only advantage to ESEARCH is getting the 
> tag, which I don't think is much of an advantage for that 
> particular command.

No. It's very unusual in practise for a UID map sent by UID SEARCH 
RETURN () ALL to equal the size of a UID SEARCH ALL. Normally, it's 
significantly smaller. You'd need to have a maximum of two contiguous 
UIDs anywhere in the mailbox to equal the size, and that's 
vanishingly rare in practise.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Date: Sat, 11 Aug 2007 18:10:28 +0100
From: Alexey Melnikov <[email-masked]>
To: Mark Crispin <[email-masked]>
CC: IMAP Extensions WG <[email-masked]>
Subject: Re: I-D ACTION:draft-ietf-imapext-i18n-12.txt
Sender: [email-masked]

Mark Crispin wrote:

> Here's my feeling about i;octet and i;ascii-casemap.  This feeling was 
> also how I implemented UW imapd.
>
> Given:
>
> i;octet is essentially strcmp() with no localization.
>
> i;ascii-casemap is what the IMAP base specification describes.  But 
> the IMAP base specification never intended to preclude 
> i;unicode-casemap, and even in the old days there were efforts in the 
> direction of what eventually became i;unicode-casemap.
>
> I see no reason why any client would deliberately choose i;octet or 
> i;ascii-casemap instead of i;unicode-casemap.

I can see reasons why people would want to use i;octet (Dave Cridland's 
example of searching for exact substring "OMA" where case-insensitive 
"oma" returns too many false positives).
But I don't see any reasons for choosing i;ascii-casemap instead of 
i;unicode-casemap

> We strongly want to encourage servers to implement i;unicode-casemap 
> instead of i;ascii-casemap as the default.

Agreed.

> Therefore, I believe that:
>
> i;octet and i;ascii-casemap are documented for the purpose of 
> documentation; to describe how old software worked.  It may also be 
> useful to have some sort of i;unknown-locale documented to describe 
> strtmp() with localization.  All of these are *optional to implement* 
> in servers that support IMAPI18N, and probably the only reason to 
> implement them is as an exercise.
>
> i;unicode-casemap is mandatory-to-implement, and default, in servers 
> that advertise IMAPI18N.

This would work for me.

> Servers that do not advertise IMAPI18N are strongly encouraged by this 
> specification to upgrade any legacy collation (i;octet, 
> i;ascii-casemap, i;unknown-locale) to i;unicode-casemap.
>
> Clients are on notice that, if the server does not advertise IMAPI18N, 
> it MAY have some legacy collation instead of i;unicode-casemap.

Agreed.

> Put another way:
>  . all IMAPI18N servers do i;unicode-casemap by default.
>  . some non-IMAPI18N do i;unicode-casemap as their sole collation; others
>    do a legacy collation.  There is no way for the client to tell.
>
> This may be an important enough issue to have a separate capability 
> specifically to say "even though I am not yet an IMAPI18N server, I do 
> i;unicode-casemap collation."

I think this would be worthwhile.

> I have not thoroughly reviewed IMAPI18N to determine the timeline to 
> make UW imapd compliant with IMAPI18N.  But I'd be willing to add a 
> capability now for the i;unicode-casemap collation, and (hint hint) 
> now would be an excellent time to do so.

Right. I think IMAP I18N is the right place for the new capability.

> A couple of months from now will be too late.

> In the long-term scheme of things, non-IMAPI18N servers will become a 
> vanished species, just as IMAP2 servers are today.

Date: Sat, 11 Aug 2007 11:39:40 +0100
From: Alexey Melnikov <[email-masked]>
To: Pawel Salek <[email-masked]>
CC: [email-masked]
Subject: Re: UID SEARCH responses
Sender: [email-masked]

Pawel Salek wrote:

> Hi,

Hi Pawel,

> I wonder which of the alternatives is a proper ESEARCH response to a  
> following search:
>
> . uid search 28:34
> * SEARCH 38782 38789 40004 40301 40421 40814 41424
> . OK UID SEARCH completed
>
> If I now issue corresponding ESEARCH command:
> . uid search return (all) 28:34
>
> What should the IMAP server answer:
> * ESEARCH (TAG ".") UID ALL 38782,38789,40004,40301,40421,40814,41424
> or
> * ESEARCH (TAG ".") UID ALL 38782:41424
> . OK UID SEARCH completed
> or maybe something else?
>
> The argument for the first form is that it represents a compressed 
> form  of the ordinary search response (OK, not very much compressed 
> this  time), and for the latter, that it can be used as the 
> sequence-set  argument in a subsequent command, as demanded by the RFC.

I agree with Arnt that both resulting UID sets can be used in subsequent 
commands and I can't think of any practical difference at the moment.

Can you show me how you are using the resulting UID set? I would like to 
better understand what you are trying to do and maybe suggest alternatives.

Date: Sat, 11 Aug 2007 11:36:10 +0100
From: Alexey Melnikov <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
CC: [email-masked], Pawel Salek <[email-masked]>
Subject: Re: UID SEARCH responses
Sender: [email-masked]

Arnt Gulbrandsen wrote:

> Pawel Salek writes:
>
>> The argument for the first form is that it represents a compressed 
>> form  of the ordinary search response (OK, not very much compressed 
>> this  time), and for the latter, that it can be used as the 
>> sequence-set argument in a subsequent command, as demanded by the RFC.
>
> So can the latter... if UIDs 1 and 3 exist, and UID does 2, then the 
> two following commands have the same result:
>
> a uid fetch 1:3 bodystructure
> b uid fetch 1,3 bodystructure
>
> I can't think of any difference. Uid store, uid fetch, uid search, 
> search, ...

I can't think of any difference for standard commands.
There might be some minor difference when this is used with QRESYNC 
extension, but I need to double check.

Subject: Re: I-D ACTION:draft-ietf-imapext-i18n-12.txt
From: Timo Sirainen <[email-masked]>
To: IMAP Extensions WG <[email-masked]>
Date: Fri, 10 Aug 2007 22:12:26 +0300
Sender: [email-masked]

--=-xLIbs8snByjq4V8NfzwS
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

On Fri, 2007-08-10 at 22:02 +0300, Timo Sirainen wrote:
> "The "i;octet" collation is a simple and fast collation intended for use
> on binary octet strings rather than on character data."
>=20
> I understand that to mean that no character set conversions are done
> before comparing the data, or possibly that if it is done it's possible
> to find e.g. invalid UTF-8 strings. If that's the case I'm strongly
> against having i;octet as a requirement.

Unless it searches only non-text MIME parts. Then it could be useful.

--=-xLIbs8snByjq4V8NfzwS
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

--=-xLIbs8snByjq4V8NfzwS--

Subject: Re: I-D ACTION:draft-ietf-imapext-i18n-12.txt
From: Timo Sirainen <[email-masked]>
To: Mark Crispin <[email-masked]>
Cc: IMAP Extensions WG <[email-masked]>
Date: Fri, 10 Aug 2007 22:02:45 +0300
Sender: [email-masked]

--=-O/5+K7Wbik+nzHqr+36O
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

On Fri, 2007-08-10 at 10:37 -0700, Mark Crispin wrote:
> i;octet is essentially strcmp() with no localization.

"The "i;octet" collation is a simple and fast collation intended for use
on binary octet strings rather than on character data."

I understand that to mean that no character set conversions are done
before comparing the data, or possibly that if it is done it's possible
to find e.g. invalid UTF-8 strings. If that's the case I'm strongly
against having i;octet as a requirement.

> I have not thoroughly reviewed IMAPI18N to determine the timeline to make=
=20
> UW imapd compliant with IMAPI18N.  But I'd be willing to add a capability=
=20
> now for the i;unicode-casemap collation, and (hint hint) now would be an=20
> excellent time to do so.  A couple of months from now will be too late.

Same here. I've implemented i;unicode-casemap as default for upcoming
Dovecot v1.1 but not i18n.

BTW. Does anyone have test cases to verify i;unicode-casemap
implementation correctness?

--=-O/5+K7Wbik+nzHqr+36O
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

--=-O/5+K7Wbik+nzHqr+36O--

Date: Fri, 10 Aug 2007 10:37:16 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: IMAP Extensions WG <[email-masked]>
Subject: Re: I-D ACTION:draft-ietf-imapext-i18n-12.txt 
Sender: [email-masked]

Here's my feeling about i;octet and i;ascii-casemap.  This feeling was 
also how I implemented UW imapd.

Given:

i;octet is essentially strcmp() with no localization.

i;ascii-casemap is what the IMAP base specification describes.  But the 
IMAP base specification never intended to preclude i;unicode-casemap, and 
even in the old days there were efforts in the direction of what 
eventually became i;unicode-casemap.

I see no reason why any client would deliberately choose i;octet or 
i;ascii-casemap instead of i;unicode-casemap.

We strongly want to encourage servers to implement i;unicode-casemap 
instead of i;ascii-casemap as the default.

Therefore, I believe that:

i;octet and i;ascii-casemap are documented for the purpose of 
documentation; to describe how old software worked.  It may also be useful 
to have some sort of i;unknown-locale documented to describe strtmp() with 
localization.  All of these are *optional to implement* in servers that 
support IMAPI18N, and probably the only reason to implement them is as an 
exercise.

i;unicode-casemap is mandatory-to-implement, and default, in servers that 
advertise IMAPI18N.

Servers that do not advertise IMAPI18N are strongly encouraged by this 
specification to upgrade any legacy collation (i;octet, i;ascii-casemap, 
i;unknown-locale) to i;unicode-casemap.

Clients are on notice that, if the server does not advertise IMAPI18N, it 
MAY have some legacy collation instead of i;unicode-casemap.

Put another way:
  . all IMAPI18N servers do i;unicode-casemap by default.
  . some non-IMAPI18N do i;unicode-casemap as their sole collation; others
    do a legacy collation.  There is no way for the client to tell.

This may be an important enough issue to have a separate capability 
specifically to say "even though I am not yet an IMAPI18N server, I do 
i;unicode-casemap collation."

I have not thoroughly reviewed IMAPI18N to determine the timeline to make 
UW imapd compliant with IMAPI18N.  But I'd be willing to add a capability 
now for the i;unicode-casemap collation, and (hint hint) now would be an 
excellent time to do so.  A couple of months from now will be too late.

In the long-term scheme of things, non-IMAPI18N servers will become a 
vanished species, just as IMAP2 servers are today.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

To: [email-masked]
Cc: [email-masked]
From: [email-masked]
Subject: I-D ACTION:draft-ietf-imapext-i18n-12.txt 
Date: Fri, 10 Aug 2007 13:15:01 -0400
Sender: [email-masked]

--NextPart

A New Internet-Draft is available from the on-line Internet-Drafts 
directories.
This draft is a work item of the Internet Message Access Protocol Extension Working Group of the IETF.

	Title		: Internet Message Access Protocol Internationalization
	Author(s)	: A. Gulbrandsen, C. Newman
	Filename	: draft-ietf-imapext-i18n-12.txt
	Pages		: 16
	Date		: 2007-8-10
	
Internet Message Access Protocol (IMAP) version 4rev1 has basic
    support for non-ASCII characters in mailbox names and search
    substrings.  It also supports non-ASCII message headers and content
    encoded as specified by Multipurpose Internet Mail Extensions
    (MIME).  This specification defines a collection of IMAP extensions
    which improve international support including comparator negotiation
    for search, sort and thread, language negotiation for international
    error text, and translations for namespace prefixes.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-imapext-i18n-12.txt

To remove yourself from the I-D Announcement list, send a message to 
[email-masked] with the word unsubscribe in the body of 
the message. 
You can also visit https://www1.ietf.org/mailman/listinfo/I-D-announce 
to change your subscription settings.

Internet-Drafts are also available by anonymous FTP. Login with the 
username "anonymous" and a password of your e-mail address. After 
logging in, type "cd internet-drafts" and then 
"get draft-ietf-imapext-i18n-12.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html 
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

Internet-Drafts can also be obtained by e-mail.

Send a message to:
	[email-masked].
In the body type:
	"FILE /internet-drafts/draft-ietf-imapext-i18n-12.txt".
	
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

--NextPart
Content-Type: Multipart/Alternative; Boundary="OtherAccess"

--OtherAccess
Content-Type: Message/External-body;
	access-type="mail-server";
	server="[email-masked]"

Content-Type: text/plain
Content-ID:	<[email-masked]>

ENCODING mime
FILE /internet-drafts/draft-ietf-imapext-i18n-12.txt

--OtherAccess
Content-Type: Message/External-body;
	name="draft-ietf-imapext-i18n-12.txt";
	site="ftp.ietf.org";
	access-type="anon-ftp";
	directory="internet-drafts"

Content-Type: text/plain
Content-ID:	<[email-masked]>

--OtherAccess--

--NextPart--

Date: Fri, 10 Aug 2007 13:52:27 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: i18n-12, WGLC
Sender: [email-masked]

Hi,

Pete, please issue a WGLC when you see -12. I posted it.

I'll be away for two weeks starting tomorrow morning. I will not read 
mail during the first week, and only occasionally during the second.

-12 says this:

     A server that advertises this extension MUST implement the
     i;ascii-casemap and i;octet comparators, as defined in [RFC4790].  A
     server intended to be deployed globally MUST implement the
     i;unicode-casemap comparator, as defined in [UCM].

     A server that advertises this extension SHOULD use i;ascii-casemap as
     the default comparator.  The selection of
     the default comparator MAY be adjustable by the server administrator,
     and MAY be sensitive to the current user.  Once the IMAP connection
     enters authenticated state, the default comparator MUST remain static
     for the remainder of that connection.

There are problems.

1. UW doesn't use that collation as default, and site administrators can 
change it too. It seems vaguely desirable that IMAP has a default and 
clients can know what it is, though...?

2. Dan can't easily implement i;ascii-casemap, and wants it out. Timo 
(less strongly?) wants both i;ascii-casemap and i;octet out.

3. In 4790, I (stupidly) already said that i;ascii-casemap is the 
default for IMAP. I didn't know about the UW change to 
i;unicode-casemap yet then.

In consequence, I believe that the two paragraphs have to be replaced 
with new text. But I can't make up my mind about what to write.

I don't have strong viewpoints myself, and cannot see any pleasant 
solution. A little bit of WG discussion and rough consensus would 
please me greatly. I'll be away at the start and around at the end of 
the WGLC period, and will type something that seems to reflect 
consensus.

Arnt

Date: Fri, 10 Aug 2007 12:11:16 +0200
From: Pawel Salek <[email-masked]>
Subject: Re: UID SEARCH responses
To: Mark Crispin <[email-masked]>
Cc: IMAP Extensions WG <[email-masked]>
Sender: [email-masked]

On 08/10/2007 07:47:40 AM, Mark Crispin wrote:
> 
> For what it's worth, UW imapd issues the
>  	* ESEARCH (TAG ".") UID ALL 38782:41424
> response instead of
>  	* ESEARCH (TAG ".") UID ALL  
> 38782,38789,40004,40301,40421,40814,41424
> because there are no other UIDs in that range.

This is the knowledge that the server has. Can client use the first type  
of response to eg. purge its cache? Such response is useful if it meant  
to send back to server but its interpretation of it on the client side  
provides less information than the second type of the response.

> Pawel thought this is wrong, and that a UID range of 38782:41424 must  
> meanall the possible values are in the map.
> 
> I suggested that he bring the matter up on this list.  I reviewed  
> theESEARCH RFC, and can find nothing to indicate that ESEARCH must  
> deliver aUID map (as SEARCH does) rather than a UID sequence set.

I agree it is important to settle this matter early on before  
incompatible client and server implementations spread.

As a client implementer, I think the single interval response  
(38782:41424) is less useful than the complete list of UIDs. The single  
interval can be achieved by other means (eg by SEARCH RETURN (MIN MAX)  
28:34, or in this very case a plain FETCH 28,34 UID) and only the  
complete list of UIDs provides new, orthogonal functionality as the  
method to fetch the UIDs of messages in given range.

> If Pawel is correct on the intent, then the ESEARCH RFC is broken since  
> itis ambiguous on that point; either live with the ambiguity, or  
> deprecateESEARCH and replace it with some other syntax.  The horse is  
> already outof the barn.
> 
> I contend that if a client wants the map, then "UID SEARCH ALL" is  
> thecorrect command.  The only advantage to ESEARCH is getting the tag,  
> whichI don't think is much of an advantage for that particular command.

There is a second - bandwidth - advantage connected with the use of  
ESEARCH the way I interpret it, as stated in the introduction:
"These result options allow clients to get SEARCH results in more  
convenient forms, while also saving bandwidth required to transport the  
results"

In the example I have given, the compression of the map is not present  
but it may be quite substantial for archive-type, append-only folders.

All in all, I will be happy with any interpretation as long as it is  
unique.

Pawel

Date: Fri, 10 Aug 2007 10:33:37 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: UID SEARCH responses
Cc: Pawel Salek <[email-masked]>
Sender: [email-masked]

Pawel Salek writes:
> The argument for the first form is that it represents a compressed 
> form  of the ordinary search response (OK, not very much compressed 
> this  time), and for the latter, that it can be used as the 
> sequence-set argument in a subsequent command, as demanded by the 
> RFC.

So can the latter... if UIDs 1 and 3 exist, and UID does 2, then the two 
following commands have the same result:

a uid fetch 1:3 bodystructure
b uid fetch 1,3 bodystructure

I can't think of any difference. Uid store, uid fetch, uid search, search, ...

Arnt

Date: Thu, 9 Aug 2007 22:47:40 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: IMAP Extensions WG <[email-masked]>
Subject: Re: UID SEARCH responses
Sender: [email-masked]

For what it's worth, UW imapd issues the
 	* ESEARCH (TAG ".") UID ALL 38782:41424
response instead of
 	* ESEARCH (TAG ".") UID ALL 38782,38789,40004,40301,40421,40814,41424
because there are no other UIDs in that range.

Pawel thought this is wrong, and that a UID range of 38782:41424 must mean 
all the possible values are in the map.

I suggested that he bring the matter up on this list.  I reviewed the 
ESEARCH RFC, and can find nothing to indicate that ESEARCH must deliver a 
UID map (as SEARCH does) rather than a UID sequence set.

If Pawel is correct on the intent, then the ESEARCH RFC is broken since it 
is ambiguous on that point; either live with the ambiguity, or deprecate 
ESEARCH and replace it with some other syntax.  The horse is already out 
of the barn.

I contend that if a client wants the map, then "UID SEARCH ALL" is the 
correct command.  The only advantage to ESEARCH is getting the tag, which 
I don't think is much of an advantage for that particular command.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Date: Fri, 10 Aug 2007 07:27:04 +0200
From: Pawel Salek <[email-masked]>
Subject: UID SEARCH responses
To: [email-masked]
Sender: [email-masked]

Hi,

I wonder which of the alternatives is a proper ESEARCH response to a  
following search:

. uid search 28:34
* SEARCH 38782 38789 40004 40301 40421 40814 41424
. OK UID SEARCH completed

If I now issue corresponding ESEARCH command:
. uid search return (all) 28:34

What should the IMAP server answer:
* ESEARCH (TAG ".") UID ALL 38782,38789,40004,40301,40421,40814,41424
or
* ESEARCH (TAG ".") UID ALL 38782:41424
. OK UID SEARCH completed
or maybe something else?

The argument for the first form is that it represents a compressed form  
of the ordinary search response (OK, not very much compressed this  
time), and for the latter, that it can be used as the sequence-set  
argument in a subsequent command, as demanded by the RFC.

Pawel Salek

Date: Thu, 2 Aug 2007 06:34:21 -0700 (PDT)
From: Dan Karp <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: Timo Sirainen <[email-masked]>, [email-masked]
Subject: Re: I-D ACTION:draft-ietf-imapext-i18n-11.txt
Sender: [email-masked]

> > As I've mentioned earlier, that depends on where you're starting
> > from.  Supporting i;ascii-casemap -- especially supporting it in
> > anything resembling an efficient manner -- would require a large
> > amount of work for me.
> 
> Don't you use either UCS-2 or UCS-4? Isn't that just mapping A-Z to
> a-z and comparing the resulting code points?

Essentially it's an upcased UCS-2.  To do an octet compare would require opening every message file containing a potential match.

Date: Thu, 2 Aug 2007 15:26:11 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: I-D ACTION:draft-ietf-imapext-i18n-11.txt
Cc: Dan Karp <[email-masked]>, Timo Sirainen <[email-masked]>
Sender: [email-masked]

Dan Karp writes:
>>  My current inclination is to keep i;ascii-casemap a MUST and thereby 
>>  avoid a discrepancy between the two documents. After all it's not 
>>  difficult to implement i;ascii-casemap.
>
> As I've mentioned earlier, that depends on where you're starting from. 
>  Supporting i;ascii-casemap -- especially supporting it in anything 
> resembling an efficient manner -- would require a large amount of 
> work for me.

Don't you use either UCS-2 or UCS-4? Isn't that just mapping A-Z to a-z 
and comparing the resulting code points?

Arnt

Date: Thu, 2 Aug 2007 05:01:25 -0700 (PDT)
From: Dan Karp <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: Timo Sirainen <[email-masked]>, [email-masked]
Subject: Re: I-D ACTION:draft-ietf-imapext-i18n-11.txt
Sender: [email-masked]

> My current inclination is to keep i;ascii-casemap a MUST and thereby 
> avoid a discrepancy between the two documents. After all it's not 
> difficult to implement i;ascii-casemap.

As I've mentioned earlier, that depends on where you're starting from.  Supporting i;ascii-casemap -- especially supporting it in anything resembling an efficient manner -- would require a large amount of work for me.

Date: Thu, 2 Aug 2007 12:59:07 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: I-D ACTION:draft-ietf-imapext-i18n-11.txt
Cc: Timo Sirainen <[email-masked]>
Sender: [email-masked]

Timo Sirainen writes:
> The only reason I see for i;ascii-casemap is if server developer is 
> lazy and doesn't want to bother implementing it. I don't see why a 
> client would want to use it if i;unicode-casemap is also available. I 
> would be happy if it was a "server MUST implement either one or both 
> of i;ascii-casemap and i;unicode-casemap".

Unfortunately RFC 4790 says this:

    In IMAP, the default collation is i;ascii-casemap, because its
    operations are understood to match IMAP's built-in operations.

I asked and tested before writing that (sometime between IETF LC and 
IESG review), but I didn't ask and test enough, so I didn't know that 
Mark's server changed to a different default collation some time ago.

My current inclination is to keep i;ascii-casemap a MUST and thereby 
avoid a discrepancy between the two documents. After all it's not 
difficult to implement i;ascii-casemap.

Arnt

Subject: Re: I-D ACTION:draft-ietf-imapext-i18n-11.txt
From: Timo Sirainen <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: [email-masked]
Date: Thu, 02 Aug 2007 13:42:10 +0300
Sender: [email-masked]

--=-IHqMYogKK8QzvPfaq7BO
Content-Type: text/plain; charset=ISO-8859-15
Content-Transfer-Encoding: quoted-printable

On Thu, 2007-08-02 at 12:25 +0200, Arnt Gulbrandsen wrote:
> Timo Sirainen writes:
> > Two things:
> >
> > 1. I thought i;unicode-casemap was supposed to be the only MUST impleme=
nt:
> > http://www1.ietf.org/mail-archive/web/lemonade/current/msg03559.html
> >
> > Is this still going to change?
>=20
> Do you mean that i;ascii-casemap and i;octet should not be MUSTs?

Yes.

> I=20
> don't care about i;octet, but leaving out i;ascii-casemap seems wrong=20
> somehow.

The only reason I see for i;ascii-casemap is if server developer is lazy
and doesn't want to bother implementing it. I don't see why a client
would want to use it if i;unicode-casemap is also available. I would be
happy if it was a "server MUST implement either one or both of
i;ascii-casemap and i;unicode-casemap".

> > What about matching invalid input with SEARCH? I think it should be=20
> > handled in implementation-specific way. I still receive ISO-8859-1=20
> > mails without charset header and I'd like to be able to SEARCH them.
..
> I have one concern which is not reflected in the text as it stands:  I=20
> think it would be bad and ugly if the success of a search depends on=20
> factors the user doesn't control. If you (as user) search for the word=20
> m=FCnchen, the success of your search should not depend on whether your=20
> IMAP client chooses to send 'charset utf-8 body m=FCnchen' or 'charset=20
> iso-8859-4 body m=FCnchen' (presuming the word is correctly encoded in=20
> both cases, with literals and blah).

I wouldn't mind about that, as long as it's not illegal for me to use a
configurable default charset in case it's missing. Dovecot isn't doing
that now, but probably in future when I get annoyed enough at this.

--=-IHqMYogKK8QzvPfaq7BO
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

--=-IHqMYogKK8QzvPfaq7BO--

Date: Thu, 2 Aug 2007 12:33:25 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: I-D ACTION:draft-ietf-imapext-i18n-11.txt
Cc: Mark Crispin <[email-masked]>
Sender: [email-masked]

Mark Crispin writes:
> The LANGUAGE extension is technically l10n and not i18n.  I won't go 
> as far as to suggest that this document be broken up into a separate 
> l10n and i18n document.  However, the document should not blur the 
> distinction and imply that l10n issues are i18n issues.  They aren't.

While I may agree, I'm not entirely sure, I would strongly prefer not to 
do that. Hardly anyone distinguishes between l10n and i18n anyway, and 
many people don't even know what l10n is. I'd rather be slightly fuzzy 
than use words the audience doesn't know.

Arnt

Date: Thu, 2 Aug 2007 12:27:18 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: I-D ACTION:draft-ietf-imapext-i18n-11.txt
Cc: Mark Crispin <[email-masked]>, Timo Sirainen <[email-masked]>
Sender: [email-masked]

Mark Crispin writes:
> As for codepoints that are not defined in the charset; this is 
> nebulous and fungible over time.  A codepoint which is undefined 
> today may be defined tommorrow by an update to the charset.  Thus, 
> nobody should depend upon an undefined codepoint remaining undefined; 
> nor should depend upon any particular error condition in that case.

Bad example. Sorry. ISO-2002-JP ESC $ B B ESC ( B or UTF-8 N A 0xCF V E 
are better. I've seen both in real mail, and I do not think IMAP 
clients should be permitted to send either with impunity. But it's a 
base IMAP issue and out of scope for this document, so I deleted the 
text.

Arnt

Date: Thu, 2 Aug 2007 12:25:50 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: I-D ACTION:draft-ietf-imapext-i18n-11.txt
Cc: Timo Sirainen <[email-masked]>
Sender: [email-masked]

Timo Sirainen writes:
> Two things:
>
> 1. I thought i;unicode-casemap was supposed to be the only MUST implement:
> http://www1.ietf.org/mail-archive/web/lemonade/current/msg03559.html
>
> Is this still going to change?

Do you mean that i;ascii-casemap and i;octet should not be MUSTs? I 
don't care about i;octet, but leaving out i;ascii-casemap seems wrong 
somehow.

> 2. "Strings encoded using unknown character encodings should never  
> match when using the SEARCH command, and should sort together with 
> invalid input"
>
> What about matching invalid input with SEARCH? I think it should be 
> handled in implementation-specific way. I still receive ISO-8859-1 
> mails without charset header and I'd like to be able to SEARCH them.

I started by explaining to you why you're wrong and ended by explaining 
to myself why that text is inappropriate ;) The problem is means to 
solve is out of scope for this document. I've deleted it now.

I have one concern which is not reflected in the text as it stands:  I 
think it would be bad and ugly if the success of a search depends on 
factors the user doesn't control. If you (as user) search for the word 
münchen, the success of your search should not depend on whether your 
IMAP client chooses to send 'charset utf-8 body münchen' or 'charset 
iso-8859-4 body münchen' (presuming the word is correctly encoded in 
both cases, with literals and blah).

Arnt

Date: Mon, 27 Aug 2007 11:31:20 +0100
From: Tony Finch <[email-masked]>
To: [email-masked], IMAP Protocol Interest List <[email-masked]>
Subject: Re: [Imap-protocol] Private FETCH items
Sender: [email-masked]

On Thu, 23 Aug 2007, Ken Murchison wrote:
>
> I probably wasn't clear in the use case.  Cyrus now has the ability to
> replicate a mailstore in near realtime.  Several sites (including CMU in the
> near future) are using this to have redundant mailstores in case of a hardware
> failure.  Rather than create a new tool or protocol to verify the consistency
> of the mailstore pairs, extending IMAP with a couple of new FETCH items seems
> like a reasonable thing to do.

I should note that the MD5 replica consistency check has been very useful
in finding lurking bugs in the code, both the original version and across
versions as the code has evolved. It's also good for discovering bit-
smashing hardware failures that haven't (yet) affected lower levels of the
os/driver/raid stack. You can only be sure of data integrity if you test
it end-to-end.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
IRISH SEA: SOUTHERLY, BACKING NORTHEASTERLY FOR A TIME, 3 OR 4. SLIGHT OR
MODERATE. SHOWERS. MODERATE OR GOOD, OCCASIONALLY POOR.

Date: Sun, 26 Aug 2007 16:12:41 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: UID SEARCH responses
Cc: Alexey Melnikov <[email-masked]>, Pawel Salek <[email-masked]>
Sender: [email-masked]

Alexey Melnikov writes:
> I can't think of any difference for standard commands.

I thought of two differences, both insignificant.

1. If the client has a cache and doesn't know which cached UIDs 
represent extant messages, it cannot use an ESEARCH response to 
retrieve messages from its cache. It has to clean its cache of expunged 
messages (as it should do anyway).

2. SEARCH implicitly returns a count of results, ESEARCH ALL does not, 
so clients who want to display a count of results have to use both 
COUNT and ALL.

Someday I'm going to check what my server does. I suspect it may return 
the shorter response where UID ranges may include nonexistsent 
messages, but I don't know. Or care. IMO this is a nonissue.

Arnt

Date: Fri, 24 Aug 2007 16:42:18 -0400
From: Ken Murchison <[email-masked]>
Organization: Carnegie Mellon University
To: Arnt Gulbrandsen <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: Private FETCH items
Sender: [email-masked]

Arnt Gulbrandsen wrote:
> Ken Murchison writes:
>> Arnt Gulbrandsen wrote:
>>> What does the GUID do?
>>
>> Uniquely identifies a message so that if the same message is present 
>> in more than one mailbox, its contents only gets replicated once.
> 
> I'm afraid I phrased the question badly. What does the code do when 
> asked for a GUID?

Pulls the message GUID out of the index file for the mailbox.

-- 
Kenneth Murchison
Systems Programmer
Project Cyrus Developer/Maintainer
Carnegie Mellon University

To: IMAP Extensions WG <[email-masked]>
From: Lisa Dusseault <[email-masked]>
Subject: Fwd: Last Call: draft-saintandre-jabberid (The Jabber-ID Header  Field) to Proposed Standard 
Date: Fri, 24 Aug 2007 09:56:20 -0700
Sender: [email-masked]

--Apple-Mail-10--786017334
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
	charset=US-ASCII;
	delsp=yes;
	format=flowed

FYI on this last call document relating to email header fields.   
Please note that some review on the Apps-review team list has already  
suggested this go to Experimental instead  (http://www1.ietf.org/mail- 
archive/web/apps-review/current/msg00062.html ).

Thanks,
Lisa

Begin forwarded message:
From: The IESG <[email-masked]>
Date: August 24, 2007 6:51:05 AM PDT
To: IETF-Announce <[email-masked]>
Subject: Last Call: draft-saintandre-jabberid (The Jabber-ID Header   
Field) to Proposed Standard
Reply-To: [email-masked]

The IESG has received a request from an individual submitter to consider
the following document:

- 'The Jabber-ID Header Field '
    <draft-saintandre-jabberid-05.txt> as a Proposed Standard

The IESG plans to make a decision in the next few weeks, and solicits
final comments on this action.  Please send substantive comments to the
[email-masked] mailing lists by 2007-09-21. Exceptionally,
comments may be sent to [email-masked] instead. In either case, please
retain the beginning of the Subject line to allow automated sorting.

The file can be obtained via
http://www.ietf.org/internet-drafts/draft-saintandre-jabberid-05.txt

IESG discussion can be tracked via
https://datatracker.ietf.org/public/pidtracker.cgi? 
command=view_id&dTag=14260&rfc_flag=0

_______________________________________________
IETF-Announce mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf-announce

--Apple-Mail-10--786017334
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=ISO-8859-1

<HTML><BODY style=3D"word-wrap: break-word; -khtml-nbsp-mode: space; =
-khtml-line-break: after-white-space; "><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">FYI on this =
last call document relating to email header fields.=A0 Please note that =
some review on the Apps-review team list has already suggested this go =
to Experimental instead=A0 (<A =
href=3D"http://www1.ietf.org/mail-archive/web/apps-review/current/msg00062=
.html"><FONT class=3D"Apple-style-span" =
color=3D"#0020E2">http://www1.ietf.org/mail-archive/web/apps-review/curren=
t/msg00062.html</FONT></A> ).</DIV><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal =
normal normal 12px/normal Helvetica; min-height: 14px; "><BR></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; ">Thanks,</DIV><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
">Lisa</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
margin-bottom: 0px; margin-left: 0px; font: normal normal normal =
12px/normal Helvetica; min-height: 14px; "><BR></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; ">Begin forwarded message:</DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; "><B>From: </B><FONT class=3D"Apple-style-span" =
color=3D"#001ED2">The IESG &lt;</FONT><A =
href=3D"mailto:[email-masked]"><FONT class=3D"Apple-style-span" =
color=3D"#0020E2">[email-masked]</FONT></A><FONT =
class=3D"Apple-style-span" color=3D"#001ED2">&gt;</FONT></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; "><B>Date: </B><FONT class=3D"Apple-style-span" =
color=3D"#001ED2">August 24, 2007 6:51:05 AM PDT</FONT></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; "><B>To: </B><FONT class=3D"Apple-style-span" =
color=3D"#001ED2">IETF-Announce &lt;</FONT><A =
href=3D"mailto:[email-masked]"><FONT class=3D"Apple-style-span" =
color=3D"#0020E2">[email-masked]</FONT></A><FONT =
class=3D"Apple-style-span" color=3D"#001ED2">&gt;</FONT></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; "><B>Subject: </B><FONT class=3D"Apple-style-span" =
color=3D"#001ED2"><B>Last Call: draft-saintandre-jabberid (The Jabber-ID =
Header</B></FONT><FONT class=3D"Apple-style-span" color=3D"#001ED2"><B>=A0=
 </B></FONT><FONT class=3D"Apple-style-span" color=3D"#001ED2"><B>Field) =
to Proposed Standard</B></FONT><FONT class=3D"Apple-style-span" =
color=3D"#001ED2"><B>=A0</B></FONT></DIV><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><B>Reply-To: =
</B><A href=3D"mailto:[email-masked]"><FONT class=3D"Apple-style-span" =
color=3D"#0020E2">[email-masked]</FONT></A></DIV><DIV style=3D"margin-top: =
0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: =
normal normal normal 12px/normal Helvetica; min-height: 14px; "><FONT =
class=3D"Apple-style-span" color=3D"#001ED2"><BR></FONT></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; "><FONT class=3D"Apple-style-span" color=3D"#001ED2">The=
 IESG has received a request from an individual submitter to =
consider</FONT><FONT class=3D"Apple-style-span" =
color=3D"#001ED2">=A0</FONT></DIV><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT =
class=3D"Apple-style-span" color=3D"#001ED2">the following =
document:</FONT></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
margin-bottom: 0px; margin-left: 0px; font: normal normal normal =
12px/normal Helvetica; min-height: 14px; "><FONT =
class=3D"Apple-style-span" color=3D"#001ED2"><BR></FONT></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; "><FONT class=3D"Apple-style-span" color=3D"#001ED2">- =
'The Jabber-ID Header Field '</FONT></DIV><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT =
class=3D"Apple-style-span" color=3D"#001ED2">=A0=A0 </FONT><FONT =
class=3D"Apple-style-span" =
color=3D"#001ED2">&lt;draft-saintandre-jabberid-05.txt&gt; as a Proposed =
Standard</FONT></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
margin-bottom: 0px; margin-left: 0px; font: normal normal normal =
12px/normal Helvetica; min-height: 14px; "><FONT =
class=3D"Apple-style-span" color=3D"#001ED2"><BR></FONT></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; "><FONT class=3D"Apple-style-span" color=3D"#001ED2">The=
 IESG plans to make a decision in the next few weeks, and =
solicits</FONT></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
margin-bottom: 0px; margin-left: 0px; "><FONT class=3D"Apple-style-span" =
color=3D"#001ED2">final comments on this action.</FONT><FONT =
class=3D"Apple-style-span" color=3D"#001ED2">=A0 </FONT><FONT =
class=3D"Apple-style-span" color=3D"#001ED2">Please send substantive =
comments to the</FONT></DIV><DIV style=3D"margin-top: 0px; margin-right: =
0px; margin-bottom: 0px; margin-left: 0px; "><A =
href=3D"mailto:[email-masked]">[email-masked]</A><FONT =
class=3D"Apple-style-span" color=3D"#001ED2"> mailing lists by =
2007-09-21. Exceptionally,</FONT><FONT class=3D"Apple-style-span" =
color=3D"#001ED2">=A0</FONT></DIV><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT =
class=3D"Apple-style-span" color=3D"#001ED2">comments may be sent to =
</FONT><A href=3D"mailto:[email-masked]"><FONT class=3D"Apple-style-span" =
color=3D"#0020E2">[email-masked]</FONT></A><FONT class=3D"Apple-style-span"=
 color=3D"#001ED2"> instead. In either case, please</FONT><FONT =
class=3D"Apple-style-span" color=3D"#001ED2">=A0</FONT></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; "><FONT class=3D"Apple-style-span" =
color=3D"#001ED2">retain the beginning of the Subject line to allow =
automated sorting.</FONT></DIV><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal =
normal normal 12px/normal Helvetica; min-height: 14px; "><FONT =
class=3D"Apple-style-span" color=3D"#001ED2"><BR></FONT></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; "><FONT class=3D"Apple-style-span" color=3D"#001ED2">The=
 file can be obtained via</FONT></DIV><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><A =
href=3D"http://www.ietf.org/internet-drafts/draft-saintandre-jabberid-05.t=
xt">http://www.ietf.org/internet-drafts/draft-saintandre-jabberid-05.txt</=
A></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: =
0px; margin-left: 0px; font: normal normal normal 12px/normal Helvetica; =
min-height: 14px; "><FONT class=3D"Apple-style-span" =
color=3D"#001ED2"><BR></FONT></DIV><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal =
normal normal 12px/normal Helvetica; min-height: 14px; "><FONT =
class=3D"Apple-style-span" color=3D"#001ED2"><BR></FONT></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; "><FONT class=3D"Apple-style-span" =
color=3D"#001ED2">IESG discussion can be tracked via</FONT></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; "><A =
href=3D"https://datatracker.ietf.org/public/pidtracker.cgi?command=3Dview_=
id&dTag=3D14260&rfc_flag=3D0">https://datatracker.ietf.org/public/pidtrack=
er.cgi?command=3Dview_id&amp;dTag=3D14260&amp;rfc_flag=3D0</A></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; font: normal normal normal 12px/normal Helvetica; =
min-height: 14px; "><FONT class=3D"Apple-style-span" =
color=3D"#001ED2"><BR></FONT></DIV><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal =
normal normal 12px/normal Helvetica; min-height: 14px; "><FONT =
class=3D"Apple-style-span" color=3D"#001ED2"><BR></FONT></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; "><FONT class=3D"Apple-style-span" =
color=3D"#001ED2">_______________________________________________</FONT></=
DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: =
0px; margin-left: 0px; "><FONT class=3D"Apple-style-span" =
color=3D"#001ED2">IETF-Announce mailing list</FONT></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; "><A =
href=3D"mailto:[email-masked]">[email-masked]</A></DIV><DI=
V style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; "><A =
href=3D"https://www1.ietf.org/mailman/listinfo/ietf-announce">https://www1=
.ietf.org/mailman/listinfo/ietf-announce</A></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; "><BR =
class=3D"khtml-block-placeholder"></DIV></BODY></HTML>=

--Apple-Mail-10--786017334--

Date: Fri, 24 Aug 2007 11:41:18 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: Private FETCH items
Cc: Ken Murchison <[email-masked]>, [email-masked]
Sender: [email-masked]

Ken Murchison writes:
> Arnt Gulbrandsen wrote:
>> What does the GUID do?
>
> Uniquely identifies a message so that if the same message is present 
> in more than one mailbox, its contents only gets replicated once.

I'm afraid I phrased the question badly. What does the code do when 
asked for a GUID?

Arnt

Date: Thu, 23 Aug 2007 10:34:06 -0400
From: Ken Murchison <[email-masked]>
Organization: Carnegie Mellon University
To: Arnt Gulbrandsen <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: Private FETCH items
Sender: [email-masked]

Arnt Gulbrandsen wrote:
> Sounds to me as the patch author really wants a FETCH COOKIE instead of 
> MD5/FILESIZE, returning an opaque cookie which should be the same if the 
> same message is present in two mailboxes/servers and different otherwise.

Correct.

> What does the GUID do?

Uniquely identifies a message so that if the same message is present in 
more than one mailbox, its contents only gets replicated once.

-- 
Kenneth Murchison
Systems Programmer
Project Cyrus Developer/Maintainer
Carnegie Mellon University

Date: Thu, 23 Aug 2007 10:32:53 -0400
From: Ken Murchison <[email-masked]>
Organization: Carnegie Mellon University
To: Lyndon Nerenberg <[email-masked]>
CC: IMAP Protocol Interest List <[email-masked]>, [email-masked]
Subject: Re: [Imap-protocol] Private FETCH items
Sender: [email-masked]

Lyndon Nerenberg wrote:
> 
> On Aug 22, 2007, at 12:05 PM, Ken Murchison wrote:
> 
>> FYI, the difference between RFC822.SIZE and RFC822.FILESIZE is that 
>> the former uses the cached size in the message index file, and the 
>> latter stat()s for the file size at the time of the request.
> 
> .FILESIZE is just  asking for trouble.  Nobody cares what the on-disk 
> representation is, and comparing file sizes is NOT the same as comparing 
> message content.

True.  The author of the patch may be using this for something other 
than what I was thinking.  I'll have to ask.

> .MD5 is useful in the general sense as a fast way to compare messages 
> for equality, although I think it would be better served in that context 
> if you could MD5 the body content without the 822 headers to be able to 
> handle Received: differences for the same message delivered to two or 
> more folders via different [SL]MTP transactions.  It would probably make 
> more sense to be able to generically MD5 by MIME body part.

I probably wasn't clear in the use case.  Cyrus now has the ability to 
replicate a mailstore in near realtime.  Several sites (including CMU in 
the near future) are using this to have redundant mailstores in case of 
a hardware failure.  Rather than create a new tool or protocol to verify 
the consistency of the mailstore pairs, extending IMAP with a couple of 
new FETCH items seems like a reasonable thing to do.

> What's a GUID?

As Mark deduced, its Global Unique ID

-- 
Kenneth Murchison
Systems Programmer
Project Cyrus Developer/Maintainer
Carnegie Mellon University

Date: Thu, 23 Aug 2007 09:03:30 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: Private FETCH items
Cc: Ken Murchison <[email-masked]>, [email-masked]
Sender: [email-masked]

Sounds to me as the patch author really wants a FETCH COOKIE instead of 
MD5/FILESIZE, returning an opaque cookie which should be the same if 
the same message is present in two mailboxes/servers and different 
otherwise.

What does the GUID do?

Arnt

Date: Wed, 22 Aug 2007 14:14:14 -0700
From: Mark Crispin <[email-masked]>
To: Lyndon Nerenberg <[email-masked]>
cc: Ken Murchison <[email-masked]>, [email-masked], IMAP Protocol Interest List <[email-masked]>
Subject: Re: [Imap-protocol] Private FETCH items
Organization: Networks & Distributed Computing
Sender: [email-masked]

On Wed, 22 Aug 2007, Lyndon Nerenberg wrote:
> .FILESIZE is just  asking for trouble.  Nobody cares what the on-disk 
> representation is, and comparing file sizes is NOT the same as comparing 
> message content.

I agree.  The c-client library (the beating heart of UW imapd, Pine, etc.) 
has a concept of an internal size, but that depends upon the mail store 
and thus is only used by mail store drivers.  It never gets as far as 
imapd (or other application), much less exported in the protocol.

Let us be clear here; any .FILESIZE value in any server which supports 
multiple mail store formats could (and WOULD) vary depending upon the 
mailbox format with no obvious indication to the user as to why.  This, by 
itself, should be enough cause to reject it.

> .MD5 is useful in the general sense as a fast way to compare messages for 
> equality, although I think it would be better served in that context if you 
> could MD5 the body content without the 822 headers to be able to handle 
> Received: differences for the same message delivered to two or more folders 
> via different [SL]MTP transactions.  It would probably make more sense to be 
> able to generically MD5 by MIME body part.

IMHO, this functionality belongs as part of CONVERT.  An MD5 checksum is, 
in effect, a conversion.

> What's a GUID?

Presumably this is a global UID which uniquely and permanently identifies 
the message on the server.

I think that IMAP should eventually have a GUID, but it needs careful 
design and review.  I don't think that we should take the first patch that 
comes along.

> But overall I guess I have to question why it's necessary to burden the 
> protocol with this just because somebody wrote some broken code.  I have done 
> an awful lot of server migrations over the years and I have never yet come up 
> against the problem this is trying to detect. I realize you intend this as 
> private data, but once it's in the wild, it will get used, for better and 
> worse.

I agree with this sentiment.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Subject: Re: [Imap-protocol] Private FETCH items
Date: Wed, 22 Aug 2007 22:13:42 +0100
From: Dave Cridland <[email-masked]>
To: Lyndon Nerenberg <[email-masked]>
Cc: <[email-masked]>, <[email-masked]>, Ken Murchison <[email-masked]>
Sender: [email-masked]

On Wed Aug 22 20:27:15 2007, Lyndon Nerenberg wrote:
> What's a GUID?
> 
> 
Past participle of the verb "to gooey".

> But overall I guess I have to question why it's necessary to burden 
>  the protocol with this just because somebody wrote some broken 
> code.   I have done an awful lot of server migrations over the 
> years and I  have never yet come up against the problem this is 
> trying to detect.  I realize you intend this as private data, but 
> once it's in the wild,  it will get used, for better and worse.

I can understand why adding private fetchable data would be in some 
way useful, though, irrespective of this particular case.

I'd personally opt for an X- prefix, at the very least. X-CMU- might 
be better. Then you have X-CMU-UID, X-CMU-CRC, X-CMU-STATSIZE, 
perhaps.

Note that I've proposed "CRC" instead of explicitly naming MD5, 
partly in case people are tempted to use it in clients, and partly to 
stop the flurry of emails you'll otherwise get telling you that MD5 
is insecure.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Cc: IMAP Protocol Interest List <[email-masked]>, [email-masked]
From: Lyndon Nerenberg <[email-masked]>
Subject: Re: [Imap-protocol] Private FETCH items
Date: Wed, 22 Aug 2007 12:27:15 -0700
To: Ken Murchison <[email-masked]>
Sender: [email-masked]

On Aug 22, 2007, at 12:05 PM, Ken Murchison wrote:

> FYI, the difference between RFC822.SIZE and RFC822.FILESIZE is that  
> the former uses the cached size in the message index file, and the  
> latter stat()s for the file size at the time of the request.

.FILESIZE is just  asking for trouble.  Nobody cares what the on-disk  
representation is, and comparing file sizes is NOT the same as  
comparing message content.

.MD5 is useful in the general sense as a fast way to compare messages  
for equality, although I think it would be better served in that  
context if you could MD5 the body content without the 822 headers to  
be able to handle Received: differences for the same message  
delivered to two or more folders via different [SL]MTP transactions.   
It would probably make more sense to be able to generically MD5 by  
MIME body part.

What's a GUID?

But overall I guess I have to question why it's necessary to burden  
the protocol with this just because somebody wrote some broken code.   
I have done an awful lot of server migrations over the years and I  
have never yet come up against the problem this is trying to detect.  
I realize you intend this as private data, but once it's in the wild,  
it will get used, for better and worse.

--lyndon

Date: Wed, 22 Aug 2007 15:05:18 -0400
From: Ken Murchison <[email-masked]>
Organization: Carnegie Mellon University
To: [email-masked], IMAP Extensions WG <[email-masked]>
Subject: Private FETCH items
Sender: [email-masked]

Folks,

I was recently given a patch to Cyrus IMAP which adds 3 new FETCH items 
(GUID, RFC822.MD5, and RFC822.FILESIZE) for a message which helps with 
consistency checking when a message store is replicated.

I'm considering applying this patch, since it will be useful for CMU and 
others, but I'd like to solicit opinions regarding adding new FETCH 
items vs. making these items new ANNOTATE entries (although Cyrus 
doesn't currently implement ANNOTATE).

FYI, the difference between RFC822.SIZE and RFC822.FILESIZE is that the 
former uses the cached size in the message index file, and the latter 
stat()s for the file size at the time of the request.  I'm not thrilled 
with the name of this item, so I'd be open to suggestions.

-- 
Kenneth Murchison
Systems Programmer
Project Cyrus Developer/Maintainer
Carnegie Mellon University

Date: Wed, 15 Aug 2007 00:12:18 -0400
From: Tony Hansen <[email-masked]>
To: ietf-822 mailing list <[email-masked]>
CC: SMTP Interest Group <[email-masked]>, LEMONADE WG <[email-masked]>, IMAP extensions mailing list <[email-masked]>, EAI WG <[email-masked]>, POP3 extensions mailing list <[email-masked]>, USEFOR WG <[email-masked]>, SIMPLE WG <[email-masked]>
Subject: Re: Mailing List Last Call for 2822 update internet-draft
Sender: [email-masked]

The mailing list last call for the 2822 update is now over.

I believe we have mailing list consensus on the issues that have been
raised during the last call. After an updated draft is posted by Pete,
we'll be moving the draft forward to IETF last call.

	Tony

Tony Hansen wrote:
> There have been some comments on draft-resnick-2822upd-* but they have
> dwindled down to none.
> 
> This is a "formal" Mailing List Last Call on
> draft-resnick-2822upd-02.txt. The last call will last for two weeks
> time, ending on August 10, 2007.
> 
> The document will be discussed on the 822 mailing list,
> <[email-masked]>. Please send your comments there.

Date: Mon, 13 Aug 2007 16:20:40 -0700 (Pacific Daylight Time)
From: Mark Crispin <[email-masked]>
To: Alexey Melnikov <[email-masked]>
cc: IMAP Extensions WG <[email-masked]>
Subject: Re: UID SEARCH responses
Organization: Networks & Distributed Computing
Sender: [email-masked]

Is there a problem with this plan as outlined below?  If not, then let's 
just do it and be done with the discussion.  It is undoubtably getting 
dreary to most people.

You need to do this work anyway, due to what I coyly allued-to as the 
"second reason" why UIDPLUS could not use the sequence-set rule. 
Currently, the following response is valid:
 	* ESEARCH (TAG "A283") UID ALL 2,10:*
Oops.

On Mon, 13 Aug 2007, Mark Crispin wrote:
> On Mon, 13 Aug 2007, Mark Crispin wrote:
>>> 2). Clarify which of the two choices should be used by ALL and add a new 
>>> return option for the other one.
>
> On second thought, this could work if you proceed as follows:
>
> Update RFC 4466 to include uid-set from RFC 4315.  Be sure to highlight that 
> uid-set MUST NOT include non-existent UIDs, whereas sequence-set does.
>
> Consider a possible sytax change for uid-set, e.g., using hyphens, to reduce 
> confusion.  That orphans UIDPLUS syntax, but the benefit may be worth not 
> adding another subtlety in IMAP.
>
> Define the new return option (and capability indicating its presence) to 
> return uid-set.
>
> As a quality of implementation note, a server which supports the new return 
> option maximally compresses in ALL.
>
> This maintains compatibility for all existing servers.  As a gesture of good 
> faith, I will pledge to implement the new return value as soon as it is 
> published as an RFC.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Date: Mon, 13 Aug 2007 16:05:08 -0700 (Pacific Daylight Time)
From: Mark Crispin <[email-masked]>
To: Dave Cridland <[email-masked]>
cc: Pawel Salek <[email-masked]>, [email-masked], Peter Coates <[email-masked]>
Subject: RE: UID SEARCH responses
Organization: Networks & Distributed Computing
Sender: [email-masked]

On Mon, 13 Aug 2007, Dave Cridland wrote:
> On Mon Aug 13 18:29:48 2007, Mark Crispin wrote:
>> 
>> On Mon, 13 Aug 2007, Dave Cridland wrote:
>>> Non-existent messages do not satisfy the SEARCH criteria
>> 
>> Why not in the case of UIDs?  They satisfy the criteria when they are 
>> arguments to the SEARCH commands.  Why don't they satisfy the criteria when 
>> the server returns it as a value?
>> 
>> 
> Eh? No, they don't. No message will satisfy a UID criteria containing as 
> argument a non-existent UID.

However, what we are talking about is not a single UID, but rather a 
range.  By definition in RFC 3501, messages can match a UID range, even if 
neither UID in the range currently exists.

> "UID SEARCH UID 1:*" gives you back all extant UIDs in the mailbox, not all 
> possible UIDs in the range. (Which'd be 1 through the highest UID existent at 
> the time of the execution of the command, of course).

No.  It would be the first assigned UID (not 1) through the highest UID 
existent at the time of the execution of the command.

> No, there's no inconsistency. Non-existent UIDs don't affect SEARCH's UID 
> criteria, and are explicitly ignored by the UID family of commands. 
> Otherwise, they mean what they say.

Where, in any of the IMAP specifications, is this "otherwise" stated? 
There is no such meaning in RFC 3501.  RFC 4731 is the first, and as yet 
only, document in which anyone claims that UID ranges work any differently 
in the SEARCH and UID commands in RFC 3501.

>> The ranges in question do not contain any messages which do not satisfy the 
>> search criteria.
> Sure. But two wrongs don't make a right, and in this case, the ranges do 
> contain UIDs which do not relate to messages which satisfy the search 
> criteria.

They may contain UIDs which do not relate to messages, just as the usage 
of UID sequence-set in RFC 3501.  Nonetheless, all UIDs which relate to 
messages satisfy the search criteria.

You're trying to have the same protocol element do two entirely different 
things.

>> RFC 3501 section 6.4.4 describes a command that takes a UID sequence-set 
>> argument with the same properties as the sequence-set in the UID commands 
>> (UID FETCH, UID STORE, UID SEARCH).  It explicitly states that ranges are 
>> permitted.
> No, sorry. It actually says:
>[snip]

The text that you quoted is from section 6.4.8, not 6.4.4.

RFC 3501 section 6.4.4 describes a command other than UID that takes UID 
ranges, but it does not separately state the sematics that are in 6.4.8. 
Either 6.4.4 does not have those semantics (and I think that a lot of 
people will complain should that suddenly happen), or the semantics of 
6.4.8 apply to more than just the UID command.

Your next argument will probably be that 6.4.4 inherits the semantics from 
6.4.8 because it is a command, and that server responses do not inherit 
semantics from commands.  If so, that raises quite a few interesting 
issues.  I don't think that you want to go there.

>> That may be just rationale.  Nonetheless, the ranges that my server returns 
>> are in complete compliance with that text, and there is NO text that 
>> prohibits those ranges.
> No, there's nothing that says that non-existent UIDs MAY be included, either.

Correct.  But, as we have seen many many times, the lack of specific 
permission to do something does not constitute prohibition.

> And if they aren't included, as I maintain the specification hints closest to 
> and intends, then they still can be used in this way.

I saw no such hints in RFC 4731.  I can't read minds; without an explicit 
statement of intentions I have to use my judgement.  Thus:

I read nothing in RFC 4731 about the ability to get the map.  The fact 
that the non-ESEARCH form of UID SEARCH is an accident; FETCH 1:* UID was 
the intended way to accomplish that.  It seemed pointless to have a third 
way to get the map.

All that was clear in RFC 4731 is that it attempts to compress the search 
results to be less chatty.  This is a real concern with mobile devices and 
users who have INBOXes with tens of thousands of messages.

It consider it absurd that you would intentionally make the mechanism 
*more* chatty in order to provide a triplicate of functionality.  Not only 
that, but with typical INBOXes (and *all* mailboxes on a server which does 
not assign consecutive UID values to a mailbox), ESEARCH ends up being 
more chatty than the non-ESEARCH!

>> I
>> see nothing in RFC 4315, other than the general principle of "don't do 
>> silly/harmful behavior", that precludes
>> 	[COPYUID 38201 1:5 4483:9857]
>> That needs to be fixed in that document.
> No, it doesn't - you called that one out explicitly, by stating that no 
> extraneous UIDs could be included.

That isn't what RFC 4315 prohibition on extraneous UIDs actually says, 
although clearly that was what it intended to mean.  This needs to be 
remedied.

Fortunately, there is a clear interpretation in RFC 4315 of what is 
correct.  I don't think that anyone can argue the benefits of any other 
interpretation.

Nonetheless, it is ill-specified/ambiguous and should be fixed.

> RFC2359 has similar language. That said, 
> it also states (in both cases) "[...] a UID set containing the UIDs [...]", 
> so it really needs that language, since otherwise you could include other 
> existent UIDs.

Again, there is no dispute as to what is correct; but the current wording 
in RFC 4315 is ill-specified/ambiguous.

>> However, in spite of RFC 4315's unfortunate use of the same syntax as 
>> sequence-set, it does NOT use the sequence-set rule.  It uses its own 
>> uid-set and uid-range rules.  There is a reason for that: it divorces 
>> UIDPLUS's return values from the semantics of sequence-set.
> And I still contend that there are no such semantics. The semantics you're 
> talking about are related to the behaviour of the UID command, and not the 
> sequence-set syntactic construct.

What, then, is the point for RFC 4315 to use uid-set?

Think carefully about this before you answer.  There are actually two 
reasons why RFC 4315 has its own uid-set rule.  I have stated one, but 
there is another.

That other reason also applies to ESEARCH.  That reason, if nothing else, 
should indicate to you why sequence-set is the wrong rule.

> That leaves the extended UID SEARCH as the tool for performing searches 
> (or obtaining the UID mapping) for areas where the UID mapping is not known, 
> but that's only useful if and only if the ALL result data does not contain 
> non-existent UIDs.

I agree with the former, but not the latter.  I can envision a UID only 
client doing ESEARCH and taking the returned sequence-set and feeding it 
straight to a UID FETCH command.

> SEARCH's UID criteria matches messages which have a UID which falls within 
> the set. I don't see why this mandates that clients must include any 
> non-existent UIDs in the set, although quite obviously doing so will not 
> effect the result.

Clients don't process the UID criterion in SEARCH; servers do.

Are you saying that SEARCH UID 18:23 must return an error if UID 21 does 
not exist?  That seems like a very bad idea to me.

If not, then the UID criterion in SEARCH handles sequence-set exactly as 
the UID command.  Why, now, is ESEARCH different?  What other sequence-set 
in IMAP behaves the same way.

> The UID command (or commands, depending on your thinking) provides explicit 
> behaviour in the presence of non-existent UIDs. It does not provide any 
> semantics on the sequence-set syntactic construct itself.

If that were the case, then it would not be possible for a invalid 
sequence-set with an invalid msgno to return a BAD response (and RFC 3501 
would be internally contradictory).  BAD indicates a protocol syntax 
error.  Command execution errors return a NO.

>>> It happens that non-existent UIDs are ignored by the server in commands, 
>>> but that's explicitly mentioned in the definitions of those commands.
>>> Server-provided sequence-sets never contain *, or non-existent UIDs.
>> Nowhere is this stated.  4466/4731 is the first occurrence.
>> UIDPLUS is not a precedent, since it returns uid-set instead of 
>> sequence-set.
> But RFC2359 did not, and managed just fine.

RFC 2359 returned RFC 2060's set (which is now sequence-set in RFC 3501), 
which was clearly a bug in RFC 2359 since a UIDPLUS did not, and could 
not, return a RFC 2060 set.

I made a point about there being two reasons why sequence-set (set) was 
wrong for UIDPLUS, and why it is wrong for ESEARCH.  Think about what may 
be the second reason.

Now, it is possible to claim that this second reason is not a bug in 
ESEARCH.  If so, then ESEARCH can't be used as a map-obtaining operation.

> But assuming I'm wrong, you're arguing that somehow, this search criterion 
> would cause only messages which satisfy the search criteria to be included in 
> results. Right?

Basically, this would have to work by explicitly defining(!!) that it 
switches on magic behavior to prevent collapsing of contiguous messages 
with non-contiguous UID values into ranges.  As this magic is explicitly 
defined, it can do as it wishes.

But on subsequent reflection, it's probably just to do Alexey's idea and 
have a return value that has this lesser form of collapse.  Magic is icky.

> Well, I'm not saying that the future is fully known to me, but I think a GUID 
> space is likely to be distinct from a UID space, too.

Why?  Is it to retain UID/UIDVALIDITY as a handle on a message that is 
shorter than a GUID?  IMHO, three handles on a message is at least one too 
many.

POP3 seems to have no problem with UIDs which are much longer than any 
IMAP UIDs and lack the strictly-ascending property of IMAP UIDs.  POP3's 
only UID operation is to fetch the map.  POP3 is far from dead (sigh!) and 
remains far more ubiquitous than IMAP (double sigh).

> either way, non-existent messages, 
> or references to messages which don't exist, are being included in the result 
> as passed to the client.

This is not an unheard-of occurance in real life.  "The university 
district encompasses all the streets from 37th to 55th" does not in any 
way imply that there is a 46th Street (there isn't, at least not in the 
University District in Seattle...46th Street does exist elsewhere).

"All meetings from July 2 through July 14 are cancelled" doesn't imply 
that there was a meeting scheduled on July 4 that is now cancelled.

Unless it is explicitly stated, I don't see how the usage of a range in 
sparse data in any way implies that all possible values within that range 
exist within the sparse data.

> There's a handful of cases where ESEARCH comes out bigger by a small number 
> of octets (the size of the correlator, possibly UID, and an extra E). In the 
> majority of cases, though, it's smaller, in many cases I've experienced, it's 
> drastically smaller, especially with the larger mailboxes we seem to be 
> dealing with these days.

It would certainly do so with non-INBOX mailboxes; or more specifically 
mailboxes that are mostly the targets of COPY/APPEND operations and are 
generally never pruned.

However, INBOX stands out as the 800 pound gorilla, especially for those 
users who don't use non-INBOX mailboxes.

> Oh, well I've only got 51069 messages right now, and my UID range isn't all 
> that sparse. Maybe my spam filtering is working.

How tolerant are you of false positives, and how much spam hits you?

With filtering, I see have about a 20-to-1 spam/non-spam ratio in my 
INBOX, and it's getting worse all the time.  I shudder to think of how 
much gets blocked by the filters.

Consequently, consecutive non-spam messages are a rarity in my mailbox. 
When they occur they are usually hot-topic threads that I end up deleting 
after reading.

That's the problem with dealing with anecdotal evidence.  You and I each 
have anecdotes, but that doesn't necessarily mean that either one is valid 
for the general case.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Date: Mon, 13 Aug 2007 13:46:23 -0700 (Pacific Daylight Time)
From: Mark Crispin <[email-masked]>
To: Dave Cridland <[email-masked]>
cc: [email-masked]
Subject: Re: 1:4294967295 (was Re: UID SEARCH responses)
Organization: Networks & Distributed Computing
Sender: [email-masked]

On Mon, 13 Aug 2007, Dave Cridland wrote:
>> On further study:
> No, please don't over analyze my reductios ad absurda (if you'll pardon my 
> butchering of Latin) - isn't that as bad as answering my rhetorical 
> questions?

Actually, it wasn't an absurd argument at all, and it raises some good 
points.

4294967295 is in many, but NOT all, ways equivalent to *.  Similarly, 1 is 
in many, but NOT all, ways equivalent to the UID associated with msgno 1.

The ways that they are not equivalent are interesting and unique; and the 
author of an IMAP client should be aware of these properties.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Subject: Re: 1:4294967295 (was Re: UID SEARCH responses)
Date: Mon, 13 Aug 2007 21:38:38 +0100
From: Dave Cridland <[email-masked]>
To: Mark Crispin <[email-masked]>
Cc: <[email-masked]>
Sender: [email-masked]

On Mon Aug 13 19:49:05 2007, Mark Crispin wrote:
> 
> On further study:

No, please don't over analyze my reductios ad absurda (if you'll 
pardon my butchering of Latin) - isn't that as bad as answering my 
rhetorical questions?

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Subject: RE: UID SEARCH responses
Date: Mon, 13 Aug 2007 21:38:15 +0100
From: Dave Cridland <[email-masked]>
To: Mark Crispin <[email-masked]>
Cc: Pawel Salek <[email-masked]>, <[email-masked]>, Peter Coates <[email-masked]>
Sender: [email-masked]

On Mon Aug 13 18:29:48 2007, Mark Crispin wrote:
> 
> On Mon, 13 Aug 2007, Dave Cridland wrote:
>> Non-existent messages do not satisfy the SEARCH criteria
> 
> Why not in the case of UIDs?  They satisfy the criteria when they 
> are arguments to the SEARCH commands.  Why don't they satisfy the 
> criteria when the server returns it as a value?
> 
> 
Eh? No, they don't. No message will satisfy a UID criteria containing 
as argument a non-existent UID.

"UID SEARCH UID 1:*" gives you back all extant UIDs in the mailbox, 
not all possible UIDs in the range. (Which'd be 1 through the highest 
UID existent at the time of the execution of the command, of course).

> That is an inconsistency between client-end and server-end.  Such 
> inconsistencies MUST be documented, carefully and explicitly.
> 
> 
No, there's no inconsistency. Non-existent UIDs don't affect SEARCH's 
UID criteria, and are explicitly ignored by the UID family of 
commands. Otherwise, they mean what they say.

> The ranges in question do not contain any messages which do not 
> satisfy the search criteria.
> 
> 
Sure. But two wrongs don't make a right, and in this case, the ranges 
do contain UIDs which do not relate to messages which satisfy the 
search criteria. The ALL result data isn't specified as "Does not 
return any messages which do not satisfy the SEARCH criteria", 
otherwise you'd be right on the money.

> You are being hung up on the assumption that there is an implicit 
> search criteria of UIDEXISTS.  But the documents do not state that 
> anywhere.
> 
> A search criteria of UIDEXISTS may be a way out of the problem.  I 
> would not object to adding that search criteria within the existing 
> framework.
> 
> 
Hang on. This is where you lost me, but more on this below.

>> First off, RFC3501 clearly states that it's the UID command which 
>> ignores any non-existent messages - that is, it's a behaviour of 
>> the command, and not a property of the syntax. In fact, page 61 of 
>> RFC3501 merely states that a UID sequence-set is identical to a 
>> message number sequence-set except it uses UIDs not message 
>> numbers. If you disagree, please quote the relevant parts to 
>> enlighten me - I'm always happy to be proven wrong.
> 
> RFC 3501 section 6.4.4 describes a command that takes a UID 
> sequence-set argument with the same properties as the sequence-set 
> in the UID commands (UID FETCH, UID STORE, UID SEARCH).  It 
> explicitly states that ranges are permitted.
> 
> Is that evidence enough?
> 
> 
No, sorry. It actually says:

      appropriate for the associated command.  However, the numbers in
      the sequence set argument are unique identifiers instead of
      message sequence numbers.  Sequence set ranges are permitted, 
but
      there is no guarantee that unique identifiers will be 
contiguous.

      A non-existent unique identifier is ignored without any error
      message generated.  Thus, it is possible for a UID FETCH command
      to return an OK without any data or a UID COPY or UID STORE to
      return an OK without performing any operations.

Note that the commands ignore them, it's not a product of the syntax 
that they exist. As it happens, this can be taken advantage of (and 
my own client does this) - a client may delibrately extend ranges 
over non-existent UIDs in order to gain a more compact 
representation. But it can do that because the command will ignore 
them without error, not becausethe syntactic construct is in some way 
magical.

>> Secondly, the ALL result mentions usage in subsequent commands, 
>> but it's not given as the reason for its existence. It actually 
>> says:
>>        criteria.  Unlike regular (unextended) SEARCH, the messages 
>> are
>>        always returned using the sequence-set syntax.  A 
>> sequence-set
>>        representation may be more compact and can be used as is in 
>> a
>>        subsequent command that accepts sequence-set.  Note, the 
>> client
>> Taking this reference to "a subsequent command" to be the purpose 
>> for its existence is stretching a point beyond its elastic limit. 
>> The phrase used merely provides rationale behind the use of a 
>> server-provided sequence-set
> 
> That may be just rationale.  Nonetheless, the ranges that my server 
> returns are in complete compliance with that text, and there is NO 
> text that prohibits those ranges.
> 
> 
No, there's nothing that says that non-existent UIDs MAY be included, 
either. And if they aren't included, as I maintain the specification 
hints closest to and intends, then they still can be used in this way.

>> (which had, incidentally, only been used by UIDPLUS prior to this, 
>> which itself obviously does not include non-existent UIDs).
> 
> That's a bug in RFC 4315.  It does not contain the word 
> "monotonically". I see nothing in RFC 4315, other than the general 
> principle of "don't do silly/harmful behavior", that precludes
> 	[COPYUID 38201 1:5 4483:9857]
> That needs to be fixed in that document.
> 
> 
No, it doesn't - you called that one out explicitly, by stating that 
no extraneous UIDs could be included. RFC2359 has similar language. 
That said, it also states (in both cases) "[...] a UID set containing 
the UIDs [...]", so it really needs that language, since otherwise 
you could include other existent UIDs.

> However, in spite of RFC 4315's unfortunate use of the same syntax 
> as sequence-set, it does NOT use the sequence-set rule.  It uses 
> its own uid-set and uid-range rules.  There is a reason for that: 
> it divorces UIDPLUS's return values from the semantics of 
> sequence-set.
> 
> 
And I still contend that there are no such semantics. The semantics 
you're talking about are related to the behaviour of the UID command, 
and not the sequence-set syntactic construct.

> The fact that there is no such divorce in RFC 4466/4731 is another 
> nail in the coffin.  How many more nails do I need to find?
> 
> 
No nails, but a coffin would be handy. Oh, and it's traditional to 
wait until the victim is actually dead.

> You can't use the "don't do silly/harmful behavior" principle in 
> the case of ESEARCH, since the behavior in question is neither 
> silly (it has demonstrably greater compression) nor harmful (there 
> are two other ways to get the map).  In fact, a mobile client that 
> has the map and only wants to do bandwidth-effective searches will 
> benefit from this behavior.
> 
> 
It'll benefit more from using SEARCH, instead of UID SEARCH. Timo 
pointed this out already, though, and I agree - it's what I've been 
using for some time. That leaves the extended UID SEARCH as the tool 
for performing searches (or obtaining the UID mapping) for areas 
where the UID mapping is not known, but that's only useful if and 
only if the ALL result data does not contain non-existent UIDs.

(Timo didn't mention, I think, that message sequence numbers are, by 
definition, always smaller than or equal to UIDs, and therefore can, 
sometimes, also have a smaller ASCII representation. Gotta get those 
octet counts down, y'know.)

>> Nope, it uses the same sequence-sets that every other IMAP usage 
>> does.
> 
> There are only two other usages of sequence-set: SEARCH and UID.
> 
> 
SEARCH's UID criteria matches messages which have a UID which falls 
within the set. I don't see why this mandates that clients must 
include any non-existent UIDs in the set, although quite obviously 
doing so will not effect the result.

The UID command (or commands, depending on your thinking) provides 
explicit behaviour in the presence of non-existent UIDs. It does not 
provide any semantics on the sequence-set syntactic construct itself.

>> It happens that non-existent UIDs are ignored by the server in 
>> commands, but that's explicitly mentioned in the definitions of 
>> those commands.
>> Server-provided sequence-sets never contain *, or non-existent 
>> UIDs.
> 
> Nowhere is this stated.  4466/4731 is the first occurrence.
> 
> UIDPLUS is not a precedent, since it returns uid-set instead of 
> sequence-set.
> 
> 
But RFC2359 did not, and managed just fine.

>>>  . forces all ESEARCH results to have longer, more 
>>> bandwidth-consuming,
>>>    results even if the client has the UID map and has no need for 
>>> such
>>>    longer results
>> I'd be happy to entertain a proposal for an alternate to ALL which 
>> had this behaviour.
> 
> The right way to do this is probably a new SEARCH criterion called 
> UIDEXISTS.  That way, you don't have to change the definition of 
> sequence-set and you don't have to change 4466/4731.
> 
> 
I'm still lost after much thought.

I'm assuming that this search criterion will only be satisfied by 
messages that have a UID that exists. By inference, this means that 
all messages will match, and, therefore, that this is equivalent to 
ALL.

But assuming I'm wrong, you're arguing that somehow, this search 
criterion would cause only messages which satisfy the search criteria 
to be included in results. Right?

But, you also argue that non-existent UIDs can also be included in 
results, anyway. So surely we're right back to square one? Or is 
there something I'm missing?

>> I certainly feel that servers that do not allocate UIDs 
>> contiguously are fairly rare, and you're absolutely correct in 
>> your assertion that ESEARCH performs much worse in those, and this 
>> will mean it's a handful of octets larger than an unextended 
>> SEARCH.
> 
> I am aware of multiple servers which do not allocate UIDs 
> contiguously. I don't know what I can say about these servers due 
> to NDA considerations. I will say that my comments about global 
> UIDs and 32-bits being doomed were not based upon flights of fancy.
> 
> 
Well, I'm not saying that the future is fully known to me, but I 
think a GUID space is likely to be distinct from a UID space, too.

>> Your reading of the specification implies that you think all 
>> non-existent messages match all possible SEARCH criteria.
> 
> If that were the case, then it would be possible to get the 
> UID/msgno map by doing UID SEARCH RETURN () NOT ALL!
> 
> 
Or NOT UIDEXISTS?

> You're misstating my position.  I claim that non-existent UIDs (not 
> "non-existent messages"; there is no such thing) do not figure into 
> the calculation at all.
> 
> 
Well, I'll readily agree that non-existent messages don't exist. But 
the SEARCH command conceptually passes back a list of messages, the 
client's just getting to pick the reference used to indicate which 
ones. So while you can split hairs and otherwise wax philosophical 
about whether your implementation is including non-existent UIDs or 
non-existent messages in the results, I don't think it's making much 
difference - either way, non-existent messages, or references to 
messages which don't exist, are being included in the result as 
passed to the client.

>> ESEARCH works with servers that do not allocate UIDs sequentially 
>> and contiguously. Just not as well.
> 
> "not as well" -> "worse than SEARCH"
> 
> That's not a good design.
> 
> 
There's a handful of cases where ESEARCH comes out bigger by a small 
number of octets (the size of the correlator, possibly UID, and an 
extra E). In the majority of cases, though, it's smaller, in many 
cases I've experienced, it's drastically smaller, especially with the 
larger mailboxes we seem to be dealing with these days.

If you can find something which contains the same amount of 
information, yet is consistently smaller, I'll be very impressed - 
I'd long since decided this was probably similar to finding a 
compression algorithm that reduced the size of all possible inputs. 
(Which is tricky, unless you just use smaller bits, of course.)

>> Similarly, it works with mailboxes than have had every second or 
>> third message expunged - again, this is a worst-case. Thankfully 
>> equally rare.
> 
> Rare?  RARE?!?!?!?!?!!
> 
> 
Rare! RARE!!!!!! R A R E ! ! ! !

Yes, I did say rare. Although not as dramatically, which I hope to 
have rectified. :-)

> "A mailbox which has every second or third message expunged" 
> describes INBOX for the great majority of users.  They have NO 
> other mailbox other than INBOX, they delete their spam and they 
> keep other messages.
> 
> 
That's me.

> Some of these guys have thousands, even tens of thousands, of 
> messages in their INBOX, all with sparse UIDs due to spam.
> 
> 
Oh, well I've only got 51069 messages right now, and my UID range 
isn't all that sparse. Maybe my spam filtering is working. Then 
again, I've found that the bigger a mailbox, the less sparse the UID 
mapping, as a general rule. A couple of thousand seems to be the 
worst case, for no reason I can readily make out.

But it's not a matter of saying "it's pretty sparse", or even "almost 
every range is only two long", you need to be thinking in terms of 
"no range exists which is longer than two", and that is rare in my 
experience, at least on servers which allocate their UIDs 
contiguously. You don't need a lot of ranges of three or more before 
ESEARCH covers its overhead. Also note that the higher the UID, the 
faster the overhead is covered.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Date: Mon, 13 Aug 2007 14:39:48 -0500
To: Mark Crispin <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: Re: UID SEARCH responses
Cc: Alexey Melnikov <[email-masked]>, Dave Cridland <[email-masked]>, [email-masked]
Sender: [email-masked]

So, I'm happy to have technical discussion of IMAP that are not WG 
related items (i.e., ESEARCH) continue here. (At some point today, 
I'll get the last call for i18n in among all of these messages, so 
please keep an eye out for it.) As ESEARCH is not a WG product, I 
don't have to be involved in any of the decisions here, except 
insofar as someone might ask my opinion about what the IMAPEXT WG 
thinks of it.

That said, I want to comment on one item, wearing only my "process geek" hat:

On 8/13/07 at 12:01 PM -0700, Mark Crispin wrote:

>On Mon, 13 Aug 2007, Alexey Melnikov wrote:
>>>I will vigorously oppose such a "clarification", including IESG 
>>>appeals if necessary.
>>
>>You must be kidding.
>
>I'm not kidding.  Such a clarification constitutes an ex post facto 
>declaration of deployed server non-compliance.  You can't do that in 
>a clarification.

You will be hard pressed to find the word "compliance" in RFC 2026. 
(It occurs in the places, none of which are terribly interesting for 
the discussion at hand.) What you will find is section 4.1.1, which 
says in part:

    A Proposed Standard specification is generally stable, has resolved
    known design choices, is believed to be well-understood, has received
    significant community review, and appears to enjoy enough community
    interest to be considered valuable.  However, further experience
    might result in a change or even retraction of the specification
    before it advances.

and:

    Implementors should treat Proposed Standards as immature
    specifications.  It is desirable to implement them in order to gain
    experience and to validate, test, and clarify the specification.
    However, since the content of Proposed Standards may be changed if
    problems are found or better solutions are identified, deploying
    implementations of such standards into a disruption-sensitive
    environment is not recommended.

Proposed Standards might have more done to them than "clarification". 
They might "change". And the fact that a particular implementation of 
an old version of a specification is not "compliant" with a new 
version of a specification is not a basis of an appeal according to 
section 6.5 of that document (as far as I read it).

The idea of our standards process has always been to get people to 
come to (sometimes rough) consensus so that implementations can 
interoperate, not to anoint implemenations with "compliance" or 
"non-compliance". I can think of several times in the past where my 
implementation did not agree with what a new version of a 
specification said to do, even if what I was doing was arguably a 
valid interpretation of the old specification. The specification was 
clarified/changed to meet the consensus of what most folks thought 
the old specification meant, and I fixed my code (sometimes making my 
old code non-conformant to the new specification). If the community 
(as judged by the IESG) decides that the best way to achieve 
interoperability is to change a specification such that one (or a 
few) implementations have to change their code to get into line, I 
think one would be hard-pressed to convince the IESG that this is a 
process violation.

Now back to your regularly scheduled technical discussion.

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

Date: Mon, 13 Aug 2007 12:37:59 -0700 (Pacific Daylight Time)
From: Mark Crispin <[email-masked]>
To: Alexey Melnikov <[email-masked]>
cc: IMAP Extensions WG <[email-masked]>
Subject: Re: UID SEARCH responses
Organization: Networks & Distributed Computing
Sender: [email-masked]

On Mon, 13 Aug 2007, Mark Crispin wrote:
>> 2). Clarify which of the two choices should be used by ALL and add a new 
>> return option for the other one.

On second thought, this could work if you proceed as follows:

Update RFC 4466 to include uid-set from RFC 4315.  Be sure to highlight 
that uid-set MUST NOT include non-existent UIDs, whereas sequence-set 
does.

Consider a possible sytax change for uid-set, e.g., using hyphens, to 
reduce confusion.  That orphans UIDPLUS syntax, but the benefit may be 
worth not adding another subtlety in IMAP.

Define the new return option (and capability indicating its presence) to 
return uid-set.

As a quality of implementation note, a server which supports the new 
return option maximally compresses in ALL.

This maintains compatibility for all existing servers.  As a gesture of 
good faith, I will pledge to implement the new return value as soon as it 
is published as an RFC.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Date: Mon, 13 Aug 2007 12:01:11 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: Alexey Melnikov <[email-masked]>
cc: Dave Cridland <[email-masked]>, [email-masked]
Subject: Re: UID SEARCH responses
Sender: [email-masked]

On Mon, 13 Aug 2007, Alexey Melnikov wrote:
>> I will vigorously oppose such a "clarification", including IESG appeals if 
>> necessary.
> You must be kidding.

I'm not kidding.  Such a clarification constitutes an ex post facto 
declaration of deployed server non-compliance.  You can't do that in a 
clarification.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Date: Mon, 13 Aug 2007 11:59:20 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: Alexey Melnikov <[email-masked]>
cc: IMAP Extensions WG <[email-masked]>
Subject: Re: UID SEARCH responses
Sender: [email-masked]

On Mon, 13 Aug 2007, Alexey Melnikov wrote:
> "deprecate and replace" is a non-starter, as far as I am concerned. There are 
> several other extensions reusing the ESEARCH response.

OK.

> But there are other choices which you've ignored:
> 1). Get rough consensus among implementors and update the spec to mandate 
> that servers implement one of the two choices.
> 2). Clarify which of the two choices should be used by ALL and add a new 
> return option for the other one.

I consider both of these to be non-starters, as it will result in a 
deployed compliant implementation being broken.

> 3). Deprecate ALL and replace it with 2 other return options

You would also have to deprecate (), which I think would be a good thing.

There are other choices:

4) Leave everything as-is; and take it as a learning experience.
5) Clarify in the document that either way is possible, and note that
     either FETCH or non-ESEARCH UID SEARCH is the appropriate command
     to get the UID map.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Date: Mon, 13 Aug 2007 19:51:18 +0100
From: Alexey Melnikov <[email-masked]>
To: Mark Crispin <[email-masked]>
CC: Dave Cridland <[email-masked]>, [email-masked]
Subject: Re: UID SEARCH responses
Sender: [email-masked]

Mark Crispin wrote:

> On Mon, 13 Aug 2007, Dave Cridland wrote:
>
>> RFC4731 defines MIN, MAX, and COUNT - none of which are part of this 
>> discussion - and ALL - which you are contending is broken. ALL can be 
>> deprecated and replaced by something else quite easily without 
>> affecting MIN, MAX, COUNT, or the ESEARCH response as a whole.
>
> You'll still have to create a new capability.  Otherwise the client 
> won't know the difference between RFC 4731 compliant servers and RFC 
> xxxx compliant servers.

Yes, if there is consensus to do that.

>> In addition, I still do not believe that there is consensus to do 
>> this - as far as I can tell, there is only one voice in support of 
>> such an action. That voice happens to have substantial weight as the 
>> author of RFC3501, however neither author of RFC4731, nor any other 
>> implementor, has leant any support to it, whereas there appears to me 
>> to be wide support in favour of retaining the intended behaviour, and 
>> clarifying it in errata and a subsequent revision.
>
> I will vigorously oppose such a "clarification", including IESG 
> appeals if necessary.

You must be kidding.

Date: Mon, 13 Aug 2007 11:49:05 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: IMAP Extensions WG <[email-masked]>
Subject: 1:4294967295 (was Re: UID SEARCH responses)
Sender: [email-masked]

On further study:

A server can return 1:4294967295 in the ESEARCH form of UID SEARCH ALL, 
but it can NOT generally return 4294967295 as the upper end of a range in 
other searches.

There are two types of non-existent UIDs:
  . those which do not now, and will never exist
  . those which may exist in the future

Any UID value greater than or equal to UIDNEXT is in the latter category. 
Such UID values are permitted in sequence-set by RFC 3501 (the prohibition 
is on message numbers only).  However, since these UIDs reflect potential 
messages, a server can not say that a sequence-set that includes such UIDs 
will still match the SEARCH criteria when that sequence-set is applied to 
a subsequent command.

Specifically, if a message is instantiated between the UID SEARCH and the 
next command, the result of 1:4294967295 would be a false positive which 
breaks IMAP's precedent of a false negative in such circumstances.

With that said, I agree that a server should NOT send :4294967295 in any 
case.  Instead, it should send the UID of the last message in the mailbox 
if it matches the mailbox.  I am less concerned about the 1: case.

There is a similar timing race in IMAP with flags, as these can change.

Perhaps what should have been done instead of ESEARCH was to create an 
operation which did FETCH/STORE/COPY with the argument being SEARCH 
criteria instead of a sequence-set, and defining the resulting operation 
as being atomic.  Maybe that would have reduced the need for CONDSTORE.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Date: Mon, 13 Aug 2007 19:47:53 +0100
From: Alexey Melnikov <[email-masked]>
To: Mark Crispin <[email-masked]>
CC: IMAP Extensions WG <[email-masked]>
Subject: Re: UID SEARCH responses
Sender: [email-masked]

Mark Crispin wrote:

> If Pawel is correct on the intent, then the ESEARCH RFC is broken 
> since it is ambiguous on that point; either live with the ambiguity, 
> or deprecate ESEARCH and replace it with some other syntax.

"deprecate and replace" is a non-starter, as far as I am concerned. 
There are several other extensions reusing the ESEARCH response.

But there are other choices which you've ignored:
1). Get rough consensus among implementors and update the spec to 
mandate that servers implement one of the two choices.
2). Clarify which of the two choices should be used by ALL and add a new 
return option for the other one.
3). Deprecate ALL and replace it with 2 other return options

Date: Mon, 13 Aug 2007 11:31:30 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: Timo Sirainen <[email-masked]>
cc: [email-masked]
Subject: Re: UID SEARCH responses
Sender: [email-masked]

On Mon, 13 Aug 2007, Timo Sirainen wrote:
> Uncompressed UID SEARCH ranges are useful if the (full) sequence <-> UID
> map isn't known yet for the client for this session. After the map is
> known, there should be no problems with compressed UID SEARCH ranges.

Correct.

> Then again, if the map is known client can just as well use SEARCH and
> get even more compressed ranges.

Correct.

> So if UID SEARCH ranges can include expunged messages, I don't see why a
> client would ever want to do UID SEARCHes instead of SEARCHes, because
> it's required to internally check the UID <-> sequence mapping anyway to
> get a usable result.

Consider a client that wants to download all messages that match a certain 
set of search criteria and then exit.  This client does not need the map; 
it just needs a sequence-set string to feed to UID FETCH.  It does the UID 
form instead of the non-UID form so as to allow expunging.

With that said, I agree with the overall sentiment that it is more useful 
to get the UID/msgno map for synchronization and subsequently use msgno 
commands.  Almost all instances of "UID-only" clients that I have studied 
in real-world usage end up doing unnecessary work.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Date: Mon, 13 Aug 2007 11:01:34 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: Dave Cridland <[email-masked]>
cc: [email-masked], Alexey Melnikov <[email-masked]>
Subject: Re: UID SEARCH responses
Sender: [email-masked]

On Mon, 13 Aug 2007, Dave Cridland wrote:
> RFC4731 defines MIN, MAX, and COUNT - none of which are part of this 
> discussion - and ALL - which you are contending is broken. ALL can be 
> deprecated and replaced by something else quite easily without affecting MIN, 
> MAX, COUNT, or the ESEARCH response as a whole.

You'll still have to create a new capability.  Otherwise the client won't 
know the difference between RFC 4731 compliant servers and RFC xxxx 
compliant servers.

> In addition, I still do not believe that there is consensus to do this - as 
> far as I can tell, there is only one voice in support of such an action. That 
> voice happens to have substantial weight as the author of RFC3501, however 
> neither author of RFC4731, nor any other implementor, has leant any support 
> to it, whereas there appears to me to be wide support in favour of retaining 
> the intended behaviour, and clarifying it in errata and a subsequent 
> revision.

I will vigorously oppose such a "clarification", including IESG appeals if 
necessary.

I believe that I have strong arguments:
  . although RFC 4315 has a syntax for UID sequence ranges that seem to
     have the same semantics as the authors of RFC 4731 claim to have
     intended, RFC 4315 uses its own rules to describe those ranges and
     does not use sequence-set in order to indicate differing semantics.
  . unlike RFC 4315, RFC 4466/4731 uses RFC 3501's sequence-set rule.
  . there are two other ways in the base specification to accomplish
     the functionality of acquiring the UID/msgno map.
  . RFC 4731 says nothing about acquiring the UID/msgno map.
  . RFC 4731 mentions use of the returned UID sequence-set in the
     IMAP commands which have the RFC 3501 sequence-set semantics.  My
     interpretation is fully in compliance with this use.
  . RFC 4731 gives, as a motivation, the desire to return a "more
     compact" representation.  My interpretation is fully in compliance
     with this motivation, and in fact it is demonstrably more compact
     than the contrary interpretation.
  . The operation of obtaining the UID/msgno map is typically a one-time
     operation in a session.  Once the map is known to the client,
     removal from the map can be mathematically calculated from an EXPUNGE
     response, and additional to the map can be obtained with a short
     map fetch.
  . Searches typically are done many times in a session, and as noted
     above the re-fetch of the map data is unnecessary.
  . INBOX tends to have frequent expunges, and consequently the UID space
     in an INBOX can become very sparse.  Anecdotally (I can collect the
     data if you insist), some people have very large INBOXes.  In one
     interpretation, ESEARCH optimizes search results; in the other, it
     does not.
  . In one interpretation, ESEARCH optimizes search results even if the
     server does not allocate contiguous UID values in a mailbox.  In
     the other, it does not.
  . I contend that the optimizing interpretion in the two previous points
     is superior.
  . No evidence has been offered to debunk the assertation that RFC 4731
     is ambiguous.
  . Assuming that RFC 4731 is ambiguous, server implementations with
     either interpretation are fully compliant.  The only issue is that a
     client must be aware of the issue.
  . Assuming that RFC 4731 is ambiguous on this point, there is an
     existing deployed implementation that is fully compliant with RFC
     4731 and would be broken by the proposed clarification.  This is
     therefore an incompatible change, not a clarification, and must be
     designed and published as such.

I will also oppose, but less vigorously, a new specification that 
deprecates ESEARCH entirely in favor of a new capability.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Subject: Re: UID SEARCH responses
From: Timo Sirainen <[email-masked]>
To: [email-masked]
Date: Mon, 13 Aug 2007 20:45:09 +0300
Sender: [email-masked]

--=-sSux3v7X/FfMQqRyZkC8
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

On Mon, 2007-08-13 at 17:54 +0100, Dave Cridland wrote:
> In addition, I still do not believe that there is consensus to do=20
> this - as far as I can tell, there is only one voice in support of=20
> such an action.

My thoughts from usefulness point of view:

Uncompressed UID SEARCH ranges are useful if the (full) sequence <-> UID
map isn't known yet for the client for this session. After the map is
known, there should be no problems with compressed UID SEARCH ranges.
Then again, if the map is known client can just as well use SEARCH and
get even more compressed ranges.

So if UID SEARCH ranges can include expunged messages, I don't see why a
client would ever want to do UID SEARCHes instead of SEARCHes, because
it's required to internally check the UID <-> sequence mapping anyway to
get a usable result.

--=-sSux3v7X/FfMQqRyZkC8
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

--=-sSux3v7X/FfMQqRyZkC8--

Date: Mon, 13 Aug 2007 10:29:48 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: Dave Cridland <[email-masked]>
cc: Pawel Salek <[email-masked]>, [email-masked], Peter Coates <[email-masked]>
Subject: RE: UID SEARCH responses
Sender: [email-masked]

On Mon, 13 Aug 2007, Dave Cridland wrote:
> Non-existent messages do not satisfy the SEARCH criteria

Why not in the case of UIDs?  They satisfy the criteria when they are 
arguments to the SEARCH commands.  Why don't they satisfy the criteria 
when the server returns it as a value?

That is an inconsistency between client-end and server-end.  Such 
inconsistencies MUST be documented, carefully and explicitly.

>> By "stand alone", do you mean "mathematically calculate the UID/msgno map"? 
>> If so, where in RFC 4731 is this property stated?
> The property of the ALL result that interests me is that it contains only 
> messages which satisfy the search criteria.

The ranges in question do not contain any messages which do not satisfy 
the search criteria.

You are being hung up on the assumption that there is an implicit search 
criteria of UIDEXISTS.  But the documents do not state that anywhere.

A search criteria of UIDEXISTS may be a way out of the problem.  I would 
not object to adding that search criteria within the existing framework.

> First off, RFC3501 clearly states that it's the UID command which ignores any 
> non-existent messages - that is, it's a behaviour of the command, and not a 
> property of the syntax. In fact, page 61 of RFC3501 merely states that a UID 
> sequence-set is identical to a message number sequence-set except it uses 
> UIDs not message numbers. If you disagree, please quote the relevant parts to 
> enlighten me - I'm always happy to be proven wrong.

RFC 3501 section 6.4.4 describes a command that takes a UID sequence-set 
argument with the same properties as the sequence-set in the UID commands 
(UID FETCH, UID STORE, UID SEARCH).  It explicitly states that ranges are 
permitted.

Is that evidence enough?

> Secondly, the ALL result mentions usage in subsequent commands, but it's not 
> given as the reason for its existence. It actually says:
>        criteria.  Unlike regular (unextended) SEARCH, the messages are
>        always returned using the sequence-set syntax.  A sequence-set
>        representation may be more compact and can be used as is in a
>        subsequent command that accepts sequence-set.  Note, the client
> Taking this reference to "a subsequent command" to be the purpose for its 
> existence is stretching a point beyond its elastic limit. The phrase used 
> merely provides rationale behind the use of a server-provided sequence-set

That may be just rationale.  Nonetheless, the ranges that my server 
returns are in complete compliance with that text, and there is NO text 
that prohibits those ranges.

> (which had, incidentally, only been used by UIDPLUS prior to this, which 
> itself obviously does not include non-existent UIDs).

That's a bug in RFC 4315.  It does not contain the word "monotonically". 
I see nothing in RFC 4315, other than the general principle of "don't do 
silly/harmful behavior", that precludes
 	[COPYUID 38201 1:5 4483:9857]
That needs to be fixed in that document.

However, in spite of RFC 4315's unfortunate use of the same syntax as 
sequence-set, it does NOT use the sequence-set rule.  It uses its own 
uid-set and uid-range rules.  There is a reason for that: it divorces 
UIDPLUS's return values from the semantics of sequence-set.

The fact that there is no such divorce in RFC 4466/4731 is another nail in 
the coffin.  How many more nails do I need to find?

Perhaps what you wanted to do was have uid-set and uid-range defined in 
RFC 4466, and then use those rules in RFC 4731.

You can't use the "don't do silly/harmful behavior" principle in the case 
of ESEARCH, since the behavior in question is neither silly (it has 
demonstrably greater compression) nor harmful (there are two other ways to 
get the map).  In fact, a mobile client that has the map and only wants to 
do bandwidth-effective searches will benefit from this behavior.

> Nope, it uses the same sequence-sets that every other IMAP usage does.

There are only two other usages of sequence-set: SEARCH and UID.

> It 
> happens that non-existent UIDs are ignored by the server in commands, but 
> that's explicitly mentioned in the definitions of those commands.
> Server-provided sequence-sets never contain *, or non-existent UIDs.

Nowhere is this stated.  4466/4731 is the first occurrence.

UIDPLUS is not a precedent, since it returns uid-set instead of 
sequence-set.

>>  . forces all ESEARCH results to have longer, more bandwidth-consuming,
>>    results even if the client has the UID map and has no need for such
>>    longer results
> I'd be happy to entertain a proposal for an alternate to ALL which had this 
> behaviour.

The right way to do this is probably a new SEARCH criterion called 
UIDEXISTS.  That way, you don't have to change the definition of 
sequence-set and you don't have to change 4466/4731.

> I certainly feel that servers that do not allocate UIDs contiguously are 
> fairly rare, and you're absolutely correct in your assertion that ESEARCH 
> performs much worse in those, and this will mean it's a handful of octets 
> larger than an unextended SEARCH.

I am aware of multiple servers which do not allocate UIDs contiguously. 
I don't know what I can say about these servers due to NDA considerations. 
I will say that my comments about global UIDs and 32-bits being doomed 
were not based upon flights of fancy.

> Your reading of the specification implies that you think all 
> non-existent messages match all possible SEARCH criteria.

If that were the case, then it would be possible to get the UID/msgno map 
by doing UID SEARCH RETURN () NOT ALL!

You're misstating my position.  I claim that non-existent UIDs (not 
"non-existent messages"; there is no such thing) do not figure into the 
calculation at all.

> ESEARCH works with servers that do not allocate UIDs sequentially and 
> contiguously. Just not as well.

"not as well" -> "worse than SEARCH"

That's not a good design.

> Similarly, it works with mailboxes than have 
> had every second or third message expunged - again, this is a worst-case. 
> Thankfully equally rare.

Rare?  RARE?!?!?!?!?!!

"A mailbox which has every second or third message expunged" describes 
INBOX for the great majority of users.  They have NO other mailbox other 
than INBOX, they delete their spam and they keep other messages.

Some of these guys have thousands, even tens of thousands, of messages in 
their INBOX, all with sparse UIDs due to spam.

>> It is a fortunate accident that UID SEARCH returns the map as well.  But 
>> how many applications that use UID SEARCH to obtain the UID/msgno map can 
>> handle:
>> 	tag UID SEARCH ALL
>> 	* SEARCH 4929 3012
>> 	* SEARCH 12 9993
>> 	tag OK done
>> Such a scenario is completely valid in RFC 3501!
> The fact that a specification does not flatly prohibit doing something 
> idiotic does not mean that it is a good thing for servers (or clients) to do 
> the idiotic thing.

It is not idiotic.  It is an intentional part of the base specification 
from inception.

There are circumstances in which a server may need to split SEARCH results 
into multiple untagged SEARCH.  There are also circumstances (typically 
defined by a mail store designed for non-IMAP) in which a server may need 
to issue out-of-order results in an untagged SEARCH.

The example above is a worst-case, and highly unlikely, but it shows up 
what would happen if a client engine does not implement the untagged data 
model of IMAP properly.

RFC 4731 specifically requires a single ESEARCH response, but leaves open 
the out-of-order possibility.  It also contains some weasel-wording about 
how ESEARCH may be extended for multiple ESEARCH responses.

> Hey, why stop there? If the client really wanted to know the beginning and 
> end of the range, it could use MIN/MAX. No, instead:
> * ESEARCH (TAG x02) UID ALL 1:4294967295
> According to your reading, that's valid.

That is correct.  With the exception of the non-ESEARCH form of UID 
SEARCH, ALL has never been a particularly useful criterion.  The only 
reason why ALL was in the original IMAP was to aid in server debugging.

There's no NONE, but NOT ALL serves that purpose.  It's even less useful 
than ALL, but it's there.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Subject: Re: UID SEARCH responses
Date: Mon, 13 Aug 2007 17:54:53 +0100
From: Dave Cridland <[email-masked]>
To: Mark Crispin <[email-masked]>
Cc: <[email-masked]>, Alexey Melnikov <[email-masked]>
Sender: [email-masked]

On Mon Aug 13 16:46:30 2007, Mark Crispin wrote:
> 
> On Mon, 13 Aug 2007, Alexey Melnikov wrote:
>>> You have to do more than that.  You have to deprecate all of 
>>> ESEARCH, and replace it with different syntax.
>> (Irrespectively to whether you are right or wrong on the issue.)
>> I am sorry Mark, but this is nonsense. RFC 4731 does NOT define 
>> ESEARCH response or extended SEARCH, they are defined in RFC 4466.
>> Both extended SEARCH and ESEARCH response were designed to be 
>> extensible.
> 
> That just means that, in order to have this new type of UID range, 
> RFC 4466 will also have be deprecated and replaced with a new 
> document that uses a rule other than sequence-set to define these 
> new magic blurdybloops that have UID ranges that violate RFC 3501's 
> definition of UID ranges.

Again, irrespective of whether RFC3501 actually does define UID 
ranges in that manner, this is simply wrong.

RFC4466 defines the extensible SEARCH framework, in particular 
defining the ESEARCH response itself. (It also defines other 
extensible syntactic frameworks).

RFC4731 defines MIN, MAX, and COUNT - none of which are part of this 
discussion - and ALL - which you are contending is broken. ALL can be 
deprecated and replaced by something else quite easily without 
affecting MIN, MAX, COUNT, or the ESEARCH response as a whole.

In addition, I still do not believe that there is consensus to do 
this - as far as I can tell, there is only one voice in support of 
such an action. That voice happens to have substantial weight as the 
author of RFC3501, however neither author of RFC4731, nor any other 
implementor, has leant any support to it, whereas there appears to me 
to be wide support in favour of retaining the intended behaviour, and 
clarifying it in errata and a subsequent revision.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Date: Mon, 13 Aug 2007 08:46:30 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: Alexey Melnikov <[email-masked]>
cc: [email-masked]
Subject: Re: UID SEARCH responses
Sender: [email-masked]

On Mon, 13 Aug 2007, Alexey Melnikov wrote:
>> You have to do more than that.  You have to deprecate all of ESEARCH, and 
>> replace it with different syntax.
> (Irrespectively to whether you are right or wrong on the issue.)
> I am sorry Mark, but this is nonsense. RFC 4731 does NOT define ESEARCH 
> response or extended SEARCH, they are defined in RFC 4466.
> Both extended SEARCH and ESEARCH response were designed to be extensible.

That just means that, in order to have this new type of UID range, RFC 
4466 will also have be deprecated and replaced with a new document that 
uses a rule other than sequence-set to define these new magic blurdybloops 
that have UID ranges that violate RFC 3501's definition of UID ranges.

The alternative is to cut losses and leave the specification as-is. 
There *are* two other ways to get the UID/msgno map.

Don't think for a minute that I'm not sympathetic.  I am.  RFC 4466/4731 
were published as RFCs before there was thorough interoperability testing 
(particularly with implementations written by implementors that were NOT 
part of the specification design process), and got bit by the Law of 
Unintended Consequences.  This has happened to me many times; the worst 
and most embarassing being the RFC 1730/2060 fiasco.

Protocol design is a tough game.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Date: Mon, 13 Aug 2007 12:05:30 +0100
From: Alexey Melnikov <[email-masked]>
To: [email-masked]
Subject: Re: UID SEARCH responses
Sender: [email-masked]

Dave Cridland wrote:

> On Fri Aug 10 06:47:40 2007, Mark Crispin wrote:
>
>> For what it's worth, UW imapd issues the
>>     * ESEARCH (TAG ".") UID ALL 38782:41424
>> response instead of
>>     * ESEARCH (TAG ".") UID ALL 
>> 38782,38789,40004,40301,40421,40814,41424
>> because there are no other UIDs in that range.
>
> Ah. That's definitely not what I intended, and it never crossed my 
> mind that this was a possible interpretation. The second is correct.

BTW, I intended the latter, as this is what I've coded into Isode's 
implementation.

Subject: RE: UID SEARCH responses
Date: Mon, 13 Aug 2007 11:45:45 +0100
From: Dave Cridland <[email-masked]>
To: Mark Crispin <[email-masked]>
Cc: Pawel Salek <[email-masked]>, <[email-masked]>, Peter Coates <[email-masked]>
Sender: [email-masked]

On Mon Aug 13 04:21:16 2007, Mark Crispin wrote:
> 
> On Sun, 12 Aug 2007, Peter Coates wrote:
>> Personally I think that having a search result that does not stand 
>> alone (as
>> in sends a range when some messages in the range do not exist) is 
>> bogus.
> 
> Why do you think that it does not "stand alone"?  The returned 
> sequence-set is explicitly and unambiguously valid as a 
> sequence-set to any other IMAP command and will absolutely return 
> to the matching messages and no others.
> 
> 
Yes, but that's not the purpose. That's merely given as an example of 
potential use, and reasoning behind the sequence-set syntax used. It 
does not, however, satisfy the definition of ALL, which is:

         Return all message numbers/UIDs that satisfy the SEARCH
         criteria.

Non-existent messages do not satisfy the SEARCH criteria, and if they 
are included, then they mean the client cannot detirmine the results 
of the search, rendering the results only useful if the client then 
issues a further command to expand them by some means. To my mind, 
that means the results do not stand on their own.

> By "stand alone", do you mean "mathematically calculate the 
> UID/msgno map"?  If so, where in RFC 4731 is this property stated?
> 
> 
The property of the ALL result that interests me is that it contains 
only messages which satisfy the search criteria.

> The definition of sequence-set in RFC 3501 explicitly allows UID 
> ranges to be used even when there are holes in the UID space.  
> Nothing in RFC 4731 states that its definition of sequence-set is 
> different from RFC 3501. In fact, there is text in RFC 4731 that 
> clearly states that the sequence-set returned from ESEARCH is 
> intended to be used as an RFC 3501 sequence-set.
> 
> 
No, I disagree, on pretty much all counts.

First off, RFC3501 clearly states that it's the UID command which 
ignores any non-existent messages - that is, it's a behaviour of the 
command, and not a property of the syntax. In fact, page 61 of 
RFC3501 merely states that a UID sequence-set is identical to a 
message number sequence-set except it uses UIDs not message numbers. 
If you disagree, please quote the relevant parts to enlighten me - 
I'm always happy to be proven wrong.

Secondly, the ALL result mentions usage in subsequent commands, but 
it's not given as the reason for its existence. It actually says:

         criteria.  Unlike regular (unextended) SEARCH, the messages 
are
         always returned using the sequence-set syntax.  A 
sequence-set
         representation may be more compact and can be used as is in a
         subsequent command that accepts sequence-set.  Note, the 
client

Taking this reference to "a subsequent command" to be the purpose for 
its existence is stretching a point beyond its elastic limit. The 
phrase used merely provides rationale behind the use of a 
server-provided sequence-set (which had, incidentally, only been used 
by UIDPLUS prior to this, which itself obviously does not include 
non-existent UIDs).

> Nowhere in RFC 4731 does it say that an RFC 4731 sequence-set has 
> new semantics that prohibit the server from returning ranges that 
> encompass holes in monotonically assigned UID space.
> 
> 
New semantics? Show me the old ones.

> If you assert that ESEARCH sequence-set has such semantics, and 
> MUST NOT send ranges that have holes, then in effect ESEARCH:
>  . creates a new type of sequence-set that is different from all 
> other
>    IMAP use of sequence-set

Nope, it uses the same sequence-sets that every other IMAP usage 
does. It happens that non-existent UIDs are ignored by the server in 
commands, but that's explicitly mentioned in the definitions of those 
commands. Server-provided sequence-sets never contain *, or 
non-existent UIDs.

>  . forces all ESEARCH results to have longer, more 
> bandwidth-consuming,
>    results even if the client has the UID map and has no need for 
> such
>    longer results

I'd be happy to entertain a proposal for an alternate to ALL which 
had this behaviour.

>  . forces ESEARCH results to ALWAYS be longer than SEARCH results 
> in any
>    server that does not monotonically assign UID values.

By "monotonic" you mean "contiguous"? ("Monotonically" has a bunch of 
different meanings, between fields and locations.)

I certainly feel that servers that do not allocate UIDs contiguously 
are fairly rare, and you're absolutely correct in your assertion that 
ESEARCH performs much worse in those, and this will mean it's a 
handful of octets larger than an unextended SEARCH.

>  . as a consequence of the previous statement creates an implicit
>    requirement, stated nowhere in any specification, that UIDs must 
> be
>    monotonically assigned in a mailbox.
> 
> 
Nope. It does imply a recommendation to server implementors that 
allocating contiguous UIDs is a good idea. (As a brief aside, IMAP 
UIDs most certainly are strictly monotonically increasing. Or merely 
monotonically increasing, depending on where you live.)

> I assert that ESEARCH sequence-set has no such semantics, and is:
>  . compatible with RFC 3501

Yes, I think it is, too.

>  . minimizes the results so that only non-matching messages cause 
> there
>    to be multiple entries in the sequence-set

That depends on whether you believe that a non-existent message 
matches or not. Your reading of the specification implies that you 
think all non-existent messages match all possible SEARCH criteria.

>  . does so even with servers that do not monotonically assign UID 
> values

ESEARCH works with servers that do not allocate UIDs sequentially and 
contiguously. Just not as well. Similarly, it works with mailboxes 
than have had every second or third message expunged - again, this is 
a worst-case. Thankfully equally rare.

>  . as a consequence of the previous states create no new implicit
>    requirements
> 
> 
But ESEARCH absolutely does not have any other requirements.

> This does mean that ESEARCH can't be used to get the UID/msgno map 
> as Pawel complains.  But there are two other commands in RFC 3501 
> that can accomplish that: FETCH and UID SEARCH.  FETCH has always 
> been the official way.
> 
> It is a fortunate accident that UID SEARCH returns the map as well. 
>  But how many applications that use UID SEARCH to obtain the 
> UID/msgno map can handle:
> 	tag UID SEARCH ALL
> 	* SEARCH 4929 3012
> 	* SEARCH 12 9993
> 	tag OK done
> Such a scenario is completely valid in RFC 3501!

The fact that a specification does not flatly prohibit doing 
something idiotic does not mean that it is a good thing for servers 
(or clients) to do the idiotic thing.

>   If your client can't handle that, it's already broken.  Hence 
> that strange warning in RFC 4731:
>          [...]  Note, the client
>          MUST NOT assume that messages/UIDs will be listed in any
>          particular order.
> 
> 
Indeed, that's the reason. We also wanted to make it explicit that 
results such as 10,3,47 where completely valid.

> Think of the effect of what you are saying.  For the benefit of 
> clients that are not satisfied with having TWO ways of getting the 
> UID/msgno map, IMAP must have THREE ways of getting the UID/msgno 
> map; and to provide that third way, an advertised protocol 
> optimization MUST be reduced and in some cases turned into a 
> chattiness pessimization!  Given:
> 	x01 UID SEARCH ALL
> 	* SEARCH 12 3012 4929 5948 6666 7739 8199 8839 9017 9993
> 	x01 OK done
> Compare:
> 	x02 UID SEARCH RETURN () ALL
> 	* ESEARCH (TAG x02) ALL 12 3012 4929 5948 6666 7739 8199 8839 9017 
> 9993
> 	x02 OK done

[Not that this affects your point, but it'd be comma-seperated, and 
have UID present]

> with:
> 	x02 UID SEARCH RETURN () ALL
> 	* ESEARCH (TAG x02) ALL 12:9993
> 	x02 OK done
> 
> 
Hey, why stop there? If the client really wanted to know the 
beginning and end of the range, it could use MIN/MAX. No, instead:

* ESEARCH (TAG x02) UID ALL 1:4294967295

According to your reading, that's valid.

> Furthermore, this pessimization only happens with UIDs.  Message 
> numbers won't get pessimized:
> 	x03 SEARCH RETURN () ALL
> 	* ESEARCH (TAG x02) ALL 1:10
> 	x03 OK done
> 
> 
Indeed - message numbers are always contiguously allocated by 
definition, so clients which use them can gain bandwidth gains. In 
fact, I do this in my own client in some cases, for precisely this 
reason.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Date: Mon, 13 Aug 2007 11:32:35 +0100
From: Alexey Melnikov <[email-masked]>
To: Mark Crispin <[email-masked]>
CC: [email-masked]
Subject: Re: UID SEARCH responses
Sender: [email-masked]

Mark Crispin wrote:

>> I disagree, I think you've misinterpreted the document. If enough 
>> people disagree, and think that UID sets are somehow defined to 
>> include missing UIDs - which I dispute - then I suppose we could 
>> issue a revision that deprecates ALL, replacing it with (perhaps) 
>> RANGE and MAP.
>
> You have to do more than that.  You have to deprecate all of ESEARCH, 
> and replace it with different syntax.

(Irrespectively to whether you are right or wrong on the issue.)
I am sorry Mark, but this is nonsense. RFC 4731 does NOT define ESEARCH 
response or extended SEARCH, they are defined in RFC 4466.
Both extended SEARCH and ESEARCH response were designed to be extensible.

Date: Mon, 13 Aug 2007 09:02:23 +0200
From: Pawel Salek <[email-masked]>
Subject: Re: UID SEARCH responses
To: Martin Konold <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

On 08/13/2007 05:32:56 AM, Martin Konold wrote:
> 
> Am Montag 13 August 2007 schrieben Sie:
> 
> Hi Pieter,
> 
> > Personally I think that having a search result that does not stand  
> alone
> > (as in sends a range when some messages in the range do not exist) is
> > bogus.
> 
> I am wondering if this can be kept at all times. Basically I consider  
> IMAP tobe incoherent anyway.
> In other words: When returning an explicit list of UIDs like 1,3,4  
> instead of1:4 not much is gained simply because at _any_ time it is  
> allowed within theprotocol that any message might have got  
> deleted/expunged in the meantime.

Not quite so - think of the example I gave: a server returning 1,3,4  
allows the client to drop the cached information about message with uid  
2. This information is absent in the 1:4 response. Expunges arriving at  
arbitrary times do not change this picture.

Pawel

From: Martin Konold <[email-masked]>
Organization: erfrakon
To: [email-masked]
Subject: Re: UID SEARCH responses
Date: Mon, 13 Aug 2007 05:32:56 +0200
Sender: [email-masked]

Am Montag 13 August 2007 schrieben Sie:

Hi Pieter,

> Personally I think that having a search result that does not stand alone
> (as in sends a range when some messages in the range do not exist) is
> bogus.

I am wondering if this can be kept at all times. Basically I consider IMAP to 
be incoherent anyway. 

In other words: When returning an explicit list of UIDs like 1,3,4 instead of 
1:4 not much is gained simply because at _any_ time it is allowed within the 
protocol that any message might have got deleted/expunged in the meantime.

Regards,
-- martin konold

-- 
e r f r a k o n
Erlewein, Frank, Konold & Partner - Beratende Ingenieure und Physiker
Sitz: Adolfstraße 23 Stuttgart - Partnerschaftsregister Stuttgart PR 126
http://www.erfrakon.com/

Date: Sun, 12 Aug 2007 20:21:16 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: Peter Coates <[email-masked]>
cc: "'Pawel Salek'" <[email-masked]>, [email-masked]
Subject: RE: UID SEARCH responses
Sender: [email-masked]

On Sun, 12 Aug 2007, Peter Coates wrote:
> Personally I think that having a search result that does not stand alone (as
> in sends a range when some messages in the range do not exist) is bogus.

Why do you think that it does not "stand alone"?  The returned 
sequence-set is explicitly and unambiguously valid as a sequence-set to 
any other IMAP command and will absolutely return to the matching messages 
and no others.

By "stand alone", do you mean "mathematically calculate the UID/msgno 
map"?  If so, where in RFC 4731 is this property stated?

> The results that are returned as a result of a UID SEARCH RETURN (UPDATE)
> command explicitly preclude sending a range (eg 1:4) to mean 1,4 even if 2
> and 3 do not exist.

Where does it "explicitly" state that in RFC 4731?

The syntax for SEARCH results is zero or more space delimited message 
numbers or UIDs.  Note that this is a different syntax from sequence-set. 
Not only does it not have ranges, it also uses space instead of comma for 
the delimiter.

The syntax for ESEARCH results is the RFC 3501 sequence-set rule, which 
has comma-delimited message numbers, UIDs, or ranges (a range being a pair 
of colon delimited message numbers or UIDs).  It is not unreasonable to 
believe that the use of the same syntax indicates the same semantics.

The definition of sequence-set in RFC 3501 explicitly allows UID ranges to 
be used even when there are holes in the UID space.  Nothing in RFC 4731 
states that its definition of sequence-set is different from RFC 3501. 
In fact, there is text in RFC 4731 that clearly states that the 
sequence-set returned from ESEARCH is intended to be used as an RFC 3501 
sequence-set.

Nowhere in RFC 4731 does it say that an RFC 4731 sequence-set has new 
semantics that prohibit the server from returning ranges that encompass 
holes in monotonically assigned UID space.

If you assert that ESEARCH sequence-set has such semantics, and MUST NOT 
send ranges that have holes, then in effect ESEARCH:
  . creates a new type of sequence-set that is different from all other
    IMAP use of sequence-set
  . forces all ESEARCH results to have longer, more bandwidth-consuming,
    results even if the client has the UID map and has no need for such
    longer results
  . forces ESEARCH results to ALWAYS be longer than SEARCH results in any
    server that does not monotonically assign UID values.
  . as a consequence of the previous statement creates an implicit
    requirement, stated nowhere in any specification, that UIDs must be
    monotonically assigned in a mailbox.

I assert that ESEARCH sequence-set has no such semantics, and is:
  . compatible with RFC 3501
  . minimizes the results so that only non-matching messages cause there
    to be multiple entries in the sequence-set
  . does so even with servers that do not monotonically assign UID values
  . as a consequence of the previous states create no new implicit
    requirements

This does mean that ESEARCH can't be used to get the UID/msgno map as 
Pawel complains.  But there are two other commands in RFC 3501 that can 
accomplish that: FETCH and UID SEARCH.  FETCH has always been the official 
way.

It is a fortunate accident that UID SEARCH returns the map as well.  But 
how many applications that use UID SEARCH to obtain the UID/msgno map can 
handle:
 	tag UID SEARCH ALL
 	* SEARCH 4929 3012
 	* SEARCH 12 9993
 	tag OK done
Such a scenario is completely valid in RFC 3501!  If your client can't 
handle that, it's already broken.  Hence that strange warning in RFC 4731:
          [...]  Note, the client
          MUST NOT assume that messages/UIDs will be listed in any
          particular order.

> I think it is unwise for a server knowingly to emit responses that are
> likely to cause clients confusion no matter what the legalistic reading of
> the RFC might be.

I will remember that statement the next time someone claims that 
implementations must MIME-process message headers that lack a 
MIME-Version: and accept overlong MIME encoded-word tokens in a header.

Think of the effect of what you are saying.  For the benefit of clients 
that are not satisfied with having TWO ways of getting the UID/msgno map, 
IMAP must have THREE ways of getting the UID/msgno map; and to provide 
that third way, an advertised protocol optimization MUST be reduced and in 
some cases turned into a chattiness pessimization!  Given:
 	x01 UID SEARCH ALL
 	* SEARCH 12 3012 4929 5948 6666 7739 8199 8839 9017 9993
 	x01 OK done
Compare:
 	x02 UID SEARCH RETURN () ALL
 	* ESEARCH (TAG x02) ALL 12 3012 4929 5948 6666 7739 8199 8839 9017 9993
 	x02 OK done
with:
 	x02 UID SEARCH RETURN () ALL
 	* ESEARCH (TAG x02) ALL 12:9993
 	x02 OK done

Furthermore, this pessimization only happens with UIDs.  Message numbers 
won't get pessimized:
 	x03 SEARCH RETURN () ALL
 	* ESEARCH (TAG x02) ALL 1:10
 	x03 OK done

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Date: Sun, 12 Aug 2007 19:11:31 -0700
From: Peter Coates <[email-masked]>
Subject: RE: UID SEARCH responses
To: "'Mark Crispin'" <[email-masked]>, "'Pawel Salek'" <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

I am surprised at the heat of this conversation.  I would hope for
conversations that generated light, not heat.

I happen not to agree with Mark's position, but recognize that it has some
validity.  However his point about 64bit UID is overstating the case.
Having UIDs not increment without gaps is likely either to break things
(shouldn't) or at the very least invalidate assumptions built into
optimization in the protocol or people's implementations.  I would hope that
any GUID mechanism introduced to IMAP would be separate from UIDs.

Personally I think that having a search result that does not stand alone (as
in sends a range when some messages in the range do not exist) is bogus.
The results that are returned as a result of a UID SEARCH RETURN (UPDATE)
command explicitly preclude sending a range (eg 1:4) to mean 1,4 even if 2
and 3 do not exist.  I have only limited skin in this argument as I am
mostly interested in the server side.
I think it is unwise for a server knowingly to emit responses that are
likely to cause clients confusion no matter what the legalistic reading of
the RFC might be.

Peter

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of Mark Crispin
Sent: August 12, 2007 15:02
To: Pawel Salek
Cc: [email-masked]
Subject: Re: UID SEARCH responses

On Sun, 12 Aug 2007, Pawel Salek wrote:
> I wonder whether it is relevant or not but the copy of RFC3501 that I use
[1] 
> does not mention UIDs on page 61... Can you please be more specific?

Page 61 of RFC 3501 is the latter part of section 6.4.8.  That is the 
section that describes the UID command!

>> Unlike message numbers, the UID sets
>> 	38782:41424
>> and
>> 	38782,38789,40004,40301,40421,40814,41424
>> are exactly equivalent if no other UIDs exist within that range.
> "...if no other UIDs exists...". The client may not know whether they do
or 
> do not exist.

That doesn't matter.  The server knows, and that exact sequence will 
refer to the messages in question, and only the messages in question.

> The information conveyed to client is  different in these two cases. I see

> that the basically only thing the client can do with the response
generated 
> by your implementation is to send the sequence-set back to the server.

That is what RFC 4731 says as being the purpose of those ESEARCH results.

> It cannot use it to eg. purge local cache for message with UID 40000 
> because it may exist.

Hence there are two other mechanisms to get the UID map which will 
accomplish that purpose

> I cannot help the feeling you are way too dramatic here. I have no idea
how 
> IETF deals with imprecise specifications but my impression is it is not
the 
> first time there is some dispute how to interpret RFCs.

The IETF generally does not make incompatible changes to a published RFC 
(I-Ds are a different matter) that will have the effect of rendering 
previously compliant implementations non-compliant.  There have been 
lawsuits, and threats of lawsuits, over such matters in the past.

> The usefulness of the 
> response generated by current version of UW-IMAP is strongly limited and 
> considered by me, a client implementer, a bug.

That is your opinion.  I disagree.  I see no point to having the 
complexity of range syntax in returned UID results if it is limited to use 
for those cases where there are three or more monotonically consecutive 
matching UIDs.  I will vigorously fight any ex post facto incompatible 
change to the published semantics of ESEARCH.

> Please observe it does not say "usable ONLY in other IMAP commands. Users 
> must be cautious that the UID sequence sets may be collapsed by the server
if 
> separated only by non-existing UIDs". I do not think your argument is
valid.

The absence of text in an RFC can not be construed to indicate that the 
RFC takes the opposite position.

The absence of text in an RFC indicates just one thing: that the RFC is 
silent on that matter.

However (and this is important!!) if an RFC does not forbid something that 
is syntactically and semantically valid in the RFC, then that something is 
permitted EVEN IF the author of the RFC thinks that it is bad or wrong or 
stupid behavior.

It is true that RFCs can not forbid every possible bad or wrong or stupid 
behavior.  But if the RFC does not prohibit the behavior in question, and 
that behavior can be reasonably inferred from the specification, then the 
only way to deal with a conflict of interpretation is either to convince 
the other side of the rightness of ones interpretation, or to accept that 
both interpretations are valid.

I have encountered that problem innumerable times.  The majority of IMAP 
client implementations things which I consider to bad or wrong or stupid. 
But, because RFC 3501 (and its predecessors) did not specifically forbid 
the behavior in question, nothing can be done.

In this case, I consider my interpretation to be reasonably inferred from 
the specification AND reasonable in the overall context of IMAP.

>>  . my interpretation returns a valid sequence-set usable in other IMAP
>>     commands
> But your interpretation cannot be used for the same client-side tasks as 
> ordinary UID SEARCH response can.

You can use ordinary UID SEARCH if you need a UID map, and use ESEARCH to 
do traffic-efficient searches once you have the map.

>>  . your interpretation prevents optimization unless there are at least
>>     three contiguous extant UIDs matching the result; without such the
>>     ESEARCH response is larger than the SEARCH result.
> And that's quite common case, I think.

That assumes that UIDs are assigned monotonically by the server, and that 
messages are rarely expunged.

In fact, the implicit requirement in your interpretation that UIDs be 
assigned monotonically by the server is the final nail in the coffin. 
IMAP has NO such requirement.

This is on top of earlier nails, such as the assumption that sequence-set 
means one thing in RFC 3501 and something completely different in RFC 
4731.  Nothing confuses matters more than to have protocol elements that 
change their meaning depending upon context or defining document.  People 
criticize IMAP (justly, in my opinion) for being weighed down with a mass 
of complexity.

> Why not try fixing implementation in question and add a clarification in
RFC 
> 4731 errata?

That only works if the implementor agrees that it is a bug in the 
implementation.  He does not.

He thinks that including ranges in ESEARCH is pointless otherwise.

He objects strongly to the idea of IMAP having two types of UID ranges 
with a subtle semantic difference between the two, not to mention adding a 
new hidden requirement to IMAP that UIDs be assigned monotonically.  IMAP 
already has entirely too many subtle semantic differences and hidden 
requirements; this is not a case where "more is better".

> I do not see any client 
> author being happy with UID ESEARCH responses generated by current UW-IMAP
- 
> what could it be used for?

It returns shorter search results if you do not need a UID/msgno map. 
You only need to get a UID/msgno map once in a session; after that 
additions to the map are a simple operation and removals from the map are 
easily calculated.  What's more, two other ways exist in IMAP to get the 
UID/msgno map.

Searches, on the other hand, are done many times in a session.  Why should 
those results be weighed down with extra traffic in the assumption that
  (1) the client wants to get the map
  (2) the client does not want to use the two ways which exist in the base
      IMAP specification
  (3) UIDs are assigned monotonically increasing.

> I will be happy, too, with only adding a clarification to RFC 4731 that
the 
> UID ESEARCH responses may include non-existing UIDs for the benefit of 
> compression. Such interpetation makes the extension less useful for me but
is 
> not a deal-breaker.

Either that, or let it be.

I understand that you would like to have a UID map operation that 
compresses monotonically increasing UIDs into ranges, but does not 
compress UID ranges that are not monotonically increasing.  However, some 
servers do not assign monotonically increasing UIDs; and the operation is 
pointless on those servers compared to the existing UID map operations.

Furthermore, we need to recognize that IMAP's lock into a 32-bit world 
will not be forever.  We are already encountering implementations that 
incorrectly treat IMAP numbers as being signed (and thus limited to 
2GB-1).  The first domino that is likely to fall is the QUOTA extension, 
followed by the base specifications's limit of message size to 4GB-1 
octets.

32-bit UIDs are also in trouble.  Although 4GB messages in a mailbox seems 
unplausible, lots of people have expressed unhappiness with UIDs not being 
globally unique and that creates pressure on a 32-bit space.  Should the 
floodgates of 64-bit be opened, globally unique UIDs will rear their head 
and pretty much preclude monotonically increasing UIDs in a single 
mailbox.

I don't think that any IMAP extension should be designed with implicit 
assumptions that are not in the base specification, but are doomed.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Date: Sun, 12 Aug 2007 15:01:51 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: Pawel Salek <[email-masked]>
cc: [email-masked]
Subject: Re: UID SEARCH responses
Sender: [email-masked]

On Sun, 12 Aug 2007, Pawel Salek wrote:
> I wonder whether it is relevant or not but the copy of RFC3501 that I use [1] 
> does not mention UIDs on page 61... Can you please be more specific?

Page 61 of RFC 3501 is the latter part of section 6.4.8.  That is the 
section that describes the UID command!

>> Unlike message numbers, the UID sets
>> 	38782:41424
>> and
>> 	38782,38789,40004,40301,40421,40814,41424
>> are exactly equivalent if no other UIDs exist within that range.
> "...if no other UIDs exists...". The client may not know whether they do or 
> do not exist.

That doesn't matter.  The server knows, and that exact sequence will 
refer to the messages in question, and only the messages in question.

> The information conveyed to client is  different in these two cases. I see 
> that the basically only thing the client can do with the response generated 
> by your implementation is to send the sequence-set back to the server.

That is what RFC 4731 says as being the purpose of those ESEARCH results.

> It cannot use it to eg. purge local cache for message with UID 40000 
> because it may exist.

Hence there are two other mechanisms to get the UID map which will 
accomplish that purpose

> I cannot help the feeling you are way too dramatic here. I have no idea how 
> IETF deals with imprecise specifications but my impression is it is not the 
> first time there is some dispute how to interpret RFCs.

The IETF generally does not make incompatible changes to a published RFC 
(I-Ds are a different matter) that will have the effect of rendering 
previously compliant implementations non-compliant.  There have been 
lawsuits, and threats of lawsuits, over such matters in the past.

> The usefulness of the 
> response generated by current version of UW-IMAP is strongly limited and 
> considered by me, a client implementer, a bug.

That is your opinion.  I disagree.  I see no point to having the 
complexity of range syntax in returned UID results if it is limited to use 
for those cases where there are three or more monotonically consecutive 
matching UIDs.  I will vigorously fight any ex post facto incompatible 
change to the published semantics of ESEARCH.

> Please observe it does not say "usable ONLY in other IMAP commands. Users 
> must be cautious that the UID sequence sets may be collapsed by the server if 
> separated only by non-existing UIDs". I do not think your argument is valid.

The absence of text in an RFC can not be construed to indicate that the 
RFC takes the opposite position.

The absence of text in an RFC indicates just one thing: that the RFC is 
silent on that matter.

However (and this is important!!) if an RFC does not forbid something that 
is syntactically and semantically valid in the RFC, then that something is 
permitted EVEN IF the author of the RFC thinks that it is bad or wrong or 
stupid behavior.

It is true that RFCs can not forbid every possible bad or wrong or stupid 
behavior.  But if the RFC does not prohibit the behavior in question, and 
that behavior can be reasonably inferred from the specification, then the 
only way to deal with a conflict of interpretation is either to convince 
the other side of the rightness of ones interpretation, or to accept that 
both interpretations are valid.

I have encountered that problem innumerable times.  The majority of IMAP 
client implementations things which I consider to bad or wrong or stupid. 
But, because RFC 3501 (and its predecessors) did not specifically forbid 
the behavior in question, nothing can be done.

In this case, I consider my interpretation to be reasonably inferred from 
the specification AND reasonable in the overall context of IMAP.

>>  . my interpretation returns a valid sequence-set usable in other IMAP
>>     commands
> But your interpretation cannot be used for the same client-side tasks as 
> ordinary UID SEARCH response can.

You can use ordinary UID SEARCH if you need a UID map, and use ESEARCH to 
do traffic-efficient searches once you have the map.

>>  . your interpretation prevents optimization unless there are at least
>>     three contiguous extant UIDs matching the result; without such the
>>     ESEARCH response is larger than the SEARCH result.
> And that's quite common case, I think.

That assumes that UIDs are assigned monotonically by the server, and that 
messages are rarely expunged.

In fact, the implicit requirement in your interpretation that UIDs be 
assigned monotonically by the server is the final nail in the coffin. 
IMAP has NO such requirement.

This is on top of earlier nails, such as the assumption that sequence-set 
means one thing in RFC 3501 and something completely different in RFC 
4731.  Nothing confuses matters more than to have protocol elements that 
change their meaning depending upon context or defining document.  People 
criticize IMAP (justly, in my opinion) for being weighed down with a mass 
of complexity.

> Why not try fixing implementation in question and add a clarification in RFC 
> 4731 errata?

That only works if the implementor agrees that it is a bug in the 
implementation.  He does not.

He thinks that including ranges in ESEARCH is pointless otherwise.

He objects strongly to the idea of IMAP having two types of UID ranges 
with a subtle semantic difference between the two, not to mention adding a 
new hidden requirement to IMAP that UIDs be assigned monotonically.  IMAP 
already has entirely too many subtle semantic differences and hidden 
requirements; this is not a case where "more is better".

> I do not see any client 
> author being happy with UID ESEARCH responses generated by current UW-IMAP - 
> what could it be used for?

It returns shorter search results if you do not need a UID/msgno map. 
You only need to get a UID/msgno map once in a session; after that 
additions to the map are a simple operation and removals from the map are 
easily calculated.  What's more, two other ways exist in IMAP to get the 
UID/msgno map.

Searches, on the other hand, are done many times in a session.  Why should 
those results be weighed down with extra traffic in the assumption that
  (1) the client wants to get the map
  (2) the client does not want to use the two ways which exist in the base
      IMAP specification
  (3) UIDs are assigned monotonically increasing.

> I will be happy, too, with only adding a clarification to RFC 4731 that the 
> UID ESEARCH responses may include non-existing UIDs for the benefit of 
> compression. Such interpetation makes the extension less useful for me but is 
> not a deal-breaker.

Either that, or let it be.

I understand that you would like to have a UID map operation that 
compresses monotonically increasing UIDs into ranges, but does not 
compress UID ranges that are not monotonically increasing.  However, some 
servers do not assign monotonically increasing UIDs; and the operation is 
pointless on those servers compared to the existing UID map operations.

Furthermore, we need to recognize that IMAP's lock into a 32-bit world 
will not be forever.  We are already encountering implementations that 
incorrectly treat IMAP numbers as being signed (and thus limited to 
2GB-1).  The first domino that is likely to fall is the QUOTA extension, 
followed by the base specifications's limit of message size to 4GB-1 
octets.

32-bit UIDs are also in trouble.  Although 4GB messages in a mailbox seems 
unplausible, lots of people have expressed unhappiness with UIDs not being 
globally unique and that creates pressure on a 32-bit space.  Should the 
floodgates of 64-bit be opened, globally unique UIDs will rear their head 
and pretty much preclude monotonically increasing UIDs in a single 
mailbox.

I don't think that any IMAP extension should be designed with implicit 
assumptions that are not in the base specification, but are doomed.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Date: Sun, 12 Aug 2007 22:37:57 +0200
From: Pawel Salek <[email-masked]>
Subject: Re: UID SEARCH responses
To: Mark Crispin <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

On 08/12/2007 03:01:26 AM, Mark Crispin wrote:
> 
> On Sat, 11 Aug 2007, Dave Cridland wrote:
> >> For what it's worth, UW imapd issues the
> >> 	* ESEARCH (TAG ".") UID ALL 38782:41424
> >> response instead of
> >> 	* ESEARCH (TAG ".") UID ALL  
> 38782,38789,40004,40301,40421,40814,41424
> >> because there are no other UIDs in that range.
> > Ah. That's definitely not what I intended, and it never crossed my  
> mind that> this was a possible interpretation. The second is correct.
> 
> Please quote me anyplace in RFC 4731 where it says that.

> > Ouch. I'd disagree, ALL is defined as "Return all message  
> numbers/UIDs that> satisfy the SEARCH criteria", not "Return all  
> message numbers/UIDs that> either satisfy the SEARCH criteria, or  
> don't exist, if that's easier".
> 
> You didn't pay adequate attention to the definition of UIDs in RFC  
> 3501.Pay particular attention to the text on page 61.

I wonder whether it is relevant or not but the copy of RFC3501 that I  
use [1] does not mention UIDs on page 61... Can you please be more  
specific?

> Unlike message numbers, the UID sets
>  	38782:41424
> and
>  	38782,38789,40004,40301,40421,40814,41424
> are exactly equivalent if no other UIDs exist within that range.

"...if no other UIDs exists...". The client may not know whether they  
do or do not exist.
  The information conveyed to client is  different in these two cases.  
I see that the basically only thing the client can do with the response  
generated by your implementation is to send the sequence-set back to  
the server. It cannot use it to eg. purge local cache for message with  
UID 40000 because it may exist.

> Furthermore, RFC 4731 specifically refers to the ESEARCH result  
> beinginterpreted as in the base specification:
>  	 A sequence-set
>           representation may be more compact and can be used as is in  
> a
>           subsequent command that accepts sequence-set.
> 
> If it was your intention to require that ESEARCH returns the map with  
> UIDsthen RFC 4731 failed to state so.
> 
> > I disagree, I think you've misinterpreted the document. If enough  
> people> disagree, and think that UID sets are somehow defined to  
> include missing UIDs> - which I dispute - then I suppose we could  
> issue a revision that deprecates> ALL, replacing it with (perhaps)  
> RANGE and MAP.
> 
> You have to do more than that.  You have to deprecate all of ESEARCH,  
> andreplace it with different syntax.
> 
> The horse is out of the barn.  RFC 4731 does not forbid my  
> interpretation,and my interpretation is out in the field.

I cannot help the feeling you are way too dramatic here. I have no idea  
how IETF deals with imprecise specifications but my impression is it is  
not the first time there is some dispute how to interpret RFCs. The  
usefulness of the response generated by current version of UW-IMAP is  
strongly limited and considered by me, a client implementer, a bug.   
And you and I know it is not the first bug in the ESEARCH  
implementation of UW-IMAP in spite of short history of its  
implementation in this code base (ok, they are not exactly comparable:  
the first one was an obvious one!) so why not change this response as  
well and be done with it?

> > No. It's very unusual in practise for a UID map sent by UID SEARCH  
> RETURN ()> ALL to equal the size of a UID SEARCH ALL. Normally, it's  
> significantly> smaller. You'd need to have a maximum of two  
> contiguous UIDs anywhere in the> mailbox to equal the size, and  
> that's vanishingly rare in practise.
> 
> That statement makes no sense at all!
> 
> In your interpretation (that a UID range must encompass contiguous  
> extantUIDs) ESEARCH responses will always be larger than SEARCH  
> responses (dueto the additional stuff in SEARCH responses) UNLESS  
> there are at leastthree contiguous extant UIDs.  It is LESS likely  
> that there be threecontiguous extant UIDs than there be two  
> contiguous extant UIDs.
> 
> I'm taking a hard line, because:
>   . UID SEARCH ALL exists as a command to get the map

And? One of the points with ESEARCH is provide a compressed response.

>   . RFC 4731 specifically refers to the ESEARCH result being a  
> sequence-set
>      usable in other IMAP commands

Please observe it does not say "usable ONLY in other IMAP commands.  
Users must be cautious that the UID sequence sets may be collapsed by  
the server if separated only by non-existing UIDs". I do not think your  
argument is valid.

>   . my interpretation returns a valid sequence-set usable in other  
> IMAP
>      commands

But your interpretation cannot be used for the same client-side tasks  
as ordinary UID SEARCH response can.

>   . my interpretation optimizes ESEARCH return results

Your interpretation does it at the expense of usefulness.

>   . your interpretation prevents optimization unless there are at  
> least
>      three contiguous extant UIDs matching the result; without such  
> the
>      ESEARCH response is larger than the SEARCH result.

And that's quite common case, I think.

>   . the sole benefit of your interpretion is to be able to use  
> ESEARCH as
>      an alternative to get the map

It appears to me that disconnected, bandwidth-limited clients may see  
the ability to receive the compressed map as a big advantage.

>   . RFC 4731 says nothing about UID ranges in ESEARCH being different  
> from
>      IMAP ranges.  IMAP does not require that UID ranges refer to
>      contiguous extant UIDs; in fact RFC 3501 says quite the opposite.
>   . RFC 4731 uses the same syntax rule as RFC 3501 for UID ranges.

Syntax does not imply semantics. One can argue that the extension  
provided by RFC 4731 is called E-SEARCH and it would be fair to expect  
that the provided capability is not inferior to ordinary SEARCH.

> That isn't to say that I am not sympathetic.  I am.  Nonetheless, the  
> shoehas been on the other foot many times in the past, and I've had  
> to dealwith the resulting fiasco.
> 
> I recommend living with the unexpected consequence.  Trying to fix  
> ESEARCH(which I argue isn't broken) will just diminish what little  
> credibilityIMAP extensions still have.

Why not try fixing implementation in question and add a clarification  
in RFC 4731 errata? It's not like people have already coded hundreds of  
applications that rely on this specific behavior of UW-IMAP. I do not  
see any client author being happy with UID ESEARCH responses generated  
by current UW-IMAP - what could it be used for?

I will be happy, too, with only adding a clarification to RFC 4731 that  
the UID ESEARCH responses may include non-existing UIDs for the benefit  
of compression. Such interpetation makes the extension less useful for  
me but is not a deal-breaker.

Pawel

[1] http://tools.ietf.org/html/rfc3501

Date: Sat, 11 Aug 2007 18:01:26 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: Dave Cridland <[email-masked]>
cc: [email-masked]
Subject: Re: UID SEARCH responses
Sender: [email-masked]

On Sat, 11 Aug 2007, Dave Cridland wrote:
>> For what it's worth, UW imapd issues the
>> 	* ESEARCH (TAG ".") UID ALL 38782:41424
>> response instead of
>> 	* ESEARCH (TAG ".") UID ALL 38782,38789,40004,40301,40421,40814,41424
>> because there are no other UIDs in that range.
> Ah. That's definitely not what I intended, and it never crossed my mind that 
> this was a possible interpretation. The second is correct.

Please quote me anyplace in RFC 4731 where it says that.

> Ouch. I'd disagree, ALL is defined as "Return all message numbers/UIDs that 
> satisfy the SEARCH criteria", not "Return all message numbers/UIDs that 
> either satisfy the SEARCH criteria, or don't exist, if that's easier".

You didn't pay adequate attention to the definition of UIDs in RFC 3501. 
Pay particular attention to the text on page 61.

Unlike message numbers, the UID sets
 	38782:41424
and
 	38782,38789,40004,40301,40421,40814,41424
are exactly equivalent if no other UIDs exist within that range.

Furthermore, RFC 4731 specifically refers to the ESEARCH result being 
interpreted as in the base specification:
 	 A sequence-set
          representation may be more compact and can be used as is in a
          subsequent command that accepts sequence-set.

If it was your intention to require that ESEARCH returns the map with UIDs 
then RFC 4731 failed to state so.

> I disagree, I think you've misinterpreted the document. If enough people 
> disagree, and think that UID sets are somehow defined to include missing UIDs 
> - which I dispute - then I suppose we could issue a revision that deprecates 
> ALL, replacing it with (perhaps) RANGE and MAP.

You have to do more than that.  You have to deprecate all of ESEARCH, and 
replace it with different syntax.

The horse is out of the barn.  RFC 4731 does not forbid my interpretation, 
and my interpretation is out in the field.

> No. It's very unusual in practise for a UID map sent by UID SEARCH RETURN () 
> ALL to equal the size of a UID SEARCH ALL. Normally, it's significantly 
> smaller. You'd need to have a maximum of two contiguous UIDs anywhere in the 
> mailbox to equal the size, and that's vanishingly rare in practise.

That statement makes no sense at all!

In your interpretation (that a UID range must encompass contiguous extant 
UIDs) ESEARCH responses will always be larger than SEARCH responses (due 
to the additional stuff in SEARCH responses) UNLESS there are at least 
three contiguous extant UIDs.  It is LESS likely that there be three 
contiguous extant UIDs than there be two contiguous extant UIDs.

I'm taking a hard line, because:
  . UID SEARCH ALL exists as a command to get the map
  . RFC 4731 specifically refers to the ESEARCH result being a sequence-set
     usable in other IMAP commands
  . my interpretation returns a valid sequence-set usable in other IMAP
     commands
  . my interpretation optimizes ESEARCH return results
  . your interpretation prevents optimization unless there are at least
     three contiguous extant UIDs matching the result; without such the
     ESEARCH response is larger than the SEARCH result
  . the sole benefit of your interpretion is to be able to use ESEARCH as
     an alternative to get the map
  . RFC 4731 says nothing about UID ranges in ESEARCH being different from
     IMAP ranges.  IMAP does not require that UID ranges refer to
     contiguous extant UIDs; in fact RFC 3501 says quite the opposite.
  . RFC 4731 uses the same syntax rule as RFC 3501 for UID ranges.

That isn't to say that I am not sympathetic.  I am.  Nonetheless, the shoe 
has been on the other foot many times in the past, and I've had to deal 
with the resulting fiasco.

I recommend living with the unexpected consequence.  Trying to fix ESEARCH 
(which I argue isn't broken) will just diminish what little credibility 
IMAP extensions still have.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Subject: Re: UID SEARCH responses
Date: Sat, 11 Aug 2007 22:59:46 +0100
From: Dave Cridland <[email-masked]>
To: Mark Crispin <[email-masked]>
Cc: <[email-masked]>
Sender: [email-masked]

On Fri Aug 10 06:47:40 2007, Mark Crispin wrote:
> 
> For what it's worth, UW imapd issues the
> 	* ESEARCH (TAG ".") UID ALL 38782:41424
> response instead of
> 	* ESEARCH (TAG ".") UID ALL 
> 38782,38789,40004,40301,40421,40814,41424
> because there are no other UIDs in that range.
> 
> 
Ah. That's definitely not what I intended, and it never crossed my 
mind that this was a possible interpretation. The second is correct.

> Pawel thought this is wrong, and that a UID range of 38782:41424 
> must mean all the possible values are in the map.
> 
> 
It does. Or, more accurately, it must mean that all those UIDs match 
the SEARCH criteria.

> I suggested that he bring the matter up on this list.  I reviewed 
> the ESEARCH RFC, and can find nothing to indicate that ESEARCH must 
> deliver a UID map (as SEARCH does) rather than a UID sequence set.
> 
> 
Ouch. I'd disagree, ALL is defined as "Return all message 
numbers/UIDs that satisfy the SEARCH criteria", not "Return all 
message numbers/UIDs that either satisfy the SEARCH criteria, or 
don't exist, if that's easier".

UID set syntax expands to all UIDs. The fact that UID FETCH etc 
happen to ignore UIDs means that clients MAY include non-existent 
UIDs as a method of reducing the bandwidth (or command length), and 
has nothing whatsoever to do with the set syntax itself. I can't find 
anything in RFC3501 which implies otherwise.

RFC4315 happens to insist on not including "extraneous UIDs", but 
that's belt and braces - I'd be happy to include such a phrase in a 
revision if it'd make people happier.

> If Pawel is correct on the intent, then the ESEARCH RFC is broken 
> since it is ambiguous on that point; either live with the 
> ambiguity, or deprecate ESEARCH and replace it with some other 
> syntax.  The horse is already out of the barn.
> 
> 
I disagree, I think you've misinterpreted the document. If enough 
people disagree, and think that UID sets are somehow defined to 
include missing UIDs - which I dispute - then I suppose we could 
issue a revision that deprecates ALL, replacing it with (perhaps) 
RANGE and MAP.

> I contend that if a client wants the map, then "UID SEARCH ALL" is 
> the correct command.  The only advantage to ESEARCH is getting the 
> tag, which I don't think is much of an advantage for that 
> particular command.

No. It's very unusual in practise for a UID map sent by UID SEARCH 
RETURN () ALL to equal the size of a UID SEARCH ALL. Normally, it's 
significantly smaller. You'd need to have a maximum of two contiguous 
UIDs anywhere in the mailbox to equal the size, and that's 
vanishingly rare in practise.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Date: Sat, 11 Aug 2007 18:10:28 +0100
From: Alexey Melnikov <[email-masked]>
To: Mark Crispin <[email-masked]>
CC: IMAP Extensions WG <[email-masked]>
Subject: Re: I-D ACTION:draft-ietf-imapext-i18n-12.txt
Sender: [email-masked]

Mark Crispin wrote:

> Here's my feeling about i;octet and i;ascii-casemap.  This feeling was 
> also how I implemented UW imapd.
>
> Given:
>
> i;octet is essentially strcmp() with no localization.
>
> i;ascii-casemap is what the IMAP base specification describes.  But 
> the IMAP base specification never intended to preclude 
> i;unicode-casemap, and even in the old days there were efforts in the 
> direction of what eventually became i;unicode-casemap.
>
> I see no reason why any client would deliberately choose i;octet or 
> i;ascii-casemap instead of i;unicode-casemap.

I can see reasons why people would want to use i;octet (Dave Cridland's 
example of searching for exact substring "OMA" where case-insensitive 
"oma" returns too many false positives).
But I don't see any reasons for choosing i;ascii-casemap instead of 
i;unicode-casemap

> We strongly want to encourage servers to implement i;unicode-casemap 
> instead of i;ascii-casemap as the default.

Agreed.

> Therefore, I believe that:
>
> i;octet and i;ascii-casemap are documented for the purpose of 
> documentation; to describe how old software worked.  It may also be 
> useful to have some sort of i;unknown-locale documented to describe 
> strtmp() with localization.  All of these are *optional to implement* 
> in servers that support IMAPI18N, and probably the only reason to 
> implement them is as an exercise.
>
> i;unicode-casemap is mandatory-to-implement, and default, in servers 
> that advertise IMAPI18N.

This would work for me.

> Servers that do not advertise IMAPI18N are strongly encouraged by this 
> specification to upgrade any legacy collation (i;octet, 
> i;ascii-casemap, i;unknown-locale) to i;unicode-casemap.
>
> Clients are on notice that, if the server does not advertise IMAPI18N, 
> it MAY have some legacy collation instead of i;unicode-casemap.

Agreed.

> Put another way:
>  . all IMAPI18N servers do i;unicode-casemap by default.
>  . some non-IMAPI18N do i;unicode-casemap as their sole collation; others
>    do a legacy collation.  There is no way for the client to tell.
>
> This may be an important enough issue to have a separate capability 
> specifically to say "even though I am not yet an IMAPI18N server, I do 
> i;unicode-casemap collation."

I think this would be worthwhile.

> I have not thoroughly reviewed IMAPI18N to determine the timeline to 
> make UW imapd compliant with IMAPI18N.  But I'd be willing to add a 
> capability now for the i;unicode-casemap collation, and (hint hint) 
> now would be an excellent time to do so.

Right. I think IMAP I18N is the right place for the new capability.

> A couple of months from now will be too late.

> In the long-term scheme of things, non-IMAPI18N servers will become a 
> vanished species, just as IMAP2 servers are today.

Date: Sat, 11 Aug 2007 11:39:40 +0100
From: Alexey Melnikov <[email-masked]>
To: Pawel Salek <[email-masked]>
CC: [email-masked]
Subject: Re: UID SEARCH responses
Sender: [email-masked]

Pawel Salek wrote:

> Hi,

Hi Pawel,

> I wonder which of the alternatives is a proper ESEARCH response to a  
> following search:
>
> . uid search 28:34
> * SEARCH 38782 38789 40004 40301 40421 40814 41424
> . OK UID SEARCH completed
>
> If I now issue corresponding ESEARCH command:
> . uid search return (all) 28:34
>
> What should the IMAP server answer:
> * ESEARCH (TAG ".") UID ALL 38782,38789,40004,40301,40421,40814,41424
> or
> * ESEARCH (TAG ".") UID ALL 38782:41424
> . OK UID SEARCH completed
> or maybe something else?
>
> The argument for the first form is that it represents a compressed 
> form  of the ordinary search response (OK, not very much compressed 
> this  time), and for the latter, that it can be used as the 
> sequence-set  argument in a subsequent command, as demanded by the RFC.

I agree with Arnt that both resulting UID sets can be used in subsequent 
commands and I can't think of any practical difference at the moment.

Can you show me how you are using the resulting UID set? I would like to 
better understand what you are trying to do and maybe suggest alternatives.

Date: Sat, 11 Aug 2007 11:36:10 +0100
From: Alexey Melnikov <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
CC: [email-masked], Pawel Salek <[email-masked]>
Subject: Re: UID SEARCH responses
Sender: [email-masked]

Arnt Gulbrandsen wrote:

> Pawel Salek writes:
>
>> The argument for the first form is that it represents a compressed 
>> form  of the ordinary search response (OK, not very much compressed 
>> this  time), and for the latter, that it can be used as the 
>> sequence-set argument in a subsequent command, as demanded by the RFC.
>
> So can the latter... if UIDs 1 and 3 exist, and UID does 2, then the 
> two following commands have the same result:
>
> a uid fetch 1:3 bodystructure
> b uid fetch 1,3 bodystructure
>
> I can't think of any difference. Uid store, uid fetch, uid search, 
> search, ...

I can't think of any difference for standard commands.
There might be some minor difference when this is used with QRESYNC 
extension, but I need to double check.

Subject: Re: I-D ACTION:draft-ietf-imapext-i18n-12.txt
From: Timo Sirainen <[email-masked]>
To: IMAP Extensions WG <[email-masked]>
Date: Fri, 10 Aug 2007 22:12:26 +0300
Sender: [email-masked]

--=-xLIbs8snByjq4V8NfzwS
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

On Fri, 2007-08-10 at 22:02 +0300, Timo Sirainen wrote:
> "The "i;octet" collation is a simple and fast collation intended for use
> on binary octet strings rather than on character data."
>=20
> I understand that to mean that no character set conversions are done
> before comparing the data, or possibly that if it is done it's possible
> to find e.g. invalid UTF-8 strings. If that's the case I'm strongly
> against having i;octet as a requirement.

Unless it searches only non-text MIME parts. Then it could be useful.

--=-xLIbs8snByjq4V8NfzwS
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

--=-xLIbs8snByjq4V8NfzwS--

Subject: Re: I-D ACTION:draft-ietf-imapext-i18n-12.txt
From: Timo Sirainen <[email-masked]>
To: Mark Crispin <[email-masked]>
Cc: IMAP Extensions WG <[email-masked]>
Date: Fri, 10 Aug 2007 22:02:45 +0300
Sender: [email-masked]

--=-O/5+K7Wbik+nzHqr+36O
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

On Fri, 2007-08-10 at 10:37 -0700, Mark Crispin wrote:
> i;octet is essentially strcmp() with no localization.

"The "i;octet" collation is a simple and fast collation intended for use
on binary octet strings rather than on character data."

I understand that to mean that no character set conversions are done
before comparing the data, or possibly that if it is done it's possible
to find e.g. invalid UTF-8 strings. If that's the case I'm strongly
against having i;octet as a requirement.

> I have not thoroughly reviewed IMAPI18N to determine the timeline to make=
=20
> UW imapd compliant with IMAPI18N.  But I'd be willing to add a capability=
=20
> now for the i;unicode-casemap collation, and (hint hint) now would be an=20
> excellent time to do so.  A couple of months from now will be too late.

Same here. I've implemented i;unicode-casemap as default for upcoming
Dovecot v1.1 but not i18n.

BTW. Does anyone have test cases to verify i;unicode-casemap
implementation correctness?

--=-O/5+K7Wbik+nzHqr+36O
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

--=-O/5+K7Wbik+nzHqr+36O--

Date: Fri, 10 Aug 2007 10:37:16 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: IMAP Extensions WG <[email-masked]>
Subject: Re: I-D ACTION:draft-ietf-imapext-i18n-12.txt 
Sender: [email-masked]

Here's my feeling about i;octet and i;ascii-casemap.  This feeling was 
also how I implemented UW imapd.

Given:

i;octet is essentially strcmp() with no localization.

i;ascii-casemap is what the IMAP base specification describes.  But the 
IMAP base specification never intended to preclude i;unicode-casemap, and 
even in the old days there were efforts in the direction of what 
eventually became i;unicode-casemap.

I see no reason why any client would deliberately choose i;octet or 
i;ascii-casemap instead of i;unicode-casemap.

We strongly want to encourage servers to implement i;unicode-casemap 
instead of i;ascii-casemap as the default.

Therefore, I believe that:

i;octet and i;ascii-casemap are documented for the purpose of 
documentation; to describe how old software worked.  It may also be useful 
to have some sort of i;unknown-locale documented to describe strtmp() with 
localization.  All of these are *optional to implement* in servers that 
support IMAPI18N, and probably the only reason to implement them is as an 
exercise.

i;unicode-casemap is mandatory-to-implement, and default, in servers that 
advertise IMAPI18N.

Servers that do not advertise IMAPI18N are strongly encouraged by this 
specification to upgrade any legacy collation (i;octet, i;ascii-casemap, 
i;unknown-locale) to i;unicode-casemap.

Clients are on notice that, if the server does not advertise IMAPI18N, it 
MAY have some legacy collation instead of i;unicode-casemap.

Put another way:
  . all IMAPI18N servers do i;unicode-casemap by default.
  . some non-IMAPI18N do i;unicode-casemap as their sole collation; others
    do a legacy collation.  There is no way for the client to tell.

This may be an important enough issue to have a separate capability 
specifically to say "even though I am not yet an IMAPI18N server, I do 
i;unicode-casemap collation."

I have not thoroughly reviewed IMAPI18N to determine the timeline to make 
UW imapd compliant with IMAPI18N.  But I'd be willing to add a capability 
now for the i;unicode-casemap collation, and (hint hint) now would be an 
excellent time to do so.  A couple of months from now will be too late.

In the long-term scheme of things, non-IMAPI18N servers will become a 
vanished species, just as IMAP2 servers are today.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

To: [email-masked]
Cc: [email-masked]
From: [email-masked]
Subject: I-D ACTION:draft-ietf-imapext-i18n-12.txt 
Date: Fri, 10 Aug 2007 13:15:01 -0400
Sender: [email-masked]

--NextPart

A New Internet-Draft is available from the on-line Internet-Drafts 
directories.
This draft is a work item of the Internet Message Access Protocol Extension Working Group of the IETF.

	Title		: Internet Message Access Protocol Internationalization
	Author(s)	: A. Gulbrandsen, C. Newman
	Filename	: draft-ietf-imapext-i18n-12.txt
	Pages		: 16
	Date		: 2007-8-10
	
Internet Message Access Protocol (IMAP) version 4rev1 has basic
    support for non-ASCII characters in mailbox names and search
    substrings.  It also supports non-ASCII message headers and content
    encoded as specified by Multipurpose Internet Mail Extensions
    (MIME).  This specification defines a collection of IMAP extensions
    which improve international support including comparator negotiation
    for search, sort and thread, language negotiation for international
    error text, and translations for namespace prefixes.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-imapext-i18n-12.txt

To remove yourself from the I-D Announcement list, send a message to 
[email-masked] with the word unsubscribe in the body of 
the message. 
You can also visit https://www1.ietf.org/mailman/listinfo/I-D-announce 
to change your subscription settings.

Internet-Drafts are also available by anonymous FTP. Login with the 
username "anonymous" and a password of your e-mail address. After 
logging in, type "cd internet-drafts" and then 
"get draft-ietf-imapext-i18n-12.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html 
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

Internet-Drafts can also be obtained by e-mail.

Send a message to:
	[email-masked].
In the body type:
	"FILE /internet-drafts/draft-ietf-imapext-i18n-12.txt".
	
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

--NextPart
Content-Type: Multipart/Alternative; Boundary="OtherAccess"

--OtherAccess
Content-Type: Message/External-body;
	access-type="mail-server";
	server="[email-masked]"

Content-Type: text/plain
Content-ID:	<[email-masked]>

ENCODING mime
FILE /internet-drafts/draft-ietf-imapext-i18n-12.txt

--OtherAccess
Content-Type: Message/External-body;
	name="draft-ietf-imapext-i18n-12.txt";
	site="ftp.ietf.org";
	access-type="anon-ftp";
	directory="internet-drafts"

Content-Type: text/plain
Content-ID:	<[email-masked]>

--OtherAccess--

--NextPart--

Date: Fri, 10 Aug 2007 13:52:27 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: i18n-12, WGLC
Sender: [email-masked]

Hi,

Pete, please issue a WGLC when you see -12. I posted it.

I'll be away for two weeks starting tomorrow morning. I will not read 
mail during the first week, and only occasionally during the second.

-12 says this:

     A server that advertises this extension MUST implement the
     i;ascii-casemap and i;octet comparators, as defined in [RFC4790].  A
     server intended to be deployed globally MUST implement the
     i;unicode-casemap comparator, as defined in [UCM].

     A server that advertises this extension SHOULD use i;ascii-casemap as
     the default comparator.  The selection of
     the default comparator MAY be adjustable by the server administrator,
     and MAY be sensitive to the current user.  Once the IMAP connection
     enters authenticated state, the default comparator MUST remain static
     for the remainder of that connection.

There are problems.

1. UW doesn't use that collation as default, and site administrators can 
change it too. It seems vaguely desirable that IMAP has a default and 
clients can know what it is, though...?

2. Dan can't easily implement i;ascii-casemap, and wants it out. Timo 
(less strongly?) wants both i;ascii-casemap and i;octet out.

3. In 4790, I (stupidly) already said that i;ascii-casemap is the 
default for IMAP. I didn't know about the UW change to 
i;unicode-casemap yet then.

In consequence, I believe that the two paragraphs have to be replaced 
with new text. But I can't make up my mind about what to write.

I don't have strong viewpoints myself, and cannot see any pleasant 
solution. A little bit of WG discussion and rough consensus would 
please me greatly. I'll be away at the start and around at the end of 
the WGLC period, and will type something that seems to reflect 
consensus.

Arnt

Date: Fri, 10 Aug 2007 12:11:16 +0200
From: Pawel Salek <[email-masked]>
Subject: Re: UID SEARCH responses
To: Mark Crispin <[email-masked]>
Cc: IMAP Extensions WG <[email-masked]>
Sender: [email-masked]

On 08/10/2007 07:47:40 AM, Mark Crispin wrote:
> 
> For what it's worth, UW imapd issues the
>  	* ESEARCH (TAG ".") UID ALL 38782:41424
> response instead of
>  	* ESEARCH (TAG ".") UID ALL  
> 38782,38789,40004,40301,40421,40814,41424
> because there are no other UIDs in that range.

This is the knowledge that the server has. Can client use the first type  
of response to eg. purge its cache? Such response is useful if it meant  
to send back to server but its interpretation of it on the client side  
provides less information than the second type of the response.

> Pawel thought this is wrong, and that a UID range of 38782:41424 must  
> meanall the possible values are in the map.
> 
> I suggested that he bring the matter up on this list.  I reviewed  
> theESEARCH RFC, and can find nothing to indicate that ESEARCH must  
> deliver aUID map (as SEARCH does) rather than a UID sequence set.

I agree it is important to settle this matter early on before  
incompatible client and server implementations spread.

As a client implementer, I think the single interval response  
(38782:41424) is less useful than the complete list of UIDs. The single  
interval can be achieved by other means (eg by SEARCH RETURN (MIN MAX)  
28:34, or in this very case a plain FETCH 28,34 UID) and only the  
complete list of UIDs provides new, orthogonal functionality as the  
method to fetch the UIDs of messages in given range.

> If Pawel is correct on the intent, then the ESEARCH RFC is broken since  
> itis ambiguous on that point; either live with the ambiguity, or  
> deprecateESEARCH and replace it with some other syntax.  The horse is  
> already outof the barn.
> 
> I contend that if a client wants the map, then "UID SEARCH ALL" is  
> thecorrect command.  The only advantage to ESEARCH is getting the tag,  
> whichI don't think is much of an advantage for that particular command.

There is a second - bandwidth - advantage connected with the use of  
ESEARCH the way I interpret it, as stated in the introduction:
"These result options allow clients to get SEARCH results in more  
convenient forms, while also saving bandwidth required to transport the  
results"

In the example I have given, the compression of the map is not present  
but it may be quite substantial for archive-type, append-only folders.

All in all, I will be happy with any interpretation as long as it is  
unique.

Pawel

Date: Fri, 10 Aug 2007 10:33:37 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: UID SEARCH responses
Cc: Pawel Salek <[email-masked]>
Sender: [email-masked]

Pawel Salek writes:
> The argument for the first form is that it represents a compressed 
> form  of the ordinary search response (OK, not very much compressed 
> this  time), and for the latter, that it can be used as the 
> sequence-set argument in a subsequent command, as demanded by the 
> RFC.

So can the latter... if UIDs 1 and 3 exist, and UID does 2, then the two 
following commands have the same result:

a uid fetch 1:3 bodystructure
b uid fetch 1,3 bodystructure

I can't think of any difference. Uid store, uid fetch, uid search, search, ...

Arnt

Date: Thu, 9 Aug 2007 22:47:40 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: IMAP Extensions WG <[email-masked]>
Subject: Re: UID SEARCH responses
Sender: [email-masked]

For what it's worth, UW imapd issues the
 	* ESEARCH (TAG ".") UID ALL 38782:41424
response instead of
 	* ESEARCH (TAG ".") UID ALL 38782,38789,40004,40301,40421,40814,41424
because there are no other UIDs in that range.

Pawel thought this is wrong, and that a UID range of 38782:41424 must mean 
all the possible values are in the map.

I suggested that he bring the matter up on this list.  I reviewed the 
ESEARCH RFC, and can find nothing to indicate that ESEARCH must deliver a 
UID map (as SEARCH does) rather than a UID sequence set.

If Pawel is correct on the intent, then the ESEARCH RFC is broken since it 
is ambiguous on that point; either live with the ambiguity, or deprecate 
ESEARCH and replace it with some other syntax.  The horse is already out 
of the barn.

I contend that if a client wants the map, then "UID SEARCH ALL" is the 
correct command.  The only advantage to ESEARCH is getting the tag, which 
I don't think is much of an advantage for that particular command.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Date: Fri, 10 Aug 2007 07:27:04 +0200
From: Pawel Salek <[email-masked]>
Subject: UID SEARCH responses
To: [email-masked]
Sender: [email-masked]

Hi,

I wonder which of the alternatives is a proper ESEARCH response to a  
following search:

. uid search 28:34
* SEARCH 38782 38789 40004 40301 40421 40814 41424
. OK UID SEARCH completed

If I now issue corresponding ESEARCH command:
. uid search return (all) 28:34

What should the IMAP server answer:
* ESEARCH (TAG ".") UID ALL 38782,38789,40004,40301,40421,40814,41424
or
* ESEARCH (TAG ".") UID ALL 38782:41424
. OK UID SEARCH completed
or maybe something else?

The argument for the first form is that it represents a compressed form  
of the ordinary search response (OK, not very much compressed this  
time), and for the latter, that it can be used as the sequence-set  
argument in a subsequent command, as demanded by the RFC.

Pawel Salek

Date: Thu, 2 Aug 2007 06:34:21 -0700 (PDT)
From: Dan Karp <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: Timo Sirainen <[email-masked]>, [email-masked]
Subject: Re: I-D ACTION:draft-ietf-imapext-i18n-11.txt
Sender: [email-masked]

> > As I've mentioned earlier, that depends on where you're starting
> > from.  Supporting i;ascii-casemap -- especially supporting it in
> > anything resembling an efficient manner -- would require a large
> > amount of work for me.
> 
> Don't you use either UCS-2 or UCS-4? Isn't that just mapping A-Z to
> a-z and comparing the resulting code points?

Essentially it's an upcased UCS-2.  To do an octet compare would require opening every message file containing a potential match.

Date: Thu, 2 Aug 2007 15:26:11 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: I-D ACTION:draft-ietf-imapext-i18n-11.txt
Cc: Dan Karp <[email-masked]>, Timo Sirainen <[email-masked]>
Sender: [email-masked]

Dan Karp writes:
>>  My current inclination is to keep i;ascii-casemap a MUST and thereby 
>>  avoid a discrepancy between the two documents. After all it's not 
>>  difficult to implement i;ascii-casemap.
>
> As I've mentioned earlier, that depends on where you're starting from. 
>  Supporting i;ascii-casemap -- especially supporting it in anything 
> resembling an efficient manner -- would require a large amount of 
> work for me.

Don't you use either UCS-2 or UCS-4? Isn't that just mapping A-Z to a-z 
and comparing the resulting code points?

Arnt

Date: Thu, 2 Aug 2007 05:01:25 -0700 (PDT)
From: Dan Karp <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: Timo Sirainen <[email-masked]>, [email-masked]
Subject: Re: I-D ACTION:draft-ietf-imapext-i18n-11.txt
Sender: [email-masked]

> My current inclination is to keep i;ascii-casemap a MUST and thereby 
> avoid a discrepancy between the two documents. After all it's not 
> difficult to implement i;ascii-casemap.

As I've mentioned earlier, that depends on where you're starting from.  Supporting i;ascii-casemap -- especially supporting it in anything resembling an efficient manner -- would require a large amount of work for me.

Date: Thu, 2 Aug 2007 12:59:07 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: I-D ACTION:draft-ietf-imapext-i18n-11.txt
Cc: Timo Sirainen <[email-masked]>
Sender: [email-masked]

Timo Sirainen writes:
> The only reason I see for i;ascii-casemap is if server developer is 
> lazy and doesn't want to bother implementing it. I don't see why a 
> client would want to use it if i;unicode-casemap is also available. I 
> would be happy if it was a "server MUST implement either one or both 
> of i;ascii-casemap and i;unicode-casemap".

Unfortunately RFC 4790 says this:

    In IMAP, the default collation is i;ascii-casemap, because its
    operations are understood to match IMAP's built-in operations.

I asked and tested before writing that (sometime between IETF LC and 
IESG review), but I didn't ask and test enough, so I didn't know that 
Mark's server changed to a different default collation some time ago.

My current inclination is to keep i;ascii-casemap a MUST and thereby 
avoid a discrepancy between the two documents. After all it's not 
difficult to implement i;ascii-casemap.

Arnt

Subject: Re: I-D ACTION:draft-ietf-imapext-i18n-11.txt
From: Timo Sirainen <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: [email-masked]
Date: Thu, 02 Aug 2007 13:42:10 +0300
Sender: [email-masked]

--=-IHqMYogKK8QzvPfaq7BO
Content-Type: text/plain; charset=ISO-8859-15
Content-Transfer-Encoding: quoted-printable

On Thu, 2007-08-02 at 12:25 +0200, Arnt Gulbrandsen wrote:
> Timo Sirainen writes:
> > Two things:
> >
> > 1. I thought i;unicode-casemap was supposed to be the only MUST impleme=
nt:
> > http://www1.ietf.org/mail-archive/web/lemonade/current/msg03559.html
> >
> > Is this still going to change?
>=20
> Do you mean that i;ascii-casemap and i;octet should not be MUSTs?

Yes.

> I=20
> don't care about i;octet, but leaving out i;ascii-casemap seems wrong=20
> somehow.

The only reason I see for i;ascii-casemap is if server developer is lazy
and doesn't want to bother implementing it. I don't see why a client
would want to use it if i;unicode-casemap is also available. I would be
happy if it was a "server MUST implement either one or both of
i;ascii-casemap and i;unicode-casemap".

> > What about matching invalid input with SEARCH? I think it should be=20
> > handled in implementation-specific way. I still receive ISO-8859-1=20
> > mails without charset header and I'd like to be able to SEARCH them.
..
> I have one concern which is not reflected in the text as it stands:  I=20
> think it would be bad and ugly if the success of a search depends on=20
> factors the user doesn't control. If you (as user) search for the word=20
> m=FCnchen, the success of your search should not depend on whether your=20
> IMAP client chooses to send 'charset utf-8 body m=FCnchen' or 'charset=20
> iso-8859-4 body m=FCnchen' (presuming the word is correctly encoded in=20
> both cases, with literals and blah).

I wouldn't mind about that, as long as it's not illegal for me to use a
configurable default charset in case it's missing. Dovecot isn't doing
that now, but probably in future when I get annoyed enough at this.

--=-IHqMYogKK8QzvPfaq7BO
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

--=-IHqMYogKK8QzvPfaq7BO--

Date: Thu, 2 Aug 2007 12:33:25 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: I-D ACTION:draft-ietf-imapext-i18n-11.txt
Cc: Mark Crispin <[email-masked]>
Sender: [email-masked]

Mark Crispin writes:
> The LANGUAGE extension is technically l10n and not i18n.  I won't go 
> as far as to suggest that this document be broken up into a separate 
> l10n and i18n document.  However, the document should not blur the 
> distinction and imply that l10n issues are i18n issues.  They aren't.

While I may agree, I'm not entirely sure, I would strongly prefer not to 
do that. Hardly anyone distinguishes between l10n and i18n anyway, and 
many people don't even know what l10n is. I'd rather be slightly fuzzy 
than use words the audience doesn't know.

Arnt

Date: Thu, 2 Aug 2007 12:27:18 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: I-D ACTION:draft-ietf-imapext-i18n-11.txt
Cc: Mark Crispin <[email-masked]>, Timo Sirainen <[email-masked]>
Sender: [email-masked]

Mark Crispin writes:
> As for codepoints that are not defined in the charset; this is 
> nebulous and fungible over time.  A codepoint which is undefined 
> today may be defined tommorrow by an update to the charset.  Thus, 
> nobody should depend upon an undefined codepoint remaining undefined; 
> nor should depend upon any particular error condition in that case.

Bad example. Sorry. ISO-2002-JP ESC $ B B ESC ( B or UTF-8 N A 0xCF V E 
are better. I've seen both in real mail, and I do not think IMAP 
clients should be permitted to send either with impunity. But it's a 
base IMAP issue and out of scope for this document, so I deleted the 
text.

Arnt

Date: Thu, 2 Aug 2007 12:25:50 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: I-D ACTION:draft-ietf-imapext-i18n-11.txt
Cc: Timo Sirainen <[email-masked]>
Sender: [email-masked]

Timo Sirainen writes:
> Two things:
>
> 1. I thought i;unicode-casemap was supposed to be the only MUST implement:
> http://www1.ietf.org/mail-archive/web/lemonade/current/msg03559.html
>
> Is this still going to change?

Do you mean that i;ascii-casemap and i;octet should not be MUSTs? I 
don't care about i;octet, but leaving out i;ascii-casemap seems wrong 
somehow.

> 2. "Strings encoded using unknown character encodings should never  
> match when using the SEARCH command, and should sort together with 
> invalid input"
>
> What about matching invalid input with SEARCH? I think it should be 
> handled in implementation-specific way. I still receive ISO-8859-1 
> mails without charset header and I'd like to be able to SEARCH them.

I started by explaining to you why you're wrong and ended by explaining 
to myself why that text is inappropriate ;) The problem is means to 
solve is out of scope for this document. I've deleted it now.

I have one concern which is not reflected in the text as it stands:  I 
think it would be bad and ugly if the success of a search depends on 
factors the user doesn't control. If you (as user) search for the word 
münchen, the success of your search should not depend on whether your 
IMAP client chooses to send 'charset utf-8 body münchen' or 'charset 
iso-8859-4 body münchen' (presuming the word is correctly encoded in 
both cases, with literals and blah).

Arnt