Subject: [lemonade] New draft: SEARCH return options

From: Alexey Melnikov <[email-masked]>
Date: Mon, 14 Feb 2005 14:56:25 +0000

[email-masked] wrote:

>A New Internet-Draft is available from the on-line Internet-Drafts directories.
>
>
>	Title		: IMAP4 extension to SEARCH command for controlling what 
>			  kind of information is returned
>	Author(s)	: A. Melnikov, D. Cridland
>	Filename	: draft-melnikov-imap-search-ret-00.txt
>	Pages		: 0
>	Date		: 2005-2-11
>	
>This document extends SEARCH and UID SEARCH commands with result
>   specifier, which can control what kind of information is returned.
>   Several result specifiers are defined: minimal value, maximal
>   value, all found messages and number of found messages. A new
>   response ESEARCH is also specified.
>
>A URL for this Internet-Draft is:
>http://www.ietf.org/internet-drafts/draft-melnikov-imap-search-ret-00.txt
>
Based on the recent discussion of the 
draft-melnikov-imap-expunged-00.txt, Dave and I wrote a draft that adds 
return options to the SEARCH command. Currently defined return options 
are: COUNT (number of found messages), MIN (the lowest message 
number/UID found), MAX (the highest message number/UID found) and ALL 
(return all found messages as a message/UID set).

Dave has implemented the extension in his Polymer email client.

I am also proud to announce the support for this extension in Isode IMAP 
server.

Alexey

P.S. If you are interested to test this extension against Isode server, please contact me directly.

 __________________________________________
IETF standard related pages:
http://www.melnikov.ca/mel/devel/Links.html
__________________________________________

_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade

---

From: Mark Crispin <[email-masked]>
Date: Mon, 14 Feb 2005 10:12:00 -0800 (PST)

I've reviewed this document and while I have document nits the protocol 
proposed looks OK to me.

Is there much feeling as to how useful it would be?  I'm not talking about 
Dave and Alexey since I'm sure they are already convinced.  It's more of a 
question of whether the benefit is worth the added complexity (don't 
forget, it does represent an additional task for server implementors!).

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: Cyrus Daboo <[email-masked]>
Date: Mon, 14 Feb 2005 15:01:30 -0500

Hi Alexey,

--On February 14, 2005 2:56:25 PM +0000 Alexey Melnikov 
<[email-masked]> wrote:

> are: COUNT (number of found messages), MIN (the lowest message number/UID
> found), MAX (the highest message number/UID found) and ALL (return all
> found messages as a message/UID set).
>

One thing you might want to consider adding to this is a 'ID' return option 
that takes an atom as an argument. The server would be required to return 
that ID and atom in the ESEARCH response for the corresponding command. 
This will solve the problem of multiple pipelined search commands and 
figuring out which search response corresponds to which command.

e.g.

C: a SEARCH RETURN (ID 1 ALL) UNSEEN
C: b SEARCH RETURN (ID 2 ALL) DELETED
S: * ESEARCH ID 1 ALL 1:100
S: * ESEARCH ID 1 ALL 200:1000
S: * ESEARCH ID 2 ALL 150
S: a OK
S: b OK

The IDs make it clear which response corresponds to each search.

-- 
Cyrus Daboo

---

From: Cyrus Daboo <[email-masked]>
Date: Mon, 14 Feb 2005 15:09:28 -0500

Hi Alexey,
I think you need to have some mention of how this interacts with 
draft-melnikov-imap-search-res-01.txt. Or that document should mention how 
it interacts with draft-melnikov-imap-search-ret-00.txt. One issue is:

a SEARCH RETURN (COUNT) UNSEEN

Strictly speaking the above command does not generate a message sequence 
set, so is the last search result $ item set or not?

There is also an interesting possibility of tying the two together via the 
ID return option I proposed. For example:

C: a SEARCH RETURN (ID 1 ALL) UNSEEN
C: a SEARCH RETURN (ID 2 ALL) DELETED
...

C: FETCH $[1] (UID INTERNALDATE FLAGS RFC822.HEADER)

C: FETCH $[2] (UID INTERNALDATE FLAGS RFC822.HEADER)

The fetch commands act on the result of the different searches, referenced 
by the ID option value. This means that a client could use multiple search 
result sets simultaneously, as opposed to in series, as they would have to 
right now.

This could prove very powerful for clients that want to present multiple 
simultaneous 'views' on the same mailbox.

-- 
Cyrus Daboo

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 14 Feb 2005 21:19:16 +0100

> The IDs make it clear which response corresponds to each search.

IMAP already has a mechanism to handle that: tagged responses.

Arnt

---

From: Cyrus Daboo <[email-masked]>
Date: Mon, 14 Feb 2005 15:30:57 -0500

Hi Arnt,

> IMAP already has a mechanism to handle that: tagged responses.

Search results are always untagged.

-- 
Cyrus Daboo

---

From: Mark Crispin <[email-masked]>
Date: Mon, 14 Feb 2005 12:43:34 -0800

> Search results are always untagged.

Yes, but instead of an ESEARCH untagged response this could be a status 
response in the tagged OK.  I think that this is the way to go.

-- Mark --

---

From: Cyrus Daboo <[email-masked]>
Date: Mon, 14 Feb 2005 15:45:24 -0500

Hi Mark,

> response in the tagged OK.  I think that this is the way to go.

Can you explain with some examples...

-- 
Cyrus Daboo

---

From: Mark Crispin <[email-masked]>
Date: Mon, 14 Feb 2005 12:51:02 -0800

> Can you explain with some examples...

tag SEARCH extended-poop
tag OK [SEARCH extended-poop-response] extended SEARCH OK

as opposed to

tag SEARCH extended-poop
* ESEARCH extended-poop-response
tag OK SEARCH completed

Note that this precludes any sort of "watching" SEARCH extension, in which 
you have a standing SEARCH and get unsolicited responses.  It has been 
nearly 20 years and we have done no such thing, so maybe we don't care.

-- Mark --

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 14 Feb 2005 21:58:42 +0100

> Search results are always untagged.

Why?

If you want ESEARCH to be bound to the command, you can either use the 
IMAP tag mechanism or reinvent it.

Arnt

---

From: Cyrus Daboo <[email-masked]>
Date: Mon, 14 Feb 2005 16:04:10 -0500

Hi Mark,

--On February 14, 2005 12:51:02 PM -0800 Mark Crispin 
<[email-masked]> wrote:

> Note that this precludes any sort of "watching" SEARCH extension, in
> which you have a standing SEARCH and get unsolicited responses.  It has
> been nearly 20 years and we have done no such thing, so maybe we don't
> care.
>

I guess I dislike the idea of having actual results come back in the 
extended response code.

Right now I don't see why we need the new ESEARCH response at all, as 
opposed to just redefining the SEARCH response (which is fine because the 
client knows what it is going to get having used the extended SEARCH 
command). Perhaps Alexey can explain why we have to have a new response?

-- 
Cyrus Daboo

---

From: Cyrus Daboo <[email-masked]>
Date: Mon, 14 Feb 2005 16:07:05 -0500

Hi Arnt,

> IMAP tag mechanism or reinvent it.

OK, so are you suggesting that the tag for the SEARCH command is used as 
the 'ID' I was proposing, e.g.:

a1234 SEARCH RETURN (TAG ALL) UNSEEN
* ESEARCH TAG a1234 ALL 1:100
a1234 OK

-- 
Cyrus Daboo

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 14 Feb 2005 22:22:58 +0100

Cyrus Daboo writes:
> OK, so are you suggesting that the tag for the SEARCH command is used 
> as the 'ID' I was proposing, e.g.:
>

I'm suggesting that IF you want to tag ESEARCH responses (whether you 
call them "tagged searches" or "searches with ID" is no matter to me), 
then the most honest response format is

a1234 ESEARCH ALL 1:100

There are alternatives, such as sneaking the tag in the back door:

Or such as using an Elvis lookalike instead of the real thing:

* ESEARCH ID a1234 ALL 1:100

Whether ESEARCH is tagged or not is no matter to me. I'm happy either way.

But if it is tagged, I'd like to see IMAP stay with just one tag 
mechanism (unless there are pressing reasons for having multiple 
similar mechanisms, of course).

Arnt

---

From: Cyrus Daboo <[email-masked]>
Date: Mon, 14 Feb 2005 16:28:32 -0500

Hi Arnt,

> then the most honest response format is

As far as I recall the IMAP base-spec only allows one tagged response to 
any command, so the above would not work without redefining IMAP's entire 
response behaviour...

> mechanism (unless there are pressing reasons for having multiple similar
> mechanisms, of course).

Ok - command tags would be sufficient.

-- 
Cyrus Daboo

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 14 Feb 2005 22:33:49 +0100

Cyrus Daboo writes:
> As far as I recall the IMAP base-spec only allows one tagged response 
> to any command, so the above would not work without redefining IMAP's 
> entire response behaviour...

That's an excellent argument, but where in the RFC does it say so?

Arnt

---

From: Cyrus Daboo <[email-masked]>
Date: Mon, 14 Feb 2005 16:37:18 -0500

Hi Arnt,

> That's an excellent argument, but where in the RFC does it say so?

The formal syntax for the 'response' element. If you follow that through 
you will see that you can only have one 'response-tagged' item in 
'response'.

-- 
Cyrus Daboo

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 14 Feb 2005 23:49:14 +0100

Cyrus Daboo writes:
> The formal syntax for the 'response' element. If you follow that 
> through you will see that you can only have one 'response-tagged' 
> item in 'response'.

Right. And I found another bit near the bottom of page 62.

I have no opinion, then. All the alternatives are fine with me.

Arnt

---

From: Pete Maclean <[email-masked]>
Date: Wed, 16 Feb 2005 11:10:53 -0500

I am wondering why you are not tempted to generalize the options a bit 
more.  I am thinking along the lines of:

	DIRECTION UP or DOWN (meaning search up/down with respect to message 
numbers/UIDs)

	MATCHES n (meaning return a maximum of n matches)

This would allow a client that wanted to find at most a fixed number of 
matches to fill a window, form or Web page to do so more efficiently.  For 
example,

	<tag> SEARCH RETURN (DIRECTION DOWN MATCHES 20) <criteria>

These could even be combined by having MATCHES +n and MATCHES -n.

Then MIN would be replaced by (DIRECTION UP MATCHES 1) and so on.

Pete Maclean

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 16 Feb 2005 17:04:59 +0000

Pete Maclean wrote:

>     DIRECTION UP or DOWN (meaning search up/down with respect to 
> message numbers/UIDs)

Isn't this part of the SORT extension? IMHO, the search order is an 
internal server business. Some servers can optimize if the client 
specified something like "RETURN (MIN)".

>     MATCHES n (meaning return a maximum of n matches)

Yes, we've just discussed this with Dave :-).

> This would allow a client that wanted to find at most a fixed number 
> of matches to fill a window, form or Web page to do so more 
> efficiently.  For example,
>

---

From: Pete Maclean <[email-masked]>
Date: Wed, 16 Feb 2005 12:27:08 -0500

At 12:04 PM 2/16/2005, Alexey Melnikov wrote:
>Isn't this part of the SORT extension?

Ah well, it may be so.  I must acknowledge that I have not acquainted 
myself with that extension.  But perhaps if a server chooses to implement 
the ESEARCH extension and not the SORT extension then it is relevant.

>  IMHO, the search order is an internal server business. Some servers can 
> optimize if the client specified something like "RETURN (MIN)".

I totally agree that the search order should be an internal server business 
by default.  However, for the type of situation I described, it does seem 
valuable for the client to be able override that.  Incidentally, I do not 
mean that to imply that the results should necessarily be returned in the 
specified order.  Furthermore, by "order", I guess I mean "effective order" 
such that the results returned would be the first N matches in the 
specified ordering.  Whether or not the server internally actually searches 
messages in that order is irrelevant.

>Yes, we've just discussed this with Dave :-).

Good show!

Pete