Subject: [lemonade] CONTEXT

From: Timo Sirainen <[email-masked]>
Date: Tue, 03 Jun 2008 17:35:09 +0300

1) If ADDTO refers to newly added messages, shouldn't it be required to
be sent after EXISTS, just like REMOVETO must be sent before EXPUNGE? So
an example session would look like:

B01 SEARCH RETURN (UPDATE) ALL
* ESEARCH (TAG "B01") 1
B01 OK
..
* 1 EXPUNGE
* ESEARCH (TAG "B01") UID REMOVEFROM (0 1)
* 1 EXISTS
* ESEARCH (TAG "B01") UID ADDTO (0 1)

2) Is the context position really all that useful in ESEARCH replies?
With sequence numbers they're completely useless and with UIDs I'd think
it's better to optimize for bandwidth usage rather than for the tiny
help in client's CPU usage.

3) ESORT description is somewhat ambiguous.

> Servers advertising the capability "ESORT" support the return options
>    specified in [ESEARCH]

Support where? Apparently in SORT command, but could be explicit.

> These update sets are carried in ADDTO and REMOVEFROM data items in
>    ESEARCH/ESORT responses.

What ESORT responses? Wasn't SORT also replying with ESEARCH replies?
Would ESORT replies be better? If 2) gets changed, it would make it
clearer that ADDTO/REMOVEFROM has the context position.

Also the context position description could use several examples to make
sure the implementors get it right.
_______________________________________________
lemonade mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/lemonade
Supplemental Web Site:
http://www.standardstrack.com/ietf/lemonade

---

From: Timo Sirainen <[email-masked]>
Date: Tue, 03 Jun 2008 19:57:48 +0300

> help in client's CPU usage.

How about making the context positions completely optional, even for
SORT? I was just thinking that my virtual mailbox implementation should
probably do the initial build by sorting matching messages by
INTERNALDATE, but all newly seen messages would be added to end of the
mailbox. So I would want to use SORT RETURN (UPDATE), but I wouldn't
want to see context positions.

So something like:

SORT RETURN (UPDATE POSITIONS) .. 

But since they're pointless for SEARCH sequences (but not UID SEARCH):

SEARCH RETURN (UPDATE POSITIONS) ..
BAD

If POSITIONS is specified, the server MUST return them correctly.
Special casing position 0 could be removed then.
_______________________________________________
lemonade mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/lemonade
Supplemental Web Site:
http://www.standardstrack.com/ietf/lemonade

---

From: Dave Cridland <[email-masked]>
Date: Tue, 03 Jun 2008 18:31:02 +0100

> * ESEARCH (TAG "B01") UID ADDTO (0 1)
REMOVEFROM only needs to be sent prior to EXPUNGE if it refers to   
MSNs, since otherwise it is impossible to reference the message to be  
  removed from the context - it's not a matter of event ordering,  
both  the REMOVEFROM and EXPUNGE occur simultaneously by definition -  
it's  a matter of when they can be reported.

> help in client's CPU usage.
No, they have low utility in both SEARCH and UID SEARCH. By "low   
utility", I mean marginally more useful than marginal utility - I   
wouldn't get too exicited by them.

The bulk of the utility is in figuring out where, in the context, the  
  results you have are, given the change. For results you had before   
the change, this is always possible for a SEARCH-derived context,   
whereas for the results you get in an ADDTO, the context position can  
  tell you. (Or in some cases you can logically determine where the   
results lie).

It's never needed by clients for REMOVEFROM, but I personally suspect  
  that given availability of ESORT, client implementors would  
generally  be using it.

> Support where? Apparently in SORT command, but could be explicit.
Agreed, I can change that in AUTH48.

> Would ESORT replies be better?

Ooops. Thanks for spotting that. Yes, the extended SORT replies with   
ESEARCH responses, since the two turned out to be semantically   
equivalent, and distinguishable. (Much like ACAP has only SEARCH,   
which can optionally SORT).

In fact, in IMAP, we declare that even a SEARCH is actually just a   
SORT-by-UID.

>  If 2) gets changed, it would make it

Possibly, but this might be going beyond what I can safely do in   
AUTH48 - however, as it's not yet gone through the IESG telechat,   
I'll ask Chris. I'd note that I'm unlikely to introduce new examples   
unless they're more or less written for me and posted to the list for  
  people to check.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade
_______________________________________________
lemonade mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/lemonade
Supplemental Web Site:
http://www.standardstrack.com/ietf/lemonade

---

From: Dave Cridland <[email-masked]>
Date: Tue, 03 Jun 2008 18:31:30 +0100

> want to see context positions.
You're talking about a client, here, or the server? If you're   
describing client behaviour, the problem is that you're going to have  
  to obtain a full result set initially, otherwise you'll have no  
idea  "where" the results are that you need to remove.

> But since they're pointless for SEARCH sequences (but not UID   
Ah, no, they're just less useful, because of what the natural sort   
order is. SEARCH is not unordered with CONTEXT.

The reason it's always present, but optional for servers in the case   
of SEARCH, is that it's just a "nice to have" for clients requesting   
SEARCH, whereas it can be quite painful to implement for servers, so   
it's nice to leave it optional.

Dave.

---

From: Timo Sirainen <[email-masked]>
Date: Wed, 04 Jun 2008 19:30:11 +0300

> it's  a matter of when they can be reported.

Yes, but I think it's a bit dirty to report sequences that haven't been
announced yet with EXISTS. It's the same as if server replies:

* 1 FETCH (FLAGS (\Seen))
* 1 EXISTS

Sure the client could understand that the EXISTS probably arrives soon,
but it complicates the code and error handling. What if the server is
buggy and the EXISTS never arrives? It's more difficult to catch bugs if
you have to look at future untagged replies to see if something is a bug
or not.

> wouldn't get too exicited by them.

Right, I somehow thought the position would be the same as the sequence,
but obviously they're (usually) different.

>   tell you. (Or in some cases you can logically determine where the   

Since the messages are sorted by UID/MSN in SEARCH result, I'd think you
could always quite easily determine where the results are?

> unless they're more or less written for me and posted to the list for  

I'll probably implement it soon and could then give examples (and see
that they give the same results in your implementation).
_______________________________________________
lemonade mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/lemonade
Supplemental Web Site:
http://www.standardstrack.com/ietf/lemonade

---

From: Timo Sirainen <[email-masked]>
Date: Wed, 04 Jun 2008 19:46:12 +0300

On Tue, 2008-06-03 at 18:31 +0100, Dave Cridland wrote:
> You're talking about a client, here, or the server? 

I'll implement them internally to server, but since there's a lot of
common functionality with SEARCH=CONTEXT, I thought I'd implement both
using the same backend functions. And that means IMAP clients could
implement virtual mailboxes the same way.

> If you're   

So you're thinking that it would be more useful if clients used a
PARTIAL initial search? Yes, I guess that would be better.

> it's nice to leave it optional.

I guess I've three problems with it (with SEARCH, not with SORT):

1) It wastes bandwidth and server resources a bit for clients that
don't care about it. Should I even bother implementing it? If I did,
does it hurt clients more (by wasted bandwidth) than not implementing
it? I'm not sure.

2) Complicates client code. They have to implement two ways to do the
same thing. Using the position could give a tiny bit of better
performance, but if I implemented such code I'd still add a check to
verify that the server's position actually was correct (there are so
many bugs in server implementations already..). So would a client
implementor even bother to implement this or just use the same code for
all servers?

3) It makes it more difficult to write generic test cases, since there
are two just as acceptable ways to reply.

So I guess my preferred way of making it work would be to never send
context position with SEARCH (more important), and make it
client-configurable for SORT (less important).
_______________________________________________
lemonade mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/lemonade
Supplemental Web Site:
http://www.standardstrack.com/ietf/lemonade

---

From: Timo Sirainen <[email-masked]>
Date: Thu, 05 Jun 2008 01:56:28 +0300

Is tag matching in CANCELUPDATE case-sensitive? Probably would be better
if it was, but since IMAP is usually case-insensitive this should be
mentioned in the spec.

---

From: Dave Cridland <[email-masked]>
Date: Thu, 05 Jun 2008 09:54:19 +0100

On Wed Jun  4 17:46:12 2008, Timo Sirainen wrote:
> > > > it's better to optimize for bandwidth usage rather than for  
> the
> > > tiny
> > > INTERNALDATE, but all newly seen messages would be added to end  
> of
> > > the
IMAP client can already implement virtual mailboxes using CONTEXT  
with either of SEARCH or SORT - it's just that messages wouldn't  
always appear at the end - this is okay for a SORTed view, although a  
little surprising for one based on a SEARCH - luckily, it's also very  
unlikely.

> PARTIAL initial search? Yes, I guess that would be better.
Or just a COUNT, yes. You can only entirely drop the positions for a  
SEARCH, and only when the number of results is small enough.

A client could make a guess at that - for example, if the mailbox  
size is small, then there cannot be sufficient matching messages, but  
I've not done that in my implementation - I do happen to do PARTIALs  
that extend across the entire range as a byproduct, but that's it.

> does it hurt clients more (by wasted bandwidth) than not  
I don't think bandwidth usage is significant here - we're talking in  
terms of perhaps four octets per message. You can, incidentally,  
always set position to zero for REMOVEFROM, and then there's no  
bandwidth hit, nor any loss in functionality to a client.

> implementor even bother to implement this or just use the same code  
Well, the idea is that a client can use essentially the same code for  
processing SEARCH and SORT - the difference is that the client has to  
cope with missing positions for SEARCH, but there's no difference to  
the parser code, nor the processing if the position is supplied -  
hence fewer code paths. For the cases where the position isn't  
supplied, the REMOVEFROM will end up being okay - delete and/or move  
- the ADDTO is rather more painful, and involves throwing some  
results away sometimes.

So in an ideal world, I'd mandate position be present in all cases,  
and remove another special-case, but I got a fair degree of push-back  
from server implementors on that front.

> are two just as acceptable ways to reply.
Ah, but the test case can be equally generic either way, it's just  
that this way, if position isn't set, you first have to determine  
what it ought to be.

> client-configurable for SORT (less important).

No, can be more efficient to send the position for ADDTO on SEARCHes.

Dave.

---

From: Timo Sirainen <[email-masked]>
Date: Fri, 6 Jun 2008 05:52:27 +0300

Looks like NOTIFY spec agrees with me:

> If a search context is in effect as specified in [CONTEXT], an  
> ESEARCH ADDTO will also be generated, if appropriate. In this case,  
> the EXISTS response MUST precede the ESEARCH response.

Since it's already specified there, it would be clearer to specify it  
in CONTEXT as well.
_______________________________________________
lemonade mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/lemonade
Supplemental Web Site:
http://www.standardstrack.com/ietf/lemonade

---

From: Timo Sirainen <[email-masked]>
Date: Sat, 07 Jun 2008 00:12:24 +0300

> modifier-partial    = "PARTIAL" SP seq-range
>        ;; <seq-range> from [IMAP]

'*' is allowed in [IMAP] seq-range. I'm guessing it shouldn't be used
though? Or if it is allowed, is it returned as '*' or as
highest-sequence in ret-data-partial?

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 09 Jun 2008 10:31:10 +0100

>highest-sequence in ret-data-partial?
I don't think "*" should be allowed, Isode's implementation doesn't 
accept it.