Subject: [lemonade] draft-gulbrandsen-imap-notify-01 use cases

From: Martin Konold <[email-masked]>
Date: Wed, 22 Nov 2006 18:36:20 +0100

Hi,

I try to understand the use cases and purpose of 
draft-gulbrandsen-imap-notify-01.

Is it correct that the primary goals are mainly the two following points?

1. being able to restrict the amount of data IDLE [RFC2177] would send 
downstream by issueing the NOTIFY command with appropriate parameters

2. theses restrictions can be used during IDLE or always

3. allow for reducing the load on the server by avoid unnecessary notification 
messages

To me this looks like it is mainly emphasizing on improving the downstream 
bandwidth usage while helping the server to avoid doing unnecessary work.

Did I miss further goals/use cases?

In addition I want to solve some further goals with IDLEPLUS. The general 
use-cases which IDLEPLUS tries to solve are:

1. improve the performance with high latency and asymmetric TCP/IP 
connections. A typical use case would be IMAP over an UMTS link with 
typically 384 k/Bit downstream but often only 64 k/Bit upstream.

2. avoid transfering upstream (C->S) the redundant data and therefore make use 
of server side configuration store (subscribe db and METADATA annotations if 
available)

3. allow to monitor a potentially high number of mailboxes while leveraging on 
the fact that generally most mailboxes are idle (*)

4. allow for efficient implementation on both server and client. (Actually I 
am currently working on an implementation for Cyrus Imapd)

I can easily imagine to merge these two efforts. Especially I am more than 
willing to add support for being able to make the type of notifications 
configurable. This can be either done like in NOTIFY using parameters or 
preferably using annotations and leveraging upon the METADATA extension. 

After reading the recent minutes 
http://tools.ietf.org/wg/lemonade/minutes?item=minutes67.html I got the 
impression that actually the scope of NOTIFY shall be extended to allow 
monitoring multiple mailboxes in parallel. I guess that this change is in the 
pending -02 version of draft-gulbrandsen-imap-notify.

On the other hand I am wondering if the use cases being addressed in IDLEPLUS 
can also be implemented in NOTIFY?

Opinions welcome!

(*) a typical use case is a CRM system with a subfolder for each customer. 
This can result easily into many thousand folders though most of the time 
nothing actually changes.

http://www.erfrakon.de/konold/draft-konold-imap-idleplus-04.txt

Yours,
-- martin

-- 
http://www.erfrakon.com/
Erlewein, Frank, Konold & Partner - Beratende Ingenieure und Physiker

_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 22 Nov 2006 18:42:41 +0000

>draft-gulbrandsen-imap-notify-01.
Hi Martin,

>3. allow for reducing the load on the server by avoid unnecessary notification 
This looks more or less correct.

>of server side configuration store (subscribe db and METADATA annotations if 
There are several pieces of information specified in parameters to 
NOTIFY command:
1). List of events the client is interested in - I don't think this is 
going to be huge.
2). List of mailboxes the client is interested in - the client can use 
mailbox patterns to specify multiple mailboxes. I don't think this is 
going to be a big list, unless you hit some pathological case
3). What kind of information needs to be returned for new messages - I 
doubt this is going to be huge.
4). List of filters (search criteria). I've written 
draft-melnikov-imapext-filters-00.txt that allows them to be stored on 
the server. This will co-exist peacefully with NOTIFY.

>the fact that generally most mailboxes are idle (*)
This is still possible with NOTIFY, see above.

>am currently working on an implementation for Cyrus Imapd)
[Making NOTIFY difficult to implement for servers and/or clients was 
certainly a no goal ;-)]

>I can easily imagine to merge these two efforts.
>
Your draft is not sufficient different ours, so this would make sense.

>Especially I am more than 
As I mentioned above, I am not convinced that the effort is worth the 
gain, in this case.

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 22 Nov 2006 19:00:53 +0000

Arnt Gulbrandsen wrote:

> Alexey Melnikov writes:
>> 1). List of events the client is interested in - I don't think this 
>> is going to be huge.
>> 2). List of mailboxes the client is interested in - the client can 
>> use mailbox patterns to specify multiple mailboxes. I don't think 
>> this is going to be a big list, unless you hit some pathological case
>> 3). What kind of information needs to be returned for new messages - 
>> I doubt this is going to be huge.
>> draft-melnikov-imapext-filters-00.txt that allows them to be stored 
>> on the server. This will co-exist peacefully with NOTIFY.
>
> Do you think anyone will send NOTIFY more than once during an IMAP 
> connection?

Unlikely.

> Assuming that it's sent once, shortly after connection establishment, 
> the only way to make it very large would seem to be sending a long 
> list of mailboxes. In that case, there's an o(size of mailbox names) 
> uplink cost and O(number of events*size of each event) downlink cost.
>
> We could store the mailbox list on the server,

Yes, but I think the existing list of mailbox patterns is good enough 
(and compact enough). I surely hope the client is not going to list all 
user mailboxes when it can just use "*" or similar.

We can create a new pattern for "all subscribed mailboxes", but because 
handling of subscribed mailboxes is already broken (inconsistent across 
clients), I doubt this is going to be very useful. But I can be 
convinced otherwise.

> of course, but in that case, a well-connected desktop client and a 
> GPRS-connected mobile phone will tend to monitor the same set of 
> mailboxes.

---

From: "Ben Last" <[email-masked]>
Date: Thu, 23 Nov 2006 08:29:47 -0000

Some overdue notes on the draft: sorry for the delay. 

In 5.1, FlagChange & AnnotationChange:
>Open issue: Which mailboxes does this cover? Those owned by the
>user, those accessible to the user, and if the latter, then which
>ACL right is needed?
How about "all mailboxes for which the client may issue a successful
SELECT"?

In 5.3, ExpungeMessage:
I don't see a huge client benefit in the UIDs of the neighbours being
sent via unsolicited FETCH after the EXPUNGE.  At best, it allows a
client to establish that it is now out of sync with the server if the
UID of the neighbour isn't that which is expected; that's really an edge
case.

>If the expunged message(s) is/are in another mailbox, the server
>sends an unsolicited STATUS (EXISTS) response for the relevant
>mailbox. If CONDSTORE is supported and enabled, HIGHESTMODSEQ should
>be included in the STATUS response.
Thus a STATUS (EXISTS) means "the number of messages in another mailbox
has changed".  Is it worth clarifying this for the reader?

Much applause and votes for the CreateMailbox, DeleteMailbox and
RenameMailbox notifications; I think it's important that the client can
easily distinguish a Rename from a Create or Delete.  Also, is it likely
that clients will be coded to expect unsolicited LIST responses?  Would
it not be more appropriate to create a new response?

I'd like an efficient way for the client to express interest in commonly
used subsets of events, rather than just all or a full list.  If it's
reasonable to allow wildcards for selection of keywords, then renaming
the events so that all mailbox event keywords start with "Mailbox" (ie,
"MailboxCreate, MailboxDelete, MailboxRename"), all message event
keyword start with "Message" (ie, "MessageNew, MessageExpunge,
MessageFlagChange"), etc, then one might be able to express a notify
command as in this example:
<tag> NOTIFY (* <searchkey> (Mailbox* Message*))
...which indicates that the client is not interested in quota events.

I acknowledge this may be an inappropriate use of wildcards: if so, how
about "MessageAll", "MailboxAll", etc, to imply all events of that
class?  The drawback with using "*" for all events is that if future
revisions introduce new event classes beyond Message, Mailbox and Quota,
a client will receive notifications of events that it may not be able to
handle.

Regards
Ben

Ben Last
R&D Manager
EMCC Software

---

From: "Ben Last" <[email-masked]>
Date: Thu, 23 Nov 2006 08:37:11 -0000

>From: Alexey Melnikov [mailto:[email-masked]] 
>Arnt Gulbrandsen wrote:

Unless, during a persistent connection, the user changes the set of
mailboxes that the client is to synchronize.  That would require
re-issuing a NOTIFY.
Ah: just occurred to me that when a client sees a new mailbox created, a
reasonable behaviour would be to start synchronizing that new mailbox.
Which would imply issuing a new NOTIFY to include it in the set of
mailboxes for which events are expected.
Along the same lines: if a mailbox is renamed such that its new name
does not match a wildcard expression used in an intial NOTIFY, does the
server continue to send notifications for that mailbox?  Ie, is the
wildcard matching done by the server at the time the NOTIFY is
processed, or when events are generated?

To be specific about that last point: consider I have the mailboxes
"Arnt", "Alexey" and "Andrew" and issue:
<tag> NOTIFY ("A*" <searchtag> *)

The server issues notifications for all three mailboxes.  I then rename
"Andrew" to "Eric".  Does the server continue to issue events for Eric?

b

Ben Last
R&D Manager
EMCC Software

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 23 Nov 2006 11:38:36 +0100

Martin Konold writes:
> impression that actually the scope of NOTIFY shall be extended to 
> allow monitoring multiple mailboxes in parallel. I guess that this 
> change is in the pending -02 version of 
> draft-gulbrandsen-imap-notify.

I think that's already in -01. The syntax uses mbox-or-pat from LISTEXT, 
so you can use a list of mailboxes or a mailbox wildcard specification.

Armt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 23 Nov 2006 11:41:19 +0100

Martin Konold writes:
> (*) a typical use case is a CRM system with a subfolder for each 
> customer. This can result easily into many thousand folders though 
> most of the time nothing actually changes.

I thought about this, and I'd say that NOTIFY solves this use case very 
badly and IDLEPLUS has a race condition. Assume that there's exactly 
1000 subfolders and fifty changes per day total, including two which 
happen at almost the same time, and that the RTT is one second, and 
that processing SELECT takes 0.250 seconds and FETCH 0.050. (The 
message interval should follow a Poisson distribution, right, so short 
intervals will occur every so often.)

The NOTIFY command handles that nicely, just one wildcard and hardly any 
responses, but the client also has to send 1000 STATUS commands 
immediately after starting NOTIFY, so for a two-hour IMAP connection 
that's 1001 commands and 2006 responses in order to notify the client 
of 4 events.

IDLEPLUS needs the same STATUS commands, so the total response count is 
effectively equal (I don't care abotu the DONE packet, it doesn't 
matter). The difference is that if a message arrives at 10:00:00.000 
and another in another mailbox at 10:00:02.000, the second one is lost. 
At 10:00:00.000 the server sends an EXISTS to notify the client, which 
the client receives at 10:00.00.500, and it immediately sends DONE and 
SELECT, which the server accept at 10:00:01.000. Sometime around 
10:00:01.250 the server is done processing select, and sometime around 
10:00:01.750 the client receives the responses for the SELECT, and 
sends a FETCH for the message that arrived at 10:00:00.000. At 
10:00:02:250 the server receives a FETCH, which completes at 
10:00:02.300, and the client receives the FETCH response very shortly 
after 10:00:02.800, and immediately sends another IDLEPLUS. The server 
goes into IDLEPLUS mode very shortly after 10:00:03.300, and announces 
any messages that arrive after that time, but the message that arrived 
at 10:00:02.000 is not announced.

If the RTT is smaller the problem is smaller. On ethernet it almost 
disappears. If the client is more primitive and doesn't pipeline, the 
problem is bigger. (IDLE doesn't have this race, it doesn't occur for 
the selected mailbox.)

I can think of ways to solve this both for NOTIFY and IDLEPLUS.

For NOTIFY I thought of a fairly pretty solution of sorts. It involves 
using the notification framework Ray spoke of and configuring it to 
e.g. "notify me of new messages in these 1000 mailboxes via IMAP/NOTIFY 
if I'm logged in, else send me an SMS". If that's done the 1000 STATUS 
commands aren't necessary, so NOTIFY works well. Just 5 responses for 4 
events. The user gets a notification via NOTIFY if logged in, else via 
SMS.

For IDLEPLUS that approach works less well - it would switch to SMS 
notifications for 2.3 seconds every time the client sends an IMAP 
command. Very unpredictable for the user. It would also be possible to 
give IDLEPLUS transaction/checkpoint ability, client identity and a 
notification spool, so the client could say "IDLEPLUS 1111111232" to 
identify itself, receive its spooled events and wait for more, and 
"NOOPPLUS 423142342" to tell the server that all events up to number 
423142342 have been received.

Arnt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 23 Nov 2006 11:50:51 +0100

Ben Last writes:
>> From: Alexey Melnikov [mailto:[email-masked]] Arnt 
>> Gulbrandsen wrote:
> mailboxes that the client is to synchronize.

IMO that case has to be handled correctly, but it's not a common case 
and does not warrant careful optimisation. My preference is to optimise 
for a single NOTIFY and make sure that if several are sent, the results 
is well-defined and useful.

> ...

I ran across that this morning too, in a different context: If I ask for 
create/rename events for a* and a new mailbox is created, named 
"Andrew", or if "Eric" is renamed back to "Andrew", what happens? 
Obviously some corner cases need much careful definition than the -01 
provides.

The two issues you raise interact, too. Fsck. I'll go brew some espresso.

Arnt

---

From: Dave Cridland <[email-masked]>
Date: Thu, 23 Nov 2006 11:47:52 +0000

On Thu Nov 23 10:41:19 2006, Arnt Gulbrandsen wrote:
> IDLEPLUS needs the same STATUS commands, so the total response 
> count is effectively equal (I don't care abotu the DONE packet, it 
> doesn't matter). The difference is that if a message arrives at 
> 10:00:00.000 and another in another mailbox at 10:00:02.000, the 
> second one is lost. At 10:00:00.000 the server sends an EXISTS to 
> notify the client, which the client receives at 10:00.00.500, and 
> it immediately sends DONE and SELECT, which the server accept at 
> 10:00:01.000. Sometime around 10:00:01.250 the server is done 
> processing select, and sometime around 10:00:01.750 the client 
> receives the responses for the SELECT, and sends a FETCH for the 
> message that arrived at 10:00:00.000. At 10:00:02:250 the server 
> receives a FETCH, which completes at 10:00:02.300, and the client 
> receives the FETCH response very shortly after 10:00:02.800, and 
> immediately sends another IDLEPLUS. The server goes into IDLEPLUS 
> mode very shortly after 10:00:03.300, and announces any messages 
> that arrive after that time, but the message that arrived at 
> 10:00:02.000 is not announced.

Ah, I see - if a message arrives in a different mailbox when IDLEPLUS 
is not active, then yes, you lose data.

This is why IDLEPLUS is designed for use in a different connection 
than that used for "real work".

The problem then is that you have to decide whether IDLEPLUS gets 
notifications for mailboxes the other session has selected. If you 
do, then you have essentially doubled the bandwidth cost of 
notifications, since they'll get transmitted once on the mailbox 
connection and once on the IDLEPLUS connection. Moreover, you can get 
notified about events which haven't "happened" yet for the other 
session, which is perhaps even more confusing. So the client has to 
ignore notifications on the IDLEPLUS connection for selected 
mailboxes in other sessions anyway.

But, if you don't send duplicate notifications, it's going to be very 
hard to decide whether you've received all the notifications for the 
mailbox as you select it. That's going to be painful. Moreover, you 
need to identify that the two sessions are from the same client, 
which currently we can't do.

This duplication problem doesn't occur for NOTIFY, even if it's used 
in the same way, because the client isn't relying on the responses to 
keep it in synch, it's relying on the messages to tell it it's out of 
synch. Losing one, if the client thinks it's out of synch, makes no 
difference. Similarly, duplicating one makes no difference. It's only 
in the case where the client is using NOTIFY-via-IDLE within the same 
session *and* it misses a notification *and* it previously thought it 
was in synch *and* it can service any requests from the mailbox cache 
that it'll make a difference.

> command. Very unpredictable for the user. It would also be possible 
> to give IDLEPLUS transaction/checkpoint ability, client identity 
> and a notification spool, so the client could say "IDLEPLUS 
> 1111111232" to identify itself, receive its spooled events and wait 
> for more, and "NOOPPLUS 423142342" to tell the server that all 
> events up to number 423142342 have been received.

... and thus become even more like CLEARIDLE, as I recall.

Losing synch isn't a major problem if regaining it is cheap, and it 
is per-mailbox. Right now, I'm somewhat more concerned with how to 
deal with regaining mailbox-level synch across large numbers of 
mailboxes at connection time.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 23 Nov 2006 13:22:48 +0100

Dave Cridland writes:
> Losing synch isn't a major problem if regaining it is cheap, and it is 
> per-mailbox. Right now, I'm somewhat more concerned with how to deal 
> with regaining mailbox-level synch across large numbers of mailboxes 
> at connection time.

You're assuming much local client storage and little bandwidth, as might 
the case for a modern laptop with a mobile phone card?

If condstore's modseq is global, we could give notify a modseq argument 
and it would send immediate STATUS responses for each maillbox whose 
highestmodseq is larger... but that won't work if condstore "next 
modseq" variable is per-mailbox, which I believe 4551 allows.

Arnt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 23 Nov 2006 15:00:11 +0100

> Some overdue notes on the draft: sorry for the delay.

I wish you'd sent that yesterday or tomorrow ;)

I will pretend that you sent it tomorrow, so that I can get -02 
published, and deal with your issues in -03. OK?

Arnt

---

From: Alexey Melnikov <[email-masked]>
Date: Thu, 23 Nov 2006 15:25:44 +0000

Arnt Gulbrandsen wrote:

> and does not warrant careful optimisation.

Exactly my thought.

> My preference is to optimise for a single NOTIFY and make sure that if 
> several are sent, the results is well-defined and useful.

Yes.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 23 Nov 2006 11:58:11 +0100

Arnt Gulbrandsen writes:
> IDLEPLUS needs the same STATUS commands, so the total response count 
> is effectively equal (I don't care abotu the DONE packet, it doesn't 
> and another in another mailbox at 10:00:02.000, the second one is 
> lost. At 10:00:00.000 the server sends an EXISTS to notify the 
> client, which the client receives at 10:00.00.500, and it immediately 
> sends DONE and SELECT, which the server accept at 10:00:01.000. 
> Sometime around 10:00:01.250 the server is done processing select, 
> and sometime around 10:00:01.750 the client receives the responses 
> for the SELECT, and sends a FETCH for the message that arrived at 
> 10:00:00.000. At 10:00:02:250 the server receives a FETCH, which 
> completes at 10:00:02.300, and the client receives the FETCH response 
> very shortly after 10:00:02.800, and immediately sends another 
> IDLEPLUS. The server goes into IDLEPLUS mode very shortly after 
> 10:00:03.300, and announces any messages that arrive after that time, 
> but the message that arrived at 10:00:02.000 is not announced.

For "very shortly after", please read "at", otherwise the transmission 
time of the FETCH response is partly accounted for twice. (0.050 
seconds is reasonable only if the transmission time is included.)

Arnt

---

From: Martin Konold <[email-masked]>
Date: Mon, 27 Nov 2006 14:44:38 +0100

Am Donnerstag, 23. November 2006 12:47 schrieb Dave Cridland:

Hi

> connection and once on the IDLEPLUS connection.

IDLEPLUS emphasizes optimization for upload not download stream. The upload 
stream is the bottleneck in most modern high latency networks like ADSL and 
UMTS.

> mailboxes in other sessions anyway.

The client has to ignore the unsolicited notification on the non IDLEPLUS 
selection as these messages are unreliable anyway! This allows for simpler 
client implementation.

The extra bandwidth can imho be neglected. Due to the fact that they are 
unreliable anyway the server might optimize them away anyway.

Yours,
-- martin