Subject: [Lemonade]About draft-gulbrandsen-imap-notify-01

From: Lingyan Wu <[email-masked]>
Date: Wed, 15 Nov 2006 17:32:58 +0800

Hi Curtis King and All,

In 5.1.(FlagChange and AnnotationChange) in this draft there is such a sentence: If so, the server sends a STATUS (HIGHESTMODSEQ) response.

What about to add some content(in []) to the sentence as follows?
If so, the server sends a STATUS (HIGHESTMODSEQ) response [with mailbox, UID and flags].

Reasons: 
The client could utilize the mailbox, UID and flags in STATUS response to determine which IMAP mailbox the event occurred in, for which message, and what the event is, so that the client could synchronize the flags of the message in client accordingly. Maybe it'll be more convenient to synchronize flags of emails if the STATUS response are extended to contain such informations.

Best Regards,
Lingyan
_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 15 Nov 2006 11:46:31 +0100

Lingyan Wu writes:
> Hi Curtis King and All,
>
> In 5.1.(FlagChange and AnnotationChange) in this draft there is such a 
> sentence: If so, the server sends a STATUS (HIGHESTMODSEQ) response.
>
> What about to add some content(in []) to the sentence as follows? If 
> so, the server sends a STATUS (HIGHESTMODSEQ) response [with mailbox, 
> UID and flags].

Not using STATUS. STATUS returns information about a mailbox, not about 
a particular message. This change would very big change to STATUS.

It would be possible to send an URLFETCH response containing the 
information, though. But that would add quite a bit to the bandwidth 
requirement, and seems unjustified. How big is the chance that the 
client and will can use this information? Seems small to me, 
considering that the client doesn't have the mailbox seelcted and that 
many/most LEMONADE devices will have a smallish cache (<10MB).

(In -02 I was going to add some text about how the server can limit the 
bandwidth requirements, btw.)

Arnt

---

From: Lingyan Wu <[email-masked]>
Date: Thu, 16 Nov 2006 20:15:38 +0800

----- Original Message ----- 
From: "Arnt Gulbrandsen" <[email-masked]>
To: <[email-masked]>
Cc: "Curtis King" <[email-masked]>; "Lingyan Wu" 
<[email-masked]>
Sent: Wednesday, November 15, 2006 6:46 PM
Subject: Re: [Lemonade]About draft-gulbrandsen-imap-notify-01

>> What about to add some content(in []) to the sentence as follows? If so, 
>> the server sends a STATUS (HIGHESTMODSEQ) response [with mailbox, UID and 
>> flags].
>
> Not using STATUS. STATUS returns information about a mailbox, not about a 
> particular message. This change would very big change to STATUS.

[Lingyan]----------------------------------------------------------------
Only adding UID and FLAGS (or MSGEVENT,e.g. ReadMessages) to STATUS 
response.

> information, though.

[Lingyan]-----------------------------------------------------------------
Maybe URLFETCH response couldn't contain FLAGS (or MSGEVENT), because the 
URL (the fourth form in RFC2192 which refers to a specific message or 
message part.) refers to the object returned by a "UID FETCH <uid> 
BODY.PEEK[<section>]" command while the UID FETCH Command doesn't return 
flags of emails.

I think extended STATUS (only adding UID and FLAGS) is more effective for 
unselected mailbox. It's just like an unsolicited FETCH response containing 
UID and FLAGS for the selected mailbox.

>But that would add quite a bit to the bandwidth requirement, and seems 
>unjustified. How big is the chance that the client and will can use this 
>information?

[Lingyan]-----------------------------------------------------------------
The client could use the FLAGS in the response to update the flags of the 
corresponding email in client after receiving the response. Through 
unsolicited FETCH response and STATUS response, all the mailboxes (selected 
and unselected mailboxes) could be synchronized expediently.
I think such a response would not add quite a bit to the bandwidth 
requirement.

> Seems small to me, considering that the client doesn't have the mailbox 
> seelcted and that many/most LEMONADE devices will have a smallish cache 
> (<10MB).
>

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 16 Nov 2006 14:22:30 +0100

Lingyan Wu writes:
> Only adding UID and FLAGS (or MSGEVENT,e.g. ReadMessages) to STATUS response.

I meant that it would be big conceptually, because it would extend 
STATUS to be about messages.

> Maybe URLFETCH response couldn't contain FLAGS (or MSGEVENT), because 
> the URL (the fourth form in RFC2192 which refers to a specific 
> message or message part.) refers to the object returned by a "UID 
> FETCH <uid> BODY.PEEK[<section>]" command while the UID FETCH Command 
> doesn't return flags of emails.

You're absolutely correct.

> I think extended STATUS (only adding UID and FLAGS) is more effective 
> for unselected mailbox. It's just like an unsolicited FETCH response 
> containing UID and FLAGS for the selected mailbox.

Except that it's not for a selected mailbox.

> The client could use the FLAGS in the response to update the flags of 
> the corresponding email in client after receiving the response. 
> Through unsolicited FETCH response and STATUS response, all the 
> mailboxes (selected and unselected mailboxes) could be synchronized 
> expediently.

Assuming the message is in the client's cache, yes it would, but what's 
the benefit of that expedient update? If the client wants to use that 
mailbox, it will select the mailbox and run the resync algorithm 
anyway.

> I think such a response would not add quite a bit to the bandwidth 

That depends on the number of flag changes in unselected mailboxes, 
which is unbounded. I presume none of us want our mobile phone to 
receive 50-100 bytes every time we read a message on our desktops (and 
thus cause a flag change). That's the gentle case. It's much worse for 
shared mailboxes.

Arnt

---

From: Lingyan Wu <[email-masked]>
Date: Fri, 17 Nov 2006 12:16:18 +0800

----- Original Message ----- 
From: "Arnt Gulbrandsen" <[email-masked]>
To: <[email-masked]>
Cc: "Curtis King" <[email-masked]>; "Lingyan Wu" 
<[email-masked]>; "Alexey Melnikov" <[email-masked]>
Sent: Thursday, November 16, 2006 9:22 PM
Subject: Re: [Lemonade]About draft-gulbrandsen-imap-notify-01

> I meant that it would be big conceptually, because it would extend STATUS 
> to be about messages.
>
>> unselected mailbox. It's just like an unsolicited FETCH response 
>> unsolicited FETCH response and STATUS response, all the mailboxes 
>> (selected and unselected mailboxes) could be synchronized expediently.
>
> Assuming the message is in the client's cache, yes it would,

Almost all of the emails in the client's cache are the emails passing the 
filters (related with the client) in the server, and only the emails in the 
client's cache (passing filters) need to be synchronized, so we could 
synchronize all of the filtered emails in selected and unselected mailboxes 
through unsolicited FETCH response and STATUS response.

>but what's the benefit of that expedient update? If the client wants to use 
>that mailbox, it will select the mailbox and run the resync algorithm 
>anyway.

The benefit is to furthest keep the consistent flags of emails between the 
client and server. When the client create a session with INBOX to do some 
operations, the server could take the chance to send the changed flags of 
filtered emails in other mailboxes through the extended STATUS response so 
that all of the emails in the client's catch could keep latest flags.
When the user accesses his emails in the disconnected client he could know 
the latest flags of the emails and could do some operations accordingly if 
filtered emails in selected and unselected mailboxes could be synchronized.

> That depends on the number of flag changes in unselected mailboxes, which 
> is unbounded. I presume none of us want our mobile phone to receive 50-100 
> bytes every time we read a message on our desktops (and thus cause a flag 
> change). That's the gentle case. It's much worse for shared mailboxes.
>
> Arnt
>

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Sat, 18 Nov 2006 11:38:13 +0100

Lingyan Wu writes:
>>> The client could use the FLAGS in the response to update the flags 
>>> of the corresponding email in client after receiving the response. 
> Almost all of the emails in the client's cache are the emails passing 
> the filters (related with the client) in the server, and only the 
> emails in the client's cache (passing filters) need to be 
> synchronized, so we could synchronize all of the filtered emails in 
> selected and unselected mailboxes through unsolicited FETCH response 
> and STATUS response.

Not quite. We can only synchronize those which are modified while the 
client has an IMAP connection to the server. And the cost of 
synchronizing those is that we also send updates for all messages which 
are NOT in the client's cache.

>> but what's the benefit of that expedient update? If the client wants 
>> to use that mailbox, it will select the mailbox and run the resync 
>> algorithm anyway.
>
> The benefit is to furthest keep the consistent flags of emails between 
> the client and server.

That would be desirable, but it cannot really work, since the client is 
not connected all the time.

> When the client create a session with INBOX to do some operations, the 
> server could take the chance to send the changed flags of filtered 
> emails in other mailboxes through the extended STATUS response so 

Wouldn't that duplicate the functionality of CONDSTORE and quick 
resynchronization?

Arnt

---

From: Lingyan Wu <[email-masked]>
Date: Tue, 21 Nov 2006 19:34:17 +0800

----- Original Message ----- 
From: "Arnt Gulbrandsen" <[email-masked]>
To: <[email-masked]>
Cc: "Lingyan Wu" <[email-masked]>
Sent: Saturday, November 18, 2006 6:38 PM
Subject: Re: [Lemonade]About draft-gulbrandsen-imap-notify-01

>>>> the corresponding email in client after receiving the response. Through 
>> filters (related with the client) in the server, and only the emails in 
>> the client's cache (passing filters) need to be synchronized, so we could 
>> synchronize all of the filtered emails in selected and unselected 
>> mailboxes through unsolicited FETCH response and STATUS response.
>
> client has an IMAP connection to the server.

Assuming only synchronize those which are modified while connection, how to 
notify the client those modified in unselected mailboxes? Is it appropriate 
through STATUS response?

>And the cost of synchronizing those is that we also send updates for all 
>messages which are NOT in the client's cache.

What I wanted to say is, what should be synchronized are those updates 
passing filters related with the client.

>>> but what's the benefit of that expedient update? If the client wants to 
>>> use that mailbox, it will select the mailbox and run the resync 
>>> algorithm anyway.
>>

Yes, the client is not connected all the time. Because of this, it would be 
better to take any chance to synchronize.

>> server could take the chance to send the changed flags of filtered emails 
>> in other mailboxes through the extended STATUS response so that all of 
>> the emails in the client's catch could keep latest flags.
>

If CONSTORE is not supportted, the server could send the tracked changes to 
the client through FETCH and STATUS response to avoid that "the client must 
SELECT each folder separately to receive the changes for that folder".
If CONSTORE is supportted, the server could send the changes since some 
known moment (could be according to the HIGHESTMODSEQ catched in client) or 
just send those which are modified while connection just as what you said 
(If I misunderstand you, please correct me:)).

---

From: Dave Cridland <[email-masked]>
Date: Tue, 21 Nov 2006 12:30:54 +0000

On Tue Nov 21 11:34:17 2006, Lingyan Wu wrote:
> Assuming only synchronize those which are modified while 
> connection, how to notify the client those modified in unselected 
> mailboxes? Is it appropriate through STATUS response?
> 
> 
Yes, I believe so.

The STATUS response can carry sufficient data to indicate whether 
synchronization is required, whether new messages have arrived, and 
whether messages have been deleted.

The specifics of these events can then be fetched by selecting the 
mailbox and synchronizing - where QRESYNC is supported, this is at 
most two round-trips - one for synchronizing FLAG changes and EXPUNGE 
events, one for downloading new message data. If there are no new 
messages - and the client can detirmine this via the information in 
the SELECT (whether enhanced by QRESYNC or not) then the second 
round-trip is not needed.

>> And the cost of synchronizing those is that we also send updates 
>> for all messages which are NOT in the client's cache.
> 
> What I wanted to say is, what should be synchronized are those 
> updates passing filters related with the client.
> 
> 
But if a user is interested in the changes to a mailbox, the user 
will look at the mailbox. If the user is looking at a mailbox, it 
should be SELECTed.

> If CONSTORE is not supportted, the server could send the tracked 
> changes to the client through FETCH and STATUS response to avoid 
> that "the client must SELECT each folder separately to receive the 
> changes for that folder".

If CONDSTORE is not supported, then synchronization is going to be 
much more expensive. There's little that can be done to avoid that.

> If CONSTORE is supportted, the server could send the changes since 
> some known moment (could be according to the HIGHESTMODSEQ catched 
> in client) or just send those which are modified while connection 
> just as what you said (If I misunderstand you, please correct me:)).
> 
> 
And the client, with NOTIFY and QRESYNC, can not only know that such 
changes exist, but can obtain them as needed in one or two round-trip.

This does two things. From the client perspective, it allows for a 
client to alter only the mailbox-level data in its cache, avoiding 
the need to open the message-level cache itself. On resynch, it can 
then write larger chunks of data at once, which is more efficient for 
almost any client, but certainly for PDAs and mobile handsets, for 
which writes are both slow and power-hungry.

Ideally, the simplest method for handling this data is to have a 
single database which maintains mailbox level data, and a second 
database per-mailbox to handle message data. This means that the 
client only has two databases open at once, causing far fewer writes.

 From the server perspective, it's much the same - servers can 
optimize passing around HIGHESTMODSEQ, EXISTS, and possibly UIDNEXT 
changes much more easily that by passing around FETCH data.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: "Ben Last" <[email-masked]>
Date: Tue, 21 Nov 2006 13:54:22 -0000

>From: Dave Cridland [mailto:[email-masked]] 
>If CONDSTORE is not supported, then synchronization is going to be much
more expensive. There's little that can be done to avoid that.

Well said.  Also; if CONDSTORE is supported, it allows the client to
control exactly what mailbox it synchronizes, and when.  I believe that
the client is best placed to make decisions about when (and how often)
it retrieves new mailbox state, since it is the client that is aware of
the state of the mobile device, network connection, etc.  STATUS gives a
way for the client to know what mailboxes have experienced state
changes.

This approach (STATUS and CONDSTORE) seems, to me, to bring IMAP closer
to the delta-notification approach one sees in other push-mail
protocols, which provide both:
a) notification of events in real time
b) a time-stamp-based way to retrieve changes

>which writes are both slow and power-hungry.

Absolutely.  We're finding that we need to keep the WiFi connections
idle as much as possible to save battery life.  The chipsets and drivers
involved usually have timeouts that power down in stages when the
connection is idle.  Consequently, it's significantly more efficient to
retrieve more data in a few consecutive fetches than to spread that over
a series of smaller fetches widely separated by time.

>client only has two databases open at once, causing far fewer writes.

That presupposes some elements of client design which aren't really true
on Symbian-based platforms :)  There's one mail database for everything
there.

Regards to all
Ben 

Ben Last
R&D Manager
EMCC Software Ltd

---

From: Lingyan Wu <[email-masked]>
Date: Wed, 22 Nov 2006 16:54:23 +0800

----- Original Message ----- 
  From: Dave Cridland 
  To: Lingyan Wu 
  Cc: Enhancements to Internet email to support diverse service enivronments 
  Sent: Tuesday, November 21, 2006 8:30 PM
  Subject: Re: [Lemonade]About draft-gulbrandsen-imap-notify-01

  Yes, I believe so.

  The STATUS response can carry sufficient data to indicate whether 
  synchronization is required, whether new messages have arrived, and 
  whether messages have been deleted.
  [Lingyan] Yes, the STATUS response could indicate if there are some events (Message Addition and Deletion events, Message Flags events etc ) in the mailbox . 
  I think when Message Flags events happen while connection , the server could send the events directly to the client to ask the client to update the flags of the corresponding emails so that the client could furthest keep the consistent flags. Thus although the client doesn't select the mailbox it could get the latest flags of the emails in that mailbox. 

  [Lingyan] Just for FLAG changes, the server could notify the client what FLAG changes happened while connection and the client updates the flags of emails in the client accordingly. Doesn't this more simple? I don't think this method could replace what you mentioned but just could provide an accessorial and simple function.

  Is the simple flow as follows right?
  C -->S: select mailboxA and do some operation
       S: a FLAG change happens in mailboxB
  C<-- S: send a STATUS response to tell the client some events have happened in mailboxB
  C<-->S: select mailboxB to synchronize FLAG changes 
  C -->S: select mailboxA to continue the former operation

  The flow could be simplified to:
  C-->S: select mailboxA and do some operation
      S: a FLAG change happens in mailboxB
  C<--S: send the event directly through extended STATUS response

  But if a user is interested in the changes to a mailbox, the user 
  will look at the mailbox. If the user is looking at a mailbox, it 
  should be SELECTed.
  [Lingyan] Maybe as a user I want to keep the latest flags of emails in the client , so I think when I select a mailbox to do some operation it's appropriate for the server to take the chance to send FLAG changes in both selected and unselected mailboxes after receiving IDLE command.

  Dave.

---

From: Dave Cridland <[email-masked]>
Date: Wed, 22 Nov 2006 10:31:42 +0000

Although I'm far from a fanatic on correct quoting, please consider 
configuring more traditional quoting at least for IETF lists, this 
style makes it very much harder to read and reply to. Clients which 
support format=flowed are freely available, and well worth it at that 
price.

I find it difficult to distinguish which paragraphs you wrote from 
those you're replying to, and I have the advantage that you're 
replying to a message I wrote.

On Wed Nov 22 08:54:23 2006, Lingyan Wu wrote:
>   [Lingyan] Yes, the STATUS response could indicate if there are 
> some events (Message Addition and Deletion events, Message Flags 
> events etc ) in the mailbox .   I think when Message Flags events 
> happen while connection , the server could send the events directly 
> to the client to ask the client to update the flags of the 
> corresponding emails so that the client could furthest keep the 
> consistent flags. Thus although the client doesn't select the 
> mailbox it could get the latest flags of the emails in that 
> mailbox. 
> 
But that's what SELECT is for. You can turn this around - if a client 
knows it has all the latest flags for a mailbox, it need not select 
it.

>   [Lingyan] Just for FLAG changes, the server could notify the 
> client what FLAG changes happened while connection and the client 
> updates the flags of emails in the client accordingly. Doesn't this 
> more simple? I don't think this method could replace what you 
> mentioned but just could provide an accessorial and simple function.
> 
>   C<-->S: select mailboxB to synchronize FLAG changes   C -->S: 
> select mailboxA to continue the former operation
> 
> 
No, that's a poorly designed client - there's no need to react 
immediately to the STATUS by resynchronizing the mailbox.

What the client ought to be doing is merely note that its cache for 
mailboxB is out of date, and therefore the next time the user 
requests operations on mailboxB, it needs to resynchronize. It might 
resynchronize without any direct need during a quiescent period, 
perhaps when the client has been idle for a second or two, perhaps, 
but resynchronizing merely because it can is definitely a poor design.

The more interesting case is when the client wants to perform 
operations on mailboxB and there haven't been any events since the 
last time it selected it - in this case, it only needs to select 
mailboxB when it cannot service the request from the cache, and 
moreover need only perform a simple SELECT rather than a QRESYNC 
enhanced one (which is slightly smaller).

In many ways, this is how I see NOTIFY being most interesting - it's 
removing the need to SELECT the mailbox at all, in some cases, and in 
others simplifies things.

>   >> And the cost of synchronizing those is that we also send 
> updates   >> for all messages which are NOT in the client's cache.
>   >   > What I wanted to say is, what should be synchronized are 
> those   > updates passing filters related with the client.
>   >   >   But if a user is interested in the changes to a mailbox, 
> the user   will look at the mailbox. If the user is looking at a 
> mailbox, it   should be SELECTed.
>   [Lingyan] Maybe as a user I want to keep the latest flags of 
> emails in the client , so I think when I select a mailbox to do 
> some operation it's appropriate for the server to take the chance 
> to send FLAG changes in both selected and unselected mailboxes 
> after receiving IDLE command.
> 
> 
But when you SELECT a mailbox, you get all the flag changes you want 
for that mailbox, and the server sends you updates.

There's nothing stopping you from maintainig multiple connections for 
particularly critical mailboxes, of course, but if you're trying to 
get all the data all the time, you'd be best off running an SMTP 
server on the mobile.

Dave.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 22 Nov 2006 11:50:16 +0100

> others simplifies things.

Should NOTIFY send an immediate STATUS (UIDNEXT HIGHESTMODSEQ) the first 
time it's run? (The server has to know whether it's the first time, 
unfortunately.)

C: a login dave acap
S: OK yep, you know the password, you must be dave
C: b notify (mailbox1 mailbox2) ...
S: * STATUS mailbox1 (HIGHESTMODSEQ 3213 UIDNEXT 13)
S: * STATUS mailbox2 (HIGHESTMODSEQ 421342314 UIDNEXT 243124)
S: b OK

Maybe this is worthless complexity. This is lemonade, so 
COMPRESS=DEFLATE is available, and this alternative is just a few bytes 
more expensive (five bytes, maybe? ten?):

C: a login dave acap
S: a OK yep, you know the password, you must be dave
C: b notify (mailbox1 mailbox2) ...
C: c status mailbox1 (uidnext highestmodseq)
C: d status mailbox2 (uidnext highestmodseq)
S: ...

Arnt

---

From: "Ben Last" <[email-masked]>
Date: Wed, 22 Nov 2006 10:55:31 -0000

>From: Arnt Gulbrandsen [mailto:[email-masked]] 
>Should NOTIFY send an immediate STATUS (UIDNEXT HIGHESTMODSEQ) the
first time it's run?
>(The server has to know whether it's the first time, unfortunately.)

> [ useful examples snipped ]

I think that's an excellent idea; it allows the client to see,
immediately after login, what the current modseq and uidnext of all
mailboxes is, and that allows it to determine immediately whether
anything has changed in those mailboxes since the last time it was
synchronized.  Loss of synchronization due to loss of network connection
is, after all, inevitable in mobile.  Thus I imagine that any client
would want to know the status, so it's efficient to have it provided
automatically.

Regards
Ben

Ben Last
R&D Manager
EMCC Software Ltd