Subject: Re: [lemonade] WGLC NOTIFY

From: Alexey Melnikov <[email-masked]>
Date: Wed, 04 Jun 2008 09:51:49 +0100

Hi Timo,

Timo Sirainen wrote:

> On May 27, 2008, at 9:33 AM, Alexey Melnikov wrote:
>
>> Timo Sirainen wrote:
>>
>>> One more thing I thought of:
>>>
>>>> If IDLE (as well as this extension) is supported, while processing
>>>>    IDLE the server MUST send the same events as instructed by the
>>>>    client using the NOTIFY command.
>>>
>>> Does this mean EXPUNGEs must not be sent if MessageExpunge isn't
>>> included in NOTIFY SET?
>>
>> Correct.
>>
>>> That would make it impossible to create clients
>>> that use MSNs and want to be immediately notified of EXPUNGEs (while
>>> IDLEing).
>>
>> Correct. The client can either chose not to use NOTIFY at all (in which
>> case it will get expunges with MSNs), or it can chose to specify both
>> MessageNew & MessageExpunge (they are tied together as per section 5),
>> in which case it will get back UIDs.
>
This statement was incorrect (and I think your statement on which I 
commented is also incorrect). When MessageExpunge is requested, the 
server will either send EXPUNGE (with message numbers) or VANISHED (with 
UIDs). VANISHED is only sent if QRESYNC was also enabled.

Having said that, if the client specifies MessageExpunge and QRESYNC was 
not enabled, it will get MSNs immediately.

> "get back UIDs"? Do you mean VANISHED? Doesn't that depend on the 
> server if it or EXPUNGE is sent?
>
> What happens if NOTIFY SET is used, but SELECTED isn't specified? I 
> can't seem to find this from the draft. I'm guessing it'll use the 
> defaults? Or does it use the first other matching rule?

Good question. I think this needs to be clarified.

> Anyway I think this is too restrictive if using NOTIFY basically 
> requires the client to drop using MSNs.

After rereading some portions of draft-ietf-lemonade-imap-notify, I 
would like to correct this: a client using NOTIFY can't use MSNs in 
commands it is issuing. However it will still be using MSNs to process 
server responses.

> Some use cases for clients which want to keep using MSNs and IDLE:
>
> 1. Client just wants to use NOTIFY to see changes in other mailboxes, 
> so it specifies NOTIFY SET (personal (messagenew messageexpunge)).

I assume the client still wants to immediately see changes in the 
current mailbox? In this case it should also include (selected 
(MessageNew MessageExpunge)) and things should work as before.

> 2. Client wants to use NOTIFY to ignore annotation changes, so it uses 
> NOTIFY SET (selected (FlagChange)).

FlagChange is tied to MessageNew and MessageExpunge, so this should 
really be "NOTIFY SET (selected (FlagChange MessageNew MessageExpunge))"

> EXISTS and EXPUNGEs are still sent in command replies, but not during 
> IDLE?

They will be sent in IDLE as well, as per my previous comment.

> 3. Client wants to get pushed information about new messages' contents 
> (MessageNew (uid ..)), but it still wants to keep using MSNs.

Can you elaborate on this use case?
Your MessageNew already contains "uid", so clearly the client is already 
capable of using UIDs anyway.

> Since MessageNew and MessageExpunge must specified together, this is 
> impossible .
>
> Is there even any other reason for client to even run IDLE unless it 
> wants to keep using MSNs + see EXPUNGEs immediately? I think IDLE 
> could be specified as sending EXPUNGEs immediately regardless of 
> whether MessagExpunge has been specified.

I would rather not special case IDLE, so that we can keep server logic 
consistent.
In light of my corrected statement above, can you double check that 
there is any issue?

> 3. is also problematic. Is it really necessary to give MessageExpunge 
> if MessageNew is given? The (well-behaving) servers already have to 
> handle the case when a message is expunged and a new message arrives 
> during FETCH/STORE/SEARCH command. The client is notified about the 
> new message but not about the expunge.

True.
The reason why we have MessageExpunge and MessageNew tied together is 
because representative of a certain company kept moaning that he can't 
implement them separately. The WG couldn't find any use cases where this 
would make a difference for a client.

>> It is possible that some servers that support IDLE don't send
>> MessageExpunge events while idling, but I haven't seen any such server.
>> We could mandate that all servers advertising the NOTIFY capability send
>> MessageExpunge during IDLE if NOTIFY wasn't issued on the connection?!
>
> I don't think that really matters since well-behaving servers already 
> do that. I'm more concerned that well-behaving servers aren't allowed 
> to do that when NOTIFY is used..

_______________________________________________
lemonade mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/lemonade
Supplemental Web Site:
http://www.standardstrack.com/ietf/lemonade

---

From: Timo Sirainen <[email-masked]>
Date: Wed, 04 Jun 2008 19:01:18 +0300

> commands it is issuing. However it will still be using MSNs to process 

I'd hate to see that happen. It took me a while to understand that MSNs
can be useful and I don't want to support an extension that forces
clients to stop sending them in commands.

Also many existing clients could benefit a lot from NOTIFY, since their
"STATUS each mailbox every n minutes" could be replaced by a single
NOTIFY command. But if this same command requires rewriting their
command sending to use only UIDs, that could be a much larger job.

> (MessageNew MessageExpunge)) and things should work as before.

Not if it can't send MSNs in commands.

> really be "NOTIFY SET (selected (FlagChange MessageNew MessageExpunge))"

And again it couldn't send MSNs in commands.

> capable of using UIDs anyway.

That was just an example. The fields could have been anything.

But anyway just because it uses UIDs in some places doesn't mean it even
could use UIDs everywhere. The best example is a Pine/webmail-like
application that fetches information only about the visible part of the
mailbox (in message list view). Those are the only kinds of IMAP clients
that are able to access huge mailboxes efficiently. In them when a user
scrolls a pageful up in message list, the client calculates how many
messages it needs to fetch and uses a FETCH with a sequence range. This
is impossible to do with UIDs unless the client fetches all UIDs, which
in turn makes it inefficient (especially with huge mailboxes) and waste
bandwidth for no good reason.

> In light of my corrected statement above, can you double check that 

If special casing IDLE made client able to send MSNs, I think it would
be useful to change it. But that also requires decoupling MessageExpunge
from other events.

> would make a difference for a client.

Again it makes sending MSNs impossible, which is bad.

I'd like it if all existing clients (which may be sending MSNs) could
easily replace their STATUS-polling with a single notify command:

1 NOTIFY SET (selected (MessageNew FlagChange) personal (MessageNew MessageExpunge))

---

From: "Jackson Chan" <[email-masked]>
Date: Wed, 4 Jun 2008 14:13:39 -0400

Quick question:

What is the status of inclusion of CONVERT in Lemonade Profile Bis?
_______________________________________________
lemonade mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/lemonade
Supplemental Web Site:
http://www.standardstrack.com/ietf/lemonade

---

From: Timo Sirainen <[email-masked]>
Date: Wed, 04 Jun 2008 21:17:46 +0300

On Wed, 2008-06-04 at 09:51 +0100, Alexey Melnikov wrote:
> implement them separately.

I forgot to mention in my previous mail: I don't think it's unreasonable
for an extension to require servers to implement the base IMAP4 protocol
correctly. Untied MessageNew and MessageExpunge handling is already
required for FETCH/STORE/SEARCH handling. If a server has trouble
implementing them untied, it's already either broken or nonoptimal. If
the server doesn't do EXISTS notifications for FETCH/STORE/SEARCH, it
could implement NOTIFY the exact same way: EXISTS notifications are sent
only when EXPUNGE notifications are also legal.

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 04 Jun 2008 19:29:02 +0100

Hi Jackson,

>What is the status of inclusion of CONVERT in Lemonade Profile Bis?
CONVERT is a mandatory feature in Lemonade Profile Bis and there is no 
plan to take CONVERT out.

Regards,
Alexey

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 5 Jun 2008 12:24:01 +0200

Timo Sirainen writes:
> for an extension to require servers to implement the base IMAP4 
> protocol correctly. Untied MessageNew and MessageExpunge handling is 
> already required for FETCH/STORE/SEARCH handling.

How?

> If a server has trouble implementing them untied, it's already either 
> broken or nonoptimal.

Peter's code is certainly not optimal. I wish he would stop trying to 
use MSNs as if they were UIDs.

Arnt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 5 Jun 2008 12:24:02 +0200

Timo Sirainen writes:
> It took me a while to understand that MSNs can be useful and I don't 
> want to support an extension that forces clients to stop sending them 
> in commands.

But Notify doesn't do that.

What Notify does is different. It lets clients say "tell me about 
expunges as they happen", "tell me about mailbox creations as they 
happen" and many other things.

If a client uses Notify to say "tell me about expunges as they happen", 
then that client can't use MSNs in commands any more (except in the 
sort of corner cases Dave loves). But that limitation is not due to 
Notify, it's an intrinsic consequence of what the client wants the 
server to do.

Arnt

---

From: Timo Sirainen <[email-masked]>
Date: Thu, 05 Jun 2008 17:25:16 +0300

> Notify, it's an intrinsic consequence of what the client wants the 

Currently there's no way to specify the default IMAP server behavior
with NOTIFY: Send MessageNew and FlagChange as soon as they happen, but
send MessageExpunge only when it's safe to do so. If MessageExpunge
wasn't tied to MessageNew I'd be happier.

---

From: Dave Cridland <[email-masked]>
Date: Thu, 05 Jun 2008 20:59:24 +0100

> happen" and many other things.
Actually, not *quite*. It allows these events to occur at any time -  
the distinction is subtle, I admit.

But IMAP4rev1 essentially states that events always happen when  
they're reported. (It's actually one of the cleverer bits of design  
there, for all that it causes problems in the expunge case). This  
means that, in particular, commands are always predictable - in the  
base specification, at least.

> Notify, it's an intrinsic consequence of what the client wants the

Well, the client is now saying it wants events to occur at any time -  
it's lifting restrictions on them.

As such, it can use MSNs, as long as its aware that, being unstable  
references, they will only become "fixed" when the command is  
processed, and in the intervening time they may have changed - almost  
certainly not what's needed.

That said, I think many cases will work fine with MSN even so,  
including cases using FETCH, STORE, etc.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade
_______________________________________________
lemonade mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/lemonade
Supplemental Web Site:
http://www.standardstrack.com/ietf/lemonade

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 5 Jun 2008 22:06:34 +0200

Timo Sirainen writes:
> with NOTIFY: Send MessageNew and FlagChange as soon as they happen, 
> but send MessageExpunge only when it's safe to do so.

That sort of wondered me, but I couldn't find any concrete ill effects 
even though I tried hard, so I eventually told my intuition "you're 
wrong too sometimes, so be quiet now".

> If MessageExpunge wasn't tied to MessageNew I'd be happier.

Yes, so would I.

Arnt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 5 Jun 2008 22:12:25 +0200

Arnt Gulbrandsen writes:
> That sort of wondered me,

I think I wanted ot write "that worried me" and "I wondered about that" 
and the two expressions mated on the way to the keyboard and produced 
the offspring you see.

Arnt

---

From: Timo Sirainen <[email-masked]>
Date: Fri, 6 Jun 2008 05:49:11 +0300

On Jun 5, 2008, at 5:25 PM, Timo Sirainen wrote:

> but

If we assumed nothing was tied together, the current NOTIFY behavior  
would be:

  - (MessageNew MessageExpunge FlagChange) : Everything sent immediately
  - (MessageNew FlagChange) : EXISTS and flags sent immediately,  
expunges delayed
  - (MessageNew) : EXISTS sent immediately, expunges delayed, flag  
changes not sent at all
  - () : EXISTS and expunges delayed, flag changes not sent at all

The closest default IMAP behavior would be (MessageNew FlagChange).  
Most servers delay sending the changes until a command is started, but  
that's not required by RFC 3501 so I don't think it matters.

Perhaps MessageNew should be always enabled? EXISTS replies can't be  
disabled entirely in any case and I don't see any benefit in delaying  
them (if client didn't want to see new events immediately, it wouldn't  
have called NOTIFY in the first place). With this done and decoupling  
MessageExpunge there would be no tied events left anymore. That would  
nicely simplify the specification.

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 11 Jun 2008 23:05:12 +0100

Timo Sirainen wrote:

>On Thu, 2008-05-22 at 23:56 +0400, Alexey Melnikov wrote:
>
 [...]

>>>a notify set (selected (MessageNew MessageExpunge)) (personal
>>>(SubscriptionChange))
>>>
>>This would result in MessageNew and MessageExpunge events for the
>>currently selected mailbox (whether it is a personal one or not) and
>>subscription change notifications for all personal mailboxes except for
>>the currently selected one (if it is a personal mailbox).
>>
>>Hmm, this raises an interesting question: is the "except for the
>>currently selected" part in the last sentence is what we actually want?
>>Should "Selected"only apply to message level events?
>>    
>>
>I think that would be better.
>
Based on private replies from Arnt and Peter, I will be changing the 
document to reflect that. If anybody objects, please speak up now.

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 11 Jun 2008 23:22:22 +0100

>  - () : EXISTS and expunges delayed, flag changes not sent at all

The idea behind NOTIFY NONE or NOTIFY SET (SELECTED NONE) is to disable 
any type of notifications whatsoever. This would allow the client to 
work on a snapshot of a mailbox and avoid receiving any extra traffic on 
a connection for which the client doesn't want to receive any 
notifications (This is analogous to signal blocking for a thread).

> Most servers delay sending the changes until a command is started, 
> but  that's not required by RFC 3501 so I don't think it matters.
>
> Perhaps MessageNew should be always enabled?

This would disallow the NOTIFY NONE behavior.

> EXISTS replies can't be  disabled entirely in any case and I don't see 
> any benefit in delaying  them (if client didn't want to see new events 
> immediately, it wouldn't  have called NOTIFY in the first place). With 
> this done and decoupling  MessageExpunge there would be no tied events 
> left anymore. That would  nicely simplify the specification.

---

From: Timo Sirainen <[email-masked]>
Date: Thu, 12 Jun 2008 13:44:08 +0300

> notifications (This is analogous to signal blocking for a thread).

I hadn't noticed NONE before. That makes this situation even more
complex..

> This would disallow the NOTIFY NONE behavior.

What if there was first a big switch between "NONE" vs. "something" and
then some finer switches under "something":

a) NONE - nothing sent at all.

b)
- MessageNew always sent immediately
- MessageExpunge always sent, but could be either delayed or immediate
- Flags, annotations, etc. are optionally sent, but if they are they're
always immediate

I guess the b) option could be done by specifying that unless NONE is
used, MessageNew is always used regardless of whether it's explicitly
specified or not.

It probably would be cleaner if the entire syntax was changed a bit:

NOTIFY SELECTED <expunge-type> [(optionals) [(fetch attributes)]]

NOTIFY SELECTED none
 - nothing sent
NOTIFY SELECTED delayed
 - delayed expunges, immediate EXISTS, no flag changes
NOTIFY SELECTED delayed (FlagChange)
 - delayed expunges, immediate EXISTS and flag changes
NOTIFY SELECTED immediate (FlagChange) (UID FLAGS)
 - immediate expunges, EXISTS and flag changes, new messages return
FETCH (UID FLAGS)
NOTIFY SELECTED immediate () (UID FLAGS)
 - no flags, immediate others

---

From: Chris Newman <[email-masked]>
Date: Fri, 13 Jun 2008 14:46:53 -0700

Since my co-worker was not participating in this conversation, I'm going to 
jump in and explain the reason to link new & expunge events in NOTIFY.

[I'm speaking as a technical participant, not as an area director.  As AD I 
will support advancement of IMAP NOTIFY regardless of the resolution of 
this issue if the WG chair believes it has WG rough-consensus.]

An IMAP server has to keep a mapping from sequence numbers to physical 
messages.

The Cyrus IMAP implementation (at least the older versions I've worked 
with) does this via a file descriptor to a memory-mapped file with 
fixed-length records.  That's a fairly elegant implementation because the 
scalability is limited by VM address space (rather than actual RAM) and the 
most current mapping is in the on-disk file, while older mappings are 
retained only as long as a process has the fd open and are automatically 
cleaned up by the filesystem when unused.

Within that model, as soon as a MessageExpunge event actually occurs, the 
MessageNew events stop coming to MSN-only server processes until they issue 
a non-MSN sensitive command.  So there is an inherent linkage between the 
two event types.

In order to truly decouple the events with that server architecture, the 
server would have to create a new temporary pre-expunge-MSN to 
post-expunge-MSN mapping table between the time the expunge happened and 
the client issued a non-MSN sensitive command in order to continue 
generating MessageNew events.  That's quite a bit of new server complexity 
(with scalability implications) for very limited benefit.

Now I can't speak for the current Cyrus Server maintainers although I'd 
like to hear from them on this topic.  But the IMAP server Sun sells 
largely retains this part of the design and the additional complexity of 
implementing the full de-coupling is likely to make us choose not to 
implement NOTIFY.

I did think of a compromise to de-coupling the events.  Introduce a new 
"ExpungePending" event.  If you ask for MessageNew without MessageExpunge, 
then the server can send ExpungePending to the client and cease sending 
MessageNew events until the client triggers the EXPUNGE responses.  I'm 
describing the compromise not because I think it's a good idea (linking the 
events is much simpler architecturally), but because it shows the edge of 
the server implementation issue.

Any comments from other Cyrus-based implementers?

		- Chris

--On June 6, 2008 5:49:11 +0300 Timo Sirainen <[email-masked]> wrote:

>   - (MessageNew FlagChange) : EXISTS and flags sent immediately, expunges
> delayed
> The closest default IMAP behavior would be (MessageNew FlagChange). Most
> servers delay sending the changes until a command is started, but that's
> not required by RFC 3501 so I don't think it matters.
>

---

From: Timo Sirainen <[email-masked]>
Date: Sat, 14 Jun 2008 08:23:11 +0300

On Jun 14, 2008, at 12:46 AM, Chris Newman wrote:

> I did think of a compromise to de-coupling the events.  Introduce a  
> new "ExpungePending" event.  If you ask for MessageNew without  
> MessageExpunge, then the server can send ExpungePending to the  
> client and cease sending MessageNew events until the client triggers  
> the EXPUNGE responses.  I'm describing the compromise not because I  
> think it's a good idea (linking the events is much simpler  
> architecturally), but because it shows the edge of the server  
> implementation issue.

All I actually care about is that NOTIFY can be fully used by clients  
that want to use MSNs. Such clients would still want to see expunges,  
so I think they'd basically be sending IDLE whenever they're done  
sending commands. So I wouldn't mind if NOTIFY had a big switch that  
worked like:

a) Immediate: Send expunges, exists and everything else immediately to  
client

b) Delayed: Server is allowed (but not required) to delay all events  
until a command is executed that allows returning expunges (usually  
IDLE, but also others).

---

From: Alexey Melnikov <[email-masked]>
Date: Sat, 21 Jun 2008 13:39:48 +0100

Hi Timo:

 [...]

Timo Sirainen wrote:

>So I wouldn't mind if NOTIFY had a big switch that 
>worked like:
>
>IDLE, but also others)
I've changed the definition of "SELECTED" to "SELECT immediate" as you 
defined above. I've also added SELECTED-DELAYED, which corresponds to 
"SELECTED delayed" in your proposal.

---

From: Timo Sirainen <[email-masked]>
Date: Sun, 22 Jun 2008 02:27:08 +0300

On Jun 21, 2008, at 3:39 PM, Alexey Melnikov wrote:

> Timo Sirainen wrote:
>
>> So I wouldn't mind if NOTIFY had a big switch that worked like:
>>
>> a) Immediate: Send expunges, exists and everything else immediately  
>> to client
>>
>> b) Delayed: Server is allowed (but not required) to delay all  
>> events until a command is executed that allows returning expunges  
>> (usually IDLE, but also others)
>>
> I've changed the definition of "SELECTED" to "SELECT immediate" as  
> you defined above. I've also added SELECTED-DELAYED, which  
> corresponds to "SELECTED delayed" in your proposal.

Sounds good. Is there also something about MessageExpunge being  
unnecessary (or even illegal?) in SELECTED/SELECTED-DELAYED and  
MessageNew being unnecessary (or even illegal?) unless it specifies  
FETCH options?