Subject: [lemonade] VFOLDER and mutable fields/WITHIN

From: Dan Karp <[email-masked]>
Date: Sat, 18 Nov 2006 13:27:32 -0800 (PST)

I've been reading the most recent VFOLDER draft, and it looks pretty good.  By making the VFOLDER explicitly a view onto another folder, the message UIDs line up and so a VFOLDER-capable client can correlate (and hence not re-download) messages in the VFOLDER with messages in the backing folder.

Unfortunately, this breaks down somewhat when the VFOLDER search-program involves WITHIN and/or dynamic message attributes.  The draft states

      If a server does support search on dynamic attributes, the 
   possibility arises that a message may be included in a vfolder, 
   excluded, and then included again. When this happens, the server MUST 
   generate a new UID each time, and this implies that the order of 
   messages in a vfolder need not match the underlying order of the 
   backing mailbox. VFOLDER aware clients may wish to try and detect 
   this case and prevent duplicate downloads. 

This issue can even arise just using WITHIN-based searches -- for example, "OR OLDER 5 YOUNGER 3" or even "OR OLDER 5 SUBJECT Lemonade".  The MUST in the draft requires either that VFOLDER UIDs diverge from backing folder UIDs (not good) or that backing folder UIDs need to change as well (just plain undesirable due to the effect on other clients).  And I really don't see how the client could "detect this case" and correlate the new-UID message with the appropriate old-UID message in the backing mailbox.

Would it make sense to change the draft such that the contents of a VFOLDER are set at SELECT time?  That way the server isn't responsible for aging messages in/out due to WITHIN criteria and it doesn't have to keep adding and removing (and renumbering) messages if the VFOLDER criteria reference mutable message attributes.

Messages added to the underlying mailbox (via COPY, APPEND, or external delivery) would be evaluated when they were inserted into the mailbox and either included or excluded from the VFOLDER at that time.  If the client wanted to refresh the VFOLDER, it could be as simple as issuing a new SELECT on the VFOLDER.

_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade

---

From: Dave Cridland <[email-masked]>
Date: Sat, 18 Nov 2006 22:07:45 +0000

On Sat Nov 18 21:27:32 2006, Dan Karp wrote:
> I've been reading the most recent VFOLDER draft, and it looks 
> pretty good.  By making the VFOLDER explicitly a view onto another 
> folder, the message UIDs line up and so a VFOLDER-capable client 
> can correlate (and hence not re-download) messages in the VFOLDER 
> with messages in the backing folder.
> 
> 
As you say, this is only in the case where a virtual mailbox is 
defined using purely static criteria, or, as per your suggestion, we 
"fix" dynamic criteria to be static - and even in this latter case 
I'm unclear if we could rely on UIDs being the same, since UIDs must 
strictly increase - you can't UNEXPUNGE, even when the client isn't 
looking.

As a client author, I want dynamic virtual folders which back onto 
multiple mailboxes. As a server developer, I shudder at the mere 
thought.

For single mailbox backed virtual folders, I can emulate it fairly 
closely using a series of SEARCH commands, plus local evaluation of 
criteria based on update information, but they'd still be better 
handled server-side if possible. Singly backed virtual mailboxes are 
also somewhat easier on the server developer.

But then I look at how my client uses this capability, and I note two 
things:

1) My client has two forms of viewing filter, essentially. The first 
is predefined filters, stored in ACAP as IMAP SEARCH program text. 
The one I use more than any other is "UNDELETED UNKEYWORD $Junk 
UNSEEN", which is of course dynamic. The second form is ad-hoc, and 
these are usually based on static data for me (typically FROM or 
SUBJECT searches).

2) I don't use these on one mailbox exclusively - they're not tied to 
that. Instead, I use these on whichever mailbox I'm reading. 
Typically, I'll use my Unread filter when catching up email, for 
instance, whichever folder I'm catching up on.

These two factors suggested to me that what we need, in fact, is a 
dynamically updated SEARCH, or in the parlance of ACAP, a "Context". 
(ACAP allows you to save the results of a search as a named context, 
and optionally have these dynamically updated.)

Hence draft-cridland-imap-context-00

In this proposal, searches may be made to be dynamically updated - 
the server doesn't add or remove messages from a mailbox, instead it 
adds and removes results from a SEARCH. This means that UIDs stay 
constant, and because no SELECT takes place at all, the client is 
never in danger of double-download.

As a bonus, the general design works with SORT (but not THREAD).

There are several other factors in its favour too, but mostly minor.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: Dan Karp <[email-masked]>
Date: Sat, 18 Nov 2006 16:04:08 -0800 (PST)

> strictly increase - you can't UNEXPUNGE, even when the client isn't 

Well, there are ways around that.  If the client is VFOLDER-aware and does something like

   C: A005 SELECT (VFOLDER) "unread msgs"
   S: * LIST (\Subscribed) "/" "unread msgs" (VFOLDER (INBOX "UNSEEN"))
   S: * OK [UIDVALIDITY 3857529045] new uidvalidity each time vfolder is selected
   S: ...

you can change the UIDVALIDITY each time the VFOLDER is SELECTed, as the client knows about the backing store folder.  (You'd want to send along an LIST-EXTENDED-style unsolicited response to make absolutely sure the client knows.)

Or you could simply not return a UIDNEXT value on the VFOLDER and effectively do an "UNEXPUNGE" while the client's not looking.  Clients are built defensively enough that introducing a new message whose UID isn't the highest in the mailbox will still work.  (It's my secret shame -- trust me on this.)  You'd still be abiding by the defining rule for UIDs:

   A 32-bit value assigned to each message, which when used with the
   unique identifier validity value (see below) forms a 64-bit value
   that MUST NOT refer to any other message in the mailbox or any
   subsequent mailbox with the same name forever.

> multiple mailboxes. As a server developer, I shudder at the mere 

For us, IMAP is a secondary view onto our mailstore.  Our internal search *defaults* to applying to all folders owned by the user (except Trash, its subfolders, and Junk), and I'm racking my brain trying to figure out how to shoehorn that into any of the pending VFOLDER-like drafts.  And if you think draft-melnikov-imapext-multimailbox-search-01 or something like it has a chance to become reality, I'm not the only one who's going to have to do some shoehoring.

> never in danger of double-download.

This definitely solves some of the problems, but I'm not convinced it's significantly better than the client just doing a SEARCH and then using notifications to manage the results client-side.  But I'm definitely open to convincing.

---

From: Dave Cridland <[email-masked]>
Date: Sun, 19 Nov 2006 11:37:53 +0000

On Sun Nov 19 00:04:08 2006, Dan Karp wrote:
> > > I've been reading the most recent VFOLDER draft, and it looks > 
> > pretty good.  By making the VFOLDER explicitly a view onto 
> another > > folder, the message UIDs line up and so a 
> VFOLDER-capable client > > can correlate (and hence not 
> re-download) messages in the VFOLDER > > with messages in the 
> backing folder.
> > defined using purely static criteria, or, as per your suggestion, 
> we > "fix" dynamic criteria to be static - and even in this latter 
> case > I'm unclear if we could rely on UIDs being the same, since 
> UIDs must > strictly increase - you can't UNEXPUNGE, even when the 
> client isn't > looking.
> 
> you can change the UIDVALIDITY each time the VFOLDER is SELECTed, 
> as the client knows about the backing store folder.  (You'd want to 
> send along an LIST-EXTENDED-style unsolicited response to make 
> absolutely sure the client knows.)
> 
> 
Well... In fact, I think current proposals prevent selection of 
virtual mailboxes by naïve clients anyway. (Which is their main 
advantage, IMHO, removed).

> Or you could simply not return a UIDNEXT value on the VFOLDER and 
> effectively do an "UNEXPUNGE" while the client's not looking.  
> Clients are built defensively enough that introducing a new message 
> whose UID isn't the highest in the mailbox will still work.  (It's 
> my secret shame -- trust me on this.)  You'd still be abiding by 
> the defining rule for UIDs:
> 
But then you're going be to breaking the mailbox semantics. (And 
trust me, there are clients for which reintroducing UIDs will break 
them. I believe my client is not the only one.) Removing UIDNEXT 
*and* removing strictly increasing UIDs is just a pain.

If you're going to break mailbox semantics, you might as well do away 
with them entirely.

> > > In this proposal, searches may be made to be dynamically 
> updated - > the server doesn't add or remove messages from a 
> mailbox, instead it > adds and removes results from a SEARCH. This 
> means that UIDs stay > constant, and because no SELECT takes place 
> at all, the client is > never in danger of double-download.
> 
> This definitely solves some of the problems, but I'm not convinced 
> it's significantly better than the client just doing a SEARCH and 
> then using notifications to manage the results client-side.  But 
> I'm definitely open to convincing.

Well, it's better in as much as it's a hell of a lot easier to do, 
and it can be used with SORT, which client-side emulation can't. 
Also, there's quite some difficulty with trying to manage this 
without doing a SEARCH command per notification - in many cases, the 
client simply doesn't have sufficient data, which introduces 
additional round-trips *and* additional transmission data.

Dave.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Sun, 19 Nov 2006 13:43:46 +0100

Dave Cridland writes:
> virtual mailboxes by naïve clients anyway.

How so?

>> my secret shame -- trust me on this.)  You'd still be abiding by the 
>> defining rule for UIDs:
>>
> But then you're going be to breaking the mailbox semantics. (And trust 
> me, there are clients for which reintroducing UIDs will break them. I 
> believe my client is not the only one.)

I suspect (without concrete evidence) this might be the case for many 
users of c-client.

Arnt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Sun, 19 Nov 2006 14:34:17 +0100

Dan Karp writes:
> I've been reading the most recent VFOLDER draft, and it looks pretty 
> good. By making the VFOLDER explicitly a view onto another folder, 
> the message UIDs line up and so a VFOLDER-capable client can 
> correlate (and hence not re-download) messages in the VFOLDER with 
> messages in the backing folder.

The experience I have with this is that people hate it when the client 
redownloads, and they also dislike reading both the vfolder and the 
backing mailbox. If you try to solve the first problem on its own, 
you're optimising for a case people don't want. You're optimising for 
minimal traffic from server to client and disregarding the traffic from 
screen to eye.

What people mostly want is to see the rest of the messages from the 
backing folder, ie. all those which aren't in a specific vfolder. My 
suggested solution to that is a new search key, INVFOLDER. (I made a 
concrete suggestion for that about five months ago, not sure if I cc'd 
the list.)

> Would it make sense to change the draft such that the contents of a 
> VFOLDER are set at SELECT time?

And add a sentence like this in draft-*-vfolder: "Clients SHOULD send an 
UNSELECT/SELECT pair every two minutes if they want to be informed of 
updates"?

> That way the server isn't responsible for aging messages in/out due to 
> WITHIN criteria and it doesn't have to keep adding and removing (and 
> renumbering) messages if the VFOLDER criteria reference mutable 
> message attributes.

No, but it gets to do the same work in the guise of SELECT, UNSELECT, 
SELECT, UNSELECT, SELECT, UNSELECT (repeat ad infinitum).

Arnt

---

From: Dan Karp <[email-masked]>
Date: Sun, 19 Nov 2006 09:54:30 -0800 (PST)

> And add a sentence like this in draft-*-vfolder: "Clients SHOULD send
> an UNSELECT/SELECT pair every two minutes if they want to be informed
> of updates"?

Well, the client would be informed of updates to messages that were in the VFOLDER at SELECT time, and could assumedly figure out if a message had left the VFOLDER, if that's what you want.

But I really think it's better to have the view remain static for the period of the SELECT -- can you imagine having a VFOLDER (INBOX "UNSEEN"), doing a FETCH BODY[] on a message, and getting an immediate EXPUNGE because the implicit STORE +FLAGS \DELETED had disqualified the message from inclusion in the VFOLDER?

> > That way the server isn't responsible for aging messages in/out due
> > to WITHIN criteria and it doesn't have to keep adding and removing
> > (and renumbering) messages if the VFOLDER criteria reference
> > mutable message attributes.
> 

You could do this with SELECT, SELECT, SELECT as well, I think.  But this way, the client doesn't have to keep redownloading the messages, right?

---

From: Dan Karp <[email-masked]>
Date: Sun, 19 Nov 2006 10:10:41 -0800 (PST)

> But I really think it's better to have the view remain static for the
> period of the SELECT -- can you imagine having a VFOLDER (INBOX
> "UNSEEN"), doing a FETCH BODY[] on a message, and getting an immediate
> EXPUNGE because the implicit STORE +FLAGS \DELETED had disqualified
> the message from inclusion in the VFOLDER?

Crud.  That should read "implicit STORE +FLAGS \SEEN".  That's what I get for replying to mail before my morning coffee <grumble, grumble>.

---

From: Mark Crispin <[email-masked]>
Date: Sun, 19 Nov 2006 10:27:14 -0800 (PST)

On Sun, 19 Nov 2006, Dave Cridland wrote:
>> effectively do an "UNEXPUNGE" while the client's not looking.  Clients are 
>> built defensively enough that introducing a new message whose UID isn't the 
>> highest in the mailbox will still work.  (It's my secret shame -- trust me 
>> on this.)

Any server that does this is broken.  Completely, utterly, broken.

I am highly offended by the idea that it is is "alright" for a server to 
do this because some clients may overlook this egregious violation of UID 
semantics.

> But then you're going be to breaking the mailbox semantics. (And trust me, 
> there are clients for which reintroducing UIDs will break them. I believe my 
> client is not the only one.) Removing UIDNEXT *and* removing strictly 
> increasing UIDs is just a pain.
>
> If you're going to break mailbox semantics, you might as well do away with 
> them entirely.

Dave is completely, absolutely, 100% correct on this.  If a developer does 
not like IMAP's UID semantics, that developer needs to get out of the IMAP 
business and spawn a new competing protocol.

The corrollary is that if one chooses to be in the IMAP business, then one 
MUST comply with the requirements of IMAP and not warp (or disregard) 
those requirements that prove to be inconvenient for a particular 
implementation.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

---

From: Dan Karp <[email-masked]>
Date: Sun, 19 Nov 2006 10:43:09 -0800 (PST)

"Mark Crispin" <[email-masked]> wrote:
> >> my secret shame -- trust me on this.)
> 
> I am highly offended by the idea that it is is "alright" for a server
> to do this because some clients may overlook this egregious violation
> of UID semantics.

Mark, the hyperbole is unnecessary.  I was only being half-serious about it.

But you're still right, it's a bad idea.

---

From: Dan Karp <[email-masked]>
Date: Sun, 19 Nov 2006 10:45:57 -0800 (PST)

> additional round-trips *and* additional transmission data.

You've got a point.  That may indeed be a better solution, as it doesn't require any UID magic and clients may not view messages that transition from matching to non-matching state as EXPUNGEs (which would cause re-downloads if they transition right back to matching).

I think that my main sticking point is client-side, not server-side.  Whether the client always remains sync'ed with the state of a persistent search or it refreshes only when the user requests it, I'm still a proponent of presenting the user with static views of search results.

We've got a product that's fundamentally search-based, and representing searches on dynamic attributes has been one of the more vexing issues.  We've found that people would rather run a search once and then work their way through a static snapshot of the result set than have the results vary in real time with changes in the backing store.  Your mileage may vary, but our users seemed put off when anything other than a real "delete" caused search results to disappear from their view, or when anything other than newly-deposited mail caused new search results to appear.  They'd rather explicitly re-run the search than have it change out from under them.

So I'm fine with a protocol that always keeps the client up to date, and I'm equally fine with a protocol that refreshes a client view only when the user revisits a "saved search".  I just think that a protocol that EXPUNGEs when a message no longer fits the search criteria will lead to bad user experience.

---

From: Mark Crispin <[email-masked]>
Date: Sun, 19 Nov 2006 11:31:12 -0800 (PST)

> But you're still right, it's a bad idea.

Humor, sarcasm, and half-seriousness does not transmit very well in email. 
The smiley ":-)" was introduced in an attempt to ameliorate the problems 
brought about by ill-advised levity.

Nor is it hyperbole for someone who must deal on a daily basis with people 
who seriously believe that IMAP's definitions can be so compromised, 
and/or:
  . "blurdybloop server works fine with Outlook, therefore it's a bug in
    your client."
  . "blurdybloop client works fine with Courier, therefore it's a bug in
    your server."
  . "blurdybloop is a commercial product, therefore it's a bug in your
    product."

I had to take the comment seriously; and a serious claim that "Clients are 
built defensively enough that introducing a new message whose UID isn't 
the highest in the mailbox will still work" can not go unchallenged.

-- Mark --

http://panda.com/mrc

---

From: Dave Cridland <[email-masked]>
Date: Sun, 19 Nov 2006 22:29:48 +0000

>> virtual mailboxes by naïve clients anyway.
I thought there was talk of to require clients to show knowledge of 
virtual mailboxes before they appear in LIST. Of course, I wrote this 
without checking the current draft, being the weekend and all, so 
I've not checked which way the debate settled.

IIRC, that debate revolved around naïve clients potentially 
automatically synchronizing all mailboxes without realizing they're 
downloading many messages multiple times in the case of virtual 
mailboxes.

Dave.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Sun, 19 Nov 2006 23:50:46 +0100

> I've not checked which way the debate settled.

It didn't settle, AFAICR.

> downloading many messages multiple times in the case of virtual 

Any lemonade client doing that has a bigger problem than just vfolder ;) 
I suppose I've mentioned the PDA that ran LIST "" * as the first 
command after login? It (the PDA) crashed in five seconds on connecting 
to uw-imapd.

Arnt

---

From: Dave Cridland <[email-masked]>
Date: Sun, 19 Nov 2006 23:24:51 +0000

On Sun Nov 19 18:27:14 2006, Mark Crispin wrote:
>> trust me, there are clients for which reintroducing UIDs will 
>> break them. I believe my client is not the only one.) Removing 
>> UIDNEXT *and* removing strictly increasing UIDs is just a pain.
>> 
>> If you're going to break mailbox semantics, you might as well do 
>> away with them entirely.
> 
> Dave is completely, absolutely, 100% correct on this.  If a 
> developer does not like IMAP's UID semantics, that developer needs 
> to get out of the IMAP business and spawn a new competing protocol.

That's not quite what I meant - I only meant to imply that if you 
cannot model a continuously (or discretely) updated SEARCH result set 
as a mailbox efficiently, then the obvious way forward would be to 
model it as something else. Like a continuously (or discretely) 
updated SEARCH result, as I propose.

Dave.

---

From: Dave Cridland <[email-masked]>
Date: Sun, 19 Nov 2006 23:25:53 +0000

On Sun Nov 19 18:45:57 2006, Dan Karp wrote:
> We've got a product that's fundamentally search-based, and 
> representing searches on dynamic attributes has been one of the 
> more vexing issues.  We've found that people would rather run a 
> search once and then work their way through a static snapshot of 
> the result set than have the results vary in real time with changes 
> in the backing store.  Your mileage may vary, but our users seemed 
> put off when anything other than a real "delete" caused search 
> results to disappear from their view, or when anything other than 
> newly-deposited mail caused new search results to appear.  They'd 
> rather explicitly re-run the search than have it change out from 
> under them.
> 
> 
Yes, I get annoyed when I'm using a SEARCH based on UNSEEN, such that 
messages would vanish as I read them. So I instruct the client, by a 
checkbox, not to dynamically remove messages from the display which 
cease to match. It can be done client-side, even with CONTEXTs.

On the other hand, I always want to see messages which become 
matching, whether due to delivery or changes in circumstance.

> So I'm fine with a protocol that always keeps the client up to 
> date, and I'm equally fine with a protocol that refreshes a client 
> view only when the user revisits a "saved search".  I just think 
> that a protocol that EXPUNGEs when a message no longer fits the 
> search criteria will lead to bad user experience.

One aspect of virtual mailboxes is that should a message cease to 
match, it not only vanishes in a way that's indistinguishable from 
the message being EXPUNGEd, but also it vanishes in such a way that 
it's no longer accessible. In other words, the client cannot choose 
the behaviour above. This is particularly annoying if the client 
relies on the implicit \Seen flag setting behaviour of a non-PEEK 
FETCH, but might actually be frustrating with WITHIN, too.

Dave.