Subject: [MORG] Why IMAP extensions are (not) used

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 8 Apr 2009 14:38:54 +0200

Arnt Gulbrandsen writes:
> Curtis King writes:
>> I'm sure there are some more clients which use a few more extensions, 
>> but that still means ~16 extensions we (Isode) have implemented are 
>> not being used. Not a very good ratio. Yet, here we are planning on 
>> adding more.
>
> A bit of discussion of the reasons seems like a good idea. I have 
> opinions but it's late and I'm too tired now. I'll try to post 
> something during the coming week.

Here goes. I'm sure someone will tell me I'm all wrong. Go on, but 
please do say something about what's right too.

I believe that IMAP extensions aren't used as much as they should be for 
a combination of five reasons. I believe that all five are factors in 
every case, but the weight varies widely.

Further, I believe that all five can be mitigated, in some cases quite 
easily. Mitigating is not the same as eliminating, and of course 
mitigating the known problems is enough to ensure takeup of a 
particular new extension.

1. RFCs written with the wrong audience in mind.

Many of the RFCs don't make much effort to serve client authors or 
answer their questions, such as «how should I use this?», «how should I 
fall back when it isn't supported?». Some RFCs have few examples, some 
have many, but «many» is not the same as «useful for a particular 
audience».

(Without making a detailed survey, I'd guess that the ones with many are 
fairly good for server implementers, less so for clients. Server 
implementers need more text and examples about corner cases, clients 
more that's related to their use cases.)

2. RFCs can be hard to find.

We don't have a list mapping from problem/benefit to the relevant 
extension(s), and googling doesn't solve that problem well.

3. What if/ifnot.

Clients generally need to implement stuff both for servers that support 
a given extensions, and for servers that don't. We haven't given much 
thought to minimising the duplicated code paths that result, or do it 
cleanly.

(I don't mean textually this time, I mean functionally.)

4. Uncertainty about compliant servers.

It's needlessly difficult for a client to find out which servers support 
something, which makes it harder to test with/without a particular 
extension than it should be, and also harder to know whether 
implementing an extension will have a benefit in the wild.

Another aspect of this is interoperation.

5. Combinatorial behaviour.

Several non-IETF people I've spoken to complain about the number of 
combinations.

The number of combinations is effectively higher than the number of 
extensions, since some extensions interact in tricky ways. This strikes 
me as wrong, and most of the interactions are IMO just neat-to-have.

It could be lower in practice, since a client can always do
   if ( server supports a, b and c)
      use all three
   else
      use none

We could try harder to avoid the interactions, and perhaps occasionally 
call attention to the a && b && c trick. (That trick also needs 
information about how many servers implement a and b but not c, etc.)

Arnt

(PS: I've tried to avoid digression in this little missive, as I think 
we would otherwise get bogged down the third sentence of RFC x section 
y.x, instead of why IMAP discussions aren't used as much as we'd like 
and how MORG can have a good fate.)

---

From: Mark Crispin <[email-masked]>
Date: Wed, 8 Apr 2009 12:25:35 -0700 (PDT)

IMAP extensions are not used much for the following reasons:

[1] Almost all IMAP extensions are worthless garbage.

Oh, they may be valuable to some limited constituency, but not generally.

Some servers may do UNSELECT or NAMESPACE or CHILDREN because they are 
low-hanging fruit, or IDLE for mobile devices.  It is nearly impossible to 
make a business case for the rest, especially the recent batch.

[2] Non-compliant servers destroy what limited value there is to an 
extension.

Any client developer who depends upon an extension quickly discovers that 
the client fails when talking to some server.  No amount of "it's the 
server's fault" will help; the customer will invariably say "it works with 
Outlook and Thunderbird, so it MUST be a bug in your client."

To some extent, the customer is right; the client author depended upon an 
extension.  That is a bug.

Courier is particularly ruthless in destroying by mis-implementation.

[3] Combinatorial behavior.

This is more a matter of perception than a real technical problem.

Nonetheless the people who do QA and regression testing are alarmed by the 
2^n combinations that need testing.

You should be too.

Last but not least...

[4] The refusal to allow IMAP Extensions to die.

IMAPEXT was dragged out long beyond the point where it should have been 
killed.

When IMAPEXT was finally killed, IMAP Extensions moved to LEMONADE. 
LEMONADE completely missed its window of opportunity, due in part to 
everybody sneaking in their favorite IMAP extension as a LEMONADE 
requirement.  Deep down inside, they knew that this extension didn't stand 
a snowball's chance in hell of any market penetration.

Now that LEMONADE is dying, along comes MORG which is simply a new name 
for IMAPEXT

Slay this dragon.   By whatever means necessary.  Do it now.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

---

From: Barry Leiba <[email-masked]>
Date: Wed, 8 Apr 2009 15:29:17 -0400

> Many of the RFCs don't make much effort to serve client authors or answer
> their questions, such as «how should I use this?», «how should I fall back
> when it isn't supported?». Some RFCs have few examples, some have many, but
> «many» is not the same as «useful for a particular audience».

I suspect the number of examples is not a factor at all in decisions
whether to implement an extension or not.

In fact, I suspect that how the RFC is written is, in general, a minor point.

> extension(s), and googling doesn't solve that problem well.

This amounts to "publicity", and, surely, the most implemented ones
may correlate with the most publicized ones.  Alexey once has a web
page with a list of which implementations support which extensions.
That also addresses your point 4.

> 3. What if/ifnot.
>
> Clients generally need to implement stuff both for servers that support a
> given extensions, and for servers that don't. We haven't given much thought
> to minimising the duplicated code paths that result, or do it cleanly.

This is really the nut of it, I think.
Servers will implement an extension if (1) it's so easy that they
might as well, or (2) they perceive a demand for it (from clients or
from customers -- the latter is sometimes just a box they have to
tick... yes, we support that).

Clients will implement an extension if (1) it really saves them a lot
of programming work, (2) it really saves a lot of performance, (3) it
provides a feature that they can't get without it, or (4) they
perceive a demand for it (see above).

Your point 3 shows that (1) is basically not possible the way
extensions are implemented in IMAP.  For POP, the TOP and UIDP
extensions, for example, are essentially required.  Any server that
doesn't support them will fail, and clients regularly just assume
they're available and refuse to work with servers that don't have
them.  So it's easy.

Lacking that, though, client developers have to do *extra* programming
to support extensions, because things have to work both ways, as you
say.

(2) mostly coves things like searching, sorting, and threading on the
server, and a few things that save a lot of bandwidth or round trips
for situations with slow connections or ones with high latency.

(3) mostly covers IDLE.  Pretty much anything else can be done another way.

What it comes down to is that (1), above, is the critical point.  As
long as the extensions are "extensions", sparsely (and perhaps
buggily) supported, things will be difficult.

There was a small effort started to trim things from IMAP that might
be better done in other ways (like folders) and to fold in the
"important" extensions to a new base.  The effort was called "IMAP5",
and it seems to have died down.

Barry

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 8 Apr 2009 22:42:39 +0200

Thank you for this very good response.

Barry Leiba writes:
>>  answer their questions, such as «how should I use this?», «how 
>>  should I fall back when it isn't supported?». Some RFCs have few 
>>  examples, some have many, but «many» is not the same as «useful for 
>>  a particular audience».
>

Maybe you're right. But I know two cases where a perceived lack of 
readability/blah delayed implementation for literally years, and my 
part of the world isn't very big.

(And yes, I'm as guilty as anyone else.)

> That also addresses your point 4.

I know the page and think not, but it's close-ish. This would be more 
helpful, e.g.:

   [ extension name ]
      known supporting servers: [ list ]
      known unsupporting: [ list ]
      test accounts/downloads with: [ list ]

>> 3. What if/ifnot.
>>
>>  Clients generally need to implement stuff both for servers that 
>>  support a given extensions, and for servers that don't. We haven't 
>>  given much thought to minimising the duplicated code paths that 
>>  result, or do it cleanly.
>

or, pedantically, (3) they need the functionality and providing an IMAP 
interface is essentially cost-free.

> extensions are implemented in IMAP.

I think you're right, at least for all the relevant extensions I can 
think of now.

Do me a favour. Repost that list at each WGLC.

> ...

A couple of hours ago someone sent me mail saying essentially «blah 
issues 350 SELECTs and 350 SEARCHes and it's unpleasantly slow». I 
think the sender might just possibly consider multimailbox search to 
fit group 3 ;)

INTHREAD searches are a bit like that too. With that you can add a 
checkbox in the search dialog:

    [ ] return entires conversations

That's a pure added feature. It's also possible to use INTHREAD 
differently in clients (e.g. send 350 THREAD commands in addition to 
the 350 SELECTs above), of course.

> and it seems to have died down.

Yes. I though that was a little too... shall we say, the likelihood of 
highly unpleasant discussions seemed high and the advantage for me 
small.

Arnt

---

From: Mark Crispin <[email-masked]>
Date: Wed, 8 Apr 2009 14:18:40 -0700 (PDT)

> interface is essentially cost-free.

There is no such thing as "cost-free" for features, at least not if you 
have a real company with QA and support.  Freeware server implementors may 
have the luxury of implementing random stuff for fun.

> A couple of hours ago someone sent me mail saying essentially «blah issues 
> 350 SELECTs and 350 SEARCHes and it's unpleasantly slow». I think the sender 
> might just possibly consider multimailbox search to fit group 3 ;)

If 350 SELECTs and 350 SEARCHes are unpleasantly slow, what makes you 
think that multimailbox search will be different?

It will save 700 RTTs, which can certainly be unpleasant.  But, unless the 
server has instantaneous SELECT and SEARCH, there are more substantial 
costs elsewhere.

The advantage of the 700 RTTs it is clear to anyone looking at the 
resulting network traffic that what the client is doing is idiotic. 
Compare this to a server command that takes 20 minutes to complete.

> It's also possible to use INTHREAD differently 
> in clients (e.g. send 350 THREAD commands in addition to the 350 SELECTs 
> above), of course.

It's also possible to have a more intelligent client that uses the 
facilities that exist already.

-- Mark --

http://panda.com/mrc

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 8 Apr 2009 23:36:26 +0200

Mark Crispin writes:
>>  think the sender might just possibly consider multimailbox search 
>>  to fit group 3 ;)
>

Not «will be», «is».

The speedup (up to orders of magnitude depending on the search 
expression) comes from two factors: The search itself is much better 
optimised, and there's no need to compute those 350 UNSEENs, EXISTSs 
and RECENTs.

Arnt

---

From: Mark Crispin <[email-masked]>
Date: Wed, 8 Apr 2009 14:46:35 -0700 (PDT)

On Wed, 8 Apr 2009, Arnt Gulbrandsen wrote:
>> If 350 SELECTs and 350 SEARCHes are unpleasantly slow, what makes you think 
>> that multimailbox search will be different?
> Not «will be», «is».
> The speedup (up to orders of magnitude depending on the search expression) 
> comes from two factors: The search itself is much better optimised, and 
> there's no need to compute those 350 UNSEENs, EXISTSs and RECENTs.

You are engaged in the fallacy of believing that an implementation aspect 
of your server is common to all other servers.

-- Mark --

http://panda.com/mrc

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 8 Apr 2009 23:59:19 +0200

Mark Crispin writes:
>>>  If 350 SELECTs and 350 SEARCHes are unpleasantly slow, what makes 
>>>  you think that multimailbox search will be different?
>>  Not «will be», «is».
>>  expression) comes from two factors: The search itself is much 
>>  better optimised, and there's no need to compute those 350 UNSEENs, 
>>  EXISTSs and RECENTs.
>
> You are engaged in the fallacy of believing that an implementation 
> aspect of your server is common to all other servers.

Locking the mailbox in order to compute RECENT isn't free for anyone. 
Scanning for UNSEEN and EXISTS isn't free for anyone. Many databases 
and libraries that will use indexes to search quickly are available to 
everyone. Maybe the query planner I use is better than most. So what?

Arnt

---

From: Barry Leiba <[email-masked]>
Date: Wed, 8 Apr 2009 18:37:10 -0400

> Scanning for UNSEEN and EXISTS isn't free for anyone. Many databases and
> libraries that will use indexes to search quickly are available to everyone.
> Maybe the query planner I use is better than most. So what?

If you have (say) a DB2 back-end to your mail store, a search of 350
mailboxes might be very efficient.

If you have (say) a set of Unix mailboxes as the back-end to your mail
store, a search of 350 mailboxes might be pretty much the same,
whether you do it in one command or 700, apart from the protocol
overhead.

Mark's point is that for a lot of these sorts of things, you kind of
need to know something about the server to know whether doing [X] is a
good idea or not.

Perhaps one might say that a server SHOULD only implement multimailbox
search if it has the sort of back-end that makes it significantly more
efficient.  Or perhaps there should also be some sort of clue to the
client as to the efficiency.  Or perhaps clients and servers should do
what they like, and the market will sort out which ones match users'
expectations best.

Whatever the decision, Mark's point, which is correct, is that you
can't make assumptions.  A client that wants to let a user search 350
mailboxes will do it, one way or another.  But it might or might not
perform badly with or without the extension, and you can't know in
advance which it'll be.

Barry

---

From: Mark Crispin <[email-masked]>
Date: Wed, 8 Apr 2009 15:41:06 -0700 (PDT)

> Locking the mailbox in order to compute RECENT isn't free for anyone.

What is "locking the mailbox in order to compute RECENT"?

> Scanning for UNSEEN and EXISTS isn't free for anyone.

What is "scanning for UNSEEN and EXISTS"?

You are assuming, falsely, that concepts that exist in your implementation 
are common to all other implementations.

-- Mark --

http://panda.com/mrc

---

From: Timo Sirainen <[email-masked]>
Date: Wed, 08 Apr 2009 19:14:03 -0400

>       test accounts/downloads with: [ list ]

I started such a list, add your own servers (and their "home pages"):
http://imapwiki.org/Extensions

---

From: Timo Sirainen <[email-masked]>
Date: Wed, 08 Apr 2009 19:43:58 -0400

> Oh, they may be valuable to some limited constituency, but not generally.

I guess that depends on your target audience. For traditional desktop
IMAP clients most extensions are worthless. For mobile clients I guess
many would be valuable in theory, but since the server support for
extensions is (still) so low I have some doubts as to if/when they start
supporting e.g. Lemonade extensions.

Anyway, most of the extensions I implement nowadays are going to be used
by webmails. Webmail users want to do everything and they want
everything to work immediately, so anything that helps there is useful.
I guess those features could also be implemented in a non-standard way,
but I do see it useful to standardize them so that webmails aren't tied
to specific server implementations.

---

From: Mark Crispin <[email-masked]>
Date: Wed, 8 Apr 2009 17:30:10 -0700 (PDT)

> mailboxes might be very efficient.

At some other costs... ;-)

> whether you do it in one command or 700, apart from the protocol

Note that to do IMAP SEARCH right, you have to be cognizant of the MIME 
structure, select only the textual parts, and search the textual data 
using the i;unicode-casemap algorithm.  "Just grep through the mailboxes" 
doesn't cut it any more.  grep doesn't know about a string that may be in 
a non-UTF8 charset within BASE64 encoded content, or about text vs. image 
MIME parts.

Either you do the necessary computations on the fly at demand, or do the 
calculations in advance and store the results separate from the message 
data, or do something in between.

No database can somehow make this choice go away.  The form may change, 
but not the function.

Nor is it as easy as saying "we have lots of disk space, just precompute 
and store everything".  The more you store, the more complexity that you 
have to deal with.  Also, delivery time is a really bad time to be doing 
anything time-consuming.

> perform badly with or without the extension, and you can't know in

There's more than that.  A server may decide that the client that searches 
350 mailboxes is abuse, and will "implement" the multimailbox search to 
block the abuse.  Only it won't really search.

Don't think for one minute that this isn't going on now.

For better or worse, most servers are designed with one goal, and only one 
goal, in mind: to implement synchronization with Outlook.  Everything 
else, and I do mean EVERYTHING else, is secondary.

-- Mark --

http://panda.com/mrc

---

From: Mark Crispin <[email-masked]>
Date: Wed, 8 Apr 2009 18:01:11 -0700 (PDT)

On Wed, 8 Apr 2009, Timo Sirainen wrote:
> For traditional desktop
> IMAP clients most extensions are worthless.

Completely agree.  The only extensions that make a significant difference 
for Pine/Alpine are SORT/THREAD and to a lesser extent MULTIAPPEND.

The thing that strikes me with the recent batch of extensions is that they 
are all stupid little tweaks that do nothing to enhance functionality.

> For mobile clients I guess

I have $100 that says that this is not going to change.

LEMONADE missed its window of opportunity.  The LEMONADE extensions are a 
joke in the vendor community.

The BIS model makes much more sense for mobile clients than the humungous 
kludge tower of LEMONADE.  BES makes even more sense for the fully 
integrated environment (will IETF get calendaring done before the sun 
novas??).  You can damn well bet that bean counters like it too.

Windows Mobile and iPhone allow you to consume IMAP/POP directly, and 
iPhone's MobileMe is like BES.  But I won't be at all surprised if Apple 
soon offers an "optional" MobileMeLite BIS service that works much better 
(from the end user viewpoint) than directly consuming IMAP/POP.

> by webmails.

This assumes that the webmails will consume IMAP.  I see plenty of free 
webmails that do that, some (e.g., Web Alpine) smarter than others.  But 
none of the industrial webmails consume IMAP now, or have any intention of 
consuming IMAP except as part of a migration tool.

Our webmail doesn't consume IMAP.  Webmail and IMAP consume the same 
distributed store; there are operations in that store specifically for 
webmail, and others specifically for IMAP.

-- Mark --

http://panda.com/mrc

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 9 Apr 2009 10:07:16 +0200

Barry Leiba writes:
>>  Locking the mailbox in order to compute RECENT isn't free for 
>>  anyone. Scanning for UNSEEN and EXISTS isn't free for anyone. Many 
>>  databases and libraries that will use indexes to search quickly are 
>>  available to everyone. Maybe the query planner I use is better than 
>>  most. So what?
>

Yes. That's my experience. I was surprised by how fast it was, to tell 
the truth.

> whether you do it in one command or 700, apart from the protocol 

And because I was surprised by the speed, I did some analysis to find 
out what made the difference. It's not just «db good, rest of world 
bad».

If that mbox server uses no indexing, then you're right.

If it uses lucene (IIRC that's what Timo mentioned using it for his mbox 
store) for each mailbox, then the story will be different. In that case 
you may see that SELECT is as much effort as the actual searching for 
some important searches (and not much improvement for some other 
searches) and that RTTs dominate the user experience. It depends on the 
search expression, of course.

If it uses lucene, but has a per-user index instead of a per-mailbox 
one, then the best-case improvement is even bigger, particularly when 
the results are in few mailboxes.

Using a DB2-like backend will boost multmailbox searches for more and 
more varied queries than just a lucene-like index. But the lucene-like 
index will be really fast for an important subset of queries (such as a 
simple SEARCH BODY x for all the user's mailboxes).

> need to know something about the server to know whether doing [X] is 
> a good idea or not.

Yes... and I have a feeling that time spent implementing and analysing 
aids knowledge.

> search if it has the sort of back-end that makes it significantly 
> more efficient. Or perhaps there should also be some sort of clue to 
> the client as to the efficiency.

I think not: The efficiency is difficult to describe concisely, and 
people will leanr from experience which buttons are best not pressed 
when the db is big.

> Or perhaps clients and servers should do what they like, and the 
> market will sort out which ones match users' expectations best.
>

But you can write the code, run some searches, analyse the logfiles and 
then you'll know quite a bit.

This is a digression. I'll stop now.

Arnt

---

From: Timo Sirainen <[email-masked]>
Date: Thu, 09 Apr 2009 15:12:18 -0400

> simple SEARCH BODY x for all the user's mailboxes).

Just one point: Using Lucene or pretty much anything else for SEARCH
TEXT or SEARCH BODY violates RFC 3501, since they can't do substring
searches. SEARCH FUZZY helps here.