Subject: Re: ANNOTATE-EXPERIMENT-1
From: Bron Gondwana <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: Timo Sirainen <[email-masked]>, [email-masked]
Date: Mon, 13 Feb 2012 14:30:14 +0100
List-Id: <ietf-imapext.imc.org>

On Mon, Feb 13, 2012, at 02:21 PM, Arnt Gulbrandsen wrote:
> On 02/13/2012 02:04 PM, Bron Gondwana wrote:
> > And of course there's normalisation as well.
> 
> Normalisation doesn't affect meaning at all. It's just changes such as
> "o umlaut-above-previous-letter" into "o-with-umlaut" or the other way.
> Useful for text processing.

Also useful for checking if a particular keyword is already set, and merging
the various forms together into a single keyword.

Bron.
-- 
  Bron Gondwana
  [email-masked]

Date: Mon, 13 Feb 2012 14:21:59 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: Bron Gondwana <[email-masked]>
Cc: Timo Sirainen <[email-masked]>, [email-masked]
Subject: Re: ANNOTATE-EXPERIMENT-1
Sender: [email-masked]

On 02/13/2012 02:04 PM, Bron Gondwana wrote:
> And of course there's normalisation as well.

Normalisation doesn't affect meaning at all. It's just changes such as
"o umlaut-above-previous-letter" into "o-with-umlaut" or the other way.
Useful for text processing.

Arnt

From: Bron Gondwana <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: Timo Sirainen <[email-masked]>, [email-masked]
Subject: Re: ANNOTATE-EXPERIMENT-1
Date: Mon, 13 Feb 2012 14:04:11 +0100
Sender: [email-masked]

On Mon, Feb 13, 2012, at 10:49 AM, Arnt Gulbrandsen wrote:
> On 02/13/2012 10:32 AM, Bron Gondwana wrote:
> > Keywords are already case insensitive.  Which is going to be a pig in poke if we try to make UTF8 keywords.
> 
> Unicode case insensitivity isn't so bad.
> 
> One rule for Turkish, one rule of the rest of the world, and anyone who
> can define i to be equal to ı can sidestep the difference. In IMAP's
> case, these two commands could be defined to be equivalent:
> 
>   a store +flags 1 istanbul
>   b store +flags 1 ıstanbul
> 
> Not pretty but also not the end of the world.
> 
> (Neat: This spell checker, which I must find out how to turn off btw,
> thinks istanbul is wrong and ıstanbul right.)

And of course there's normalisation as well.  I really don't know as much
about this area as I should.  I guess the obvious is "require best practice
as defined <over there>".

> >> But I don't mind dropping that in the least. AFAIK noone has actually
> >> wanted to sort by an annotation.
> > 
> > If nobody wants it, it doesn't matter if it's inefficient, right?
> 
> Mostly, but not quite.
> 
> A fetch item that noone ever calls can still require copy and append to
> do extra work.

True - so long as it's stored rather than calculated on request.

> >> Why would store be able to change only some entries?
> > 
> > ACLs is the obvious case.  Another thing which the Gmail guys mentioned
> > recently is "partial storage failure" - IMAP presents a view in which
> > everything is either fully functional or fully broken, but the reality
> > under the hood is not always so.
> 
> Good point.
> 
> Still, I think that's an IMAP problem. It's not made worse by covering
> all of IMAP, as opposed to all except ANNOTATE.

Agree.

Bron.
-- 
  Bron Gondwana
  [email-masked]

Date: Mon, 13 Feb 2012 10:49:44 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: Bron Gondwana <[email-masked]>
Cc: Timo Sirainen <[email-masked]>, [email-masked]
Subject: Re: ANNOTATE-EXPERIMENT-1
Sender: [email-masked]

On 02/13/2012 10:32 AM, Bron Gondwana wrote:
> Keywords are already case insensitive.  Which is going to be a pig in poke if we try to make UTF8 keywords.

Unicode case insensitivity isn't so bad.

One rule for Turkish, one rule of the rest of the world, and anyone who
can define i to be equal to ı can sidestep the difference. In IMAP's
case, these two commands could be defined to be equivalent:

  a store +flags 1 istanbul
  b store +flags 1 ıstanbul

Not pretty but also not the end of the world.

(Neat: This spell checker, which I must find out how to turn off btw,
thinks istanbul is wrong and ıstanbul right.)

>> But I don't mind dropping that in the least. AFAIK noone has actually
>> wanted to sort by an annotation.
> 
> If nobody wants it, it doesn't matter if it's inefficient, right?

Mostly, but not quite.

A fetch item that noone ever calls can still require copy and append to
do extra work.

>> Why would store be able to change only some entries?
> 
> ACLs is the obvious case.  Another thing which the Gmail guys mentioned
> recently is "partial storage failure" - IMAP presents a view in which
> everything is either fully functional or fully broken, but the reality
> under the hood is not always so.

Good point.

Still, I think that's an IMAP problem. It's not made worse by covering
all of IMAP, as opposed to all except ANNOTATE.

Arnt

From: Bron Gondwana <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>, Timo Sirainen <[email-masked]>
Cc: [email-masked]
Subject: Re: ANNOTATE-EXPERIMENT-1
Date: Mon, 13 Feb 2012 10:32:40 +0100
Sender: [email-masked]

On Mon, Feb 13, 2012, at 10:19 AM, Arnt Gulbrandsen wrote:
> 
> On 02/13/2012 01:45 AM, Timo Sirainen wrote:
> > 
> > I was thinking about finally implementing METADATA + per-mail annotations to Dovecot so I read through the RFCs. There are mainly two things I'd do differently in ANNOTATE-EXPERIMENT-2:
> > 
> > 1. METADATA has case-insensitive entry names, ANNOTATE has case-sensitive. Probably better to keep them case-insensitive in both.
> 
> I wouldn't mind.

Keywords are already case insensitive.  Which is going to be a pig in poke if we try to make UTF8 keywords.

> > 2. SORT (ANNOTATION) - is this really worth the trouble? There's really no good way for servers to implement this in an efficient way. It would pretty much require building and maintaining an index for every single annotation entry name.
> 
> Sorting at read time isn't bad, considering the very low number of
> annotations used.
> 
> But I don't mind dropping that in the least. AFAIK noone has actually
> wanted to sort by an annotation.

If nobody wants it, it doesn't matter if it's inefficient, right?

If people do want it, then you can track the annotations they are actually
using to sort by and index those... at least that's the theory behind having
adaptive indexing.

(just being contrary now!)

> > And other things I wondered about:
> > 
> >  - If APPEND has annotations, and annotation limits are reached, the whole APPEND fails with TOOBIG/TOOMANY error? I guess it has to, although that's a bit annoying.
> 
> Append is transactional. I agree it can be annoying sometimes, such as
> when ten-thousand-message multiappends break. But IMAP already has
> enough funny little exceptions, please don't try to add a rule such as
> "append is transactional except in regard to annotations".

Agree - if it's transactional, it's transactional.

> >  - ANNOTATE doesn't specify what happens if STORE can change only some of the entries. Should it rollback the changes to all the previous entries as well? Including changes to previous messages if multiple messages were specified? Would have been simpler if both METADATA and ANNOTATE supported changing only one annotation at a time.
> 
> My code is transactional, so I don't have the problem.
> 
> Why would store be able to change only some entries?

ACLs is the obvious case.  Another thing which the Gmail guys mentioned
recently is "partial storage failure" - IMAP presents a view in which
everything is either fully functional or fully broken, but the reality
under the hood is not always so.

More simply, could just be permission problems on a file somewhere, or
partial database corruption.  Shit happens.

Bron.
-- 
  Bron Gondwana
  [email-masked]

Date: Mon, 13 Feb 2012 10:19:17 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: Timo Sirainen <[email-masked]>
Cc: [email-masked]
Subject: Re: ANNOTATE-EXPERIMENT-1
Sender: [email-masked]

On 02/13/2012 01:45 AM, Timo Sirainen wrote:
> 
> I was thinking about finally implementing METADATA + per-mail annotations to Dovecot so I read through the RFCs. There are mainly two things I'd do differently in ANNOTATE-EXPERIMENT-2:
> 
> 1. METADATA has case-insensitive entry names, ANNOTATE has case-sensitive. Probably better to keep them case-insensitive in both.

I wouldn't mind.

> 2. SORT (ANNOTATION) - is this really worth the trouble? There's really no good way for servers to implement this in an efficient way. It would pretty much require building and maintaining an index for every single annotation entry name.

Sorting at read time isn't bad, considering the very low number of
annotations used.

But I don't mind dropping that in the least. AFAIK noone has actually
wanted to sort by an annotation.

> And other things I wondered about:
> 
>  - If APPEND has annotations, and annotation limits are reached, the whole APPEND fails with TOOBIG/TOOMANY error? I guess it has to, although that's a bit annoying.

Append is transactional. I agree it can be annoying sometimes, such as
when ten-thousand-message multiappends break. But IMAP already has
enough funny little exceptions, please don't try to add a rule such as
"append is transactional except in regard to annotations".

>  - ANNOTATE doesn't specify what happens if STORE can change only some of the entries. Should it rollback the changes to all the previous entries as well? Including changes to previous messages if multiple messages were specified? Would have been simpler if both METADATA and ANNOTATE supported changing only one annotation at a time.

My code is transactional, so I don't have the problem.

Why would store be able to change only some entries?

>  - TOOBIG/TOOMANY doesn't map very nicely if I want to have global "max. annotation size" and "max. annotation count". Perhaps a global max. count shouldn't exist, since the number of mails/mailboxes could be limited instead, but a global size similar to mail quota should exist. I guess when that global size is reached it just fails with TOOBIG until some annotations are deleted.

Yes.

Arnt

From: Timo Sirainen <[email-masked]>
Subject: ANNOTATE-EXPERIMENT-1
Date: Mon, 13 Feb 2012 02:45:32 +0200
To: [email-masked]
Sender: [email-masked]

I was thinking about finally implementing METADATA + per-mail annotations to Dovecot so I read through the RFCs. There are mainly two things I'd do differently in ANNOTATE-EXPERIMENT-2:

1. METADATA has case-insensitive entry names, ANNOTATE has case-sensitive. Probably better to keep them case-insensitive in both.

2. SORT (ANNOTATION) - is this really worth the trouble? There's really no good way for servers to implement this in an efficient way. It would pretty much require building and maintaining an index for every single annotation entry name.

And other things I wondered about:

 - If APPEND has annotations, and annotation limits are reached, the whole APPEND fails with TOOBIG/TOOMANY error? I guess it has to, although that's a bit annoying.

 - ANNOTATE doesn't specify what happens if STORE can change only some of the entries. Should it rollback the changes to all the previous entries as well? Including changes to previous messages if multiple messages were specified? Would have been simpler if both METADATA and ANNOTATE supported changing only one annotation at a time.

 - TOOBIG/TOOMANY doesn't map very nicely if I want to have global "max. annotation size" and "max. annotation count". Perhaps a global max. count shouldn't exist, since the number of mails/mailboxes could be limited instead, but a global size similar to mail quota should exist. I guess when that global size is reached it just fails with TOOBIG until some annotations are deleted.

From: Thomas Koch <[email-masked]>
Reply-To: [email-masked]
To: Adrien de Croy <[email-masked]>
Subject: Re: [imap5] Designing a new replacement protocol for IMAP
Date: Fri, 10 Feb 2012 15:44:50 +0100
Cc: [email-masked], Bron Gondwana <[email-masked]>, IMAP Protocol Interest List <[email-masked]>, [email-masked]
Sender: [email-masked]

Hi Adrien,

thank you for your insights. I've some more questions inline.

Adrien de Croy:
> * updates.  Would need to poll or use long polling to get real-time
> updates (e.g. notification of incoming mail).
Could Websockets or http://en.wikipedia.org/wiki/Comet_(programming) be of 
help here?

> * complex.  In order to reduce round-trips, you'd have to do many things
> in a single transaction, which would create complexity issues in the
> server and client
I was thinking restful HTTP which by definition is not complex. The question 
is, whether a useful mail fetch protocol could be designed in the constraints 
of rest.
 
> I think it would also be very problematic in the wild, since any http
> intermediary would feel entitled to mess with traffic.  E.g. heuristic
> caching, intercepting proxies etc.
There is an HTTP header to instruct any cache to pipe the request through and 
do no caching. But anybody on any line in any protocol could mess with your 
data.

Have to go now, sorry. :-)

Best regards,

Thomas Koch, http://www.koch.ro

Date: Thu, 9 Feb 2012 12:32:08 +0100
From: Bron Gondwana <[email-masked]>
To: Giovanni Panozzo <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: [imap5] Designing a new replacement protocol for IMAP
Organization: brong.net
Sender: [email-masked]

On Thu, Feb 09, 2012 at 12:25:33PM +0100, Giovanni Panozzo wrote:
> PS: I think that crossposting in 3 ML is too much. Which one will be
> the official ML for this discussion ?

(posting to all three for this...)

Let's take the rest of the traffic to just [email-masked]

Bron.

Date: Thu, 9 Feb 2012 12:24:11 +0100
From: Bron Gondwana <[email-masked]>
To: Adrien de Croy <[email-masked]>
Cc: [email-masked], [email-masked], Bron Gondwana <[email-masked]>, IMAP Protocol Interest List <[email-masked]>, [email-masked]
Subject: Re: [imap5] Designing a new replacement protocol for IMAP
Organization: brong.net
Sender: [email-masked]

On Thu, Feb 09, 2012 at 09:05:53PM +1300, Adrien de Croy wrote:
> 
> Whilst HTTP could certainly be used to put together an email system,
> I don't think it would perform well enough to replace existing
> protocols on desktops.
> 
> The main issues I see are:
> 
> * updates.  Would need to poll or use long polling to get real-time
> updates (e.g. notification of incoming mail).

Yes - long polling or "out of band" notification systems.  We have
an HTTP protocol defined kind of "on top of" IMAP plus our own
extensions here at FastMail/Opera (mail.opera.com if you want to
play with it)

We are currently using EventSource to notify the client that
"there is something new", after which the client queries to
find what's updated.

This is possible to be efficient because we use a single counter
for uidvalidity and a single counter for highestmodseq across the
entire user's folders - so any change to uidvalidity means the
folder list needs refreshing, and any change to highestmodseq
means the message view needs checking.

Combined with efficient "what's changed in this view since my
previous modseq value" queries, it's very bandwidth efficient.

> * slow.  Gut feel based on 17 years writing http proxies is that it
> would be too slow

Not really actually - so long as you can query multiple things in
a single "transaction"

> * complex.  In order to reduce round-trips, you'd have to do many
> things in a single transaction, which would create complexity issues
> in the server and client

But here's the rub.  Yes, complex.

> * protocol bloat.  if you think IMAP is bloated, try HTTP.
> * fundamentally an off-line style protocol for mail, which has
> strong incentives to be online (esp for in-office use).

Mostly facebook's "live feed" is plenty fast enough.  People are
doing all sorts of close-enough-to-real-time stuff on top of HTTP.
It doesn't mean it's the right choice, but I wouldn't dismiss it
out of hand either.

> I think it would also be very problematic in the wild, since any
> http intermediary would feel entitled to mess with traffic.  E.g.
> heuristic caching, intercepting proxies etc.

That's a problem for sure.  Mind you, proxyability is high on my
list of goals.  Immutable stuff should be cachable.

> It's almost too ubiquitous.  Everyone and their dog thinks they know
> how http works.  It's not strictly-enough defined, and has a LOT of
> optional features which aren't relevant for mail (e.g. content
> negotiation).  Interop in http is already problematic, but start
> getting people writing mail servers in php etc, and see what starts
> to happen.  I think it would be a nightmare.

Like IMAP then, except only their dogs actually understand it ;)

Good enough tests should, in theory, allow people to write their
servers in PHP if they want to.  Why you would choose to write in
PHP, I'm not sure... but still.  It takes all types.

Bron.

Date: Thu, 09 Feb 2012 21:05:53 +1300
From: Adrien de Croy <[email-masked]>
To: [email-masked]
CC: [email-masked], Bron Gondwana <[email-masked]>, IMAP Protocol Interest List <[email-masked]>, [email-masked]
Subject: Re: [imap5] Designing a new replacement protocol for IMAP
Sender: [email-masked]

Whilst HTTP could certainly be used to put together an email system, I 
don't think it would perform well enough to replace existing protocols 
on desktops.

The main issues I see are:

* updates.  Would need to poll or use long polling to get real-time 
updates (e.g. notification of incoming mail).
* slow.  Gut feel based on 17 years writing http proxies is that it 
would be too slow
* complex.  In order to reduce round-trips, you'd have to do many things 
in a single transaction, which would create complexity issues in the 
server and client
* protocol bloat.  if you think IMAP is bloated, try HTTP.
* fundamentally an off-line style protocol for mail, which has strong 
incentives to be online (esp for in-office use).

I think it would also be very problematic in the wild, since any http 
intermediary would feel entitled to mess with traffic.  E.g. heuristic 
caching, intercepting proxies etc.

It's almost too ubiquitous.  Everyone and their dog thinks they know how 
http works.  It's not strictly-enough defined, and has a LOT of optional 
features which aren't relevant for mail (e.g. content negotiation).  
Interop in http is already problematic, but start getting people writing 
mail servers in php etc, and see what starts to happen.  I think it 
would be a nightmare.

Adrien

On 9/02/2012 8:20 p.m., Thomas Koch wrote:
> Bron Gondwana:
>> Snide comments from those who don't agree with the goals are welcome too...
>> I just won't listen to you.  These goals are very important to me.
> Hi Bron,
>
> I'm not an expert in IMAP, just got here out of curiosity about Kolab[1]'s use
> of IMAP folders as data stores.
> I found some older initiatives to replace IMAP with an HTTP based approach.[2]
> What do you think of HTTP for mail access? For my bachelor thesis[3] I
> currently argue that CardDAV/CalDAV could be perfectly replaced by AtomPub
> (RFC 5023).
> The main advantage would be that most software developers on this planet have
> some vague understanding of HTTP and that imense infrastructure and software
> already exists.
>
> [1] http://wiki.kolab.org/Why_IMAP_Is_Used_For_Storage
> [2] http://www.prescod.net/rest/restmail/
>      https://www.ietf.org/mailman/listinfo/httpmail
> [3] https://github.com/thkoch2001/bachelor-thesis
>
> Best regards,
>
> Thomas Koch, http://www.koch.ro
>

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

From: Thomas Koch <[email-masked]>
Reply-To: [email-masked]
To: [email-masked]
Subject: Re: [imap5] Designing a new replacement protocol for IMAP
Date: Thu, 9 Feb 2012 08:20:27 +0100
Cc: Bron Gondwana <[email-masked]>, IMAP Protocol Interest List <[email-masked]>, [email-masked]
Sender: [email-masked]

Bron Gondwana:
> Snide comments from those who don't agree with the goals are welcome too...
> I just won't listen to you.  These goals are very important to me.
Hi Bron,

I'm not an expert in IMAP, just got here out of curiosity about Kolab[1]'s use 
of IMAP folders as data stores.
I found some older initiatives to replace IMAP with an HTTP based approach.[2] 
What do you think of HTTP for mail access? For my bachelor thesis[3] I 
currently argue that CardDAV/CalDAV could be perfectly replaced by AtomPub 
(RFC 5023).
The main advantage would be that most software developers on this planet have 
some vague understanding of HTTP and that imense infrastructure and software 
already exists.

[1] http://wiki.kolab.org/Why_IMAP_Is_Used_For_Storage
[2] http://www.prescod.net/rest/restmail/
    https://www.ietf.org/mailman/listinfo/httpmail
[3] https://github.com/thkoch2001/bachelor-thesis

Best regards,

Thomas Koch, http://www.koch.ro

From: Greg Banks <[email-masked]>
To: Lyndon Nerenberg <[email-masked]>
Subject: Re: Metadata (5654) Implementation Limits
Date: Thu, 9 Feb 2012 07:18:39 +1100
Cc: "[email-masked]" <[email-masked]>
Sender: [email-masked]

On 09/02/2012, at 5:33, Lyndon Nerenberg <[email-masked]> wrote:

>
> [I didn't get much response from the imap-protocol list, so I'm  
> trying here, too.]
>
> I'm curious to know what the implementation limits are on servers  
> that support the Metadata extension.  I would appreciate it if the  
> server authors out there could take a minute to fill out this survey.
>
> I will send a summary to the list once the responses trickle off.  
> (If you want your results to be anonymous, say so in the comments.)
>
> Thanks,
>
> --lyndon
>
> Implementation name and version:

Cyrus (METADATA support rewritten and ANNOTATE support added to future  
2.5 release).

> Do you support /private?:

Yes

> Do you support /private/vendor?:

Yes, with a subset of that space reserved for the server itself. Users  
cannot create new entries in that space, but they can create /private/ 
vendor/theirname/*.

> Do you support unsolicited Metadata responses without values?:

No

> Do you support unsolicited Metadata responses with values?:

No - no unsolicited responses.

> Maximum number of annotations per mailbox:

Not explicitly limited.

> Maximum size of an <entry>:

Not explicitly limited, but database keys include entry names and are  
limited to 4KiB.

> Maximum size of a <value>:

Not explicitly limited.

The old code had an undocumented limit of around 2KiB (iirc) at which  
it would silently truncate data in a stack buffer. The new code deals  
with values dynamically.

There is presumably a limit to the size of the database values.

For ANNOTATE support we report a 64KiB limit at SELECT time but this  
is a lie.

> Maximum depth supported in <entries>:

Not limited except by the length.

> Other comments:
>

By default, Cyrus only accepts annotations whose entries are hardcoded  
or explicitly listed in a Cyrus configuration file (which is empty by  
default). A config option allows users to set arbitrary entries.

We thought carefully about limits when implementing ANNOTATE as the  
per-message annotations pose a more extreme challenge. The solution we  
came up with was to track and limit the total size of all values of  
annotations on mailboxes and on messages, using a new QUOTA resource  
which we called X-ANNOTATION-STORAGE, with semantics similar to  
STORAGE. Per-server annotations are not tracked but can only be set by  
admin users.

Greg.

From: Bron Gondwana <[email-masked]>
To: IMAP Protocol Interest List <[email-masked]>, "Discussion on drastically slimming-down IMAP." <[email-masked]>, [email-masked]
Subject: Designing a new replacement protocol for IMAP
Date: Wed, 08 Feb 2012 21:15:26 +0100
Sender: [email-masked]

I've given up on the IMAP5 mailing list going endlessly around in circles,
though I'm definitely going to read through the backlogs and remind myself
again what everyone spoke about, because there was some excellent discussion.

So - I will build something and prove that it's good by showing improved
user experiences.

And I'm happy to keep talking here on these mailing lists too.

I've started a wiki page:

http://imapwiki.org/ReplacementProtocol

and plan to blog about what I'm doing on our own (Opera's) blogging platform,
to practice using it:

http://my.opera.com/brongondwana/blog/

I plan to make on a protocol which can replace IMAP in most of its current
uses.  I want to get (at least) two working server implementations, and
two working clients.  I have at least in-principle support from enough
project maintainers to do that.

My strong philosophical ideals are:

* Simple to build clients - morons welcome. If it's not clear the right way
  to do something to someone just reading a protocol dump, we haven't done a
  good enough job.  We can't expect you to read 30 RFCs and resolve their
  conflicts yourself before starting coding, and we won't insult you if you
  didn't get it right.  Where there is necessary complexity, the server
  authors bear the brunt.

* Server side data model compatible with IMAP4 + extensions; we need to
  be able to co-exist.

* Handle disconnection more gracefully - able to cheaply resync state (I
  plan to reuse the QRESYNC/CONDSTORE logic here - the model is sane).
  Getting reliable updates on mailbox state changes should not require a
  continuous connection.

* Easy to proxy - regular syntax (UTF-8 for all communications where
  possible, but won't screw about trying to "fix" MIME - message file
  format will remain unchanged)

* NO MAGIC - everything is explicitly requested.  No UNSELECT/CLOSE/
  SELECT another mailbox special behaviours, no implicit FETCH + SET \Seen,
  no pre-calculated "UNSEEN X" that wasn't asked for. (This probably means
  some way to compose actions and fetches, but it's better than being
  complex and magical.  Remember our friends the morons.)

* COMPLETE TEST SUITE that exercises every constraint in the spec, both
  positive and negative constraints.  An implementation which passes the
  tests is compliant.  An implementation which doesn't is not.  This is
  much more likely to produce inter-operable implementations than a wordy
  spec ...

  ... if the authors want inter-operability that is.  Nothing can
  protect against a deliberate embrace and extend - though negative
  constraint tests (MUST NOT) should help.

I would love input.  I spent most of the time at FOSDEM this year talking to
other people involved in email about these ideas.  I quite like the mailbox
model behind IMAP, but I want to address the pain points I have seen so many
times over the years.

Snide comments from those who don't agree with the goals are welcome too...
I just won't listen to you.  These goals are very important to me.

Regards,

Bron.
-- 
  Bron Gondwana
  [email-masked]

From: Lyndon Nerenberg <[email-masked]>
Subject: Metadata (5654) Implementation Limits
Date: Wed, 8 Feb 2012 10:33:13 -0800
To: [email-masked]
Sender: [email-masked]

[I didn't get much response from the imap-protocol list, so I'm trying here, too.]

I'm curious to know what the implementation limits are on servers that support the Metadata extension.  I would appreciate it if the server authors out there could take a minute to fill out this survey.

I will send a summary to the list once the responses trickle off. (If you want your results to be anonymous, say so in the comments.)

Thanks,

--lyndon

Implementation name and version:
Do you support /private?:
Do you support /private/vendor?:
Do you support unsolicited Metadata responses without values?:
Do you support unsolicited Metadata responses with values?:
Maximum number of annotations per mailbox:
Maximum size of an <entry>:
Maximum size of a <value>:
Maximum depth supported in <entries>:
Other comments:

From: Bron Gondwana <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: Timo Sirainen <[email-masked]>, [email-masked]
Subject: Re: ANNOTATE-EXPERIMENT-1
Date: Mon, 13 Feb 2012 14:30:14 +0100
Sender: [email-masked]

On Mon, Feb 13, 2012, at 02:21 PM, Arnt Gulbrandsen wrote:
> On 02/13/2012 02:04 PM, Bron Gondwana wrote:
> > And of course there's normalisation as well.
> 
> Normalisation doesn't affect meaning at all. It's just changes such as
> "o umlaut-above-previous-letter" into "o-with-umlaut" or the other way.
> Useful for text processing.

Also useful for checking if a particular keyword is already set, and merging
the various forms together into a single keyword.

Bron.
-- 
  Bron Gondwana
  [email-masked]

Date: Mon, 13 Feb 2012 14:21:59 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: Bron Gondwana <[email-masked]>
Cc: Timo Sirainen <[email-masked]>, [email-masked]
Subject: Re: ANNOTATE-EXPERIMENT-1
Sender: [email-masked]

On 02/13/2012 02:04 PM, Bron Gondwana wrote:
> And of course there's normalisation as well.

Normalisation doesn't affect meaning at all. It's just changes such as
"o umlaut-above-previous-letter" into "o-with-umlaut" or the other way.
Useful for text processing.

Arnt

From: Bron Gondwana <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: Timo Sirainen <[email-masked]>, [email-masked]
Subject: Re: ANNOTATE-EXPERIMENT-1
Date: Mon, 13 Feb 2012 14:04:11 +0100
Sender: [email-masked]

On Mon, Feb 13, 2012, at 10:49 AM, Arnt Gulbrandsen wrote:
> On 02/13/2012 10:32 AM, Bron Gondwana wrote:
> > Keywords are already case insensitive.  Which is going to be a pig in poke if we try to make UTF8 keywords.
> 
> Unicode case insensitivity isn't so bad.
> 
> One rule for Turkish, one rule of the rest of the world, and anyone who
> can define i to be equal to ı can sidestep the difference. In IMAP's
> case, these two commands could be defined to be equivalent:
> 
>   a store +flags 1 istanbul
>   b store +flags 1 ıstanbul
> 
> Not pretty but also not the end of the world.
> 
> (Neat: This spell checker, which I must find out how to turn off btw,
> thinks istanbul is wrong and ıstanbul right.)

And of course there's normalisation as well.  I really don't know as much
about this area as I should.  I guess the obvious is "require best practice
as defined <over there>".

> >> But I don't mind dropping that in the least. AFAIK noone has actually
> >> wanted to sort by an annotation.
> > 
> > If nobody wants it, it doesn't matter if it's inefficient, right?
> 
> Mostly, but not quite.
> 
> A fetch item that noone ever calls can still require copy and append to
> do extra work.

True - so long as it's stored rather than calculated on request.

> >> Why would store be able to change only some entries?
> > 
> > ACLs is the obvious case.  Another thing which the Gmail guys mentioned
> > recently is "partial storage failure" - IMAP presents a view in which
> > everything is either fully functional or fully broken, but the reality
> > under the hood is not always so.
> 
> Good point.
> 
> Still, I think that's an IMAP problem. It's not made worse by covering
> all of IMAP, as opposed to all except ANNOTATE.

Agree.

Bron.
-- 
  Bron Gondwana
  [email-masked]

Date: Mon, 13 Feb 2012 10:49:44 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: Bron Gondwana <[email-masked]>
Cc: Timo Sirainen <[email-masked]>, [email-masked]
Subject: Re: ANNOTATE-EXPERIMENT-1
Sender: [email-masked]

On 02/13/2012 10:32 AM, Bron Gondwana wrote:
> Keywords are already case insensitive.  Which is going to be a pig in poke if we try to make UTF8 keywords.

Unicode case insensitivity isn't so bad.

One rule for Turkish, one rule of the rest of the world, and anyone who
can define i to be equal to ı can sidestep the difference. In IMAP's
case, these two commands could be defined to be equivalent:

  a store +flags 1 istanbul
  b store +flags 1 ıstanbul

Not pretty but also not the end of the world.

(Neat: This spell checker, which I must find out how to turn off btw,
thinks istanbul is wrong and ıstanbul right.)

>> But I don't mind dropping that in the least. AFAIK noone has actually
>> wanted to sort by an annotation.
> 
> If nobody wants it, it doesn't matter if it's inefficient, right?

Mostly, but not quite.

A fetch item that noone ever calls can still require copy and append to
do extra work.

>> Why would store be able to change only some entries?
> 
> ACLs is the obvious case.  Another thing which the Gmail guys mentioned
> recently is "partial storage failure" - IMAP presents a view in which
> everything is either fully functional or fully broken, but the reality
> under the hood is not always so.

Good point.

Still, I think that's an IMAP problem. It's not made worse by covering
all of IMAP, as opposed to all except ANNOTATE.

Arnt

From: Bron Gondwana <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>, Timo Sirainen <[email-masked]>
Cc: [email-masked]
Subject: Re: ANNOTATE-EXPERIMENT-1
Date: Mon, 13 Feb 2012 10:32:40 +0100
Sender: [email-masked]

On Mon, Feb 13, 2012, at 10:19 AM, Arnt Gulbrandsen wrote:
> 
> On 02/13/2012 01:45 AM, Timo Sirainen wrote:
> > 
> > I was thinking about finally implementing METADATA + per-mail annotations to Dovecot so I read through the RFCs. There are mainly two things I'd do differently in ANNOTATE-EXPERIMENT-2:
> > 
> > 1. METADATA has case-insensitive entry names, ANNOTATE has case-sensitive. Probably better to keep them case-insensitive in both.
> 
> I wouldn't mind.

Keywords are already case insensitive.  Which is going to be a pig in poke if we try to make UTF8 keywords.

> > 2. SORT (ANNOTATION) - is this really worth the trouble? There's really no good way for servers to implement this in an efficient way. It would pretty much require building and maintaining an index for every single annotation entry name.
> 
> Sorting at read time isn't bad, considering the very low number of
> annotations used.
> 
> But I don't mind dropping that in the least. AFAIK noone has actually
> wanted to sort by an annotation.

If nobody wants it, it doesn't matter if it's inefficient, right?

If people do want it, then you can track the annotations they are actually
using to sort by and index those... at least that's the theory behind having
adaptive indexing.

(just being contrary now!)

> > And other things I wondered about:
> > 
> >  - If APPEND has annotations, and annotation limits are reached, the whole APPEND fails with TOOBIG/TOOMANY error? I guess it has to, although that's a bit annoying.
> 
> Append is transactional. I agree it can be annoying sometimes, such as
> when ten-thousand-message multiappends break. But IMAP already has
> enough funny little exceptions, please don't try to add a rule such as
> "append is transactional except in regard to annotations".

Agree - if it's transactional, it's transactional.

> >  - ANNOTATE doesn't specify what happens if STORE can change only some of the entries. Should it rollback the changes to all the previous entries as well? Including changes to previous messages if multiple messages were specified? Would have been simpler if both METADATA and ANNOTATE supported changing only one annotation at a time.
> 
> My code is transactional, so I don't have the problem.
> 
> Why would store be able to change only some entries?

ACLs is the obvious case.  Another thing which the Gmail guys mentioned
recently is "partial storage failure" - IMAP presents a view in which
everything is either fully functional or fully broken, but the reality
under the hood is not always so.

More simply, could just be permission problems on a file somewhere, or
partial database corruption.  Shit happens.

Bron.
-- 
  Bron Gondwana
  [email-masked]

Date: Mon, 13 Feb 2012 10:19:17 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: Timo Sirainen <[email-masked]>
Cc: [email-masked]
Subject: Re: ANNOTATE-EXPERIMENT-1
Sender: [email-masked]

On 02/13/2012 01:45 AM, Timo Sirainen wrote:
> 
> I was thinking about finally implementing METADATA + per-mail annotations to Dovecot so I read through the RFCs. There are mainly two things I'd do differently in ANNOTATE-EXPERIMENT-2:
> 
> 1. METADATA has case-insensitive entry names, ANNOTATE has case-sensitive. Probably better to keep them case-insensitive in both.

I wouldn't mind.

> 2. SORT (ANNOTATION) - is this really worth the trouble? There's really no good way for servers to implement this in an efficient way. It would pretty much require building and maintaining an index for every single annotation entry name.

Sorting at read time isn't bad, considering the very low number of
annotations used.

But I don't mind dropping that in the least. AFAIK noone has actually
wanted to sort by an annotation.

> And other things I wondered about:
> 
>  - If APPEND has annotations, and annotation limits are reached, the whole APPEND fails with TOOBIG/TOOMANY error? I guess it has to, although that's a bit annoying.

Append is transactional. I agree it can be annoying sometimes, such as
when ten-thousand-message multiappends break. But IMAP already has
enough funny little exceptions, please don't try to add a rule such as
"append is transactional except in regard to annotations".

>  - ANNOTATE doesn't specify what happens if STORE can change only some of the entries. Should it rollback the changes to all the previous entries as well? Including changes to previous messages if multiple messages were specified? Would have been simpler if both METADATA and ANNOTATE supported changing only one annotation at a time.

My code is transactional, so I don't have the problem.

Why would store be able to change only some entries?

>  - TOOBIG/TOOMANY doesn't map very nicely if I want to have global "max. annotation size" and "max. annotation count". Perhaps a global max. count shouldn't exist, since the number of mails/mailboxes could be limited instead, but a global size similar to mail quota should exist. I guess when that global size is reached it just fails with TOOBIG until some annotations are deleted.

Yes.

Arnt

From: Timo Sirainen <[email-masked]>
Subject: ANNOTATE-EXPERIMENT-1
Date: Mon, 13 Feb 2012 02:45:32 +0200
To: [email-masked]
Sender: [email-masked]

I was thinking about finally implementing METADATA + per-mail annotations to Dovecot so I read through the RFCs. There are mainly two things I'd do differently in ANNOTATE-EXPERIMENT-2:

1. METADATA has case-insensitive entry names, ANNOTATE has case-sensitive. Probably better to keep them case-insensitive in both.

2. SORT (ANNOTATION) - is this really worth the trouble? There's really no good way for servers to implement this in an efficient way. It would pretty much require building and maintaining an index for every single annotation entry name.

And other things I wondered about:

 - If APPEND has annotations, and annotation limits are reached, the whole APPEND fails with TOOBIG/TOOMANY error? I guess it has to, although that's a bit annoying.

 - ANNOTATE doesn't specify what happens if STORE can change only some of the entries. Should it rollback the changes to all the previous entries as well? Including changes to previous messages if multiple messages were specified? Would have been simpler if both METADATA and ANNOTATE supported changing only one annotation at a time.

 - TOOBIG/TOOMANY doesn't map very nicely if I want to have global "max. annotation size" and "max. annotation count". Perhaps a global max. count shouldn't exist, since the number of mails/mailboxes could be limited instead, but a global size similar to mail quota should exist. I guess when that global size is reached it just fails with TOOBIG until some annotations are deleted.

From: Thomas Koch <[email-masked]>
Reply-To: [email-masked]
To: Adrien de Croy <[email-masked]>
Subject: Re: [imap5] Designing a new replacement protocol for IMAP
Date: Fri, 10 Feb 2012 15:44:50 +0100
Cc: [email-masked], Bron Gondwana <[email-masked]>, IMAP Protocol Interest List <[email-masked]>, [email-masked]
Sender: [email-masked]

Hi Adrien,

thank you for your insights. I've some more questions inline.

Adrien de Croy:
> * updates.  Would need to poll or use long polling to get real-time
> updates (e.g. notification of incoming mail).
Could Websockets or http://en.wikipedia.org/wiki/Comet_(programming) be of 
help here?

> * complex.  In order to reduce round-trips, you'd have to do many things
> in a single transaction, which would create complexity issues in the
> server and client
I was thinking restful HTTP which by definition is not complex. The question 
is, whether a useful mail fetch protocol could be designed in the constraints 
of rest.
 
> I think it would also be very problematic in the wild, since any http
> intermediary would feel entitled to mess with traffic.  E.g. heuristic
> caching, intercepting proxies etc.
There is an HTTP header to instruct any cache to pipe the request through and 
do no caching. But anybody on any line in any protocol could mess with your 
data.

Have to go now, sorry. :-)

Best regards,

Thomas Koch, http://www.koch.ro

Date: Thu, 9 Feb 2012 12:32:08 +0100
From: Bron Gondwana <[email-masked]>
To: Giovanni Panozzo <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: [imap5] Designing a new replacement protocol for IMAP
Organization: brong.net
Sender: [email-masked]

On Thu, Feb 09, 2012 at 12:25:33PM +0100, Giovanni Panozzo wrote:
> PS: I think that crossposting in 3 ML is too much. Which one will be
> the official ML for this discussion ?

(posting to all three for this...)

Let's take the rest of the traffic to just [email-masked]

Bron.

Date: Thu, 9 Feb 2012 12:24:11 +0100
From: Bron Gondwana <[email-masked]>
To: Adrien de Croy <[email-masked]>
Cc: [email-masked], [email-masked], Bron Gondwana <[email-masked]>, IMAP Protocol Interest List <[email-masked]>, [email-masked]
Subject: Re: [imap5] Designing a new replacement protocol for IMAP
Organization: brong.net
Sender: [email-masked]

On Thu, Feb 09, 2012 at 09:05:53PM +1300, Adrien de Croy wrote:
> 
> Whilst HTTP could certainly be used to put together an email system,
> I don't think it would perform well enough to replace existing
> protocols on desktops.
> 
> The main issues I see are:
> 
> * updates.  Would need to poll or use long polling to get real-time
> updates (e.g. notification of incoming mail).

Yes - long polling or "out of band" notification systems.  We have
an HTTP protocol defined kind of "on top of" IMAP plus our own
extensions here at FastMail/Opera (mail.opera.com if you want to
play with it)

We are currently using EventSource to notify the client that
"there is something new", after which the client queries to
find what's updated.

This is possible to be efficient because we use a single counter
for uidvalidity and a single counter for highestmodseq across the
entire user's folders - so any change to uidvalidity means the
folder list needs refreshing, and any change to highestmodseq
means the message view needs checking.

Combined with efficient "what's changed in this view since my
previous modseq value" queries, it's very bandwidth efficient.

> * slow.  Gut feel based on 17 years writing http proxies is that it
> would be too slow

Not really actually - so long as you can query multiple things in
a single "transaction"

> * complex.  In order to reduce round-trips, you'd have to do many
> things in a single transaction, which would create complexity issues
> in the server and client

But here's the rub.  Yes, complex.

> * protocol bloat.  if you think IMAP is bloated, try HTTP.
> * fundamentally an off-line style protocol for mail, which has
> strong incentives to be online (esp for in-office use).

Mostly facebook's "live feed" is plenty fast enough.  People are
doing all sorts of close-enough-to-real-time stuff on top of HTTP.
It doesn't mean it's the right choice, but I wouldn't dismiss it
out of hand either.

> I think it would also be very problematic in the wild, since any
> http intermediary would feel entitled to mess with traffic.  E.g.
> heuristic caching, intercepting proxies etc.

That's a problem for sure.  Mind you, proxyability is high on my
list of goals.  Immutable stuff should be cachable.

> It's almost too ubiquitous.  Everyone and their dog thinks they know
> how http works.  It's not strictly-enough defined, and has a LOT of
> optional features which aren't relevant for mail (e.g. content
> negotiation).  Interop in http is already problematic, but start
> getting people writing mail servers in php etc, and see what starts
> to happen.  I think it would be a nightmare.

Like IMAP then, except only their dogs actually understand it ;)

Good enough tests should, in theory, allow people to write their
servers in PHP if they want to.  Why you would choose to write in
PHP, I'm not sure... but still.  It takes all types.

Bron.

Date: Thu, 09 Feb 2012 21:05:53 +1300
From: Adrien de Croy <[email-masked]>
To: [email-masked]
CC: [email-masked], Bron Gondwana <[email-masked]>, IMAP Protocol Interest List <[email-masked]>, [email-masked]
Subject: Re: [imap5] Designing a new replacement protocol for IMAP
Sender: [email-masked]

Whilst HTTP could certainly be used to put together an email system, I 
don't think it would perform well enough to replace existing protocols 
on desktops.

The main issues I see are:

* updates.  Would need to poll or use long polling to get real-time 
updates (e.g. notification of incoming mail).
* slow.  Gut feel based on 17 years writing http proxies is that it 
would be too slow
* complex.  In order to reduce round-trips, you'd have to do many things 
in a single transaction, which would create complexity issues in the 
server and client
* protocol bloat.  if you think IMAP is bloated, try HTTP.
* fundamentally an off-line style protocol for mail, which has strong 
incentives to be online (esp for in-office use).

I think it would also be very problematic in the wild, since any http 
intermediary would feel entitled to mess with traffic.  E.g. heuristic 
caching, intercepting proxies etc.

It's almost too ubiquitous.  Everyone and their dog thinks they know how 
http works.  It's not strictly-enough defined, and has a LOT of optional 
features which aren't relevant for mail (e.g. content negotiation).  
Interop in http is already problematic, but start getting people writing 
mail servers in php etc, and see what starts to happen.  I think it 
would be a nightmare.

Adrien

On 9/02/2012 8:20 p.m., Thomas Koch wrote:
> Bron Gondwana:
>> Snide comments from those who don't agree with the goals are welcome too...
>> I just won't listen to you.  These goals are very important to me.
> Hi Bron,
>
> I'm not an expert in IMAP, just got here out of curiosity about Kolab[1]'s use
> of IMAP folders as data stores.
> I found some older initiatives to replace IMAP with an HTTP based approach.[2]
> What do you think of HTTP for mail access? For my bachelor thesis[3] I
> currently argue that CardDAV/CalDAV could be perfectly replaced by AtomPub
> (RFC 5023).
> The main advantage would be that most software developers on this planet have
> some vague understanding of HTTP and that imense infrastructure and software
> already exists.
>
> [1] http://wiki.kolab.org/Why_IMAP_Is_Used_For_Storage
> [2] http://www.prescod.net/rest/restmail/
>      https://www.ietf.org/mailman/listinfo/httpmail
> [3] https://github.com/thkoch2001/bachelor-thesis
>
> Best regards,
>
> Thomas Koch, http://www.koch.ro
>

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

From: Thomas Koch <[email-masked]>
Reply-To: [email-masked]
To: [email-masked]
Subject: Re: [imap5] Designing a new replacement protocol for IMAP
Date: Thu, 9 Feb 2012 08:20:27 +0100
Cc: Bron Gondwana <[email-masked]>, IMAP Protocol Interest List <[email-masked]>, [email-masked]
Sender: [email-masked]

Bron Gondwana:
> Snide comments from those who don't agree with the goals are welcome too...
> I just won't listen to you.  These goals are very important to me.
Hi Bron,

I'm not an expert in IMAP, just got here out of curiosity about Kolab[1]'s use 
of IMAP folders as data stores.
I found some older initiatives to replace IMAP with an HTTP based approach.[2] 
What do you think of HTTP for mail access? For my bachelor thesis[3] I 
currently argue that CardDAV/CalDAV could be perfectly replaced by AtomPub 
(RFC 5023).
The main advantage would be that most software developers on this planet have 
some vague understanding of HTTP and that imense infrastructure and software 
already exists.

[1] http://wiki.kolab.org/Why_IMAP_Is_Used_For_Storage
[2] http://www.prescod.net/rest/restmail/
    https://www.ietf.org/mailman/listinfo/httpmail
[3] https://github.com/thkoch2001/bachelor-thesis

Best regards,

Thomas Koch, http://www.koch.ro

From: Greg Banks <[email-masked]>
To: Lyndon Nerenberg <[email-masked]>
Subject: Re: Metadata (5654) Implementation Limits
Date: Thu, 9 Feb 2012 07:18:39 +1100
Cc: "[email-masked]" <[email-masked]>
Sender: [email-masked]

On 09/02/2012, at 5:33, Lyndon Nerenberg <[email-masked]> wrote:

>
> [I didn't get much response from the imap-protocol list, so I'm  
> trying here, too.]
>
> I'm curious to know what the implementation limits are on servers  
> that support the Metadata extension.  I would appreciate it if the  
> server authors out there could take a minute to fill out this survey.
>
> I will send a summary to the list once the responses trickle off.  
> (If you want your results to be anonymous, say so in the comments.)
>
> Thanks,
>
> --lyndon
>
> Implementation name and version:

Cyrus (METADATA support rewritten and ANNOTATE support added to future  
2.5 release).

> Do you support /private?:

Yes

> Do you support /private/vendor?:

Yes, with a subset of that space reserved for the server itself. Users  
cannot create new entries in that space, but they can create /private/ 
vendor/theirname/*.

> Do you support unsolicited Metadata responses without values?:

No

> Do you support unsolicited Metadata responses with values?:

No - no unsolicited responses.

> Maximum number of annotations per mailbox:

Not explicitly limited.

> Maximum size of an <entry>:

Not explicitly limited, but database keys include entry names and are  
limited to 4KiB.

> Maximum size of a <value>:

Not explicitly limited.

The old code had an undocumented limit of around 2KiB (iirc) at which  
it would silently truncate data in a stack buffer. The new code deals  
with values dynamically.

There is presumably a limit to the size of the database values.

For ANNOTATE support we report a 64KiB limit at SELECT time but this  
is a lie.

> Maximum depth supported in <entries>:

Not limited except by the length.

> Other comments:
>

By default, Cyrus only accepts annotations whose entries are hardcoded  
or explicitly listed in a Cyrus configuration file (which is empty by  
default). A config option allows users to set arbitrary entries.

We thought carefully about limits when implementing ANNOTATE as the  
per-message annotations pose a more extreme challenge. The solution we  
came up with was to track and limit the total size of all values of  
annotations on mailboxes and on messages, using a new QUOTA resource  
which we called X-ANNOTATION-STORAGE, with semantics similar to  
STORAGE. Per-server annotations are not tracked but can only be set by  
admin users.

Greg.

From: Bron Gondwana <[email-masked]>
To: IMAP Protocol Interest List <[email-masked]>, "Discussion on drastically slimming-down IMAP." <[email-masked]>, [email-masked]
Subject: Designing a new replacement protocol for IMAP
Date: Wed, 08 Feb 2012 21:15:26 +0100
Sender: [email-masked]

I've given up on the IMAP5 mailing list going endlessly around in circles,
though I'm definitely going to read through the backlogs and remind myself
again what everyone spoke about, because there was some excellent discussion.

So - I will build something and prove that it's good by showing improved
user experiences.

And I'm happy to keep talking here on these mailing lists too.

I've started a wiki page:

http://imapwiki.org/ReplacementProtocol

and plan to blog about what I'm doing on our own (Opera's) blogging platform,
to practice using it:

http://my.opera.com/brongondwana/blog/

I plan to make on a protocol which can replace IMAP in most of its current
uses.  I want to get (at least) two working server implementations, and
two working clients.  I have at least in-principle support from enough
project maintainers to do that.

My strong philosophical ideals are:

* Simple to build clients - morons welcome. If it's not clear the right way
  to do something to someone just reading a protocol dump, we haven't done a
  good enough job.  We can't expect you to read 30 RFCs and resolve their
  conflicts yourself before starting coding, and we won't insult you if you
  didn't get it right.  Where there is necessary complexity, the server
  authors bear the brunt.

* Server side data model compatible with IMAP4 + extensions; we need to
  be able to co-exist.

* Handle disconnection more gracefully - able to cheaply resync state (I
  plan to reuse the QRESYNC/CONDSTORE logic here - the model is sane).
  Getting reliable updates on mailbox state changes should not require a
  continuous connection.

* Easy to proxy - regular syntax (UTF-8 for all communications where
  possible, but won't screw about trying to "fix" MIME - message file
  format will remain unchanged)

* NO MAGIC - everything is explicitly requested.  No UNSELECT/CLOSE/
  SELECT another mailbox special behaviours, no implicit FETCH + SET \Seen,
  no pre-calculated "UNSEEN X" that wasn't asked for. (This probably means
  some way to compose actions and fetches, but it's better than being
  complex and magical.  Remember our friends the morons.)

* COMPLETE TEST SUITE that exercises every constraint in the spec, both
  positive and negative constraints.  An implementation which passes the
  tests is compliant.  An implementation which doesn't is not.  This is
  much more likely to produce inter-operable implementations than a wordy
  spec ...

  ... if the authors want inter-operability that is.  Nothing can
  protect against a deliberate embrace and extend - though negative
  constraint tests (MUST NOT) should help.

I would love input.  I spent most of the time at FOSDEM this year talking to
other people involved in email about these ideas.  I quite like the mailbox
model behind IMAP, but I want to address the pain points I have seen so many
times over the years.

Snide comments from those who don't agree with the goals are welcome too...
I just won't listen to you.  These goals are very important to me.

Regards,

Bron.
-- 
  Bron Gondwana
  [email-masked]

From: Lyndon Nerenberg <[email-masked]>
Subject: Metadata (5654) Implementation Limits
Date: Wed, 8 Feb 2012 10:33:13 -0800
To: [email-masked]
Sender: [email-masked]

[I didn't get much response from the imap-protocol list, so I'm trying here, too.]

I'm curious to know what the implementation limits are on servers that support the Metadata extension.  I would appreciate it if the server authors out there could take a minute to fill out this survey.

I will send a summary to the list once the responses trickle off. (If you want your results to be anonymous, say so in the comments.)

Thanks,

--lyndon

Implementation name and version:
Do you support /private?:
Do you support /private/vendor?:
Do you support unsolicited Metadata responses without values?:
Do you support unsolicited Metadata responses with values?:
Maximum number of annotations per mailbox:
Maximum size of an <entry>:
Maximum size of a <value>:
Maximum depth supported in <entries>:
Other comments: