Subject: [imap5] Designing a new replacement protocol for IMAP

From: Bron Gondwana <[email-masked]>
Date: Wed, 08 Feb 2012 21:15:26 +0100

I've given up on the IMAP5 mailing list going endlessly around in circles,
though I'm definitely going to read through the backlogs and remind myself
again what everyone spoke about, because there was some excellent discussion.

So - I will build something and prove that it's good by showing improved
user experiences.

And I'm happy to keep talking here on these mailing lists too.

I've started a wiki page:

http://imapwiki.org/ReplacementProtocol

and plan to blog about what I'm doing on our own (Opera's) blogging platform,
to practice using it:

http://my.opera.com/brongondwana/blog/

I plan to make on a protocol which can replace IMAP in most of its current
uses.  I want to get (at least) two working server implementations, and
two working clients.  I have at least in-principle support from enough
project maintainers to do that.

My strong philosophical ideals are:

* Simple to build clients - morons welcome. If it's not clear the right way
  to do something to someone just reading a protocol dump, we haven't done a
  good enough job.  We can't expect you to read 30 RFCs and resolve their
  conflicts yourself before starting coding, and we won't insult you if you
  didn't get it right.  Where there is necessary complexity, the server
  authors bear the brunt.

* Server side data model compatible with IMAP4 + extensions; we need to
  be able to co-exist.

* Handle disconnection more gracefully - able to cheaply resync state (I
  plan to reuse the QRESYNC/CONDSTORE logic here - the model is sane).
  Getting reliable updates on mailbox state changes should not require a
  continuous connection.

* Easy to proxy - regular syntax (UTF-8 for all communications where
  possible, but won't screw about trying to "fix" MIME - message file
  format will remain unchanged)

* NO MAGIC - everything is explicitly requested.  No UNSELECT/CLOSE/
  SELECT another mailbox special behaviours, no implicit FETCH + SET \Seen,
  no pre-calculated "UNSEEN X" that wasn't asked for. (This probably means
  some way to compose actions and fetches, but it's better than being
  complex and magical.  Remember our friends the morons.)

* COMPLETE TEST SUITE that exercises every constraint in the spec, both
  positive and negative constraints.  An implementation which passes the
  tests is compliant.  An implementation which doesn't is not.  This is
  much more likely to produce inter-operable implementations than a wordy
  spec ...

  ... if the authors want inter-operability that is.  Nothing can
  protect against a deliberate embrace and extend - though negative
  constraint tests (MUST NOT) should help.

I would love input.  I spent most of the time at FOSDEM this year talking to
other people involved in email about these ideas.  I quite like the mailbox
model behind IMAP, but I want to address the pain points I have seen so many
times over the years.

Snide comments from those who don't agree with the goals are welcome too...
I just won't listen to you.  These goals are very important to me.

Regards,

Bron.
-- 
  Bron Gondwana
  [email-masked]

---

From: Thomas Koch <[email-masked]>
Date: Thu, 9 Feb 2012 08:20:27 +0100

> I just won't listen to you.  These goals are very important to me.
Hi Bron,

I'm not an expert in IMAP, just got here out of curiosity about Kolab[1]'s use 
of IMAP folders as data stores.
I found some older initiatives to replace IMAP with an HTTP based approach.[2] 
What do you think of HTTP for mail access? For my bachelor thesis[3] I 
currently argue that CardDAV/CalDAV could be perfectly replaced by AtomPub 
(RFC 5023).
The main advantage would be that most software developers on this planet have 
some vague understanding of HTTP and that imense infrastructure and software 
already exists.

[1] http://wiki.kolab.org/Why_IMAP_Is_Used_For_Storage
[2] http://www.prescod.net/rest/restmail/
    https://www.ietf.org/mailman/listinfo/httpmail
[3] https://github.com/thkoch2001/bachelor-thesis

Best regards,

Thomas Koch, http://www.koch.ro

---

From: Bron Gondwana <[email-masked]>
Date: Thu, 9 Feb 2012 12:24:11 +0100

On Thu, Feb 09, 2012 at 09:05:53PM +1300, Adrien de Croy wrote:
> 
> Whilst HTTP could certainly be used to put together an email system,
> I don't think it would perform well enough to replace existing
> protocols on desktops.
> 
> The main issues I see are:
> 
> * updates.  Would need to poll or use long polling to get real-time
> updates (e.g. notification of incoming mail).

Yes - long polling or "out of band" notification systems.  We have
an HTTP protocol defined kind of "on top of" IMAP plus our own
extensions here at FastMail/Opera (mail.opera.com if you want to
play with it)

We are currently using EventSource to notify the client that
"there is something new", after which the client queries to
find what's updated.

This is possible to be efficient because we use a single counter
for uidvalidity and a single counter for highestmodseq across the
entire user's folders - so any change to uidvalidity means the
folder list needs refreshing, and any change to highestmodseq
means the message view needs checking.

Combined with efficient "what's changed in this view since my
previous modseq value" queries, it's very bandwidth efficient.

> * slow.  Gut feel based on 17 years writing http proxies is that it
> would be too slow

Not really actually - so long as you can query multiple things in
a single "transaction"

> * complex.  In order to reduce round-trips, you'd have to do many
> things in a single transaction, which would create complexity issues
> in the server and client

But here's the rub.  Yes, complex.

> * protocol bloat.  if you think IMAP is bloated, try HTTP.
> * fundamentally an off-line style protocol for mail, which has
> strong incentives to be online (esp for in-office use).

Mostly facebook's "live feed" is plenty fast enough.  People are
doing all sorts of close-enough-to-real-time stuff on top of HTTP.
It doesn't mean it's the right choice, but I wouldn't dismiss it
out of hand either.

> I think it would also be very problematic in the wild, since any
> http intermediary would feel entitled to mess with traffic.  E.g.
> heuristic caching, intercepting proxies etc.

That's a problem for sure.  Mind you, proxyability is high on my
list of goals.  Immutable stuff should be cachable.

> It's almost too ubiquitous.  Everyone and their dog thinks they know
> how http works.  It's not strictly-enough defined, and has a LOT of
> optional features which aren't relevant for mail (e.g. content
> negotiation).  Interop in http is already problematic, but start
> getting people writing mail servers in php etc, and see what starts
> to happen.  I think it would be a nightmare.

Like IMAP then, except only their dogs actually understand it ;)

Good enough tests should, in theory, allow people to write their
servers in PHP if they want to.  Why you would choose to write in
PHP, I'm not sure... but still.  It takes all types.

Bron.

---

From: Giovanni Panozzo <[email-masked]>
Date: Thu, 09 Feb 2012 12:25:33 +0100

> some vague understanding of HTTP and that imense infrastructure and software

Hi Bron, hi Thomas.
I'm happy to see this interesting traffic on the ML.

Just a small note: these two post are full of "programmer point of view" 
considerations and philosophical ideals. That's good.

But I'm mostly "system integrator", involved also with end-user 
helpdesk. I'm a programmer also, but only form 2% of my time, so I'm not 
a "professional programmer".
I would like to contribute with other key points useful both to system 
integrators, helpdesk and end user (moron end user) ease to use (I have 
a long list... do you want it ? :))

Here are my first notes about HTTP: I think HTTP is welcome. Many mobile 
phone carriers do offer only http/https connectivity, sometimes filtered 
by a transparent proxy too. In some enterprise LANs, users are forced to 
use an http proxy with proxy authentication to access the Internet. No 
other protocols are allowed. HTTP would allow all these users to access 
their e-mail without requiring impossible access upgrades.
And HTTP completed with digest authentication and email body encryption 
also will also help sysadmins to have more security without incurring 
into the SSL certificate infrastructure pains.
Http is also "stateless" from its birth: the advantage is that we can 
better serve clients with intermittent connections (point 3 in Bron's 
list), like mobile phones or poorly connected wifi laptops.

Giovanni

PS: I think that crossposting in 3 ML is too much. Which one will be the 
official ML for this discussion ?

---

From: Bron Gondwana <[email-masked]>
Date: Thu, 9 Feb 2012 12:32:08 +0100

On Thu, Feb 09, 2012 at 12:25:33PM +0100, Giovanni Panozzo wrote:
> PS: I think that crossposting in 3 ML is too much. Which one will be
> the official ML for this discussion ?

(posting to all three for this...)

Let's take the rest of the traffic to just [email-masked]

Bron.

---

From: Bron Gondwana <[email-masked]>
Date: Thu, 9 Feb 2012 12:42:44 +0100

On Thu, Feb 09, 2012 at 12:25:33PM +0100, Giovanni Panozzo wrote:
> Just a small note: these two post are full of "programmer point of
> view" considerations and philosophical ideals. That's good.
> 
> helpdesk. I'm a programmer also, but only form 2% of my time, so I'm
> not a "professional programmer".
> I would like to contribute with other key points useful both to
> system integrators, helpdesk and end user (moron end user) ease to
> use (I have a long list... do you want it ? :))

Yep!  I do all of those too, though programming is a larger part of
my week.  There's a lot of sysadmin and user operations - and third
level support.  I'm not doing much first level any more.

> Here are my first notes about HTTP: I think HTTP is welcome. Many
> mobile phone carriers do offer only http/https connectivity,
> sometimes filtered by a transparent proxy too. In some enterprise
> LANs, users are forced to use an http proxy with proxy
> authentication to access the Internet. No other protocols are
> allowed. HTTP would allow all these users to access their e-mail
> without requiring impossible access upgrades.
> And HTTP completed with digest authentication and email body
> encryption also will also help sysadmins to have more security
> without incurring into the SSL certificate infrastructure pains.
> Http is also "stateless" from its birth: the advantage is that we
> can better serve clients with intermittent connections (point 3 in
> Bron's list), like mobile phones or poorly connected wifi laptops.

That's certainly a strong point in HTTP's favour.  I'm going to spend
some time on the wiki tonight (Oslo time) building up a list of the
major technology choices and the advantages and disadvantages of each
one.  Protocols will basically be:

* Custom
* HTTP
* XMPP

And within HTTP: XML / JSON / other?

RESTful URIs vs batching...

One important point - I don't want to lose "literals".  RFC822 messages
should be able to zero-copy IO across the wire as the raw bytes.

Bron.

---

From: "Jeroen van Meeuwen (Kolab Systems)" <[email-masked]>
Date: Thu, 09 Feb 2012 13:38:35 +0100

On 2012-02-09 8:20, Thomas Koch wrote:
> I'm not an expert in IMAP, just got here out of curiosity about
> Kolab[1]'s use of IMAP folders as data stores.

Please, rest of list, let's not view this article as authoritative 
representation of the position of the current Kolab Groupware 
development drivers, whom are currently primarily the colleagues at the 
company I work for.

Kind regards,

Jeroen van Meeuwen

-- 
Systems Architect, Kolab Systems AG

e: vanmeeuwen at kolabsys.com
t: +44 144 340 9500
m: +44 74 2516 3817
w: http://www.kolabsys.com

pgp: 9342 BF08

---

From: Dave Cridland <[email-masked]>
Date: Thu, 09 Feb 2012 12:50:44 +0000

> Bron's list), like mobile phones or poorly connected wifi laptops.

This one's a myth.

We've had BOSH in the XMPP world for a while, and although it's used  
in the web browser, it's typically avoided in mobile because it ramps  
up the power requirements quite heavily, and turns out to to no more  
effective than the newer XEP-0198 anyway.

It'd be much better for web clients to run actual IMAP over either  
BOSH or Websocket. The difficulty with this approach is twofold:

1) IMAP transfers binary data in-line, and Javascript simply doesn't  
cope well with this.

2) Parsing IMAP in Javascript means parsing it with a pure Javascript  
parser, which will be slower and more prone to error. Using XML or  
JSON would be better here.

To address both, I think I could build a useful IMAP replacement that  
ran the metadata, push, and folder management over XMPP, and the  
content access over HTTP, quite easily. It's almost tempting, and is  
at least as perverse as the other proposals.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 9 Feb 2012 14:41:16 +0100

On 02/09/2012 01:50 PM, Dave Cridland wrote:
> ran the metadata, push, and folder management over XMPP, and the content
> access over HTTP, quite easily.

If you wish to be faithful to history, then the right thing would be to
run that on TCP ports 20 and 21. Ahem.

Arnt

---

From: Dave Cridland <[email-masked]>
Date: Thu, 09 Feb 2012 13:51:06 +0000

On Thu Feb  9 13:41:16 2012, Arnt Gulbrandsen wrote:
> content

Quite - the design is not far off.

Obviously if we did use port 20, then we could also send IAC WILL  
RANDOMLY_LOSE, and come full circle.

Dave.

---

From: Thomas Koch <[email-masked]>
Date: Fri, 10 Feb 2012 15:44:50 +0100

Hi Adrien,

thank you for your insights. I've some more questions inline.

> updates (e.g. notification of incoming mail).
Could Websockets or http://en.wikipedia.org/wiki/Comet_(programming) be of 
help here?

> * complex.  In order to reduce round-trips, you'd have to do many things
> in a single transaction, which would create complexity issues in the
> server and client
I was thinking restful HTTP which by definition is not complex. The question 
is, whether a useful mail fetch protocol could be designed in the constraints 
of rest.
 
> I think it would also be very problematic in the wild, since any http
> intermediary would feel entitled to mess with traffic.  E.g. heuristic
> caching, intercepting proxies etc.
There is an HTTP header to instruct any cache to pipe the request through and 
do no caching. But anybody on any line in any protocol could mess with your 
data.

Have to go now, sorry. :-)

Best regards,

---

From: Cyrus Daboo <[email-masked]>
Date: Fri, 10 Feb 2012 10:09:48 -0500

Hi Thomas,

--On February 10, 2012 3:44:50 PM +0100 Thomas Koch <[email-masked]> wrote:

> Could Websockets or http://en.wikipedia.org/wiki/Comet_(programming) be
> of  help here?

For CalDAV and CardDAV we (Apple, http://calendarserver.org) have used XMPP 
pubsub and our own APNS to implement a push notification service. Frankly I 
think the IETF should have developed a proper notification protocol for use 
alongside other protocols a long time ago - simply "blessing" XMPP pubsub 
as the solution for that would be fine from a standards point, but perhaps 
something else (new) coukld be done too. What then needs to be defined are 
the hooks in each underlying protocol to allow clients to discover the 
appropriate pubsub service. In CalDAV/CardDAV that is simply done using 
WebDAV properties to advertise the essential client push configuration.

So, in my opinion, whilst push notifications should be a requirement for 
imap5, we should not define that protocol and instead push the IETF to 
provide such a protocol for general use.

-- 
Cyrus Daboo

---

From: Filip Navara <[email-masked]>
Date: Fri, 10 Feb 2012 16:29:25 +0100

On Fri, Feb 10, 2012 at 4:09 PM, Cyrus Daboo <[email-masked]> wrote:

>> Could Websockets or http://en.wikipedia.org/wiki/**Comet_(programming)<http://en.wikipedia.org/wiki/Comet_(programming)>be
>> of  help here?
>>
>
> For CalDAV and CardDAV we (Apple, http://calendarserver.org) have used
> XMPP pubsub and our own APNS to implement a push notification service.
> Frankly I think the IETF should have developed a proper notification
> protocol for use alongside other protocols a long time ago - simply
> "blessing" XMPP pubsub as the solution for that would be fine from a
> standards point, but perhaps something else (new) coukld be done too. What
> then needs to be defined are the hooks in each underlying protocol to allow
> clients to discover the appropriate pubsub service. In CalDAV/CardDAV that
> is simply done using WebDAV properties to advertise the essential client
> push configuration.
>

There already exists a standardized way to implement XMPP notifications on
IMAP, although it is a bit cumbersome and I doubt anyone ever implemented
it.

http://tools.ietf.org/html/rfc5435 (Sieve Notify)
http://tools.ietf.org/html/rfc
<http://tools.ietf.org/html/rfc5435>5550<http://www.rfc-editor.org/rfc/rfc5550.txt>
(Lemonade
IMAP profile)

Best regards,
Filip Navara

---

From: Cyrus Daboo <[email-masked]>
Date: Fri, 10 Feb 2012 10:48:32 -0500

Hi Filip,

--On February 10, 2012 4:29:25 PM +0100 Filip Navara 
<[email-masked]> wrote:

> There already exists a standardized way to implement XMPP notifications
> on IMAP, although it is a bit cumbersome and I doubt anyone ever
> implemented it. 
>
>
> http://tools.ietf.org/html/rfc5550 (Lemonade IMAP profile)
>

The SIEVE notify solution only covers the case of message delivery into a 
mailstore (be it IMAP or POP). It won't cover the case of changes made 
directly to that store (e.g. and IMAP APPEND). Now there is the IMAP-SIEVE 
draft (<http://tools.ietf.org/id/draft-ietf-sieve-imap-sieve-02.txt>) which 
in theory allows notifications to be added to IMAP. That may be overboard 
for what is required here and it does not cover what needs to be done in 
other protocols.

-- 
Cyrus Daboo

---

From: Bron Gondwana <[email-masked]>
Date: Fri, 10 Feb 2012 19:22:42 +0100

On Fri, Feb 10, 2012 at 10:09:48AM -0500, Cyrus Daboo wrote:
> For CalDAV and CardDAV we (Apple, http://calendarserver.org) have
> used XMPP pubsub and our own APNS to implement a push notification
> service. Frankly I think the IETF should have developed a proper
> notification protocol for use alongside other protocols a long time
> ago - simply "blessing" XMPP pubsub as the solution for that would
> be fine from a standards point, but perhaps something else (new)
> coukld be done too. What then needs to be defined are the hooks in
> each underlying protocol to allow clients to discover the
> appropriate pubsub service. In CalDAV/CardDAV that is simply done
> using WebDAV properties to advertise the essential client push
> configuration.

I really should make sure we're compatible with what you're doing
too.  Obviously, there's a tradeoff between being able to do
everything in one protocol vs simplicity and saving the wheels
from being reinvented all over again.

I really appreciate your input here - you've got some good points,
and an actual implementation of something interesting, which trumps
talking every time!

> So, in my opinion, whilst push notifications should be a requirement
> for imap5, we should not define that protocol and instead push the
> IETF to provide such a protocol for general use.

My plan is to define in-band push if there's an active TCP connection,
plus out-of-band notifications.  Either way they would be of the
"something has changed, go request an update to the views you are
interested in"...

What we've done at mail.opera.com for a web-based system is to
use eventsource to push just a "here's a new modseq for you" to
the client, after which it can request a diff against the previous
view.  This works because we use a single HIGHESTMODSEQ counter
across all the users' mailboxes.

Obviously, it doesn't scale so nicely to shared folders.  For this
I'm seriously considering a CHANGEROOT or similar, which would be
a tree of related folders in which modseqs are serialised.  For a
64 bit counter, there's probably actually no real problem with
updating them across the entire server - other than then having to
filter the change notifications so you don't generate a zillion
spurious "something may have been updated" notification to every
client.

Bron.

---

From: Adrien de Croy <[email-masked]>
Date: Mon, 13 Feb 2012 10:56:49 +1300

On 11/02/2012 4:09 a.m., Cyrus Daboo wrote:
> standards point, but perhaps something else (new) coukld be done too. 
> What then needs to be defined are the hooks in each underlying 
> protocol to allow clients to discover the appropriate pubsub service. 
> In CalDAV/CardDAV that is simply done using WebDAV properties to 
> advertise the essential client push configuration.
>

I don't think that's a workable approach.

Getting such a protocol together, which enables notifications from any 
other application protocol I think will take a very long time, if it can 
even succeed.  It's hard enough getting consensus within one protocol 
working group, let alone all of them working together.

Also every different protocol has different notification requirements.  
trying to cover all that in a single protocol I think would be difficult.

Finally, one of the things I am keen to see disappear with IMAP5 (for 
want of a better name) is the requirement for clients, admins etc to 
deploy multiple protocols - e.g. allow submission in IMAP.  This is 
because it's frankly a large (and should be unnecessary) admin burden 
trying to support 2 protocols for mail (e.g. SMTP + POP3/IMAP4).  Mail 
users don't understand the significance, they just submit support 
tickets because they can retrieve mail but not send.  Firewalls block 
ports, ISPs intercept / block port 25 (NZ largest ISP does this), 
multiple protocols = multiple realms for credentials, multiple 
implementations of authentication and authorization etc.

Having to have another protocol to implement to get notifications back 
probably over another channel etc makes this issue worse not better.

Adrien

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com
WinGate 7 is released! - http://www.wingate.com/getlatest/

---

From: Bron Gondwana <[email-masked]>
Date: Sun, 12 Feb 2012 23:08:56 +0100

On Mon, Feb 13, 2012 at 10:56:49AM +1300, Adrien de Croy wrote:
> Finally, one of the things I am keen to see disappear with IMAP5
> (for want of a better name) is the requirement for clients, admins
> etc to deploy multiple protocols - e.g. allow submission in IMAP.
> This is because it's frankly a large (and should be unnecessary)
> admin burden trying to support 2 protocols for mail (e.g. SMTP +
> POP3/IMAP4).  Mail users don't understand the significance, they
> just submit support tickets because they can retrieve mail but not
> send.  Firewalls block ports, ISPs intercept / block port 25 (NZ
> largest ISP does this), multiple protocols = multiple realms for
> credentials, multiple implementations of authentication and
> authorization etc.

Absolutely agree.  There has to be a way to do basic "client
wants to upload a message to Sent (possibly upload it to
Drafts first and then move it to Sent, with a copy going
out to the destination too) within the one protocol.  If
you're doing something more fancy, there's still SMTP.

> Having to have another protocol to implement to get notifications
> back probably over another channel etc makes this issue worse not
> better.

Similarly, there needs to be a way to do notifications in-protocol.
I still think that being compatible with an external protocol as
well is good.  Either way the notification should need to be nothing
more than a new HIGHESTMODSEQ value from which the client can update
whichever views it's interested in.

To reduce roundtrips, it would be nice to register a view on which
you want updates pushed rather than just getting a new modseq and
having to ask for the updates... but that's a minor optimisation
compared to having to run a "STATUS loop" over all your folders or
open hundreds of separate TCP connections to get information.

Bron.

---

From: Cyrus Daboo <[email-masked]>
Date: Mon, 13 Feb 2012 10:20:45 -0500

Hi Adrien,

> trying to cover all that in a single protocol I think would be difficult.

I disagree because what I envisage for the generic notification service is 
an OS-level api (supplied by OS or 3rd party libraries) that implement the 
"internet push service protocol". What that means is that client developers 
only have to implement a simple api to get push notifications. What is more 
anyone implementing more than one protocol in their client (e.g. IMAP, 
CalDAV and CardDAV) would only have to implement that once, albeit with 
some minor differences in regard to how to get protocol specific pieces for 
registering for notifications.

Your point about actually getting this done is valid. But realistically, do 
you really think IMAP5 is going to deploy overnight? Frankly, anyone who 
has a reasonably solid IMAP4 implementation today is going to question the 
need to work on something new, particularly if that something new brings 
nothing to the table (and simply fixing interop problems will probably not 
be seen as something "new"). If you can actually show that IMAP5 adds 
significant value by doing things like helping centralize push 
notifications, simplifying submission etc, then maybe, just maybe, those 
existing implementors might actually consider throwing out their current 
investment in IMAP4 for the new thing. But, IMHO, you are really going to 
have to up-sell IMAP5 to get buy-in from the major email providers. Now 
that does not mean it is not worth doing, but it does mean having to do 
more than just fixing perceived or real deficiencies.

-- 
Cyrus Daboo

---

From: Bron Gondwana <[email-masked]>
Date: Mon, 13 Feb 2012 17:23:33 +0100

On Mon, Feb 13, 2012 at 10:20:45AM -0500, Cyrus Daboo wrote:
> I disagree because what I envisage for the generic notification
> service is an OS-level api (supplied by OS or 3rd party libraries)
> that implement the "internet push service protocol". What that means
> is that client developers only have to implement a simple api to get
> push notifications. What is more anyone implementing more than one
> protocol in their client (e.g. IMAP, CalDAV and CardDAV) would only
> have to implement that once, albeit with some minor differences in
> regard to how to get protocol specific pieces for registering for
> notifications.

That's my vision too.  Ideally not pushing a full data feed, but
just a "ping, there's new stuff for you".  Otherwise you wind up
embedding every other protocol inside this protocol, which is
too meta, even for me.

> Your point about actually getting this done is valid. But
> realistically, do you really think IMAP5 is going to deploy
> overnight? Frankly, anyone who has a reasonably solid IMAP4
> implementation today is going to question the need to work on
> something new, particularly if that something new brings nothing to
> the table (and simply fixing interop problems will probably not be
> seen as something "new"). If you can actually show that IMAP5 adds
> notifications, simplifying submission etc, then maybe, just maybe,
> those existing implementors might actually consider throwing out
> their current investment in IMAP4 for the new thing. But, IMHO, you
> are really going to have to up-sell IMAP5 to get buy-in from the
> major email providers. Now that does not mean it is not worth doing,
> but it does mean having to do more than just fixing perceived or
> real deficiencies.

Absolutely agree.  We need a hook to make it worthwhile, and I
think submission and reliable push notifications are the hook.

I've updated my RFC list from a simple text file which generates
the list.  Makes it easier to add new items :)  Of course it means
I miss any changes that anyone else makes to the wiki now - so I'll
have to write a reverse parser too so I can see a diff first.

The tricky bit is that the providers for whom it's going to be the
most work are the ones who aren't already doing the LEMONADE profile,
since that requires the sort of datastructures that would support
the things I want to do with re-syncable state.

Bron.

---

From: Anil SRIVASTAVA <[email-masked]>
Date: Mon, 13 Feb 2012 09:57:30 -0800

+1.  I could not have said this better.

Anil

On Mon Feb 13 2012 07:20:45 GMT-0800 (PST), Cyrus Daboo
<[email-masked]> wrote:
> Hi Adrien,
>
> --On February 13, 2012 10:56:49 AM +1300 Adrien de Croy
> <[email-masked]> wrote:
>
> realistically, do you really think IMAP5 is going to deploy overnight?
> Frankly, anyone who has a reasonably solid IMAP4 implementation today
> is going to question the need to work on something new, particularly
> if that something new brings nothing to the table (and simply fixing
> interop problems will probably not be seen as something "new"). If you
> can actually show that IMAP5 adds significant value by doing things
> like helping centralize push notifications, simplifying submission
> etc, then maybe, just maybe, those existing implementors might
> actually consider throwing out their current investment in IMAP4 for
> the new thing. But, IMHO, you are really going to have to up-sell
> IMAP5 to get buy-in from the major email providers. Now that does not
> mean it is not worth doing, but it does mean having to do more than
> just fixing perceived or real deficiencies.
>

-- 
~Anil SRIVASTAVA

---

From: Mark Crispin <[email-masked]>
Date: Mon, 13 Feb 2012 10:55:26 -0800 (PST)

I will say this once. Then I will go back to rolling on the floor laughing
at the extremely naive things that certain individuals post here.

On Mon Feb 13 2012 07:20:45 GMT-0800 (PST), Cyrus Daboo wrote:
> What is more anyone implementing more than one

This should be self-evident, as is this statement:

> interop problems will probably not be seen as something "new").

The IMAP5 talk is stunning and laughable in two ways. One is it that it
seeks to make IMAP "simpler". The other is that it seeks to expand its
scope to be Exchange.

Unstated, but evident, is a third aspect: the naive notion that a protocol
can be replaced without repeating the work that went into the original.
History shows that the effort in any replacement is quite a bit greater.

You can't escape that work no matter smart you think you are, or how much
better your knowledge and tools are.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 13 Feb 2012 19:00:48 +0000

> On Mon, Feb 13, 2012 at 10:20:45AM -0500, Cyrus Daboo wrote:
  [...]
>> their current investment in IMAP4 for the new thing.
I agree. But I think another important point is that Bron not suggesting 
to throw away everything and start from scratch. Maybe it is just me, 
but I am hoping that if this effort takes off, then lots of 
capabilities/commands of IMAP4+extensions will be preserved as is, so 
that upgrading an existing IMAP4 implementation would be relatively easy.
>> But, IMHO, you

---

From: Bron Gondwana <[email-masked]>
Date: Mon, 13 Feb 2012 20:42:22 +0100

On Mon, Feb 13, 2012 at 07:00:48PM +0000, Alexey Melnikov wrote:
> I agree. But I think another important point is that Bron not
> suggesting to throw away everything and start from scratch. Maybe it
> is just me, but I am hoping that if this effort takes off, then lots
> of capabilities/commands of IMAP4+extensions will be preserved as
> is, so that upgrading an existing IMAP4 implementation would be
> relatively easy.

Absolutely preserved in semantics if not in syntax.  The semantics
are pretty good, mostly - and the ones that I don't like are mostly
the "magic" bits.

Being able to plug this in as another access method to an already
existing IMAP store is a very important property - at most it
would require a little extra indexing if you want to be efficient.

Even more ideally, you could implement a connector either way at
a loss of efficiency - an (at least basic) IMAP connector in front
of this protocol, or a proxy for this protocol (caching quite a lot
more information probably - particularly if the backend didn't have
a concept of MODSEQ) in front of an existing IMAP store.

Bron.

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 09 Feb 2012 21:05:53 +1300

Whilst HTTP could certainly be used to put together an email system, I 
don't think it would perform well enough to replace existing protocols 
on desktops.

* updates.  Would need to poll or use long polling to get real-time 
updates (e.g. notification of incoming mail).
* slow.  Gut feel based on 17 years writing http proxies is that it 
would be too slow
* complex.  In order to reduce round-trips, you'd have to do many things 
in a single transaction, which would create complexity issues in the 
server and client
* protocol bloat.  if you think IMAP is bloated, try HTTP.
* fundamentally an off-line style protocol for mail, which has strong 
incentives to be online (esp for in-office use).

It's almost too ubiquitous.  Everyone and their dog thinks they know how 
http works.  It's not strictly-enough defined, and has a LOT of optional 
features which aren't relevant for mail (e.g. content negotiation).  
Interop in http is already problematic, but start getting people writing 
mail servers in php etc, and see what starts to happen.  I think it 
would be a nightmare.

Adrien

---

From: Adrien de Croy <[email-masked]>
Date: Fri, 10 Feb 2012 00:45:56 +1300

> extensions here at FastMail/Opera (mail.opera.com if you want to

if you compare the overhead / complexity in setting up the poll (on a 
separate connection), and re-sending the poll request each time it's 
completed by the server (or timed out by an intermediary etc), it's a 
particularly inelegant and wasteful way to get a notification.

It's done this way in HTTP because that's the only way to do it, but 
let's face it, it's a fairly ugly hack.

> It doesn't mean it's the right choice, but I wouldn't dismiss it
it might seem quick enough for a human, but not all mail clients are 
humans.  E.g. EDI systems etc.

I wouldn't dismiss HTTP out of hand either, but I would take some 
convincing, and the work that would be required to specify the requests 
and responses (would you use SOAP-RPC?) would be not much less (if any) 
than that required to do a nice slim efficient and blindingly quick 
specialised protocol.

In fact, since mobile devices are going to probably be the primary 
agents for mail, we really should be looking at a binary protocol?

Just not ASN.1 BER :)

> list of goals.  Immutable stuff should be cachable.

caches impose their own beliefs on what is cachable and for how long.

> PHP, I'm not sure... but still.  It takes all types.

it sure does :)

Adrien

---

From: Bron Gondwana <[email-masked]>
Date: Mon, 13 Feb 2012 20:52:53 +0100

> at the extremely naive things that certain individuals post here.

Go for it.  I said the comments of haters who want to hate were
welcome too, and I stand by it.

> seeks to make IMAP "simpler". The other is that it seeks to expand its

Simpler as in more regular data model, removing the unnecessary
complexity, not the inherent complexity.  I believe there is a
reasonable pool of unnecessary complexity involved in both ends
of IMAP which isn't inherent in the task of viewing and managing
email, but is instead caused by the layers of protocol on top -
both the irregular syntax of the different layers, and the
MSGNO model with its artifacts like 'UNSEEN' which don't match
the way in which email is being presented and managed by existing
clients today.

And then there's the need to support even more parsers and
protocol agents for SIEVE, ACAP, and good old SMTP/Submission,
plus the support headaches involved in firewall issues.  For the
99% case of a client wanting to send an email via the same
infrastructure where it stores its email, a basic submission will
be sufficient.

All of which I've said plenty of times before.  I've even added
xmove and xsend to Cyrus now.  XMOVE is already in use for our
own web interface code, removing all the quota issues we used to
have with the "Delete to Trash" model, and the crazy workarounds
like deleting a small number of messages at a time.  There's a
draft in progress, and I made sure our implementation is compatible
with all the MUSTs.

> History shows that the effort in any replacement is quite a bit greater.

I'm expecting to do a lot of work.  I'm not afraid of work.  I'm
afraid of getting so bogged down saying it can't be done that we
waste another 10 years.

> better your knowledge and tools are.

You can escape the worst parts of the work by using what already
exists, and by not re-inventing every wheel along the way, but by
looking at some of the wheels that are a lot smoother than they
were 20 years ago.

I would prefer to have you involved than sitting on the sidelines
laughing, but I'd prefer to do something than listen to you saying
it can't be done.  Maybe I'll burn out along the way, maybe not.
I'm still going to try.

Bron.

---

From: Mark Crispin <[email-masked]>
Date: Mon, 13 Feb 2012 12:26:53 -0800 (PST)

> welcome too, and I stand by it.

Bwahahahaha! Complaining about "haters" just like a 13-year-old girl!

If you really were competent enough to do what you purpose to do, you
would babble less about "haters" (and public proclamations of what you
purpose to do), and instead go off quietly on your own to do it. You'd
also stay quiet about it until you had something concrete to show for it.

"Concrete" means a written specification and interoperable code, both of
which can be reviewed. Then it goes the entire IETF process, in which
everybody and his grandmother weights in on how to "improve" what you have
done (so much for "regularity" and "consistency" as each "improvement"
goes in). Then, and only then, you try to convince the Big Players that
they is a compelling reason why they should adopt it instead of IMAP.

I know how long it took me, and I wasn't trying to replace something that
was already widely deployed.

-- Mark --

http://panda.com/mrc

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 14 Feb 2012 09:26:58 +1300

Hi Cyrus

I guess we are on different pages for what we want out of a mail 
protocol and where we want it to go.  Sorry if this mail rambles a bit.

Also taking into account Mark and others' comments.

Maybe we need to start with what we want to achieve... which actually 
Bron did do at the start.  All of us have various different histories 
when it comes to mail.

When I first implemented our IMAP server, I was firstly struck by the 
unusual syntax, parsing requirements, and then the missing bits - lack 
of completely defined folder management protocol support, and the highly 
problematic dual-indexing (no UID in expunge response!!!).   History 
obviously plays a huge part, but it's not 1998 any more.

Then I was hit with the quirks of the implementations (e.g. TB ignores 
unsolicited responses unless it issued an IDLE).  I've spent a lot of 
time evaluating IMAP clients, none of them are what I would call great 
(for my use).

In terms of features, it seems over half the feature set nowadays is 
optional protocol extensions, which basically means no-one can count on 
them being there.

So, I'm approaching this whole thing from a mail-user perspective, since 
I also am one.  I think usability could be greatly improved if the 
protocol supported it.

My issue with XMPP isn't with the protocol per-se, but just that if 
you're outside a firewall, someone needs to open another hole so you can 
access the XMPP server.  It introduces another point of failure for the 
system.  If we were to rely on XMPP for notifications, we would just be 
adding support tickets like "my mail client doesn't notify me when I get 
new mail".  I already get "I can retrieve mail but not send".  Also you 
then need to have hooks into it at both ends, and define some sort of 
addressing system so the XMPP server can know what the client wants it 
to provide notifications about, and hooks in the back end of the 
services so it can get them.  Lots of points of failure.

I don't know how many of you spend a lot of time on customer support 
desks.  I do.

So I think maybe we need to take a step back.  If we want people to 
implement something new, it has to have some compelling reasons.  I 
don't think just interop is that compelling since it's not completely 
awful at the moment, and I think that is not thinking big enough.  We 
could do so much more.  If we're going to do a new mail protocol, we 
need to think what we want it to be like, with the benefit of however 
many decades of experience.

Mark is right in that the problems that IMAP solves are often problems 
that will also need solutions in a new system.  I don't believe 
implementing a new protocol means doing all that work again.  I've 
refactored enough code to know the thing you keep is the knowledge, even 
if the syntax changes.  So, anyway for me an incremental change to IMAP 
isn't horribly interesting.  Sure we can fix some things, but the way 
the protocol is extended and requirement for backwards compatibility is 
always going to hamstring us.

For me something more interesting would be a protocol that allows a 
client to provides the best possible mail user experience. And doing 
that using a single connection.

The reason for a single connection (could maybe be relaxed to a single 
port) is simply to reduce deployment issues.  One username, one 
password, one port to open.  That's the minimum number of things to go 
wrong, and minimum number of things to support.

If a vendor knows that by moving to a new mail protocol their customer 
support issues can halve, that's very compelling.
If a vendor knows that by using a new protocol they can drastically 
improve their users' mail experience, that's very compelling.

For me, I'd like something that:

* is fast, even over low bandwidth or latent connections
* is simple to deploy for administrators and users
* gives me all the mail functions I could need, including calendar etc.
* gives me cool new functions, like telling me the mail address I just 
typed into the destination field is bad even before I finished typing 
the subject line.
* works well with multiple clients (e.g. home and office, and on the road)

And no, I don't believe for a second this will happen overnight.

Cheers

Adrien

---

From: Bron Gondwana <[email-masked]>
Date: Mon, 13 Feb 2012 21:37:57 +0100

> Bwahahahaha! Complaining about "haters" just like a 13-year-old girl!

Yo, I'm down with the kids, dawg.  Or something.

> also stay quiet about it until you had something concrete to show for it.

I want to talk to other people about what they need as well rather than
just making everything up myself.  Sorry for disturbing you with it.

> was already widely deployed.

It's true - the mountain looks very tall from the bottom - and you have
climbed a mountain, which I haven't.  It's not exactly the same mountain
though, and we don't have exactly the same equipment.  Feel free to keep
laughing, but I'm starting climbing.  And I'd rather do it in public than
in secret right up until the IETF process bit.

Do you still have your un-"improved" specification and interoperable
code sitting around somewhere that you can show us all how nice it was
before the grandmothers improved it?  I'd be interested to see so I
can tell what I'm in for.

Thanks,

Bron.

---

From: Bron Gondwana <[email-masked]>
Date: Mon, 13 Feb 2012 22:08:05 +0100

On Tue, Feb 14, 2012 at 09:26:58AM +1300, Adrien de Croy wrote:
> Maybe we need to start with what we want to achieve... which
> actually Bron did do at the start.  All of us have various different
> histories when it comes to mail.

Oh yeah, so I did... it feels like a long time ago already!

> When I first implemented our IMAP server, I was firstly struck by
> the unusual syntax, parsing requirements, and then the missing bits
> - lack of completely defined folder management protocol support, and
> the highly problematic dual-indexing (no UID in expunge
> response!!!).   History obviously plays a huge part, but it's not
> 1998 any more.

Folder management is amazingly complex now that all the LIST-EXT
stuff is in there, and interactions with "subscriptions" - complete
with the whole "subscription has to survive when folder is deleted"
funkyness.  It's amazing how the best intentions have created a mess
there.

> Then I was hit with the quirks of the implementations (e.g. TB
> ignores unsolicited responses unless it issued an IDLE).  I've spent
> a lot of time evaluating IMAP clients, none of them are what I would
> call great (for my use).

Mark will agree there I think.  Me too.  I use offlineimap and mutt
with Maildirs still.  It's not awesome, but at least I can edit my
email in vim.

> optional protocol extensions, which basically means no-one can count
> on them being there.

I covered that in detail.  Optional is useless for a client writer,
because they have to support both options, and that's twice as much
code to write and debug and keep in sync.  So they don't bother
unless the benefits are huge - they just write to the lowest common
denominator.

> [ extra protocol is adds too many support headaches ]
> 
> don't think just interop is that compelling since it's not
> completely awful at the moment, and I think that is not thinking big
> enough.  We could do so much more.  If we're going to do a new mail
> protocol, we need to think what we want it to be like, with the
> benefit of however many decades of experience.

Absolutely agree.  And my experience alone isn't enough, which is
why I don't want to go hide in a dark room and make one up out of
whole cloth.  I may have tried that a few years ago, but talking
to others at Fosdem in particular showed that I don't have all
the right answers.  I need to listen to other people who actually
believe in making this protocol too.  I'm willing to do a lot of
the hard labour in getting the implementations and TESTS done.
The tests are even more important than the code, because that's
how you keep implementations compatible.

I think that's something we have learned in the past 20 years.
Wordy specs don't do as much for compatibility as automated
tests which can be run against an implementation and show
immediately if it's doing the right thing or not.  Saying
"you fail test XYZ, and here's how you can test against it
until you have fixed your code" is much more effective than
saying "go read all the documents again you moron, and make
sure you understand them properly before you try to implement
anything"

> Mark is right in that the problems that IMAP solves are often
> problems that will also need solutions in a new system.  I don't
> believe implementing a new protocol means doing all that work again.
> I've refactored enough code to know the thing you keep is the
> knowledge, even if the syntax changes.  So, anyway for me an
> incremental change to IMAP isn't horribly interesting.  Sure we can
> fix some things, but the way the protocol is extended and
> requirement for backwards compatibility is always going to hamstring
> us.

And the first part of "not doing all the work again" is looking
at what we have now.  Every RFC and draft is the result of
somebody feeling enough pain from what's there now to sit down
and write up how they think their pain can be solved, and
shepherd it through the standards process.  That's a lot of
effort.  Those are important problems, and valuable thought
has gone into them.  Maybe some of them will be solved as a
side effect of discarding backwards compatibility at the
protocol level.  I sure hope so, because otherwise it's not
worth it.

> that using a single connection.

God yes.  That's what it's all about.

> The reason for a single connection (could maybe be relaxed to a
> single port) is simply to reduce deployment issues.  One username,
> one password, one port to open.  That's the minimum number of things
> to go wrong, and minimum number of things to support.

I think it has to be a single port.  Otherwise you wind up
having to multiplex EVERYTHING.  Ouch.  But one username,
one password, potentially a "sessionid" from the initial
login which can be used by the further connections.  I
think SCTP is not well enough supported to use, unfortunately,
because it would be a great way to deal with the multiple
connections issue otherwise.  There's a lot of potentially
great technology out there which just isn't right for the
problem.  We have to be pragmatic as well - it has to be
stupidly simple for administrators and users to deal with.

> If a vendor knows that by moving to a new mail protocol their
> customer support issues can halve, that's very compelling.

Yes.

> * is simple to deploy for administrators and users

These two are non-negotiable for me.

> * gives me all the mail functions I could need, including calendar etc.

This one is slightly more negotiable.  I think we could easily
over-reach in this.  I'd like to do a "version 1" which doesn't
go quite so far beyond what's there now.  That said, there's
the Kolab work to create a "Calendar Mailbox" which is just a
bunch of messages with a specified attachment type, one message
per calendar entry.  That plus a "\Calendar special use" or
similar could get you a working calendar system without needing
too much complexity.

> * gives me cool new functions, like telling me the mail address I
> just typed into the destination field is bad even before I finished
> typing the subject line.

That one's tricky.  I'd almost outsource that to a webservice
rather than being "in-protocol".  This being offline temporarily
is annoying, but not fatal to the user experience, and it's
cachable and doesn't necessarily need authentication.

> * works well with multiple clients (e.g. home and office, and on the road)

That's the one thing IMAP really has going for it already, but
we can get a bit better.

> And no, I don't believe for a second this will happen overnight.

Nup.

Bron.

---

From: Mark Crispin <[email-masked]>
Date: Mon, 13 Feb 2012 13:37:14 -0800 (PST)

On Mon, 13 Feb 2012, Bron Gondwana wrote:
> And I'd rather do it in public than

You will never make much progress as long as the initial design process is
burdened with that monkey on its back. The best way to destroy any project
is to subject it to review at that stage.

> can tell what I'm in for.

Knowledge of the original IMAP (before IMAP2) exists primarily in my mind
as all the original IMAP specifications and implementations were replaced
with IMAP2.

The only substantial difference between original IMAP and IMAP2 was the
introduction of tags in IMAP2. In original IMAP, a response started with
one of "OK", "NO", "BAD", "BYE", or "*".

That incompatibility is caused the "2". Since it was still possible to
replace all the "old protocol" implementations, there was no attempt at
IMAP/IMAP2 compatibility.

A secondary effect was the much-maligned advent of untagged OK/NO/BAD
responses.

As it turned out, tags never were as important for pipelining as its
advocates claimed. The vast majority of Internet protocols pipeline do
just fine without tags, although I admit ManageSieve did it in a cool way.

The motivation for tags was the quaint idea (which I never believed back
then) that an IMAP server will routinely have requests blocked while
waiting for the computer operator to mount a 9-track tape or optical disk.
Thus it was purportedly important for responses to arrive out of order for
the request, and all clients must be completely asynchronous. Of course,
that never happened...

Post-IMAP2, there are all the past IMAP RFCs. The ftp.cac.washington.edu
FTP site still has many of the IMAP2bis drafts on the mail/old/ directory,
as well as the RFC drafts. There is quite a lot to review there.

The history of IMAP2bis is particularly important, as it explains a great
deal of the process from RFC 1176 to RFC 1730 which would otherwise be an
unbridgeable gap.

In turn, RFC 1730 to RFC 2060 shows what happens when a set of last-minute
additions (to RFC 1730) are put in without adequate thought and
consideration. There is a BIG lesson to learn from that abortion.

RFC 2060 is the RFC 1176 replacement protocol in more or less final form.

-- Mark --

http://panda.com/mrc

---

From: Tony Finch <[email-masked]>
Date: Wed, 15 Feb 2012 14:11:04 +0000

> funkyness.  It's amazing how the best intentions have created a mess

Is there any reason to keep subscriptions in IMAP 5 ?

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Dogger, Fisher, German Bight, Humber, Thames: Northwest backing west 5 to 7,
occasionally gale 8 at first, decreasing 4 at times. Rough or very rough,
becoming moderate or rough. Occasional rain. Moderate or good.

---

From: Timo Sirainen <[email-masked]>
Date: Wed, 15 Feb 2012 16:18:51 +0200

> Is there any reason to keep subscriptions in IMAP 5 ?

Somewhat more generic mailbox tags would be useful. So I could configure
my IMAP client to show a view of only mailboxes related to e.g. "work",
"mobile", etc. This could be done generically by assigning some metadata
tags to mailboxes and having a LIST command that supports filtering
using metadata. Of course clients would have to support it also somehow.

---

From: Bron Gondwana <[email-masked]>
Date: Wed, 15 Feb 2012 15:19:12 +0100

> Is there any reason to keep subscriptions in IMAP 5 ?

I envisage "subscription" as either an annotation or a "Special Use" on
a folder rather than yet another axis of data.

Bron.
-- 
  Bron Gondwana
  [email-masked]

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 15 Feb 2012 14:22:28 +0000

On 15/02/2012 14:19, Bron Gondwana wrote:
>> Bron Gondwana<[email-masked]>  wrote:
Yes. I think clients frequently have a mode of only showing subscribed 
folders as opposed to everything. This seems to be useful for mobile/low 
bandwidth use.
> a folder rather than yet another axis of data.
+1.

---

From: Bron Gondwana <[email-masked]>
Date: Wed, 15 Feb 2012 15:43:01 +0100

> > a folder rather than yet another axis of data.

I was chatting to one of the other guys on our team yesterday about data
modelling in this context.

Mail is a structured data store, with some small set of types of data stored
in the IMAP "model".  The IMAP protocol is very much about the communications,
and trying to optimise specific use cases.

But at the core, it's about querying and modifying data in a particular model,
with some constraints on the valid modifications - both for integrity and for
ACL permissions reasons.

Along with a side dose of syncronising up offline modifications to that same
data model.

This is not a problem that's unique to email.  There's nothing really special
about email here unless you make it special.  Sure there's a bunch of indexed
and optimised ways of viewing that data - sort by trimmed subject, encodings,
etc.  All of which could be expressed as generic queries against the data
model with a query optimiser on the far end rather than needing a custom
syntax for everything...

It's a somewhat different view of email, but I think quite a useful perspective
to look at it from.  Just a database with some funky constraints and triggers.

So in my head, I'm kind of thinking in SQL when I think about how I would
access email.  LIMIT, OFFSET, ORDER BY, VIEW.  Particularly VIEW and ORDER BY
with a dose of 'use modseq to re-synchronise up with your last copy of a view'.

CREATE VIEW 'Inbox' AS
  SELECT * FROM Messages M JOIN Folders F USING (FolderId)
   WHERE F.SpecialUse = '\Inbox' AND M.IsExpunged = 0 ORDER BY M.Uid DESC;

CREATE VIEW Status AS
  SELECT FolderId, MAX(Uid) + 1 AS UIDNEXT, MAX(Modseq) AS HIGHESTMODSEQ,
                   COUNT(*) AS EXISTS, SUM(IsSeen) AS UNSEEN
  FROM Messages WHERE IsExpunged = 0 GROUP BY FolderId;

There exist database engines which could optimise those views up the wazoo,
rather than writing a custom optimizer for each one, and a custom protocol
element with a custom parser for each value.

----

So basically what I'd like to see us doing with email is LESS.  Less custom.
Less "it's so special".  Treat it like a generic bunch of data, and manipulate
it with generic data manipulation methods.

Bron.
-- 
  Bron Gondwana
  [email-masked]

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 15 Feb 2012 15:57:30 +0100

> it with generic data manipulation methods.

It sounds a bit like a reinvention of RFC 2244, which was too generic
for people like me. Dave liked it, I think he still likes it, but he's
really smart. Personally, I understood that 2244 could be used to do
wondrous stuff in general, but when I read the document I wasn't able to
connect it to any concrete wondrous applications for my work.

Optimising for really smart people risks leaving mortals unable to see
any reason to write code for it.

Arnt

---

From: Petite Abeille <[email-masked]>
Date: Wed, 15 Feb 2012 16:37:01 +0100

On Feb 15, 2012, at 3:43 PM, Bron Gondwana wrote:

> access email. 

Funnily enough, this is precisely how my diminutive IMAP server is implemented :D

E.g.:

SelectMailboxStatus =
[[
  select  cast( count( * ) as integer ) as messages,
          cast( 0 as integer ) as recent,
          cast( coalesce( max( uid ), 0 ) + 1 as integer ) as uidnext,
          cast( 1 as integer ) as uidvalidity,
          cast( 0 as integer ) as unseen
  from    fetch_seq
]],

FWIW, here is the DDL of the data model (a bit of a work in progress):

http://dev.alt.textdrive.com/browser/Mail/MailStore.ddl

And the DML for the IMAP store:

http://dev.alt.textdrive.com/browser/Mail/IMAPStore.dml

---

From: Bron Gondwana <[email-masked]>
Date: Wed, 15 Feb 2012 19:10:47 +0100

> connect it to any concrete wondrous applications for my work.

Oh yeah, ACAP.  It's on the list of things to look at.

> any reason to write code for it.

Obviously you need to avoid getting so over-generic that you wind up
doing an anything protocol.  The anything protocol is always tempting.

The thing is - I don't mean leaving everything super flexible and
undefined.  I'm meaning more "don't define a separate custom protocol
with custom behaviour for everything".

For example (and I quote directly from ##imap on freenode over the past
couple of hours)

<IroquoisTwist> hello, is there a simple reason why my imap connected
                clients can't distinguish unread messages? (all messages
                marked as read)
<brong_>        IroquoisTwist, probably is a simple reason, but it
                depends on a lot of factors...
<IroquoisTwist> brong_: thanks, i found out my spam guard was "reading"
                the messages - turned it off and used local span
                filtering - all is well
<brong_>        sounds like a classy piece of software.  Forgot to put
                a .peek in did they?
<brong_>        what am I saying.  imap5 people - LOOK AT THIS
<IroquoisTwist> brong_: not too sure - end of my spam knowledge - it's
                called Spamassassin (preinstalled via cPanel)

Now, you could blame the authors of the spam scanner which wrapped
Spamassassin and called it via IMAP.  They're morons, no doubt.

But .PEEK.  Seriously?  Hands up everyone who has screwed that up
themselves or seen someone else screw it up over their entire time
of working with email?  I'll own up to having done it more than
once.

.PEEK is one of the warts that is out.  No side effects.  No magic.

Along with \Recent, which is also magic - it's a read only value where
EVERYTHING ELSE in the same space is writable (ACLs permitting), it's
defined in a way that makes multi-session a pain.  It's a wart.

And UNSEEN, but I already said that.  Why is 'STATUS UNSEEN' special,
and "* UNSEEN x" - complete with the contradictory "use your common
sense" definition...

Bron.

---

From: Jan Kundrát <[email-masked]>
Date: Wed, 15 Feb 2012 19:25:52 +0100

> defined in a way that makes multi-session a pain.  It's a wart.

On the other hand, some people (you know, I keep telling myself that I
cannot possibly be alone in this) use the lack of \Seen to mean "I
should do something for this mail", which means that we have a lot of
"unread" messages in our mailboxes. Having a way of automatically
marking those messages which are *really* fresh arrivals is therefore
very convenient, and \Recent does exactly that.

Sure, we can use another flag for this, but having these flags managed
by the server means that no client has to ever touch it, and therefore
it will "just work" all the time. That's a pretty big benefit, IMHO.

(Yes, I typically run just a single instance of a MUA these days and lug
my laptop around -- that might explain why this works for me.)

Cheers,
Jan

-- 
Trojita, a fast e-mail client -- http://trojita.flaska.net/

---

From: Bron Gondwana <[email-masked]>
Date: Wed, 15 Feb 2012 19:54:03 +0100

> very convenient, and \Recent does exactly that.

Kinda, unless you lose your IMAP connection, or...

In other words, it's a lossy, partial solution - which comes at a
server end penalty of having to WRITE something when a client reads.

> it will "just work" all the time. That's a pretty big benefit, IMHO.

Except then someone says "but I want to see messages which have not
yet been seen BY THIS DEVICE", so my phone sees things as new even
when I left my desktop logged in and checking for new messages.

Special cases are great when they match your use case, for sure.

> my laptop around -- that might explain why this works for me.)

Whereas I work in a couple, and have my phone... so it's pointless
to me.

Bron ( and the management types with their crackb^WiPhones... they want
       something different again )

---

From: Mark Crispin <[email-masked]>
Date: Wed, 15 Feb 2012 11:51:39 -0800 (PST)

On Wed, 15 Feb 2012, Bron Gondwana wrote:
> No side effects.  No magic.

Your effort is doomed with that attitude.

Protocol designers do NOT drive the evolution of a protocol. Protocol
consumers do.

In order for a protocol to be used, the protocol designer must accomodate
everything that a protocol consumer identifies (whether correctly or
incorrectly) as a requirement. All too often, there is no way to do so
without adding magic and/or warts. If the designer refuses, the consumer
will go elsewhere.

In every protocol, every little wart, every bit of magic, happened for a
reason - and almost always over the protocol designer's objection. For
each of these, some consumer depends upon it.

In order to get a consumer to switch from an existing protocol, you must
accomodate ALL their dependencies, AND present a compelling reason why
they should invest in switching. Neither elegance, nor beauty, nor magic
and wart free are compelling.

It's VERY hard to accomodate all the requirements of protocol consumers,
especially when those requirements conflict (and they do conflict).

That is why you have a very small number of successful protocol designers,
and a much larger number of people who think that they can do a better
job. Every field of human endeavor demonstrates it.

The miraculous thing about IMAP is that, after nearly 25 years, it has a
surprisingly small amount of magic and warts. Even better, the majority of
magic and warts are in extensions that most people ignore.

Above all, IMAP works well enough for consumers. The older that any
protocol gets, the more difficult and expensive it becomes to deploy a new
solution within open standards. Nobody cares that an implementor may find
it to be hard work. Implementors are supposed to be Really Smart People
that are worth their exorbitant salaries; and some of the best IMAP
implementations I've seen were written by very smart people in Third World
countries that work for much less...

One last thought. The only likely "IMAP killer" is not a new open-source
protocol. It is Exchange.

-- Mark --

http://panda.com/mrc

---

From: Timo Sirainen <[email-masked]>
Date: Wed, 15 Feb 2012 22:01:08 +0200

> protocol. It is Exchange.

Well, it is open enough nowadays and has one almost complete open source implementation, so it's not all bad..

---

From: Michel Sébastien <[email-masked]>
Date: Wed, 15 Feb 2012 21:59:56 +0100

>> > I envisage "subscription" as either an annotation or a "Special Use"
>> > on a folder rather than yet another axis of data.
>> +1.
Does it works with shared folders ?

> I was chatting to one of the other guys on our team yesterday about data modelling in this context.
>
> Mail is a structured data store, with some small set of types of data stored in the IMAP "model".  The IMAP protocol is very much about the
> communications, and trying to optimise specific use cases.
>
> But at the core, it's about querying and modifying data in a particular model, with some constraints on the valid modifications - both for integrity and for ACL permissions reasons.
>
> Along with a side dose of syncronising up offline modifications to that same data model.
>
> This is not a problem that's unique to email.  There's nothing really special about email here unless you make it special.  Sure there's a bunch of indexed and optimised ways of viewing that data - sort by trimmed subject, encodings, etc.  All of which could be expressed as generic queries against the data model with a query optimiser on the far end rather than needing a custom syntax for everything...
>

Some others seems to think that webdav could be a candidate : http://www.webdav.org/other/faq.html#Q26
A new layer on top of webdav, with some keywords registered at IANA. But I just don't like the trend to use HTTP for everything... despite its interest here.

Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité d'Atos ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis.

This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Atos liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.

---

From: Bron Gondwana <[email-masked]>
Date: Wed, 15 Feb 2012 22:13:01 +0100

> Does it works with shared folders ?

Sure, it's a private annotation.

> A new layer on top of webdav, with some keywords registered at IANA. But I just don't like the trend to use HTTP for everything... despite its interest here.

Yes, webdav is tempting for a few reasons - the downside is
a relatively high overhead.

BEEP has also been mentioned.  A good advantage of both of
these is that you can transport unmodified MIME across them.
I'm wary of anything which will require the raw MIME bodies
of messages to be encoded across the wire - some sort of
length based literal syntax is very valuable.

Of course it's hard to love something with examples like this:

 S: RPY 0 1 . 221 185

 S: Content-Type: application/beep+xml
 S:
 S: <profile uri='http://iana.org/beep/SASL/CRAM-MD5'>
 S: <![CDATA[<blob>PDE4OTYuNjk3MTcwOTUyQHBvc3RvZmZpY2UucmVzdG9uLm1
                                               jaS5uZXQ+</blob>]]>
 S: </profile>
 S: END
 
It makes MIME header encoding look so lightweight in comparison.

Still... as I'm regretting learning, compatible is more important
than good.  If there exist libraries everywhere which can
reliably read and write that, and it's ugly enough that nobody
wants to do it themselves, then maybe - just maybe, you'll actually
get BETTER complience than if it's a simple enough protocol that
people roll almost-correct code by hand.

Bron.

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 16 Feb 2012 10:28:02 +1300

having dealt with support issues relating primarily to HTTP for the last 
17 years, I'd STRONGLY recommend against using anything HTTP based.  The 
number of proxies that break WebDAV makes it problematic alone.

If some clients need HTTP-based access to some IMAP function, they can 
use a gateway.

Then an administrator can choose whether to allow such access.  But 
clients using the protocol that provides it all don't need it.

If you're a system admin, and there's a product you can install where 
you install 1 service, open 1 port and it provides everything you'd go 
for that right?

If we play our cards right, it should be simple for some gateway to 
provide legacy interfaces to the new protocol.

re the discussion about richness of protocol... sure you can do things 
at a lower level.  That typically requires more round-trips to the 
server though.

unless the things are pipelined..... specifically designed to be, so a 
single meta command is sent as a bunch of micro commands... but therein 
lie a multitude of problems (e.g. enforcing security, synchronisation etc).

Adrien

On 16/02/2012 10:13 a.m., Bron Gondwana wrote:
>   S:<profile uri='http://iana.org/beep/SASL/CRAM-MD5'>
>   S:<![CDATA[<blob>PDE4OTYuNjk3MTcwOTUyQHBvc3RvZmZpY2UucmVzdG9uLm1
>                                                 jaS5uZXQ+</blob>]]>
>   S:</profile>
>   S: END
>
> _______________________________________________
> imap5 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/imap5

---

From: Bron Gondwana <[email-masked]>
Date: Wed, 15 Feb 2012 22:31:22 +0100

> Your effort is doomed with that attitude.

My effort would be even more doomed if I had your attitude,
of course.

And you're right - "no" is too strong.  There will have to
be magic.  There doesn't have to be side effects though.
More protocol consumers are aware of the downsides of them
these days.  They know there's a tradeoff, and side-effect-free
is not just the realm of ivory tower Haskell peddlers.

Magic is bound to happen, but if there's a clear way to
indicate that you want the magic, rather than a default
of magic, I contend that protocol consumers will be
pleased.  I suspect most protocol consumers use BODY.PEEK
and explicit "STORE \Seen" these days.  I could collect
some statistics amongst our users easily enough, though
I would have to exclude our web interface or it would
skew the stats.

At least there is BODY.PEEK.

Bron.

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 16 Feb 2012 11:05:15 +1300

Seems to me we're getting bogged in the implementation details, or at 
least going too far down that path.  We don't even have a clearly 
defined set of objectives for the features we want the protocol to provide.

here's my first cut off-the-cuff list:

* mail access
for obvious reasons.

* mail submission
For reasons I've stated already including ease of administrative and 
configuration and support.  This breaks down into:
- composing messages on the server
- providing delivery information (e.g. destination address)

* storage of mail client configuration
So that clients can be dumb, and you can hook a new client to your 
mailstore and you don't need to configure anything else.

* server-side mail processing rules
Save downloading headers + MOVE etc to process filtering rules.  Also 
saves synchronising such rules amongst multiple clients

* ability for server to "comment" on anything, e.g:
- that destination email address you gave me doesn't resolve (with MX or 
A) or is otherwise bad.
- that attachment you uploaded contains a virus
- that subject is offensive...
- etc etc etc
this could be part of a broader notification framework.  In the cases 
where the IMAP server co-exists (as in WinGate for example) with SMTP 
delivery agent, then delivery information could be communicated back to 
the connected client.  If we allow for it in the protocol, it can 
provide benefit where it's available, and causes no issues if it isn't.  
The general idea though is that anything should be able to be remarked 
upon by the server.

* address book
* calendaring
* Any other feature to compete with Exchange.
* Any other feature we think would give us an edge.

In terms of design, I'd propose the following:

* single port.  This minimises problems traversing firewalls etc.
* ability to access all mail (in all folders) through a single 
connection, and receive notifications about all state changes to 
anything of importance (probably with client-controlled filter)
* layered protocol.  Bottom layer deals with connection security (e.g. 
negotiating TLS) authentication, and advertisement of and connection to 
sub-services.  Next layer are the sub-services e.g. mail, calendar, 
config etc.  This eases gateway development, since a calendar gateway 
can simply connect through to the calendar sub-service.
* compact yet extensible protocol message structure.

The debate of text-based vs binary protocols will forever rage, I tend 
to prefer binary, since then you don't need to parse text which requires 
escaping of characters which would otherwise be protocol delimiters.  
Some of the downsides of text-based protocols can be averted by using 
length prefixes instead of searching for (and therefore needing to 
escape) delimiters.  Even a mixture.  Literal+ is a damn good design in 
my view, since it allows a large literal to be transferred without it 
having to survive parsing.  It also allows any character set to go 
through.  HTTP is a nightmare in this respect.

* configuration attributes to be clearly defined (e.g. take a page from 
various MIBs) so that attributes can be re-used between different 
clients to the maximum extent.  Clients can store arbitrary 
configuration attributes against the mail account.

Regards

Adrien

---

From: Dave Cridland <[email-masked]>
Date: Wed, 15 Feb 2012 22:25:33 +0000

> * Any other feature we think would give us an edge.

This mailing list is "Discussion on drastically slimming-down IMAP",  
and you've listed the properties of ACAP, SIEVE, IMAP, CalDAV,  
CardDAV *and* Submission, and then thrown in a kitchen sink too.

I'm impressed.

Dave.

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 16 Feb 2012 11:28:18 +1300

thanks :)

I don't think in the context of a discussion on "Designing a new 
replacement protocol for IMAP" that it's off topic tho'

Regards

Adrien

On 16/02/2012 11:25 a.m., Dave Cridland wrote:
> and you've listed the properties of ACAP, SIEVE, IMAP, CalDAV, CardDAV 
> *and* Submission, and then thrown in a kitchen sink too.
>
> I'm impressed.
>
> Dave.

---

From: Timo Sirainen <[email-masked]>
Date: Thu, 16 Feb 2012 00:29:20 +0200

On 16.2.2012, at 0.25, Dave Cridland wrote:

> This mailing list is "Discussion on drastically slimming-down IMAP", and you've listed the properties of ACAP, SIEVE, IMAP, CalDAV, CardDAV *and* Submission, and then thrown in a kitchen sink too.
> 
> I'm impressed.

Who wrote that description? :) I never thought this list was about that. Except maybe in some politically correct sense.

---

From: Giovanni Panozzo <[email-masked]>
Date: Wed, 15 Feb 2012 23:36:06 +0100

> If some clients need HTTP-based access to some IMAP function, they can

Such a gateway should be a mandatory part of the protocol, or we will 
end up on having a lot of servers with the http gateway and many other 
servers without it, really bad user experience.

Could we think a "skype-like" solution, where the client makes two attempts:

1) Direct socket connection on a single TCP port
  and then, in case of failure
2) https tunnel of the same protocol (thru optional client side proxy).
    https is more likely to survive across proxyes than http.

?

---

From: Dave Cridland <[email-masked]>
Date: Wed, 15 Feb 2012 22:42:10 +0000

On Wed Feb 15 22:36:06 2012, Giovanni Panozzo wrote:
> Such a gateway should be a mandatory part of the protocol, or we  
> will end up on having a lot of servers with the http gateway and  
> many other servers without it, really bad user experience.
> 

BOSH.

Dave.

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 16 Feb 2012 11:44:15 +1300

On 16/02/2012 11:36 a.m., Giovanni Panozzo wrote:
> Such a gateway should be a mandatory part of the protocol, 

I think you misunderstood what I meant by gateway.  What I was meaning 
was some other piece of software which talks HTTP to clients, and talks 
<new-protocol> to the <new-protocol> server.

So it's a separate piece of software.

> or we will end up on having a lot of servers with the http gateway and 

there's talk (OK, I admit it, I proposed it) of deprecating http CONNECT 
method.  Check the httpbis WG list archives for more details.

Regards

Adrien

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 16 Feb 2012 11:51:37 +1300

On 16/02/2012 11:42 a.m., Dave Cridland wrote:
>> end up on having a lot of servers with the http gateway and many 
>> other servers without it, really bad user experience.
>>

long polling is a hideous hack.  Proxies hate it.

It's basically designing a system to provide a TCP over multiple HTTP 
over TCP connections.  Bloat to the extreme.

I understand the reasons why it exist, due to the model of HTTP, but 
building more things on top of it heading in the wrong direction IMO.

---

From: Dave Cridland <[email-masked]>
Date: Wed, 15 Feb 2012 22:53:52 +0000

On Wed Feb 15 22:51:37 2012, Adrien de Croy wrote:
> It's basically designing a system to provide a TCP over multiple  
> HTTP over TCP connections.  Bloat to the extreme.
> 
> I understand the reasons why it exist, due to the model of HTTP,  
> but building more things on top of it heading in the wrong  
> direction IMO.

There are two options if you want to live in the web world - BOSH or  
WebSockets.

WebSocket support is *far* from universal, and BOSH works - and works  
very well with proxies, despite whatever personal feelings they may  
have.

Dave.

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 16 Feb 2012 12:00:48 +1300

> There are two options if you want to live in the web world - BOSH or 

I agree.  I guess the question is whether we want to use http for a new 
mail protocol transport.  I'd suggest not.

> very well with proxies, despite whatever personal feelings they may have.

sure, since FB adopted long polling proxies have had to begrudgingly 
support it.  It's not without issues, and as to whether it's more 
reliable to make a BOSH virtual TCP connection than just a TCP 
connection or not I don't have a feeling for.  My understanding is this 
approach is used to basically bypass firewall filtering / blocking of ports.

But it makes the task of trying to impose some meaningful restrictions 
on "web access" very difficult for customers let alone proxy vendors.

In the end we could layer everything on top of HTTP.  If you don't care 
about performance or resource consumption.

Adrien

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 16 Feb 2012 12:18:02 +1300

thanks for the pointer on BOSH.

<OT>

Some of the claims they make are patently false however.  Especially in 
http://xmpp.org/extensions/xep-0124.html Sec 4.

"Therefore, if over time the traffic to and from the client is balanced, 
bandwidth consumption will be about the same as if a standard TCP 
connection were being used"

this completely ignores the overhead of the HTTP request and HTTP 
response, which can be very large.  A protocol layered over BOSH which 
makes lots of small transmissions will have its bandwidth consumption 
greatly increased.  IME most transactions have an http overhead > 1KB.  
There are minimum requirements for headers which must be included in 
order to be valid HTTP.  So telnet over BOSH would increase bandwidth 
consumption by 500 times.

It's quite amazing this statement remains in that document.

</OT>

---

From: Curtis King <[email-masked]>
Date: Wed, 15 Feb 2012 15:50:48 -0800

On 2012-02-15, at 2:29 PM, Timo Sirainen wrote:

> Who wrote that description? :)

Good question

> I never thought this list was about that. Except maybe in some politically correct sense.

ck

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 16 Feb 2012 13:03:10 +1300

> https://www.ietf.org/mailman/listinfo/imap5

so, what's the upshot of that?

do need to change the description associated with the list / scope?

Regards

Adrien

---

From: Brandon Long <[email-masked]>
Date: Wed, 15 Feb 2012 16:26:43 -0800

This is the second time I've heard about concern that proxies would
break access to my mail data over http.

Take those issues out of the loop, use https instead.  That way, the
data's encrypted end-end anyways, which of course it should be.

I know people hate dealing with certs, but I think we're well past the
point at which people want their personal mail traveling unencrypted
from their mailstore to their client.

Brandon

On Wed, Feb 15, 2012 at 1:28 PM, Adrien de Croy <[email-masked]> wrote:
>
> having dealt with support issues relating primarily to HTTP for the last 17
> years, I'd STRONGLY recommend against using anything HTTP based.  The number
> of proxies that break WebDAV makes it problematic alone.
>
> If some clients need HTTP-based access to some IMAP function, they can use a
> gateway.
>
> Then an administrator can choose whether to allow such access.  But clients
> using the protocol that provides it all don't need it.
>
> If you're a system admin, and there's a product you can install where you
> install 1 service, open 1 port and it provides everything you'd go for that
> right?
>
> If we play our cards right, it should be simple for some gateway to provide
> legacy interfaces to the new protocol.
>
> re the discussion about richness of protocol... sure you can do things at a
> lower level.  That typically requires more round-trips to the server though.
>
> single meta command is sent as a bunch of micro commands... but therein lie
> a multitude of problems (e.g. enforcing security, synchronisation etc).
>
> Adrien
>
>
>
>>>> This is not a problem that's unique to email.  There's nothing really
>>>> special about email here unless you make it special.  Sure there's a bunch
>>>> of indexed and optimised ways of viewing that data - sort by trimmed
>>>> subject, encodings, etc.  All of which could be expressed as generic queries
>>>> against the data model with a query optimiser on the far end rather than
>>>> needing a custom syntax for everything...
>>>>
>>> A new layer on top of webdav, with some keywords registered at IANA. But
>>> I just don't like the trend to use HTTP for everything... despite its
>>> interest here.
>>

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 16 Feb 2012 13:43:59 +1300

> break access to my mail data over http.

layering other protocols over HTTP is currently an arms-race between 
client-software developers who want to do this, and proxy developers who 
want to provide the tools their customers (sys admins) ask for to allow 
them to prevent it.

the harder it gets to block something intelligently, the more likely 
admins will resort to over-blocking.  We see it every day.

Developing a new protocol for mail designed ONLY to layer over HTTP is 
therefore extremely foolhardy.  Sure by all means have options.

HTTPS is no different.  There are already proxies that scan and restrict 
https.

IMO we'd be better off designing a protocol that is clearly identifiable 
(e.g. port) and easily able to be restricted.  Then it is more likely to 
be explicitly permitted than fall victim to overblocking.  I wonder what 
will happen when malware authors start using BOSH for instance, how long 
it will be before it's locked down.  We already see malware using https 
/ CONNECT which results in those services being locked down.

I agree re encryption, but IMAP already has STARTTLS.  (what I'd like to 
see is mandatory client certs for mail sending)

Adrien

> On Wed, Feb 15, 2012 at 1:28 PM, Adrien de Croy<[email-masked]>  wrote:

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 16 Feb 2012 07:56:07 +0100

On 02/15/2012 11:25 PM, Dave Cridland wrote:
> This mailing list is "Discussion on drastically slimming-down IMAP", and
> you've listed the properties of ACAP, SIEVE, IMAP, CalDAV, CardDAV *and*
> Submission, and then thrown in a kitchen sink too.

As I see it, he's listed features which are in the Exchange protocol and
in the unnamed protocol spoken by gmail's javascript heap and its
mothership.

That makes them worthy of discussion.

I know IETF dogma is that protocols shouldn't overlap. But it's a weak
kind of dogma: IMAP overlaps with POP, POP overlaps with Submission,
various IMAP extensions with ACAP and what's that about IMSP? I could go on.

Arnt

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 16 Feb 2012 20:10:19 +1300

> various IMAP extensions with ACAP and what's that about IMSP? I could go on.

actually with a layered approach, this dogma could be maintained, as the 
facilities would be able to be specified separately.

Would be interesting to see wire-line protocol on GMail - I bet they use 
Json

Regards

Adrien

---

From: Bron Gondwana <[email-masked]>
Date: Thu, 16 Feb 2012 08:53:10 +0100

> Would be interesting to see wire-line protocol on GMail - I bet they use 

Yes they do - with the "while(1);" hack at the top to stop it being scraped
by cross site tricks.  It's quite a clever workaround - any cross site request
that tries to embed the JSON URL just freezes - so the remote site can't
actually read the data.

We (mail.opera.com) also use JSON.  Ours is probably much less efficient over
the wire, because we chose readability over optimisation at this stage, so we
have named keys.  We will hopefully have two independent clients for it soon,
which will help us keep it stable.  I'm not sure that it's an ideal place to
start either though.  At least it was designed with a direct mapping to IMAP
in mind, so the data models are as similar as we could get (though we
"extended" IMAP to support the cross folder things we needed)

Bron.
-- 
  Bron Gondwana
  [email-masked]

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 16 Feb 2012 09:21:09 +0100

Bron wrote (about exchange, gmail, mail.opera, I think):
> I'm not sure that it's an ideal place to
> start either though. 

No. IMAP4 is the default starting point. But the success of these things
is such that the difference between their designs and that of IMAP4
cannot be disregarded.

IMAP4 by design excludes (taking an example at random) mail submission.
I think if Adrian or anyone else puts forward a cogent argument that
IMAP's excluding mail submission is wrong, then the charter should
permit discussing that.

Arnt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 16 Feb 2012 09:26:46 +0100

Oh. I misspelled Adrien's name. I'm so sorry.

Arnt

---

From: Dave Cridland <[email-masked]>
Date: Thu, 16 Feb 2012 08:49:37 +0000

> That makes them worthy of discussion.

I'm mostly revelling in the irony.

> various IMAP extensions with ACAP and what's that about IMSP? I  

There's two huge problems with the approach.

Firstly, using a generic data model, or a generic protocol,  
automatically produces compromise. I've learnt to mistrust genericity  
in protocols - it all seems like such a lovely idea, and then  
everything turns into the bastard offspring of SQL. And the thing  
with SQL is that you can do useful things like indices and whotsits,  
which let you specialize the data store, but nobody ever gets that  
far. The only cases where this has worked is to partially specialize  
the datastore - LDAP/X.500 does this, as did ACAP - but only one of  
those has succeeded by any metric.

I'd note that, similarly, METADATA and ANNOTATE should have done  
well, if it weren't for the fact they expanded beyond a simple  
dumping ground for "everything else", and people tried to use them as  
the One True Datastore.

Secondly, the broader the scope, the bigger the task - I don't see  
any likelyhood of getting such a protocol sorted out before the end  
of the decade, or beyond. The phrase "boil the ocean" springs to  
mind. Remember, Exchange and the like are not successful because they  
do calendaring and mail, they're successful because they seamlessly  
blend calendaring and mail - the result is more than the sum of its  
parts, but making that blend will not be easy.

Aside from anything else - you want configuration storage services in  
$NEWPROTO? Well, I surely want these to have all the facilities that  
ACAP gives me. Calendaring? I'm sure that Cyrus will want it to have  
parity with, or exceed, CalDAV. Mail? There's any number of folk here  
who'll want their own special sauce in. And they'd be right, from  
their perspective, and the net result would be unimplementable.

Dave.

---

From: Michel Sébastien <[email-masked]>
Date: Thu, 16 Feb 2012 10:07:18 +0100

>layering other protocols over HTTP is currently an arms-race between client-software developers who want to do this, and proxy developers who want to provide the tools their customers (sys admins) ask for to allow them to prevent it.
>
>the harder it gets to block something intelligently, the more likely admins will resort to over-blocking.  We see it every day.
>
>Developing a new protocol for mail designed ONLY to layer over HTTP is therefore extremely foolhardy.  Sure by all means have options.

I'm not really convinced, does it means that CalDAV and CardDAV are on the wrong way ? I think not, even if they are not supported by all mobile platforms

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 16 Feb 2012 22:20:44 +1300

Hi Dave

I agree with the things you've said about generic protocols, and trying 
to fix all the worlds problems - e.g. scoping something too big to be 
doable.

This all started when Brom sent an email around about a replacement 
protocol for IMAP.

One of the main problems with the current suite of protocols used for 
mail, is that even if one could implement them all, providing a better 
user experience than Exchange or Gmail would not be the result.

Add to that the fact that specifying a new mail protocol is a heap of 
work, the obvious question was why limit ourselves to only providing a 
replacement.

Forever is a long time.

That's why I proposed a layered approach.

If you look at a mail client even like Thunderbird, which I'm using to 
write this mail.

There are settings for:

SMTP: specification of server, choice of authentication method, choice 
of security (SSL vs STARTTLS vs none), username and password.
IMAP: specification of server, choice of authentication method, choice 
of security (SSL vs STARTTLS vs none), username and password.
LDAP: specification of server(s), choice of authentication method, 
choice of security (SSL vs STARTTLS vs none), username and password.

If I want a calendar I need to add in something like Lightning (current 
version not compatible with TB11), or something else.  Presumably it 
uses CalDAV, which goes over WebDAV which goes over HTTP, so I've got 
credentials to deal with for that, and possibly some intermediary proxy 
with credentials as well.

This all adds up to a complete clusterxxxx - it's an appalling user 
experience, and supporting it is costly as well.

Any one of these settings if wrong is a support call.  That's why I 
proposed condensing all the choice of server, port, auth and security to 
1 protocol, and layer the facilities on top.  Even if all we did was 
layer the existing protocols on top of this, we'd be a lot further ahead.

Last time I checked it was 2012.  Email is supposed to be (or have been) 
the "killer" internet application - the most important one.

To provide such a poor user experience is shameful.  Especially after 30 
odd years.

And of course I understand it's the result of an evolutionary process 
over time.  So of course it's messy and ugly.

But unless we do something about it, it will be that way forever, or 
until someone in future deals with it.  Or it gets replaced out from 
under us by web-based services which are able to independently provide 
an acceptable user experience.

Seems to me that in the context of discussion of a new mail protocol, 
could be a good time to consider these things.

Hence the kitchen sink :)

Cheers

Adrien

On 16/02/2012 9:49 p.m., Dave Cridland wrote:
> everything turns into the bastard offspring of SQL. And the thing with 
> SQL is that you can do useful things like indices and whotsits, which 
> let you specialize the data store, but nobody ever gets that far. The 
> only cases where this has worked is to partially specialize the 
> datastore - LDAP/X.500 does this, as did ACAP - but only one of those 
> has succeeded by any metric.
>
> I'd note that, similarly, METADATA and ANNOTATE should have done well, 
> if it weren't for the fact they expanded beyond a simple dumping 
> ground for "everything else", and people tried to use them as the One 
> True Datastore.
>
> Secondly, the broader the scope, the bigger the task - I don't see any 
> likelyhood of getting such a protocol sorted out before the end of the 
> decade, or beyond. The phrase "boil the ocean" springs to mind. 
> Remember, Exchange and the like are not successful because they do 
> calendaring and mail, they're successful because they seamlessly blend 
> calendaring and mail - the result is more than the sum of its parts, 
> but making that blend will not be easy.
>

---

From: Sebastian Hagedorn <[email-masked]>
Date: Thu, 16 Feb 2012 10:35:43 +0100

--On 16. Februar 2012 22:20:44 +1300 Adrien de Croy <[email-masked]> wrote:

> SMTP: specification of server, choice of authentication method, choice of
> security (SSL vs STARTTLS vs none), username and password.
> IMAP: specification of server, choice of authentication method, choice of
> security (SSL vs STARTTLS vs none), username and password.
> LDAP: specification of server(s), choice of authentication method, choice

Another way of dealing with that particular issue is autoconfiguration. 
Unfortunately there's no accepted standard for that yet, but we (Cologne 
University) support Microsoft's and Thunderbird's mechanisms. If you enter 
a @uni-koeln.de address in the new account wizard, all settings are filled 
in automatically.
-- 
     .:.Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18.:.
                 .:.Regionales Rechenzentrum (RRZK).:.
.:.Universität zu Köln / Cologne University - ✆ +49-221-478-5587.:.

---

From: Giovanni Panozzo <[email-masked]>
Date: Thu, 16 Feb 2012 10:52:43 +0100

> University) support Microsoft's and Thunderbird's mechanisms. If you
> enter a @uni-koeln.de address in the new account wizard, all settings
> are filled in automatically.

I'm using autoconfiguration for both M$ and Thubderbird, for some 
Exchange and IMAP servers.
It's quite good, but in my opinion it has 3 major problems:

a) As  you wrote, it's not a standard :(
b) Autoconfiguration is checked only at client configuration. I would 
like to have the client reconfigured at every startup. This will let me 
do major changes at the server sides (ie: enable SSL ? Change server 
names ?), and client will be able to reconfigure themselves at next startup.
c) Autoconfiguration for thunderbird does not solve the "multiple 
channel" problem. Thunderbird is still using one channel (TCP port) for 
IMAP and one for SMTP (and optionally one for LDAP). A roaming user, 
when changing network, can find some TCP ports open and some closed. So 
he/she will be able to only send or receive e-mail. He/she will be 
confused and he will open an helpdesk ticket. This is not a good user 
experience.

I hope that newer IMAP will address all those problems.

---

From: Dave Cridland <[email-masked]>
Date: Thu, 16 Feb 2012 09:57:43 +0000

On Thu Feb 16 09:20:44 2012, Adrien de Croy wrote:
> SMTP: specification of server, choice of authentication method,  
> IMAP: specification of server, choice of authentication method,  

Well, of course, I'd argue that you could use a combination of SRV,  
common options, discovery, and ACAP to fix all that.

The problem with Thunderbird isn't that it has all these options,  
it's that it requires the user to enter them, and fails to do  
discovery properly - Tony Finch wrote a particularly good blog post  
on why it's so awful several years ago, and provided solutions, too.

Interestingly, XMPP has generally gone the discovery route, and the  
result is that you only enter a jid and a password.

For Polymer, you enter a username, ACAP server, and password - ACAP  
doesn't have the SRV option, and maybe I should just add that in -  
not that anyone but me cares anymore.

Having written multiprotocol clients, I just don't think they're as  
hard as people make them out to be.

Dave.

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 16 Feb 2012 22:59:52 +1300

> I'm not really convinced, does it means that CalDAV and CardDAV are on the wrong way ? I think not, even if they are not supported by all mobile platforms

well, deploying CalDAV to a corporate environment has all sorts of 
issues.  You want to run your CalDAV server on a computer running a web 
server?  Port conflict.  Or you need to wedge it into your web server 
somehow.

It's still another set of creds and config to access what users consider 
to just be a feature in mail.

Imagine if you needed to use 5 different keys to drive your car.  One 
for the door, a different one for the steering lock, another to start 
the engine, another to unlock the transmission etc etc.

Then you lose 1 key.

It's actually insane.

---

From: Bron Gondwana <[email-masked]>
Date: Thu, 16 Feb 2012 11:04:10 +0100

> hard as people make them out to be.

Having seen support tickets for many years, those of us in a "service
provider" role know how bad the multiple-credential, multiple service
with different uptime issues and intermediate unreliability issues is.

The autoconfiguration space is getting better, many services "just work"
with Thunderbird now, for example.

As for the ACAP server - the world is moving, for the better I think, to
"you enter your email address and password, and it just works".  That's
the minimum short of a single-signon solution.  One thing that uniquely
identifies you and one thing that's secret.  Entering the ACAP server
address as a third datapoint is 50% more overhead, and that's the 50%
that you need to look up somewhere, because you already know your
email address and password.

Which is a vote from me for "yes, SRV option makes sense".

Bron.

-- 
  Bron Gondwana
  [email-masked]

---

From: Dave Cridland <[email-masked]>
Date: Thu, 16 Feb 2012 10:04:26 +0000

On Thu Feb 16 09:59:52 2012, Adrien de Croy wrote:
> Imagine if you needed to use 5 different keys to drive your car.   
> One for the door, a different one for the steering lock, another to  
> start the engine, another to unlock the transmission etc etc.

My car has several keyholes, but due to advanced technology, it  
requires only one key!

I've dubbed this "Single Key In", and it's a fantastic new technology!

I wonder if we could devise some similar capability for  
authentication on the catenet?

Dave.

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 16 Feb 2012 23:15:04 +1300

sure, anything that can be auto discovered is great.

But SRV has issues, not every corporate runs their own DNS (at least not 
for external).

ACAP is great too, but it's another port and set of creds.  And the 
tie-in between ACAP and other services is probably manual on the 
back-end right?

Xtra is NZ's biggest ISP.  It blocks port 25.  So when I take my laptop 
home I need to reconfigure it.  At least I know how to do that.

We're techies here, we forget how lost and confused the punters get.

Adrien

On 16/02/2012 10:57 p.m., Dave Cridland wrote:
> The problem with Thunderbird isn't that it has all these options, it's 
> that it requires the user to enter them, and fails to do discovery 
> properly - Tony Finch wrote a particularly good blog post on why it's 
> so awful several years ago, and provided solutions, too.
>
> doesn't have the SRV option, and maybe I should just add that in - not 
> that anyone but me cares anymore.
>

---

From: Michel Sébastien <[email-masked]>
Date: Thu, 16 Feb 2012 11:16:07 +0100

> Which is a vote from me for "yes, SRV option makes sense".

Your dream has come true :) http://tools.ietf.org/html/rfc6186

--
Sébastien

---

From: Dave Cridland <[email-masked]>
Date: Thu, 16 Feb 2012 10:25:22 +0000

> Which is a vote from me for "yes, SRV option makes sense".

Oh, totally agree. Discovery is awesome, user-configuration is  
insane. I don't see there's any valid argument the other way.

Dave.

---

From: Dave Cridland <[email-masked]>
Date: Thu, 16 Feb 2012 10:41:39 +0000

On Thu Feb 16 10:15:04 2012, Adrien de Croy wrote:
> But SRV has issues, not every corporate runs their own DNS (at  
> least not for external).
> 
> 
Well, tough. There is a point at which we have to assume people will  
have to fix things. XMPP services get this fixed pretty quick, and  
mail is a much bigger juggernaut.

> tie-in between ACAP and other services is probably manual on the  
What tie-in? ACAP's just a simple store. The enhancement to the  
client is that a sysadmin can preconfigure the clients, and ACAP  
gives a bunch of wacky data inheritance tricks to make this easy.

I wouldn't worry too much about ACAP anyway - even I think it's  
dead-end. The key portions could be slammed into almost any other  
convenient protocol, and it's one case where I think bolting it onto  
an existing protocol makes sense.

This wasn't the case when it was the only viable solution for  
handling address books, mind, and you'd lose the multi-account  
capability, but I think people are basically OK with such things now.

> Xtra is NZ's biggest ISP.  It blocks port 25.  So when I take my  
> laptop home I need to reconfigure it.  At least I know how to do  
> that.
> 
> 
Why would blocking port 25 be a problem? Unless you're running an MTA  
on your laptop, in which case you're presumably savvy-enough to deal  
with the consequences.

> We're techies here, we forget how lost and confused the punters get.

No matter how good the protocols involved are, it comes down to how  
good the deployment and implementations are. The client is the  
punter-facing component, and without good clients you're shot  
whatever you do.

To get "good" clients, you need mind-share, and you'll only get that  
if you start off with the status quo and figure out where to go - or  
if you base on another preexisting framework. Lots of folk are doing  
this very successfully with the web, of course, but I think there are  
other options, too.

All this talk of a boil-the-ocean brave-new-world is great fun, I'll  
be the first to admit, but I really don't see how it gets us anywhere  
useful.

Dave.

---

From: Bron Gondwana <[email-masked]>
Date: Thu, 16 Feb 2012 11:58:28 +0100

> this very successfully with the web, of course, but I think there are  

Or if you provide a better-enough experience that the switching cost
is worth it to enough client authors.

> be the first to admit, but I really don't see how it gets us anywhere  

I see three choices:

a) We want to spend the rest accessing our email with IMAP + layers of
   hotfixes, with an ever more complex set of potentially partially
   implemented capabilities.
b) We want to switch to Exchange protocol (as per Mark's comment earlier)
c) Neither of the above.

If (c), what then.  It's already going to be really hard.  If we wait
longer to start, it will keep getting harder.

Bron.
-- 
  Bron Gondwana
  [email-masked]

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 16 Feb 2012 23:58:51 +1300

On 16/02/2012 11:41 p.m., Dave Cridland wrote:
>> But SRV has issues, not every corporate runs their own DNS (at least 
>> not for external).
>>
>>

how many corporates deploy XMPP services?

SRV is great, but it's only marginally better than ACAP in terms of 
conifguration points.  You still need a domain.  What happens when 
you're using hosted mail?  Does your mail host have to give you a 
sub-domain so you can have your own SRV record to point to your own ACAP 
server to get your mail config?

How many points of failure there?

That's why I keep going back to the 1 port like a broken record.  Maybe 
it should just be an SSH tunnel... but that;s back-pedalling quickly and 
reducing potential user experience with it.

> What tie-in? ACAP's just a simple store. The enhancement to the client 
> is that a sysadmin can preconfigure the clients, and ACAP gives a 
> bunch of wacky data inheritance tricks to make this easy.

as I said - manual.  You have to type in the settings, the IMAP server 
can't publish them automatically to the ACAP server.

> an existing protocol makes sense.
definitely.

>
> This wasn't the case when it was the only viable solution for handling 
> address books, mind, and you'd lose the multi-account capability, but 
> I think people are basically OK with such things now.
>
>
>> laptop home I need to reconfigure it.  At least I know how to do that.
>>
>>

there are a myriad of reasons.  My MTA is Thunderbird.  At work, I have 
my mail set to send to smtp.qbik.com with creds.  When I take my laptop 
home, I can't connect any more.  I have to send my mail through the ISP 
mail server.  Some companies don't like this.  Some users have trouble 
configuring this.  We do actually get support tickets created by this 
particular issue.

You go to a hotel, same issue.

Some companies use SPF as well, so mail starts to bounce when you can't 
use your corporate MTA to send.

> punter-facing component, and without good clients you're shot whatever 
> you do.
>
absolutely.  But good clients can be impossible to achieve if the 
protocols don't allow it.

> this very successfully with the web, of course, but I think there are 

Microsoft abandoned it all and wrote Exchange.  So did GMail.

They can afford to do that.  We aren't MS or Google.

Why do you think they did that?  Exchange server was delayed for years.  
It cost them.  But now it's almost the only game in town.

> be the first to admit, but I really don't see how it gets us anywhere 

I prefer my ocean at ATP.

We could just talk about what could be stripped out of IMAP.  But I 
don't really see how that would get us anywhere useful either.

Adrien

---

From: Dave Cridland <[email-masked]>
Date: Thu, 16 Feb 2012 11:22:24 +0000

On Thu Feb 16 10:58:51 2012, Adrien de Croy wrote:
>> Well, tough. There is a point at which we have to assume people  
>> will have to fix things. XMPP services get this fixed pretty  
>> quick, and mail is a much bigger juggernaut.
>> 
>> 
Approaching 10% and growing, at least by some metrics:

http://eggert.org/meter/xmpp.html

> sub-domain so you can have your own SRV record to point to your own  
> ACAP server to get your mail config?
> 
> 
You've gone off on one.

If you have a mail domain, then you can put in SRV records to point  
to the servers. If you use another mail domain, then they do that for  
you.

> How many points of failure there?
Lots - but no fewer than without. Unless you think that SRV might  
break when the rest of the DNS doesn't. I don't think that's been an  
issue for the past decade or so.

> That's why I keep going back to the 1 port like a broken record.   
> Maybe it should just be an SSH tunnel... but that;s back-pedalling  
> quickly and reducing potential user experience with it.
> 
> 
No, I think it's an orthogonal issue.

>>> ACAP is great too, but it's another port and set of creds.  And  
>>> the tie-in between ACAP and other services is probably manual on  
>>> the back-end right?
>>> 
>>> 
> as I said - manual.  You have to type in the settings, the IMAP  
> server can't publish them automatically to the ACAP server.
> 
> 
Oh, right.

Well, yes, it *can* - both WorldMail and CommuniGate work(ed) in this  
manner.

But given the sysadmin has to make precisely one STORE command on a  
real ACAP server, it's a bit of a non-issue - it no more manual than  
setting up any other server.

>> Why would blocking port 25 be a problem? Unless you're running an  
>> MTA on your laptop, in which case you're presumably savvy-enough  
>> to deal with the consequences.
> 
> there are a myriad of reasons.  My MTA is Thunderbird.  At work, I  
> have my mail set to send to smtp.qbik.com with creds.  When I take  
> my laptop home, I can't connect any more.  I have to send my mail  
> through the ISP mail server.  Some companies don't like this.  Some  
> users have trouble configuring this.  We do actually get support  
> tickets created by this particular issue.
> 
> Some companies use SPF as well, so mail starts to bounce when you  
> can't use your corporate MTA to send.
> 
> 
Right, sure, understood (after s/MTA/MUA/) but what has port 25 got  
to do with it?

>> No matter how good the protocols involved are, it comes down to  
>> how good the deployment and implementations are. The client is the  
>> To get "good" clients, you need mind-share, and you'll only get  
>> that if you start off with the status quo and figure out where to  
>> go - or if you base on another preexisting framework. Lots of folk  
>> are doing this very successfully with the web, of course, but I  
>> think there are other options, too.
> 
GMail did it with the web - preexisting framework. Then they  
leveraged the deployed base of mobile IMAP clients - preexisting  
framework.

> They can afford to do that.  We aren't MS or Google.
Apparently not...

> Why do you think they did that?  Exchange server was delayed for  
> years.  It cost them.  But now it's almost the only game in town.
> 
> 
Exchange was largely an X.400 system, until recently. Again, they  
didn't make any real attempt to reinvent the wheel.

What they did do was carefully market it - Outlook was a very nice  
client, and it came free with Office, and only really worked  
tolerably with Exchange - where it worked really quite well. So they  
managed to leverage from Windows to Office, and from Office to  
Exchange.

The fact that it's a monolithic protocol has little to do with it.

>> All this talk of a boil-the-ocean brave-new-world is great fun,  
>> I'll be the first to admit, but I really don't see how it gets us  
>> anywhere useful.
> 

Sure.

What we need to do is identify the core problems to be solved,  
instead of finding solutions and trying to figure out how to use them.

But it's not nearly as much fun.

Dave.

---

From: Michel Sébastien <[email-masked]>
Date: Thu, 16 Feb 2012 12:23:39 +0100

> well, deploying CalDAV to a corporate environment has all sorts of issues.  You want to run your CalDAV server on a computer running a web server?  Port conflict.  Or you need to wedge it into your web server somehow.

You're right. It's also possible to bind to a different IP, do a port forwarding. Nothing complicated for a sysadmin

>It's still another set of creds and config to access what users consider to just be a feature in mail.
>
>Imagine if you needed to use 5 different keys to drive your car.  One for the door, a different one for the steering lock, another to start the engine, another to unlock the transmission etc etc.
>

Seems inappropriate, SMTP and IMAP configurations are already barriers for end-user.

Regards,
Sébastien

---

From: Tony Finch <[email-masked]>
Date: Thu, 16 Feb 2012 11:30:15 +0000

Bron Gondwana <[email-masked]> wrote:
> special about email here unless you make it special.  Sure there's a
> bunch of indexed and optimised ways of viewing that data - sort by
> trimmed subject, encodings, etc.  All of which could be expressed as
> generic queries against the data model with a query optimiser on the far
> end rather than needing a custom syntax for everything...

This sounds to me a bit more radical than I was expecting. In particular I
thought you wanted to keep the data model so that existing servers could
support the new protocol reasonably easily. Which implies that mailboxes
remain separate silos rather than the store being a soup of messages
GMail style.

> BEEP has also been mentioned.

I think BEEP is insane. Masses of complexity just to avoid using multiple
concurrent TCP connections.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Sole: North backing west 4 or 5. Moderate. Fair. Good.

---

From: Dave McMurtrie <[email-masked]>
Date: Thu, 16 Feb 2012 06:37:24 -0500

On 2/16/12 4:59 AM, Adrien de Croy wrote:
>>> client-software developers who want to do this, and proxy developers
>>> who want to provide the tools their customers (sys admins) ask for to
>>> allow them to prevent it.
>>>
>>> Developing a new protocol for mail designed ONLY to layer over HTTP
>>> is therefore extremely foolhardy. Sure by all means have options.
>> I'm not really convinced, does it means that CalDAV and CardDAV are on
>> the wrong way ? I think not, even if they are not supported by all
>> mobile platforms
>

The Cyrus Project aims to include CalDAV support as part of the Cyrus 
IMAP server, which would make CalDAV deployment much simpler for any 
sites that are already running Cyrus.

Dave

---

From: Adrien de Croy <[email-masked]>
Date: Fri, 17 Feb 2012 00:40:37 +1300

> http://eggert.org/meter/xmpp.html

OK.. I'm struggling to understand why...

>> ACAP server to get your mail config?

hmmm, you're right I have!!!! what was I thinking...

>
> If you have a mail domain, then you can put in SRV records to point to 
> the servers. If you use another mail domain, then they do that for you.
>
>

no I wouldn't expect its reliability to differ from any other records.

> No, I think it's an orthogonal issue.

maybe, but nonetheless real, especially for ISP tech support staff.

Just trying to think of "cheap" ways to get single sign on and single 
port using the existing protocols.  But it would still be a bandaid.

Auto config is a band-aid as well to the solution of multiple sets of creds.

> setting up any other server.

ok

>>> MTA on your laptop, in which case you're presumably savvy-enough to 
>>> deal with the consequences.
>>
>> have my mail set to send to smtp.qbik.com with creds.  When I take my 
>> laptop home, I can't connect any more.  I have to send my mail 
> Right, sure, understood (after s/MTA/MUA/) but what has port 25 got to 
> do with it?

back to my point about getting everything over 1 port.  If we had that, 
then blocking 25 wouldn't have affected me.

>>> this very successfully with the web, of course, but I think there 
>>> are other options, too.
>>
> GMail did it with the web - preexisting framework. Then they leveraged 
> the deployed base of mobile IMAP clients - preexisting framework.
>
>
> client, and it came free with Office, and only really worked tolerably 
> with Exchange - where it worked really quite well. So they managed to 
> leverage from Windows to Office, and from Office to Exchange.

Actually last time I compared prices of versions of Office, the Outlook 
component accounted for like $500 NZ.

It's also possibly the worst IMAP implementation out there.  But they 
don't _want_ you to use IMAP.   They want you to buy Exchange.

>>> be the first to admit, but I really don't see how it gets us 
>>> anywhere useful.
>>
> What we need to do is identify the core problems to be solved, instead 
> of finding solutions and trying to figure out how to use them.

* SPAM
* UDNs
* configuration issues

solve any one of these and you'll make a lot of people happy.

Adrien

---

From: Adrien de Croy <[email-masked]>
Date: Fri, 17 Feb 2012 00:44:56 +1300

> sites that are already running Cyrus.

as someone looking to add CalDAV to a mail server, wouldn't it be nice 
if you didn't have to

a) write a web server
b) write DAV extensions
c) layer XML on top of that
d) debug / support all of the above

just to get a calendar?

Adrien

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 16 Feb 2012 12:54:17 +0100

> OK.. I'm struggling to understand why...

You may be too young to remember the ascent of SMTP twenty years ago. MX
records were an insuperable hurdle, corporates preferred other mail
universes, and so on and so forth.

Googletalk's adding an XMPP gateway rather reminds me of how Compuserve
added an SMTP gateway.

Arnt

---

From: Dave McMurtrie <[email-masked]>
Date: Thu, 16 Feb 2012 06:55:53 -0500

Yes.

I don't view Cyrus CalDAV as a replacement for coming up with new, 
simplified protocols.  It's an attempt to provide a viable open-source 
alternative to Exchange.  In the near term, it may at best offer 
existing Cyrus sites another option if they're seeking integrated mail 
and calendar.

Dave

---

From: Bron Gondwana <[email-masked]>
Date: Thu, 16 Feb 2012 12:56:35 +0100

> But it's not nearly as much fun.

I'm working on that!  Have updated a reasonable amount of the wiki page with a
list of RFCs now... more later.

I absolutely agree that finding solutions first is a lousy approach, and
I'm as guilty as anyone here.  Let's get back to working on the core problem(s).

Bron.
-- 
  Bron Gondwana
  [email-masked]

---

From: Adrien de Croy <[email-masked]>
Date: Fri, 17 Feb 2012 00:56:43 +1300

>> http://eggert.org/meter/xmpp.html

that measures deployment only amongst the top 500 sites per territory as 
per alexa.  I think the true deployment would be much much lower.

Adrien

---

From: Dave Cridland <[email-masked]>
Date: Thu, 16 Feb 2012 11:50:08 +0000

> OK.. I'm struggling to understand why...
What, why corporates deploy XMPP? That's wildly off-topic for this  
list.

>>> Maybe it should just be an SSH tunnel... but that;s  
>>> back-pedalling quickly and reducing potential user experience  
>>> with it.
>>> 
>>> 
> Just trying to think of "cheap" ways to get single sign on and  
> single port using the existing protocols.  But it would still be a  
> bandaid.
> 
It's just part of reducing manual configuration, which we're all  
agreed is a good thing.

Using multiple ports has no bearing, though.

>> Right, sure, understood (after s/MTA/MUA/) but what has port 25  
>> got to do with it?
> 
> back to my point about getting everything over 1 port.  If we had  
> that, then blocking 25 wouldn't have affected me.

But my point is that blocking port 25 shouldn't be affecting you  
anyway.

Submission runs on port 587, has done for years.

>> tolerably with Exchange - where it worked really quite well. So  
>> they managed to leverage from Windows to Office, and from Office  
>> to Exchange.
> 
> Actually last time I compared prices of versions of Office, the  
> Outlook component accounted for like $500 NZ.
> 
> 
Yes, indeed - now they charge for it. But originally, the Office  
suite covered everything.

> It's also possibly the worst IMAP implementation out there.  But  
> they don't _want_ you to use IMAP.   They want you to buy Exchange.
> 
> 
Right, exactly my point.

>> instead of finding solutions and trying to figure out how to use  

... an infrastructure problem, mostly.

> * UDNs

I'm not familiar with this acronym. Nor is Wikipedia or Google, I'm  
afraid.

> * configuration issues

Right, and discovery (and decent implementations using it) solves 99%  
of this without any changes to the core protocols.

Dave.

---

From: Bron Gondwana <[email-masked]>
Date: Thu, 16 Feb 2012 13:11:36 +0100

> then blocking 25 wouldn't have affected me.

If you were using port 587, it wouldn't have affected you - that's been
standardised for a while.

The more important thing is - if we were going over one port, it would
have either not affected you, or TOTALLY affected you - with nothing
working.  Either of those is significantly more understandable to the
"punter" than some things working, and others not.

The same with "Send succeeds, but then append-to-Sent-folder fails" if
your IMAP connection is unavailable but your SMTP server isn't.  Or if
you're over quota.

BURL at least solves the second one, but it solves it in a half arsed
way that the little XSEND I put together last week doesn't.

With XSEND, you upload the message to IMAP first, then you say:

TAG XSEND UID

Which sends the message UNLESS it has the \XSent flag already.

And it sets the \XSent flag as soon as the message is sent.  No
waiting on network IO.  So the only gap is a failure within the
server itself.  If the client disconnects and tries again, the
flag will already be set, and it will not duplicate the message.

No partial failures.  No multiple systems depending on each other
to be up, configured correctly, routing to each other.

(Implementation in this case is "Call sendmail -bmi and pipe the
spool file to it")

> > of finding solutions and trying to figure out how to use them.

There's a checklist for why this one won't be solved.

> * UDNs

In the case of wingate, because of the embedded SMTP server, you could
theoretically hold the client waiting until you had tried one outbound
SMTP.  Only fixes the first hop of course.

A "ocean boiling" solution to this would involve server level magic on
control messages.  You'd want forgery protection though... gets trickier
when you consider active attacks.  Mind you, MDN is already fraught with
active attacks.

Bron.
-- 
  Bron Gondwana
  [email-masked]

---

From: Adrien de Croy <[email-masked]>
Date: Fri, 17 Feb 2012 01:19:12 +1300

On 17/02/2012 12:50 a.m., Dave Cridland wrote:
>
> What, why corporates deploy XMPP? That's wildly off-topic for this list.
>

why so many.

it's ok, I understand those stats better now.  I think they are skewed.  
Top 500 sites are quite different to the other 99% of sites.

> Using multiple ports has no bearing, though.

not true.  Ports can be (and are) individually blocked, filtered 
whathaveyou.  Therefore availability of individual features within a 
mail application can be variable rather than all or nothing.  Which is 
better is debatable, but there's more confusion associated with partial 
failure than complete failure IME.

> But my point is that blocking port 25 shouldn't be affecting you anyway.
>

ah... well I use SMTP :)

So do many clients.  Not enough years I think to make such a big impact.

> I'm not familiar with this acronym. Nor is Wikipedia or Google, I'm 

Un deliverable notifications.... pretty sure I'd seen that TLA around 
before.

by and large wrt bounce messages people

* don't read them
* don't understand them

they just ask tech support.  We get heaps of such questions.  There may 
be no helping some people, but it's not free.

> of this without any changes to the core protocols.

Discovery is a bit different to config.  It's only part of the picture.

I'd like to see it more fully fledged.  If ACAP had been widely adopted 
and deployed it would be a no-brainer.

---

From: Bron Gondwana <[email-masked]>
Date: Thu, 16 Feb 2012 13:41:43 +0100

> remain separate silos rather than the store being a soup of messages

Not really.  Existing servers would be a lot more efficient with a query
which limited to a single "folder", for sure - because they could optimise
it.  But it's no different than an SQL query across a partitioned table.
It means your in-memory-state needs to be big enough to accommodate all
the mailboxes that might be in the regular searches, of course.

But anyone accessing with an IMAP-model client would still access single
folders.  The only difference is that the model would accommodate
cross-folder requests as a first-order object, so you could sort them by
something other than "folder name first", which the existing cross-folder
solutions do.  If you're always sorting by folder first, you lose all the
benefits of having the server do it rather than the client synthesize the
results (the FastMail interface's cross-folder-search is a client
synthasize results thing - and it splits it up by folder... exactly
because the alternative means client-side sorting.  I wrote it many years
ago, and it does a "GUID" alternative which is "f${folderid}u${uid}" - kind
of ugly, and requires the "folder name to folderid" mapping in the database)

> concurrent TCP connections.

Yeah, I went and actually read the RFCs.  Inclined to agree.  It's
enterprisy in the extreme.

Now SCTP, that would be cool.  But support isn't there.

Bron.
-- 
  Bron Gondwana
  [email-masked]

---

From: Adrien de Croy <[email-masked]>
Date: Fri, 17 Feb 2012 01:57:25 +1300

> "punter" than some things working, and others not.

that's what I've noticed in support.

> way that the little XSEND I put together last week doesn't.

aieeee BURL...

until then, a mail server vendor had no need to write an IMAP client.

Would have been a trillion times simpler to implement submission from 
IMAP.  I can't even think of a mail server product with IMAP that 
doesn't have SMTP already.

Did anyone actually implement this?

It seems so fraught with problems / fragile that I'd expect very few 
implementations.  That's even apart from the complexity.

> With XSEND, you upload the message to IMAP first, then you say:

how do you provide the SMTP forward path?  Is that scraped from the headers?

maybe should be

TAG XSEND UID (addr, addr...)

> SMTP.  Only fixes the first hop of course.

sure, but if you're delivering direct to end MTA, you have pretty much 
the whole path.

> when you consider active attacks.  Mind you, MDN is already fraught with

half the problem is lack of standardisation.  MDNs should be 
machine-readable.

Adrien

---

From: Tony Finch <[email-masked]>
Date: Thu, 16 Feb 2012 13:06:11 +0000

> the mailboxes that might be in the regular searches, of course.

Have you seen UW-IMAP's in-memory state? :-)

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
South Biscay: Northerly or northeasterly, veering easterly or northeasterly 3
or 4. Slight or moderate. Mainly fair. Moderate or good.

---

From: Bron Gondwana <[email-masked]>
Date: Thu, 16 Feb 2012 15:57:45 +0100

> Have you seen UW-IMAP's in-memory state? :-)

Nup - I've seen Cyrus' though.  It could be trimmed considerably if we
had to...

Bron.

---

From: Cyrus Daboo <[email-masked]>
Date: Thu, 16 Feb 2012 10:15:08 -0500

Hi Sebastian,

--On February 16, 2012 10:35:43 AM +0100 Sebastian Hagedorn 
<[email-masked]> wrote:

> enter a @uni-koeln.de address in the new account wizard, all settings are
> filled in automatically.

Today we have SRV mechanisms for email and CalDAV/CardDAV. Whilst the 
former is relatively new and apparently not supported in any big way (*), 
the later is used by servers and clients.

That said, at the Calendaring and Scheduling Consortium, we have been 
discussing putting together a more generic "account" provisioning mechanism 
as opposed to a per-service lookup which is essentially what SRV provides. 
I am planning on starting some discussion inthe IETF apps area on this. An 
initial proposal would be to use an HTTP well-known URI to advertise a 
site's set of services and basic user account provisioning. Many of the 
vendors involved in CalConnect want this because they feel that the current 
approach of configuring "standards" based protocols is too unwieldy for 
users (e.g. having to separately setup email, calendaring, chat etc) vs the 
kind of setup one gets with e.g. Exchange.

Now one could argue that the current complexity could be hidden behind a 
simple UI on the client (where the client does all the multitude of SRV 
lookups to see what services are available) but there are downsides to that 
(multiple DNS requests, inability to tailor response on a per-user basis 
etc) - basically the process is too complex.

So, I (we - CalConnect) want to see something done in this area. Keeping 
the scope as small and simple as possible will be key though...

(*) Some services do have SRVs for email, CalDAV, CardDAV, e.g. try:

dig _imaps._tcp.gmail.com srv
dig _caldavs._tcp.gmail.com srv
dig _caldavs._tcp.icloud.com srv
dig _carddavs._tcp.icloud.com srv

-- 
Cyrus Daboo

---

From: Tony Finch <[email-masked]>
Date: Thu, 16 Feb 2012 15:21:55 +0000

> Nup - I've seen Cyrus' though.  It could be trimmed considerably if we

The problem is UW-IMAP is structured so it can only handle one mailbox at
a time.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Dogger, Fisher, German Bight, Humber, Thames, Dover: West or southwest 4 or 5,
increasing 6 for a time in Fisher. Slight or moderate, occasionally rough in
Fisher. Occasional rain or showers. Moderate or good.

---

From: Cyrus Daboo <[email-masked]>
Date: Thu, 16 Feb 2012 10:23:49 -0500

Hi Giovanni,

> names ?), and client will be able to reconfigure themselves at next

One of the things we are doing in CalDAV is supporting a server-driven 
client re-configuration mode. In large scale CalDAV deployments it is 
sometimes more efficient to "redirect" users to a specific host rather than 
rely on some internal-to-the-server reverse proxying. To deal with that we 
simply have servers change a WebDAV property from a path absolute value 
(e.g. '/calendars/users/cyrus') to one with an FQDN (e.g. 
'https://newserver.example.com/calendars/users/cyrus'). Clients are then 
expected to check that property on a regular basis and if they see it point 
to a new host, they "re-base" the user account accordingly.

And of course IMAP has a similar mechanism - RLOGIN.

Now I think I would prefer to stick to a mechanism like that - i.e. the 
service (imap, CalDAV etc) provides a "redirect" mechanism or at least a 
signal to the client, to indicate that a re-basing of the account is 
needed. But that could also be the trigger for the client to go re-do 
account provisioning. However, you need to be careful in terms of 
understanding client side "layering". i.e. there are different apps for 
each service, and possible a separate overall account configuration system. 
It may not be feasible for say the email app to tell the account config app 
to re-do all the services for that account. In any case, there are some 
interesting things to think about here.

-- 
Cyrus Daboo

---

From: Cyrus Daboo <[email-masked]>
Date: Thu, 16 Feb 2012 10:34:57 -0500

Hi Dave,

--On February 16, 2012 10:25:22 AM +0000 Dave Cridland <[email-masked]> 
wrote:

> Oh, totally agree. Discovery is awesome, user-configuration is insane. I
> don't see there's any valid argument the other way.

Practical experience with SRV has shown that, whilst it is fine for large 
service providers to do that sort of thing (c.f., Google, iCloud etc), the 
practicalities of actually being able to setup SRV records and have proper 
SSL cert validation is heard, particularly for "hosted domain" type 
applications.

Having discussed with several engineers who have implemented SRV it is 
clear that having each app do it separately is problematic because there 
are a bunch of awkward issues. Not in the least is the simple issue of even 
getting SRV from standard DNS libraries (e.g. one Android developer 
mentioned that the size of his app increased significantly when he had to 
bring in libraries to do SRV). Then there is all the certificate 
verification stuff that needs to be done to properly implement discovery. 
Overall this is much harder than it needs to be, and other non-standard 
approaches have proved that.

-- 
Cyrus Daboo

---

From: Cyrus Daboo <[email-masked]>
Date: Thu, 16 Feb 2012 10:53:15 -0500

Hi Adrien,

--On February 17, 2012 12:44:56 AM +1300 Adrien de Croy <[email-masked]> 
wrote:

> as someone looking to add CalDAV to a mail server, wouldn't it be nice if
> you didn't have to
>
> a) write a web server
> b) write DAV extensions

You fail to appreciate the hard part here - it is not the protocol (which, 
guess what, is not that hard to do as there are many off-the-shelf webdav 
implementations to pick on as a starting point - and indeed within a few 
months of the initial draft being published we had several servers and 
clients interoperating). The hard part is the semantics of calendaring. As 
someone who has lived in both the IMAP (email) world, and the 
iCalendar/CalDAV world, I can tell you that calendaring is like an order of 
magnitude more complex than email - specifically scheduling.

CalDAV servers are very "write heavy" in that there are a lot of 
modifications happening to existing data - that is something not typical of 
an IMAP server where modifications are simply metadata changes (flags) or 
actual message "injection" (delivery, APPEND) or deletion. So if you want 
to build a high performance CalDAV server you need to take that into 
account and build something whose scalability is based on a different set 
of client/server interactions than is typical for IMAP.

That is not to say that building that within IMAP or on top of an existing 
mailstore is impossible - it is. But what is more important is to fully 
understand the core use cases - or more importantly the requirements for 
the client/server api.

What I would really like us to focus on here, is not IMAP5 per se, but 
instead a "generic" mail store access API. Lets define the key operations 
needed by clients and a server API that can provide those behaviors. Once 
we have that, we can fit it into any protocol we like, be it extensions to 
IMAP4 (to make IMAP5), HTTP, XMPP whatever. The same thing can be done for 
a calendar store api (and indeed the Calendaring and Scheduling Consortium 
has been working on generic abstractions giving rise to REST and SOAP based 
protocols all built on the same store api model used by CalDAV).

-- 
Cyrus Daboo

---

From: Sebastian Hagedorn <[email-masked]>
Date: Thu, 16 Feb 2012 17:06:23 +0100

Hi Cyrus,

> --On February 16, 2012 10:25:22 AM +0000 Dave Cridland
> <[email-masked]> wrote:
>
> service providers to do that sort of thing (c.f., Google, iCloud etc),
> the practicalities of actually being able to setup SRV records and have
> proper SSL cert validation is heard, particularly for "hosted domain"
> type applications.

This is veering off-topic, but it's something I've spent quite a bit of 
time on: I suppose that you could call one of Germany's largest 
universities a large service provider. Setting up SRV records and 
certificates is no big deal for us. I have set up RC 6186 SRV records for 
the uni-koeln.de domain, but there are two problems:

• I'm not aware of a single client that actually uses those
• it's not enough.

What we need is a mechanism to map from an email address to a combo of 
server addresses and logins. Both the Microsoft approach and the 
Thunderbird model allow for that. With RFC 6186 the user still needs to 
know what his or her login is ... in our case it's often not the email 
address.
--
Sebastian Hagedorn - RZKR-R1 (Flachbau), Zi. 18, Robert-Koch-Str. 10
Regionales Rechenzentrum (RRZK)
Universität zu Köln / Cologne University - Tel. +49-221-478-5587

---

From: Tony Finch <[email-masked]>
Date: Thu, 16 Feb 2012 16:40:32 +0000

> how do you provide the SMTP forward path?  Is that scraped from the headers?

That's the right thing to do. You also need to do BCC: processing.
(sendmail -t does the right thing.)

The rationale for BURL is that there is more to the SMTP envelope than
just the sender and recipient addresses - in particular there are the DSN
attributes. There's a somewhat ugly and ill-defined split between
information for MTA processing (in the envelope) and information for MUA
processing (in the headers - see MDN for example). But in fact MTAs do
header processing too, so there no practical advantage to ESMTP envelope
extensions and a lot of complexity disadvantage.

There are a few envelope extensions: DSN, future release, message
tracking, CONNEG and CONPERM facsimile media conversion, and 8BITMIME +
BINARYMIME. If you want to eliminate BURL you need to either define a
mapping from headers to the extension parameters that you want to support,
or embed ESMTP inside IMAP.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Fair Isle: West or northwest, backing southwest, 5 to 7. Rough or very rough.
Squally wintry showers. Good, occasionally poor.

---

From: Bron Gondwana <[email-masked]>
Date: Thu, 16 Feb 2012 20:18:27 +0100

> The problem is UW-IMAP is structured so it can only handle one mailbox at

Cyrus too.  Which is a reason for not forcing the model of "one big
pool".  I'm not committed to one big pool - if there's a syntax for
joining smaller pools together to sort/search across them.  This is
one of the "more work for the server, less for the client" options,
where we assume people writing servers are slightly more clueful.

Besides, they'll have a test to run :)

Bron.

---

From: Bron Gondwana <[email-masked]>
Date: Thu, 16 Feb 2012 20:36:33 +0100

On Thu, Feb 16, 2012 at 10:53:15AM -0500, Cyrus Daboo wrote:
> What I would really like us to focus on here, is not IMAP5 per se,
> but instead a "generic" mail store access API.

Absolutely - this is what I've been saying (or at least trying to
say) all along!

Though I would say "API _and_ data model".  And of course complience
tests that ensure the API and the data model are correct.

> Lets define the key
> operations needed by clients and a server API that can provide those
> behaviors. Once we have that, we can fit it into any protocol we
> like, be it extensions to IMAP4 (to make IMAP5), HTTP, XMPP
> whatever. The same thing can be done for a calendar store api (and
> indeed the Calendaring and Scheduling Consortium has been working on
> generic abstractions giving rise to REST and SOAP based protocols
> all built on the same store api model used by CalDAV).

Yes please.  As I've said, I'm really happy to spend a lot of time
on this.  My initial posts were largely grown from the frustration
of seeing IMAP5 discussions bogging down in silly little bits of
custom syntax and bandaids.

I will continue with my initial task of accumulating all the
relevant RFCs and documenting the pain points they solved.  I
don't know all of them yet!

Bron.

---

From: Dave Cridland <[email-masked]>
Date: Thu, 16 Feb 2012 19:52:32 +0000

On Thu Feb 16 15:34:57 2012, Cyrus Daboo wrote:
>> insane. I
> Practical experience with SRV has shown that, whilst it is fine for  
> large service providers to do that sort of thing (c.f., Google,  
> iCloud etc), the practicalities of actually being able to setup SRV  
> records and have proper SSL cert validation is heard, particularly  
> for "hosted domain" type applications.
> 
> 
I agree with what you're saying, but I think it's getting  
substantially easier as time goes on.

It used to be a rarity for XMPP services to have SRV records, and  
very rare to have them relied upon, but it's now commonplace, and the  
numbers of systems that require SRV resolution for them to be  
reachable is increasing.

> Having discussed with several engineers who have implemented SRV it  
> is clear that having each app do it separately is problematic  
> because there are a bunch of awkward issues. Not in the least is  
> the simple issue of even getting SRV from standard DNS libraries  
> (e.g. one Android developer mentioned that the size of his app  
> increased significantly when he had to bring in libraries to do  
> SRV). Then there is all the certificate verification stuff that  
> needs to be done to properly implement discovery. Overall this is  
> much harder than it needs to be, and other non-standard approaches  
> have proved that.

Again, I think SRV resolution in software is increasingly common, and  
becoming generally easier.

It used to be next to impossible in many environments, whereas it's  
now merely awkward.

Dave.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 16 Feb 2012 21:25:22 +0100

On 02/16/2012 04:34 PM, Cyrus Daboo wrote:
> Not in the least is the simple issue of even getting SRV from standard
> DNS libraries (e.g. one Android developer mentioned that the size of his
> app increased significantly when he had to bring in libraries to do SRV).

The code they're complaining about was ported to android by me, and I
must say that the code isn't at all optimised for that use case. It
wouldn't be too difficult to strip out about 95% of the code and leave a
simple DNS query library.

I didn't care enough to do it, and I observe that neither did the
developer you spoke to.

Arnt

---

From: Adrien de Croy <[email-masked]>
Date: Fri, 17 Feb 2012 09:50:01 +1300

One of the things that discovery gives you is an opportunity when the 
discovery is initiated to make a decision about what will provide the 
service(s).

this means load-balancing, failover etc can be implemented.

One issue with using a well-known http URI for user account discovery, 
is that most organisations that run web services already have a website 
set up on their base domain.  So you'd have to wedge the discovery 
provision into an existing web site and infrastructure.

Another option could be to use DNS again.

e.g. do a dns SRV lookup for say _localpart._allservices._example.com

which could return multiple records, one for each available service.  
Could even use SRV records for this where the record name is _ldap._tcp 
etc.... possibly need some CNAME glue.

Adrien

On 17/02/2012 4:23 a.m., Cyrus Daboo wrote:
> sometimes more efficient to "redirect" users to a specific host rather 
> than rely on some internal-to-the-server reverse proxying. To deal 
> with that we simply have servers change a WebDAV property from a path 
> absolute value (e.g. '/calendars/users/cyrus') to one with an FQDN 
> (e.g. 'https://newserver.example.com/calendars/users/cyrus'). Clients 
> are then expected to check that property on a regular basis and if 
> they see it point to a new host, they "re-base" the user account 
> accordingly.
>
> Now I think I would prefer to stick to a mechanism like that - i.e. 
> the service (imap, CalDAV etc) provides a "redirect" mechanism or at 
> least a signal to the client, to indicate that a re-basing of the 
> account is needed. But that could also be the trigger for the client 
> to go re-do account provisioning. However, you need to be careful in 
> terms of understanding client side "layering". i.e. there are 
> different apps for each service, and possible a separate overall 
> account configuration system. It may not be feasible for say the email 
> app to tell the account config app to re-do all the services for that 
> account. In any case, there are some interesting things to think about 
> here.
>

---

From: Adrien de Croy <[email-masked]>
Date: Fri, 17 Feb 2012 09:55:58 +1300

On 17/02/2012 4:53 a.m., Cyrus Daboo wrote:
> Hi Adrien,
>
> --On February 17, 2012 12:44:56 AM +1300 Adrien de Croy 
> <[email-masked]> wrote:
>
> You fail to appreciate the hard part here - it is not the protocol 
> (which, guess what, is not that hard to do as there are many 
> off-the-shelf webdav implementations to pick on as a starting point - 
> and indeed within a few months of the initial draft being published we 
> had several servers and clients interoperating). The hard part is the 
> semantics of calendaring. As someone who has lived in both the IMAP 
> (email) world, and the iCalendar/CalDAV world, I can tell you that 
> calendaring is like an order of magnitude more complex than email - 
> specifically scheduling.

I'm not discounting the difficulty and complexity of writing 
calendaring.  It's just that putting it over HTTP adds complexity is 
all.  Not everyone can use an off-the-shelf component for that.

Writing a web server is non-trivial.  Making it hardened enough to 
withstand the internet is another issue.

It's harder to defend against http-borne attacks than IMAP attacks.

> modifications happening to existing data - that is something not 
> typical of an IMAP server where modifications are simply metadata 
> changes (flags) or actual message "injection" (delivery, APPEND) or 
> deletion. So if you want to build a high performance CalDAV server you 
> need to take that into account and build something whose scalability 
> is based on a different set of client/server interactions than is 
> typical for IMAP.
>
> That is not to say that building that within IMAP or on top of an 
> existing mailstore is impossible - it is. But what is more important 
> is to fully understand the core use cases - or more importantly the 
> requirements for the client/server api.

Yeah, I don't know that I was proposing building CalDAV into IMAP 
itself.  That's where I was going with layering.

> instead a "generic" mail store access API. Lets define the key 
> behaviors. Once we have that, we can fit it into any protocol we like, 
> be it extensions to IMAP4 (to make IMAP5), HTTP, XMPP whatever. The 
> same thing can be done for a calendar store api (and indeed the 
> Calendaring and Scheduling Consortium has been working on generic 
> abstractions giving rise to REST and SOAP based protocols all built on 
> the same store api model used by CalDAV).
>

IMAP is fairly feature-rich.  E.g. If we want to support server-side 
searching (which I hope we do), then the server needs to be able to 
unpick mime, and have a query language rich enough to be useful.  That 
seems to me (on the face of it) to be a bit beyond just a mail store.

Regards

Adrien

---

From: Adrien de Croy <[email-masked]>
Date: Fri, 17 Feb 2012 10:00:07 +1300

On 17/02/2012 5:40 a.m., Tony Finch wrote:
> Adrien de Croy<[email-masked]>  wrote:

I think the guys that developed SMTP would disagree with you.  The 
reason the envelope is even specified in SMTP rather than the receiver 
simply scraping them out of the message, is that sometimes you need to 
deliver a message somewhere other than the To: / bcc: headers.

> or embed ESMTP inside IMAP.

yep, I'd definitely go for embedding SMTP into IMAP... since

a) SMTP is a trivially simple protocol for an IMAP server to use to 
submit a message on behalf, whereas an IMAP client is much more complicated,
b) take a look at the security implications in the BURL RFC.

Adrien

---

From: Adrien de Croy <[email-masked]>
Date: Fri, 17 Feb 2012 10:00:55 +1300

one thing - what do you mean by "one big pool".

Cheers

Adrien

---

From: Bron Gondwana <[email-masked]>
Date: Thu, 16 Feb 2012 23:31:04 +0100

> one thing - what do you mean by "one big pool".

Similar to quotaroot - "messageroot".  We're going to do it
with users and the conversations database.  For now they're
actually locked to users, but it doesn't handled thousands
of shared folders well, because they would all shared one
central DB.  So one 'root' per user which shares a single
highestmodseq and uidvalidity counter.

In particularly, we could then enforce every single folder
having a different UIDVALIDITY, so we could use a single 64
bit value to uniquely identify message/folder.  It would even
allow multiple consecutive messages in the same folder to
compress into a range.

This will allow supporting shared folders quite nicely by
rooting each one in itself as a standalone folder, no shared
state.

That's the idea, anyway.

Bron.

---

From: Bron Gondwana <[email-masked]>
Date: Thu, 16 Feb 2012 23:37:24 +0100

On Fri, Feb 17, 2012 at 10:00:07AM +1300, Adrien de Croy wrote:
> reason the envelope is even specified in SMTP rather than the
> receiver simply scraping them out of the message, is that sometimes
> you need to deliver a message somewhere other than the To: / bcc:
> headers.

Rarely.  I don't even know an IMAP client which supports doing that.
It's not like SMTP would disappear if you need something more complex.

> yep, I'd definitely go for embedding SMTP into IMAP... since

Simple, heh.

> b) take a look at the security implications in the BURL RFC.

Definitely agree that

Client <-----> IMAP ------> SMTP

is much simpler dataflow than

        +--------------------+               +--------------+
        |                    | <------------ |              |
        |     MUA (M)        |               | IMAPv4Rev1   |
        |                    |               |  Server      |
        |                    | ------------> | (Server I)   |
        +--------------------+               +--------------+
               ^    |                              ^     |
               |    |                              |     |
               |    |                              |     |
               |    |                              |     |
               |    |                              |     |
               |    |                              |     |
               |    |                              |     v
               |    |                        +--------------+
               |    |----------------------> |   SMTP       |
               |                             |   Submit     |
               |-----------------------------|   Server     |
                                             |  (Server S)  |
                                             +--------------+

Half as many communication channels.

Bron.

---

From: Dan White <[email-masked]>
Date: Thu, 16 Feb 2012 16:41:24 -0600

>b) take a look at the security implications in the BURL RFC.

However, spam is a problem caused by a deficiency (or whatever you want to
call it) within the smtp protocol. imap does not directly or indirectly lead
to the generation of spam, unless you want to consider backscatter from a
sieve notification to be spam.

smtp (or submission) is one of those things that's actually easy to
configure in an email client. I fear that if the two get combined on the
same port that imap server IPs may start to get caught up in the cat
and mouse game spam parsers play.

-- 
Dan White

---

From: Mark Crispin <[email-masked]>
Date: Thu, 16 Feb 2012 14:48:41 -0800 (PST)

> Rarely.  I don't even know an IMAP client which supports doing that.

HAHAHAHAHAHAHAHAHAHAHAHAHAHA.

-- Mark --

http://panda.com/mrc

---

From: Adrien de Croy <[email-masked]>
Date: Fri, 17 Feb 2012 11:50:08 +1300

> It's not like SMTP would disappear if you need something more complex.

parsing the message file is a lot more work than the client passing the 
info over (which it already has).

just sayin'

then can handle both scenarios... list processing etc... opportunities 
open up instead of being shut down.

Adrien

---

From: Brandon Long <[email-masked]>
Date: Thu, 16 Feb 2012 14:53:18 -0800

> It's not like SMTP would disappear if you need something more complex.

mutt comes to mind, though its not doing it through SMTP but via
arguments to sendmail.  Well, maybe it does support them with the
newer built-in smtp submission.

Brandon

---

From: Brandon Long <[email-masked]>
Date: Thu, 16 Feb 2012 14:57:53 -0800

> and mouse game spam parsers play.

IMAP playing the game is no different than SMTP-MSA playing the game,
it still requires authentication.  I'm not sure how wide-spread it is,
but we certainly already have to deal with spam through spammy
accounts or hijacked accounts over SMTP-MSA.  I find it hard to
believe that IMAP doing mail submission is going to change any of
that, but perhaps you mean that you would have to have the same smarts
for protecting against bad logins in IMAP that you have in SMTP-MSA...
but I would expect anyone who's under attack for either already does
that as well (we certainly use the same auth backend for both.

Brandon

---

From: Dan White <[email-masked]>
Date: Thu, 16 Feb 2012 17:29:54 -0600

>that as well (we certainly use the same auth backend for both.

Separating imap from submission allows those functions to (easily) run on
separate servers, like where the smtp server is a black box appliance,
which is not uncommon in the service provider or enterprise space.

Focussing on changing the mail submission aspect of the picture may
introduce unexpected side effects, such as how other providers parse and
block your messages (which some do based on client *and/or* relay IP).

imap essentially already has its own mail submission component via imap
append. Users can trust who sends them messages, and can limit who can send
them messages (via enforceable acls). I just wish smtp worked more like
that, but that's a pipe dream.

-- 
Dan White

---

From: Adrien de Croy <[email-masked]>
Date: Fri, 17 Feb 2012 13:51:50 +1300

> which is not uncommon in the service provider or enterprise space.

Whichever way you do it, BURL vs SUBMIT, there is an issue of trust if 
the 2 servers are in different realms.

And those issues were addressed in the BURL RFC, and the same security 
conditions apply.

Notwithstanding all of that.  It's still easier for a developer to hook 
up a way for an IMAP server to submit mail to SMTP than it is to hook up 
SMTP to retrieve IMAP.

If the servers are on the same computer, it can be as simple as a file copy.

One option would be an extension for submission that would only be 
available if the SMTP server trusted the IMAP server.

That would then avoid the need to smuggle SMTP creds across your IMAP 
session (c.f. the way that BURL specifies IMAP creds in BURL requests).

I think in most organisations this would prove a lot simpler to manage.  
Certainly a lot simpler to code as a developer.

Maybe it should be an optional extension, usable as an alternative to BURL.

I'd still be keen on any deployment stats on BURL.

> that, but that's a pipe dream.

I don't know how you can use APPEND to send a message to another user 
unless you share a folder with them.

Adrien

---

From: Bron Gondwana <[email-masked]>
Date: Fri, 17 Feb 2012 04:58:00 +0100

> HAHAHAHAHAHAHAHAHAHAHAHAHAHA.

I have not heard of this IMAP client.  Please provide usage stats
to help a determination if it's a significant player that we need
to plan around.

Thanks,

Bron.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 17 Feb 2012 10:10:49 +0100

> Rarely.  I don't even know an IMAP client which supports doing that.

Some remove Bcc themselves, or never add it. That is, even though the
word 'Bcc' may be visible onscreen, it does not occur in the submitted
message.

Arnt

---

From: Jan Kundrát <[email-masked]>
Date: Fri, 17 Feb 2012 11:15:23 +0100

> It's not like SMTP would disappear if you need something more complex.

Even Thunderbird supports this, although through an extension nowadays
("Redirect Mail" IIRC). Unless I'm mistaken, they used to support it
out-of-the-box in the TB 2.x series.

You're probably right that "most people" have never used it, but it's a
feature that's here for a good reason and could be useful, IMHO.

Cheers,
-jkt

---

From: Tony Finch <[email-masked]>
Date: Fri, 17 Feb 2012 11:30:40 +0000

Adrien de Croy <[email-masked]> wrote:
>
> I think the guys that developed SMTP would disagree with you.  The reason the
> envelope is even specified in SMTP rather than the receiver simply scraping
> them out of the message, is that sometimes you need to deliver a message
> somewhere other than the To: / bcc: headers.

I don't believe that is true for message submission. See the discussion in
RCF 5322 about how the BCC header can turned into a message envelope (or
in some cases, several).

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Fair Isle: Southwest 6 to gale 8, veering northwest 7 to severe gale 9,
perhaps storm 10 later. Very rough or high. Rain or squally showers, wintry
later. Good, occasionally poor.

---

From: Tony Finch <[email-masked]>
Date: Fri, 17 Feb 2012 15:36:13 +0000

Adrien de Croy <[email-masked]> wrote:
>
> Whichever way you do it, BURL vs SUBMIT, there is an issue of trust if the 2
> servers are in different realms.

I don't think that setup is worth supporting in a simplified protocol.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Viking, North Utsire, South Utsire: Westerly 5 or 6, backing southwesterly 5
to 7, perhaps gale 8 later in Viking. Rough, occasionally very rough in Viking
and North Utsire. Rain or squally showers. Good, occasionally poor.

---

From: Tony Finch <[email-masked]>
Date: Fri, 17 Feb 2012 15:37:08 +0000

> out-of-the-box in the TB 2.x series.

That's what the Resent-* headers are for.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Trafalgar: Northeasterly 5 or 6, becoming variable 4. Moderate or rough. Fair.
Good.

---

From: Dan White <[email-masked]>
Date: Fri, 17 Feb 2012 11:14:57 -0600

On 02/17/12 13:51 +1300, Adrien de Croy wrote:
>>append. Users can trust who sends them messages, and can limit who can
>>send them messages (via enforceable acls). I just wish smtp worked more
>>like that, but that's a pipe dream.
>>
>

That's exactly what I want. I want to configure my ACLs to allow specific
users to connect via IMAP (or an SMTP replacement). If someone wants to
send me a message, their client connects directly to my server (why is
relay still necessary?). They authenticate over sasl using some fancy
federated authentication protocol (project moonshot) before being allowed
to post to my inbox.

1) The need for submission-and-relay goes away.
2) I can trust the identity of who's sending me a message.
3) I can fiddle with my acls bits to determine who I want to get messages
from.

When relay is *really* necessary, sasl authorization to allow servers to
act on behalf of domains/users should do the trick.

In my opinion (and I admit I'm getting off topic), spam is merely a problem
rooted in relay.

-- 
Dan White

---

From: Bron Gondwana <[email-masked]>
Date: Fri, 17 Feb 2012 20:40:59 +0100

> In my opinion (and I admit I'm getting off topic), spam is merely a problem

You have an excellent point that unavailable endpoints and incomplete
routing really are almost entirely a thing of the past.  There are still
some network structures where things aren't directly connected to the world,
but IPv6 should solve the remaining routability issues.

BUT - for me at least, I don't want to solve this problem.  It's a massive
problem for sure.  I'm not interested in your "point 3" though.  It puts the
administrative burden of adding every webshop I've ever used to my whitelist
on to me.

Sure it may be technically feasible - but it's just not the pain point that
_I_ am feeling, so it's not in my vision of a replacement protocol for IMAP.
I'm purely concerned with communications between the user agent and their
remote data store/server.

Bron.

(in this theoretical world you could talk direct SUBMISSION to the remote
 users' servers and not even involve your IMAP$n server at all, given a
 federated authentication)

---

From: Dan White <[email-masked]>
Date: Fri, 17 Feb 2012 14:05:47 -0600

>but IPv6 should solve the remaining routability issues.

I don't think the disconnected state really matters. The email could just
sit in a local queue until it's reconnected to the network, at which point
it could be send directly to the recipient's lmtp server (which makes
better sense than direct imap, now that I think about it).

>administrative burden of adding every webshop I've ever used to my whitelist

It's the same burden that Facebook users have, and points to the advantage
that Facebook and Facebook like services have, one in which they have
perfect knowledge of who's sending messages to who. It makes stopping spam
a lot easier. I suppose someone you don't know could send you 10,000
invites, but Facebook will just deliver one of those. Or one of your
friends goes crazy and annoys you will 100 messages, as which point you
remove them from your list. 

Are the younger folks coming up today going to use email if the spam
problem isn't solved? I doubt it. SMS and Facebook seem to suit most folks
under age X (where X will continue to increase linearly over time).

imap, smtp, and even exchange are pretty much doomed to role of legacy
enterprise application if that continues.

> federated authentication)

I don't really even think there needs to be a change to the basic imap/lmtp
specifications to support my grandios plan for eradicating spam (although
I'm far from the first to have such a light bulb go off). As soon as the
appropriate sasl mechanisms are in place to support some federated
authentication (think authentication against Facebook), then all that's
left is to improve ACL support in servers and clients. Simple matter...

-- 
Dan White

---

From: Mark Crispin <[email-masked]>
Date: Fri, 17 Feb 2012 13:10:37 -0800 (PST)

> better sense than direct imap, now that I think about it).

What if the network you are on requires you to use their servers, and
enforces that requirement through various evil means? Their policy may
allow external traffic going in, even from a remote IMAP server, but not
out. They even put themselves as a MITM on SSL/TLS (not that anyone pays
attention to cert validation messages anyway).

Outgoing email security: it's not just for totalitarian dictatorships any
more.

> under age X (where X will continue to increase linearly over time).

I doubt that they will use email in any case. There is simply no
compelling reason for them to do so.

> enterprise application if that continues.

I agree with this statement other than the word "legacy". Email originated
as an enterprise application and was designed as to be an enterprise
application. To date, nothing has filled that niche.

Attempts to create FB-style applications for the enterprise have not done
well. One problem is that enterprises are reluctant to turn internal data
over to "the cloud". Those that have done so (primarily academia) are in
the process of learning some hard lessons.

> I'm far from the first to have such a light bulb go off).

This is a definite "well, duh!" statement. You are correct; eliminating
spam is not a correct motivation.

> As soon as the

"Small matter of programming".

-- Mark --

http://panda.com/mrc

---

From: Bron Gondwana <[email-masked]>
Date: Fri, 17 Feb 2012 22:31:08 +0100

> better sense than direct imap, now that I think about it).

Not if you want to solve non-delivery notifications as well.  Otherwise,
store and forward isn't too bad.  I'd love if it had a return path all
the way through, so each intermediate didn't delete its local copy
until EVERYONE had confirmed it to the destination.  It would help
with retryability too.

But that depends on Message-Id being not played fast-and-loose with,
and that boat sailed :(

> remove them from your list.

I know.

> under age X (where X will continue to increase linearly over time).

I have a whole talk about this that I've given a couple of times now.
I think there are slides online from at least one set of it.

Basically, "social networks are good for fun, email is for serious
stuff".  I pointed out that I have code from 10 years ago, and emails
from even longer than that, that still works pefectly.  I can still
access it all.  No "404", no "lost in a site redesign into a new
timeline", no "deleted by sender".  It's my copy, and it's immutable.

In other words, it behaves a lot more like a piece of paper.

I use facebook for social stuff - but I email copies of our internal
IRC logs to myself, and all my instant messaging chats - because once
it's in email, it's accessible everywhere, it's searchable, and it
doesn't disappear.

> enterprise application if that continues.

That's definitely a concern.

> left is to improve ACL support in servers and clients. Simple matter...

Heh.  Go for it.  I'll support your crusade, but I won't lead the charge on
this one.  Client/Server is my baby.

Bron.

---

From: Bron Gondwana <[email-masked]>
Date: Fri, 17 Feb 2012 22:35:24 +0100

> attention to cert validation messages anyway).

Sounds pretty easy to do with a nice proxyable protocol, they just block
it, and you need to use a client which supports sending to an SMTP server.

Hopefully a relatively rare case.

Rarely enough, I agree with the rest of what you said, so I won't leave
it inline with a bunch of m3t00.

Bron.

---

From: Adrien de Croy <[email-masked]>
Date: Sat, 18 Feb 2012 20:12:35 +1300

> I don't think that setup is worth supporting in a simplified protocol.

I agree.

If a remote SMTP Server needs to be used for submission, then the IMAP 
server can be configured with some creds it can use for all submissions 
from all clients.

---

From: Adrien de Croy <[email-masked]>
Date: Sat, 18 Feb 2012 20:13:19 +1300

AFAIR Bcc should never hit the wire.  That's how it can be "blind" 
carbon copy.

Otherwise you rely on a server to strip it.

---

From: Adrien de Croy <[email-masked]>
Date: Sat, 18 Feb 2012 20:14:08 +1300

the use case I'm thinking of is when you want to forward a message
to someone after it was already sent to someone else.

On 17/02/2012 11:15 p.m., Jan Kundrát wrote:

Cheers,
-jkt

---

From: Adrien de Croy <[email-masked]>
Date: Sat, 18 Feb 2012 20:24:36 +1300

On 18/02/2012 6:14 a.m., Dan White wrote:
> relay still necessary?). 
in some parts of the world dialup is still prevalent.

We can't presume everyone has a full time internet connection.

If the sender and receiver are both in this category, (which is common 
in such places) then finding an intersection in times for delivery if we 
only have end-to-end delivery could result in significant delays.  
Therefore store-and-forward is required.

> They authenticate over sasl using some fancy

Personally I'd be tempted to mandate use of X.509 (SSL) client certs and 
TLS.

Citizens and organisations of a country would have certs which were 
issued and vouched for by their government.

Every submitter into the secure mail system would require a cert, and 
abusers would have their cert revoked (and be fined/punished).

ISPs could vouch for clients, but could have their cert revoked if they 
supported spammers.

> In my opinion (and I admit I'm getting off topic), spam is merely a 
I believe it's rooted in the lack of responsibility.  Responsibility for 
the actions of a mail sender is not enforced.

Computers can't be tried and sent to jail.  They can't be responsible 
for their actions.  Only humans can.  So the concept of a computer doing 
something where no person holds responsibility for it is nonsensical.

To hold someone accountable for an action of a computer, that person 
must be identifiable as the sponsor of the action.  Certificates can do 
that.  Hence government involvement, since governments are the 
organisations responsible for establishing the identity of their citizens.

---

From: Bron Gondwana <[email-masked]>
Date: Sat, 18 Feb 2012 08:36:13 +0100

> Otherwise you rely on a server to strip it.

I entirely disagree.  I was thinking quite a bit about the "upload to
the IMAP server and then send it via SMTP while telling it who to
send to", and I actually don't like it.

Certainly in the XSENT I implemented, once it has sent, it sets an
\XSent flag on the immutable message.  Not an \XSent-to-one-address.

The message with an \XSent flag is a record that it was sent to the
addresses listed in that header.

And the copy that went "over the wire" up to the IMAP server is your
record of what you did.  That _should_ include the BCC, so you can go
back later and check who you sent it to.  Obviously, the BCC should
be stripped as it gets handed over to the delivery agent.

Bron.
-- 
  Bron Gondwana
  [email-masked]

---

From: Giovanni Panozzo <[email-masked]>
Date: Sat, 18 Feb 2012 08:45:17 +0100

> We can't presume everyone has a full time internet connection.

100% agree. Store and forward is still required in some part of the
world. I developed XATRN (http://xatrn.panozzo.it), and there are still
very some (few, very few) users that use it with intermittent Internet
connection. Yes, I think that the future will be for always-on
connections, but there is no full world coverage of such kind of
Internet access.

> Personally I'd be tempted to mandate use of X.509 (SSL) client certs and

Maybe X509 can be one of the weapons against spam. But today spam comes
from a "stolen" webserver (injectet PHP script) or from "stolen" PC
(zombie PC, zombie network).
Spam NEVER comes from the sender itself. SPAM comes from a stolen account :(
Yes, better knowing the stolen account can help in fix the problem,
linke telling the user to run antivirus/reinstall OS, or the webmaster
to check its .PHP files. But I don't think that identifiyng the user
with X509 cert or some other federated authentication will help.

---

From: Adrien de Croy <[email-masked]>
Date: Sat, 18 Feb 2012 23:06:43 +1300

> send to", and I actually don't like it.

with respect, others may like it and want to do it.

If you don't like it, you can design your server and client to behave 
the way you like without making it impossible for others to do it in the 
protocol.

others may prefer to submit the SMTP envelope information when they tell 
the server to send a message, and it could be stored as some meta-data 
on the message.

Sure, if there's a huge desire to do this then fall-back to SMTP may be 
possible... but we're doing submission in IMAP so that client authors no 
longer have to support that protocol right?

> \XSent flag on the immutable message.  Not an \XSent-to-one-address.

server is free to do whatever else it likes, like moving it to sent, or 
making a copy or whatever.

> be stripped as it gets handed over to the delivery agent.
I guess it's a grey area there - if you consider IMAP to be a remote 
mail client that you're driving from a distance, then it's ok to store a 
BCC header, just like you would if you were storing it in a local 
store.  But you're relying on the IMAP server to strip it when it 
submits the mail for delivery, which is a bit of work for the IMAP server.

If you don't have to parse and re-build messages all the time, you'll 
scale much better.

---

From: Adrien de Croy <[email-masked]>
Date: Sat, 18 Feb 2012 23:07:07 +1300

On 18/02/2012 8:44 p.m., Giovanni Panozzo wrote:
> world. I developed XATRN (http://xatrn.panozzo.it), and there are
> still very some (few, very few) users that use it with intermittent
> Internet connection. Yes, I think that the future will be for
> always-on connections, but there is no full world coverage of such
> kind of Internet access.
>
> Maybe X509 can be one of the weapons against spam. But today spam
> comes from a "stolen" webserver (injectet PHP script) or from "stolen"
> PC (zombie PC, zombie network).

plenty of spam comes from the sender not stolen accounts.  That's why 
the spammers do things like register their own domains and SPF records.

> with X509 cert or some other federated authentication will help.

the server will have a cert.  It can be seen as spamming, and its cert 
can be revoked.  That will cut it off.

Having to get another cert will provide an incentive for the admin to 
care about it.

---

From: Adrien de Croy <[email-masked]>
Date: Sat, 18 Feb 2012 23:13:37 +1300

> 3) I can fiddle with my acls bits to determine who I want to get messages

many companies publish email addresses, e.g. for sales or support enquiries.

They need to be able to receive mail from unknown people.

This doesn't fit with a whitelist approach to incoming.

Even Facebook allows you to send a message to someone without asking 
them to friend you.

Adrien

---

From: Tony Finch <[email-masked]>
Date: Sun, 19 Feb 2012 18:34:13 +0000

> be stripped as it gets handed over to the delivery agent.

Right. That last bit (creating the message envelope from the
From/To/CC/BCC headers and stripping BCC) is explained in RFC 5321,
and it's what sendmail -t does.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Shannon, Rockall, Malin: Southwesterly 5 to 7, occasionally gale 8 in Rockall
and Malin. Rough or very rough. Occasional rain. Moderate or good,
occasionally poor.

---

From: Bron Gondwana <[email-masked]>
Date: Sun, 19 Feb 2012 20:26:04 +0100

> and it's what sendmail -t does.

I've seen the argument of "what if you want to forward a copy on to
someone else for inspection, whatever" - without forming a new copy
of the message.  That's actually really hard to do in the current
universe, because the headers will make it look extra spammy.  You're
much better off forming a new message.  Also, storage space for
"Sent Items" is almost always cheap.  The record of who you sent it
to is almost always useful.  All that's really missing is a way to
form a "new" message from an "old" message without necessarily
downloading everything first - but that's what the Lemonade CATENATE
stuff was for.  Build a new message, potentially including the old
one as a message/rfc822 attachment, and then send that.

Again, I really think it's the rare case.  I'd rather not complicate
message sending (and knowing which recipients had the message sent
to them) for this case.  There will still be SMTP.  Most clients
don't provide it for good reason - it's weird, advanced,
non-intuative stuff.  Do Exchange or Facebook provide such interfaces?

I get a feeling that protocol designers tend to be people who are
paranoid about all the edge cases, which is good, but then wind up
optimising rare edge cases at the expense of being able to do the
normal operations easily - which is bad.  I don't want to optimise
for "I want to bounce this message untouched to a new recipient
without changing the sender", which is almost always a spam scanning
nightmare anyway, at the expense of "the \XSent flag means that it
was sent to the recipients named in the message, nothing more,
nothing less".

Bron.

---

From: Sebastian Hagedorn <[email-masked]>
Date: Sun, 19 Feb 2012 20:35:57 +0100

> universe, because the headers will make it look extra spammy.

Not necessarily. I just resent your message to myself. Mulberry, IMO still 
the best email client, adds the following headers:

Resent-Date: Sun, 19 Feb 2012 20:30:30 +0100
Resent-From: Sebastian Hagedorn <[email-masked]>
Resent-To: Sebastian Hagedorn <[email-masked]>
Resent-Message-ID: <[email-masked]>
X-Resent-Mailer: Mulberry/4.1.0a1 (Mac OS X)

Doesn't look spammy at all, because it's completely clear what has 
happened. I use that feature all the time.

>  You're
> much better off forming a new message.

No, because the idea is precisely to preserve the original headers.

> non-intuative stuff.  Do Exchange or Facebook provide such interfaces?

Facebook not, but Outlook/Exchange does ... it's called "Resend This 
Message...".
--
Sebastian Hagedorn - RZKR-R1 (Flachbau), Zi. 18, Robert-Koch-Str. 10
Regionales Rechenzentrum (RRZK)
Universität zu Köln / Cologne University - Tel. +49-221-478-5587

---

From: Adrien de Croy <[email-masked]>
Date: Mon, 20 Feb 2012 09:31:03 +1300

not everyone is on sendmail

we're talking about adding 2 parameters, a return path and forward 
path.    Hardly complex.

this allows us to
- not have to parse the headers
- not have to re-write the entire message when stripping bcc

these things are not cheap.  they will affect service capacity.

Why would you incur such penalty simply to avoid transmission of a 
couple of parameters?

It's trivially simple for an MUA to supply this information when it 
wants to send the message.

There's no reason to assume this information would be lost either.  It's 
up to the server how to record such things and make them visible.  
Whether that's in the sent items folder or something associated with the 
message.

Distinction between envelope and content is a key principle of SMTP.  
One could argue it's also a cause of certain issues, but I don't think 
it should be discarded without some fairly serious thought.

Adrien

---

From: Bron Gondwana <[email-masked]>
Date: Sun, 19 Feb 2012 23:01:13 +0100

On Sun, Feb 19, 2012 at 08:35:57PM +0100, Sebastian Hagedorn wrote:
> Not necessarily. I just resent your message to myself. Mulberry, IMO
> still the best email client, adds the following headers:
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^

> happened. I use that feature all the time.

Doesn't sound like it's the same IMAP UID to me.

So it looks like using "Resent-To" rather than "To" if Resent-To
exists handily covers your usecase.

> No, because the idea is precisely to preserve the original headers.

Sounds like you're forming a new RFC822 message stored on the server
rather than redirecting the actual original bytes...

> Facebook not, but Outlook/Exchange does ... it's called "Resend This

Do they attach the original?  Is the destination address stored in
a header somewhere in the saved copy so that you can see it?  Assuming
that information is stored somewhere, requiring it to be so and
parsing from that rather than extending the command is much cleaner.
It provides an audit trail of sorts.

Bron.

---

From: Bron Gondwana <[email-masked]>
Date: Sun, 19 Feb 2012 23:08:35 +0100

> - not have to re-write the entire message when stripping bcc

Most MTAs seem to do header rewriting by appending or stripping
parts of the header on the fly.  They don't rewrite the entire
message, they print the headers down the wire, then blat the
body.  That's not expensive unless you're doing it wrong.

> these things are not cheap.  they will affect service capacity.

Seriously?  We do it to millions of emails in Postfix every day
on a single box and the CPU barely breaks a sweat.  What sort of
capacities are you considering serving on a single machine here?
Micro-optimising this... if you can save a single fsync somewhere
you'll get much better optimisation than avoiding a header removal.
Remember you're probably ADDING a header anyway to note the sending
time and user/ip of the sender at the time it happened.

> Why would you incur such penalty simply to avoid transmission of a

Because I don't believe you when you make it out to be a big penalty.
You're not rewriting the header on disk, purely on what gets written
to the wire.

> wants to send the message.

Trivially simple, yes - not so simple to track and repeat.

> There's no reason to assume this information would be lost either.
> It's up to the server how to record such things and make them
> visible.  Whether that's in the sent items folder or something
> associated with the message.

Significantly more complex than a simple keyword.  For a rare case.
Parsing "Resent-To" instead handily covers the entire use-case that
you have raised, with no loss of functionality, and without bloody
raising the edge cases to higher importance than the simple common
case which happens all the time.  Make the easy things EASY and the
hard things possible, not everything equally hard.

> One could argue it's also a cause of certain issues, but I don't
> think it should be discarded without some fairly serious thought.

Happy to have serious thought about it, but in this case the initial
envelope should IMHO be part of the content, because you're sending
something out of your "\Sent" box, near enough, and what's there
should be a record of what you sent.

SMTP will still exist for the 0.1% cases.  Resent-To will handily
cover the normal "user operated client" use-case.  This is a method
for "submission by a user operated client", that's all.

Bron ( getting bogged down in side issues, whee )

---

From: Adrien de Croy <[email-masked]>
Date: Mon, 20 Feb 2012 12:00:05 +1300

> body.  That's not expensive unless you're doing it wrong.

that depends on how your system is structured.  The MTAs I'm  familiar 
with put files into their send queues, and specify envelopes separately, 
either in a separate file or in a DB or something..

So a message file re-write would be required to strip bcc before putting 
it into the queue, or a re-factor of the SMTP sending code to allow the 
option to parse the headers on the fly (the option would need to be 
communicated out-of-band).  That is of course do-able.  But it forces a 
model on the SMTP delivery agent.

What is the concern about sending the parameters?

That they can be different to the headers in the message?

the main use-case I can think of where the recipients and return-path 
are not normally in the message are in lists.  So with your system, you 
won't be able to have an efficient list processor attach to an IMAP 
message store.  It will need to make copies of the message for each 
recipient.

Another problem is auto-responders, where commonly you specify a 
blackhole or empty return path to avoid loops.

You'll at least need a way to update certain headers into the message 
efficiently.  This would make for much more efficient forwarding as 
well... copy, rewrite To, Subject and send.  Rather than download, edit, 
upload send.  Maybe store the headers separately.

> capacities are you considering serving on a single machine here?

I always save CPU when I can.  There's always a bigger client out there 
who will need more capacity.

> time and user/ip of the sender at the time it happened.

Senders don't normally stamp Received if that's what you're referring 
to, that happens at the SMTP receiver.  our sender doesn't write to the 
message.

> You're not rewriting the header on disk, purely on what gets written

Sure if you do it all your way... but not all mail systems work that way.

> Trivially simple, yes - not so simple to track and repeat.

Sure, displaying the information later is not something currently supported.

Your way allows existing sent items mailbox to continue to perform its 
current function in the current way.

But there may be a better way.

Honestly I haven't fully convinced myself either way yet.  I am quite 
nervous about such a wide-reaching change though.

> case which happens all the time.

I must be missing something.  AFAIK the common case that happens all the 
time is that you specify forward and reverse paths separately from the 
message content.

You're proposing removing this.

I'm still open to the option, but there are a myriad of consequences 
that haven't been touched on.

Most MUAs can and will still ensure the parameters match the headers.  
Other software may have other requirements.  You're just saying they 
will need to use SMTP.  Fine, but that doesn't move us forward.

I think if we're to drop the concept of an SMTP envelope, and say you 
can't send a message where the to and from in the message don't match 
the envelope (which I'm sure many people would like) we need tomake sure 
we don't break so much stuff as to make the new protocol impossible for 
people to adopt is all.  Just seems that policy can be used to enforce 
such a requirement as desired, rather than mandating it in the protocol 
and removing everyone's choice.

> Make the easy things EASY and the
sure.  I just don't think it's hard to have a command like

SEND UID forward-path reverse-path

In fact I'd be keen on something like

SENDFLAGANDMOVE UID +\Sent "Sent Items" forward-path reverse-path

to avoid magic which you're otherwise doing with your Sent flag.

The final point I'd make about obtaining the forward path from the 
message vs a parameter.

If you're parsing the To, CC, BCC headers, you need to be able to parse 
all character sets (there are many) in a completely bullet-proof fashion.

Adrien

---

From: Bron Gondwana <[email-masked]>
Date: Mon, 20 Feb 2012 00:39:01 +0100

On Mon, Feb 20, 2012 at 12:00:05PM +1300, Adrien de Croy wrote:
> that depends on how your system is structured.  The MTAs I'm
> familiar with put files into their send queues, and specify
> envelopes separately, either in a separate file or in a DB or
> something..

Unless you're putting the pre-exiting file from store into
the queue by reference rather than by copying, you would do
the rewrite as you copy.

Putting it in the queue by reference means you will have to
guarantee that the original mailbox copy doesn't get
expunged until the send has finished.
 
> What is the concern about sending the parameters?

Basically the \Sent things below.

> the main use-case I can think of where the recipients and
> return-path are not normally in the message are in lists.  So with
> your system, you won't be able to have an efficient list processor
> attach to an IMAP message store.  It will need to make copies of the
> message for each recipient.

I wouldn't expect an efficient list processor to be a user-facing
client.  Hence it would obviously talk directly, probably LMTP, to
the MTA.

> blackhole or empty return path to avoid loops.

Not a user client.

> You'll at least need a way to update certain headers into the
> message efficiently.  This would make for much more efficient
> forwarding as well... copy, rewrite To, Subject and send.  Rather
> than download, edit, upload send.  Maybe store the headers
> separately.

Lemonade style CATENATE.

> Honestly I haven't fully convinced myself either way yet.  I am
> quite nervous about such a wide-reaching change though.

So am I.  But your arguments are convincing me more that I'm right.
The cases you're coming up with are all "agents running on the
server doing non-user-client things" or "we don't support a way to
strip headers from a message when sending".

The "we don't support a way to strip headers" is the closest to a
viable argument I've heard so far, but that's an egg I think is
worth breaking in exchange for a really clear model about who a
message was sent to.  You read the headers, you know.  The \Sent
flag has a really specific meaning - and it takes an actively
malicious client to break that.  The theoretical "moron" will
get it right, because getting it wrong is a lot more work.

> I must be missing something.  AFAIK the common case that happens all
> the time is that you specify forward and reverse paths separately
> from the message content.

Yes, you do.

> You're proposing removing this.

Yes, I am.

> Most MUAs can and will still ensure the parameters match the
> headers.  Other software may have other requirements.  You're just
> saying they will need to use SMTP.  Fine, but that doesn't move us
> forward.

I'm saying this is a protocol for MUAs to talk to mail stores.  That's
all.  It does not replace SMTP, it does not replace LMTP.  It provides
a submission path for MUAs that can and will and want to make the
parameters match because the message on the server is a record of what
was sent.

> to avoid magic which you're otherwise doing with your Sent flag.

The magic is a latch.  The latch is a protection against replay.
I'd rather see "SENDFLAGANDMOVE" as a composition of simpler commands
which can be composed than one funny-shaped lego block, so when you
want to add something else to your chain, you don't need to do
anything else.

Which would make it "MOVE (IFNOT (FLAGS (\Sent)) (REPLACEHEADERS (Envelope-To: forward-path Envelope-From: return-path) +FLAGS (\Sent) DELIVER) UID "Sent Items"

Looks a bit nasty in this syntax for sure.  But that's non-magic
then, it's a transaction including the various changes.  I would
infinitely prefer this to actual magic.  But I think side effects
on taking an action are significantly preferable to side effects
on reading.  The same way that a "STORE +FLAGS" has an implicit
side-effect of bumping the MODSEQ on the message, and you can do
an UNCHANGEDSINCE to latch that too.

> If you're parsing the To, CC, BCC headers, you need to be able to
> parse all character sets (there are many) in a completely
> bullet-proof fashion.

Now that's a better argument than anything else I've seen so far.

The flip side is, you're going to have to parse them to search and
sort on them as well - so the code already exists in your server
if it's even vaguely complete.

Bron.

---

From: Adrien de Croy <[email-masked]>
Date: Mon, 20 Feb 2012 13:34:51 +1300

> expunged until the send has finished.

i'm just talking about good old file system file copy, without parsing.

>> blackhole or empty return path to avoid loops.

check out RFC 6409 Sec 3.2 para 5 re NULL return paths.

Clients do commonly auto respond.

>> than download, edit, upload send.  Maybe store the headers

I need to take another look, last time it gave me a headache. Is anyone 
supporting this?

we may be putting the cart before the horse talking about submission 
before we even discussed message generation.

> parameters match because the message on the server is a record of what

does anyone except ISPs use LMTP?

> an UNCHANGEDSINCE to latch that too.

however you do it, it needs to be transactional.  A single command for 
this is one way to do it. Or you could create a new syntax that 
explicitly does transactions.

Coding my SENDFLAGANDMOVE function would be quite simple.  Your's with 
the conditional would be less simple.  An alternative is providing 
explicit locking methods to clients (a la webDAV).

But frankly I don't like giving control of locks to clients.  So, maybe 
the best way is

START
   SEND UID
   UID STORE UID +\Sent
   MOVE ...
END

Sent as one block.  But that's just splitting my previous example into a 
few more lines.  Gives you more combinations/flexibility I guess.

Actually I don't have a problem with a side-effect of Send being to set 
a \Sent flag.  But prohibiting subsequent re-sends I'm not so convinced 
about.  You'd need to clear that flag if the message were to be copied 
(more magic).

> if it's even vaguely complete.

failure is quite different though - failure to deliver vs failure to search.

Having had to deal with things like KoiR, Shift-JIS etc etc, it's non 
trivial.  Frankly I'd avoid parsing wherever possible.

An alternative would be to take another approach, where envelope is 
stored separately from the message itself, and combined with it at 
submission time when the copy is put into the sent folder.

I don't see a strong reason to prohibit sending the same message more 
than once.  Resending from my sent items folder is something I do quite 
often.  They end up as another message in there.  So It's implicitly a 
copy, edit, send.

Adrien

---

From: Tony Finch <[email-masked]>
Date: Mon, 20 Feb 2012 10:52:28 +0000

> universe, because the headers will make it look extra spammy.

Use the Resent-* headers. (Pine/Alpine gets this right, though it uses 822
semantics rather than the better 2822 semantics.)

> All that's really missing is a way to form a "new" message from an "old"
> message without necessarily downloading everything first - but that's
> what the Lemonade CATENATE stuff was for.  Build a new message,
> potentially including the old one as a message/rfc822 attachment, and
> then send that.

message/rfc822 is also good. IMO every message that is forwarded or
top-post replied should use message/rfc822 attachments rather than doing
some fucked up redaction of the original message.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Viking, North Utsire: Westerly or southwesterly 6 to gale 8, occasionally
severe gale 9 at first, veering northwesterly 4 or 5 later. Rough or very
rough. Rain or squally showers. Good, occasionally poor.

---

From: Tony Finch <[email-masked]>
Date: Mon, 20 Feb 2012 10:55:09 +0000

Tony Finch <[email-masked]> wrote:
>
> Use the Resent-* headers.

... in fact here's something I wrote on this topic six years ago ...
http://www-uxsup.csx.cam.ac.uk/~fanf2/hermes/doc/qsmtp/draft-fanf-smtp-rcpthdr.html

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Northwest FitzRoy, Sole: Southerly 5 or 6. Rough, becoming moderate. Showers.
Good.

---

From: Bron Gondwana <[email-masked]>
Date: Mon, 20 Feb 2012 12:39:56 +0100

> http://www-uxsup.csx.cam.ac.uk/~fanf2/hermes/doc/qsmtp/draft-fanf-smtp-rcpthdr.html

That reads like almost exactly what I have in mind.

Bron.
-- 
  Bron Gondwana
  [email-masked]

---

From: Bron Gondwana <[email-masked]>
Date: Mon, 20 Feb 2012 12:41:57 +0100

> i'm just talking about good old file system file copy, without parsing.

Just our of interest, are you doing this with BURL now?

If not, then a requirement to translate during copying is not an additional
imposition on top of the IO and CPU hit you're currently getting from
two different copies being sent through your system(s).

If so, how do you handle the case where the client wants to store a
BCC field in their "Sent Items" folder as a record of who it was
really sent to?

Bron.
-- 
  Bron Gondwana
  [email-masked]

---

From: Tony Finch <[email-masked]>
Date: Mon, 20 Feb 2012 11:56:48 +0000

> In my opinion (and I admit I'm getting off topic), spam is merely a problem

No, the spam problem comes from accepting messages from people you don't
already know. This is of course a necessary feature of mail which is what
makes spam so hard.

(Actually not just that - it also comes from accepting messages from
people you sort-of know but who don't care about wasting your time, such
as corporate marketing departments.)

Your federated authentication idea is pretty neat, and could be the basis
of a better way of classifying mail from people you know or sort-of know.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Shannon, Rockall, Malin: Southwesterly 5 to 7, occasionally gale 8 at first in
Rockall and Malin. Rough, occasionally very rough in Rockall and Malin.
Occasional rain. Moderate or good, occasionally poor.

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 21 Feb 2012 01:30:28 +1300

We didn't implement BURL (HURL).

It was just too far over the insanity horizon to write an IMAP client 
for that purpose.

Especially since I don't know of a single client that uses it.

So the MUA takes care of BCC in sent items, when it uploads the file 
there after sending with SMTP.

---

From: Filip Navara <[email-masked]>
Date: Mon, 20 Feb 2012 13:43:25 +0100

JYFI, we (eM Client, www.emclient.com) do use BURL in some cases.

F.

On Mon, Feb 20, 2012 at 1:30 PM, Adrien de Croy <[email-masked]> wrote:

> It was just too far over the insanity horizon to write an IMAP client for
> that purpose.
>
> So the MUA takes care of BCC in sent items, when it uploads the file there
> after sending with SMTP.
>
>
> ______________________________**_________________
> imap5 mailing list
> [email-masked]
> https://www.ietf.org/mailman/**listinfo/imap5<https://www.ietf.org/mailman/listinfo/imap5>
>

---

From: Bron Gondwana <[email-masked]>
Date: Mon, 20 Feb 2012 14:13:51 +0100

> JYFI, we (eM Client, www.emclient.com) do use BURL in some cases.

How do you handle BCC?

> > So the MUA takes care of BCC in sent items, when it uploads the file there

So in other words, requiring the IMAP5 server to strip BCC when sending to
the MTA would cause no more IO or CPU usage than what is currently required
to do two separate uploads, one via SMTP and one to the Sent folder.

Which has been my point all along.  This is making the server do what the
client would be doing anyway - doing it in one place rather than many -
purely for the case of a user-facing IMAP client.

Bron.
-- 
  Bron Gondwana
  [email-masked]

---

From: Bron Gondwana <[email-masked]>
Date: Mon, 20 Feb 2012 14:27:14 +0100

> Clients do commonly auto respond.

Yep, got that.  And thanks, I'll add RFC 6409 to the list of RFCs on
the wiki.  Done.

If client do send an auto-response with a NULL return path, do they
usually store a copy of that message on the server at all?  I would
imagine they don't.

I also see from the KEYWORDS rfc (5788) that $MDNSent is registered
for the MDN purpose.  It almost seems that a SENDMDN command is what
is indicated for this usage.  Again, the "client can't screw it up"
philosophy.

Bron.
-- 
  Bron Gondwana
  [email-masked]

---

From: Dave Cridland <[email-masked]>
Date: Mon, 20 Feb 2012 13:27:14 +0000

> JYFI, we (eM Client, www.emclient.com) do use BURL in some cases.
As does the Qt Messaging Framework.

(And Polymer, and Isode's M-Switch/M-Box, and ... )

That all said, I'm swinging around to the notion of sending mail  
through the message store access protocol:

If we have a command which sends a particular mail *with an  
envelope*, then I think we can map all ESMTP stuff to it - but rather  
more interestingly, we can keep the envelope around as metadata  
within the store.

This allows things which you simply cannot do with BURL, for instance:

1) We could track bounces using VERP, and annotate the message when a  
DSN is received, so you could see which sent messages have failed.

2) We could track MDNs in a similar way.

You'd want multiple envelopes on a message, to handle resends and  
redirects, and it seems sensible to capture the S/LMTP envelope on  
delivery, too, such that a client would have sufficient information  
to cause a (delayed) bounce.

I appreciate this appears to be a U-turn on my behalf - that's  
because it is. I don't see multiple protocols as being a  
show-stopper, but the comments about DSN/MDN processing sparked this  
chain of thought for me.

Dave.

---

From: Tony Finch <[email-masked]>
Date: Mon, 20 Feb 2012 13:30:19 +0000

Bron Gondwana <[email-masked]> wrote:
>
> It almost seems that a SENDMDN command is what is indicated for this
> usage.  Again, the "client can't screw it up" philosophy.

On the other hand, MDNs are an abusive misfeature that should be shunned.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Trafalgar: North 4 or 5. Moderate. Fair. Good.

---

From: Bron Gondwana <[email-masked]>
Date: Mon, 20 Feb 2012 14:33:24 +0100

> On the other hand, MDNs are an abusive misfeature that should be shunned.

Good luck selling that one.  Some sort of "it got received/seen" is wanted
by a great many users.  I'm not sure if Facebook provides it, but Exchange
certainly does.

Bron.
-- 
  Bron Gondwana
  [email-masked]

---

From: Filip Navara <[email-masked]>
Date: Mon, 20 Feb 2012 15:05:12 +0100

> > JYFI, we (eM Client, www.emclient.com) do use BURL in some cases.

We don't use BURL for mails with BCC. In theory we could use CHUNKING+BURL,
but we never implemented it.

F.

---

From: Dave Cridland <[email-masked]>
Date: Mon, 20 Feb 2012 14:15:17 +0000

> but we never implemented it.

Right - if you sent the header first with HEADER.FIELDS.NOT, and then  
send the body, you can do it.

Polymer just never puts BCC into the mail at all, which has alternate  
problems.

Dave.

---

From: Bron Gondwana <[email-masked]>
Date: Mon, 20 Feb 2012 15:35:35 +0100

> show-stopper, but the comments about DSN/MDN processing sparked this  

I'm beginning to appreciate the "multiple envelopes" idea as well
actually, particularly reading your thoughts.

It probably means storing annotations - or at least providing a way
to access this envelope data that smells remarkably like annotations.

In which case I wouldn't be adverse to having the envelope in the
protocol if it's stored with the message as well.

The hard part then is bolting it on to existing server implementations,
since we need to store it SOMEWHERE.

At FastMail we add 'X-Delivered-To' and 'X-Mail-From' headers which
capture at least that much of the envelope.  That happens during the
LMTP proxy stage, where spam scanning and cleanups are done.
(Postfix on the MX passes ORCPT via LMTP, we patch that in.)

The downside of out-of-band metadata is that everything needs to
support it.  Otherwise you can't even see it.  If it's in-band in
the message, you can transfer it via intermediate stages that don't
understand it - so you can download it via a dumb client, then
operate on the local copy.

Of course, with this theoretical protocol, annotations would have to
stay alongside.  Potentially mutable annotations.

Bron.
-- 
  Bron Gondwana
  [email-masked]

---

From: Dave Cridland <[email-masked]>
Date: Mon, 20 Feb 2012 14:43:57 +0000

> to access this envelope data that smells remarkably like  

No.

Store it as a properly modelled transport envelope - no need to turn  
it into magic soup - that's never worked.

Dave.

---

From: Bron Gondwana <[email-masked]>
Date: Mon, 20 Feb 2012 15:46:40 +0100

> it into magic soup - that's never worked.

Fine with me.  It's just yet another axis of data tied to a message, and
it needs its very own "keywords" or "annotations" as well, presumably, if
you want to note when it was accepted, or MDNed.

Bron.
-- 
  Bron Gondwana
  [email-masked]

---

From: Tony Finch <[email-masked]>
Date: Mon, 20 Feb 2012 14:49:32 +0000

> operate on the local copy.

I think this is very important.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Viking, North Utsire, South Utsire: Southwesterly 6 to gale 8, occasionally
severe gale 9 at first in Viking and North Utsire, veering northwesterly 4 or
5, becoming variable 3 or 4 later. Rough or very rough. Rain or showers. Good,
occasionally poor.

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 21 Feb 2012 08:50:40 +1300

you're correct.

But just because the current way is inefficient doesn't mean we should 
replace it with an equally inefficient method.

I'd rather do something better.

Allow the client to instruct the server to refer to the message in sent 
items, and add some meta data about its delivery envelope(s)

Adrien

On 21/02/2012 2:13 a.m., Bron Gondwana wrote:
>> On Mon, Feb 20, 2012 at 1:30 PM, Adrien de Croy<[email-masked]>  wrote:
>>

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 21 Feb 2012 08:52:14 +1300

> usually store a copy of that message on the server at all?  I would

Actually I think they do.  I've had MDNs turned off since dot, but I've 
seen such messages in sent items folders before.

> is indicated for this usage.  Again, the "client can't screw it up"

sure.  Gives the server the freedom to implement it the way they see fit 
as well.

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 21 Feb 2012 08:57:42 +1300

> (And Polymer, and Isode's M-Switch/M-Box, and ... )

OK, I'm not familiar with any of these clients.  Most of our customers 
are running windows on the desktop.

Anyone know of any windows-based clients that use BURL?

> If we have a command which sends a particular mail *with an envelope*, 
> then I think we can map all ESMTP stuff to it - but rather more 
> interestingly, we can keep the envelope around as metadata within the 
> store.
>
> delivery, too, such that a client would have sufficient information to 
> cause a (delayed) bounce.
>
> I appreciate this appears to be a U-turn on my behalf - that's because 
> it is. I don't see multiple protocols as being a show-stopper, but the 
> comments about DSN/MDN processing sparked this chain of thought for me.

People nowadays are used to instant feedback.  So a 4 hour-later 
delivery warning is not well received.  Some systems you only find out 2 
days later that your mail couldn't be delivered.  You might have missed 
the deal at that stage.

There's no technical reason why a system can't be designed and built 
that would provide pretty much immediate feedback to a mail submitter as 
to the outcome of the delivery.  And I'd like to even check parameters 
before that as well.  It won't work in all cases, since there are still 
many clients that are not connected full-time to the network.  But it 
can work in enough cases to provide real benefits.

There are of course all manner of commercial roadblocks to that, such as 
what protocols are deployed.  But there are ways to migrate to new 
protocols if there are enough compelling reasons to do so.

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 21 Feb 2012 09:23:29 +1300

the main problem with MDNs, bounce messages etc is that the network 
doesn't know what they are.

This is WAY o-t now...

With TCP or UDP etc, if you have a delivery issue (certain types), you 
get an ICMP message back.

If there was a distinction between a control message and a data message, 
then the network could know things like how to not create loops.

It could also mean we could drop the requirement to allow submission of 
mail with NULL return paths, and instead use a non-destructive (in terms 
of information) method to notify issues which retained the addressing 
information.

It would need to be clearly defined, machine readable etc.  It could be 
stored and forwarded, and moved over the same transport.  It would just 
need to be unambiguously identifiable in all cases, which IMO would 
require another verb in the mail transport to mark it as a control 
message.  I don't think message format would cut it.

In terms of privacy etc, if you send a registered letter to someone, you 
know when it is received.  You're entitled to know this, and you don't 
hear complaints about it.  So I don't have a problem with the concept of 
allowing a sender to know when a message is retrieved.  I just turned 
off MDNs because the way they are implemented is problematic esp with spam.

Adrien

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 21 Feb 2012 09:36:00 +1300

the reason I bring this up.

Even though we may not fix SMTP MDN issues etc in a long time, if we 
build protocol support for notifying a client about things relating to 
delivery etc, then we can

a) implement it initially with server-side heuristics if at all (e.g. 
recognise and strip bounce messages, or deliver them to a special folder 
and notify)
b) move to a deterministic model once it becomes available

So it affects what we do now with whatever client-server stuff we do now.

Adrien

On 21/02/2012 9:23 a.m., Adrien de Croy wrote:
> If there was a distinction between a control message and a data 
> message, then the network could know things like how to not create loops.
>
> It could also mean we could drop the requirement to allow submission 
> of mail with NULL return paths, and instead use a non-destructive (in 
> terms of information) method to notify issues which retained the 
> addressing information.
>
> It would need to be clearly defined, machine readable etc.  It could 
> be stored and forwarded, and moved over the same transport.  It would 
> just need to be unambiguously identifiable in all cases, which IMO 
> would require another verb in the mail transport to mark it as a 
> control message.  I don't think message format would cut it.
>
> In terms of privacy etc, if you send a registered letter to someone, 
> you know when it is received.  You're entitled to know this, and you 
> don't hear complaints about it.  So I don't have a problem with the 
> concept of allowing a sender to know when a message is retrieved.  I 
> just turned off MDNs because the way they are implemented is 
> problematic esp with spam.
>
> Adrien
>
>

---

From: Tony Finch <[email-masked]>
Date: Mon, 20 Feb 2012 23:53:07 +0000

Adrien de Croy <[email-masked]> wrote:
>
> If there was a distinction between a control message and a data message, then
> the network could know things like how to not create loops.

That's what null return paths are for. Exchange doesn't implement
auto-replies properly so they don't work as they should.

> It would need to be clearly defined, machine readable etc. It could be stored
> and forwarded, and moved over the same transport.

Already done.

Tony.

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 21 Feb 2012 13:12:50 +1300

> auto-replies properly so they don't work as they should.

null return paths are a hack.  And they aren't reliable.  Even though 
it's explicitly stated in many RFCs, some MTAs still reject them.

And they destroy information, unless you can put the lost information 
into a machine readable location in the message body.

>> and forwarded, and moved over the same transport.

you mean the message format is defined?  Where?  sorry if I'm being 
lazy, I didn't think that was defined in the RFC for DNs.

Adrien

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 21 Feb 2012 18:23:38 +1300

On 21/02/2012 1:43 a.m., Filip Navara wrote:
> JYFI, we (eM Client, www.emclient.com <http://www.emclient.com>) do 
> use BURL in some cases.

thanks, I'll have a look.  be nice to get a decent replacement for TB.

Adrien

>
> F.
>
> On Mon, Feb 20, 2012 at 1:30 PM, Adrien de Croy <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>
>     It was just too far over the insanity horizon to write an IMAP
>     client for that purpose.
>
>     So the MUA takes care of BCC in sent items, when it uploads the
>     file there after sending with SMTP.
>
>
>     [email-masked] <mailto:[email-masked]>

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 21 Feb 2012 18:29:11 +1300

presuming you were storing the message in a file on disk, you could 
adopt some sort of meta structure, where the message was a part, and 
envelopes became other parts (and annotations if desired).  This could 
be as simple as putting a length at the start of the message to where 
the envelopes start.  Any sort of structure I'd length-prefix, so you 
can skip data instead of having to scan it.

the server would take care of it all, and clients would just fetch the 
bits they wanted.  Being in a single file would solve issues relating to 
synchronisation / keeping it all together.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 21 Feb 2012 11:08:03 +0100

> already know. This is of course a necessary feature of mail which is what

You make it sound as if necessary features were binary.

IMO, this problem is really three disjoint problems, two of which can be
solved given architectural support. Two is a smaller number than three,
and the solutions might not be perfect, but still...

I'll post something when the current thread has subsided.

> as corporate marketing departments.)

The one thing I love about Atlassian Confluence is that I can ignore the
rubbish without causing offense to its creators, and in case I'm wrong
to ignore some particular page, someone will give me its URL.

Arnt

---

From: Brandon Long <[email-masked]>
Date: Wed, 22 Feb 2012 11:24:35 -0800

On Sat, Feb 18, 2012 at 2:07 AM, Adrien de Croy <[email-masked]> wrote:
> plenty of spam comes from the sender not stolen accounts.  That's why the
> spammers do things like register their own domains and SPF records.
>
>
> the server will have a cert.  It can be seen as spamming, and its cert can
> be revoked.  That will cut it off.
>
> Having to get another cert will provide an incentive for the admin to care
> about it.

You seem to believe that all servers can always be entirely free from
sending spam.  That's pretty funny.

Given that spam is in the eye of the beholder, there are plenty of
messages which are spam to some and not to others.  Do you consider
the latest commercial offer from Target or Amazon as spam?  Plenty of
people mark it as such, even if they opted-in to receiving it.

How about spam sent from a hijacked account?  How many hijacked
accounts a day do you think there are on a service with 1B email
users?

Or how much money do you think a spammer is willing to spend to buy an
account, even on a free service?  Or do you think its actually
possible to force everyone who wants an email account to pay for it at
this point?  And if so, how much money?  $5/year is cheap in parts of
the world, and really expensive in others, should poor parts of the
world be relegated to the email ghetto because their accounts are so
cheap that spammer abuse them constantly, while they have the least
resources to keep them out?

And do you think that every person who runs a mail server wants to
spend $100/year on a certificate?  We already do it, but its not a big
deal to us.  How many people run servers on their personal box?

Which is all pretty irrelevant, for most users today spam is already a
solved problem.  They don't see how much effort we put into it, and
they know nothing about it until their account gets hijacked or one of
their friends does and they get a mugged in London message.  Or when
some filter gets too aggressive and they don't get a message.  Or when
some company still thinks the spam world is black & white and uses a
blacklist against their server.  Any effort they would have to make to
whitelist senders before they can send them mail is something they
aren't likely to understand the need for.

As for getting the Facebooks of the world to open up their social
connection information to solve the spam problem for you, well, good
luck with that.  If you're Yahoo or Microsoft you can pay enough money
to get access to that, and maybe its in the ToS to use it that way.

Brandon

---

From: Bron Gondwana <[email-masked]>
Date: Wed, 22 Feb 2012 20:36:45 +0100

> sending spam.  That's pretty funny.

Yeah, this.  We're a lot smaller than you, and we are more strict about
spam scanning what goes OUT than what comes in, but we still wind up
with all the usual reasons why we have outbound spam :(

> [...] They don't see how much effort we put into it, and
> blacklist against their server.

Or because someone's incoming rate limits are delaying their messages
(just saying ;) - one of my major causes of notifications is someone
who forwards all their email to gmail causing a massive outbound
backlog when their monitoring service goes crazy and your inbound limits
are stricter than our inbound limits...

> Any effort they would have to make to

There's some nice stuff you can do with semi-automatic whitelisting
(up-rate things which are a response to something they have already
sent, certainly up-rate addresses in their address book) - but we
found you can't go too far there (like whitelisting "from self"
because the spammers will exploit that.  Same with whitelisting stuff
from someone who has already spoken to them, because it just means
you get targetted spam supposedly from someone else on a public
mailing list.

> to get access to that, and maybe its in the ToS to use it that way.

The more I read (particularly from Bruce Schneier) about this, the
more I realise that spam isn't solveable.  Make it harder, and it
will be more valuable to be able to get through, because the scammer
is no longer competing against a ton of other scams.  The flavour
might change to more targetted, but unsolicited rubbish will always
exist.  The internet is not a special flower here.

Bron.
-- 
  Bron Gondwana
  [email-masked]

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 23 Feb 2012 09:22:47 +1300

On 23/02/2012 8:24 a.m., Brandon Long wrote:
> On Sat, Feb 18, 2012 at 2:07 AM, Adrien de Croy<[email-masked]>  wrote:

sorry, where do I propose that?

I'm just proposing a system that allows the identification of 
organisations that inject and relay spam.  That then allows enforcement 
of accountability.

> messages which are spam to some and not to others.

of course

> Do you consider

if they opted in, they indicated a willingness to receive whatever they 
get, within the terms of their opt-in

> accounts a day do you think there are on a service with 1B email

How many other crimes are there committed a day, do you propose we don't 
go after criminals?

> resources to keep them out?

why do you assume the system would be structured like this?  Sounds like 
a system that would fail.

> deal to us.  How many people run servers on their personal box?

need to think outside the square a bit.

> solved problem.
it certainly is not a solved problem for anyone.  Ignorance is not the 
answer.

Jut because a business doesn't know how many customers they are losing 
due to over-agressive spam filtering doesn't mean it has no cost to them.

> They don't see how much effort we put into it, and
> some filter gets too aggressive and they don't get a message.

if they find out they didn't get the message.

>    Or when
> some company still thinks the spam world is black&  white and uses a

The system (and I admit it's ambitious) would need co-operation from 
governments.

there's no need for ma and pa to have a certificate, they can submit to 
their ISP.  The ISP would need a certificate.  There's no reason to 
assume the certs would be managed by the existing CA infrastructure.  
I'd propose that should be a function of Governments, and there are 
already special provisions for governments to issue certificates.  They 
could be for long periods as well.  The purpose is to identify and 
provide a means to revoke.  Renewing annually seems like a waste of time 
for that, unless you think the certificate may be breached.

Organisations wanting to deliver directly could get a certificate as well.

As to determination about whether someone spams or not.  Well most 
countries have systems to establish whether crimes are committed and go 
after and punish those responsible.  There are already spamming laws all 
over the place.  I'm proposing setting up a system that allows for 
identification of perpetrators and enforcement, and enables services to 
be set up to solve issues independently (e.g. if a government refuses to 
prosecute a spammer).  Revokation of certificates would be a function of 
government after due process.  People couldn't just buy new ones (unless 
they get them from corrupt government officials), because their previous 
spamming would be associated with them as a person.  In short, treat 
spamming like any other crime - which it certainly is.

I think if governments were aware of the costs of spamming they may take 
a different view on it.  How many hours are wasted deleting spam? How 
much money is spent on anti-spam?  How much network capacity (which 
costs money) is wasted transporting spam?  How many opportunities are 
lost due to false positives?  Personally I believe the real economic 
costs of spam are astronomical.  Someone needs to do a study, and come 
up with some numbers they can back up.

Otherwise we should just all join FB and just use that for communication 
and ditch mail altogether.

Adrien

---

From: Brandon Long <[email-masked]>
Date: Wed, 22 Feb 2012 13:28:59 -0800

On Wed, Feb 22, 2012 at 12:22 PM, Adrien de Croy <[email-masked]> wrote:
>>> care

You're proposing revoking a server's certificate for spamming.  Based
on what level?  What level of fault?  Would Gmail get its certificate
revoked because 1% of the email it sends is spam?

> I'm just proposing a system that allows the identification of organisations
> that inject and relay spam.  That then allows enforcement of accountability.

We can already do this via IP addresses and sender domains and
SPF/DKIM authentication.  Yes, its just a proxy and sometimes its
wrong, but it works fairly well.

> How many other crimes are there committed a day, do you propose we don't go
> after criminals?

Heh.  Do you know how many spam messages are sent a day?  How large an
enforcement organization do you propose to go after them all?  And how
long do you think that would take?

Not to mention that multiple people and governments have different
definitions of spam.

When we see a new spam campaign, we need to be able to shut it down in
less than hours.  A recent time that we helped the US government go
after a malware operation, it took them a year before the first
arrests.  A year where we had to leave the botnets and operations
alone so they could gather the evidence necessary to make the arrests.

Police action doesn't scale the same way that spammers do.

> why do you assume the system would be structured like this?  Sounds like a
> system that would fail.

Then who pays for this enforcement?  Who pays for the certification?

> Jut because a business doesn't know how many customers they are losing due
> to over-agressive spam filtering doesn't mean it has no cost to them.

Of course it has a cost.  I'm saying the cost of your solution is higher.

> The system (and I admit it's ambitious) would need co-operation from

As if all the governments of the world agree on anything, much less
the definition of spam.

> their ISP.  The ISP would need a certificate.  There's no reason to assume
> the certs would be managed by the existing CA infrastructure.  I'd propose
> that should be a function of Governments, and there are already special
> provisions for governments to issue certificates.  They could be for long
> periods as well.  The purpose is to identify and provide a means to revoke.
>  Renewing annually seems like a waste of time for that, unless you think the
> certificate may be breached.

And what if the CA is breached?  Ie, like the 2-3 that have happened
in the last year?

> As to determination about whether someone spams or not.  Well most countries
> have systems to establish whether crimes are committed and go after and
> punish those responsible.  There are already spamming laws all over the
> place.  I'm proposing setting up a system that allows for identification of
> perpetrators and enforcement, and enables services to be set up to solve
> issues independently (e.g. if a government refuses to prosecute a spammer).

Weee, now we're talking about extra-governmental authorities making
the rules.  Its always great to argue with an RBL maintainer about
whether or not something is spam.  Or maybe what you're proposing is
more like SOPA/PIPA, we can have an organization like the RIAA
deciding what's good.  Even better, the government of Iran can just
prevent their providers from accepting any mail certified by other
governments.

Or here's an even more fun one: We just emailed all of our users about
the changes to our privacy policy, a move we made at the request of
the US government.  And we had RBL organizations complaining that it
was spam.  Who wins?

Our answer is simple: the user decides what is spam, not someone else.
 Our job is to make our spam filter match each user's expectations.

>  Revokation of certificates would be a function of government after due
> process.  People couldn't just buy new ones (unless they get them from
> corrupt government officials), because their previous spamming would be
> associated with them as a person.  In short, treat spamming like any other
> crime - which it certainly is.

No corrupt government officials in the world, that's for sure.

And they already treat spamming as a crime, have for years.  Done a
lot of good at reducing the spam load, eh?

> I think if governments were aware of the costs of spamming they may take a
> different view on it.  How many hours are wasted deleting spam? How much
> money is spent on anti-spam?  How much network capacity (which costs money)
> is wasted transporting spam?

Not as much as you'd think, turns out spam is much smaller than
regular mail at this point, at least for consumers.  A large
percentage of mail, but on the order of 40x smaller in size (on
average).  And email in general is not generally a large user of
network capacity.  How many email messages, even at 100k average, does
it take to equal a single iphone app download?  Or a streamed video
from Youtube?

> How many opportunities are lost due to false
> positives?  Personally I believe the real economic costs of spam are
> astronomical.  Someone needs to do a study, and come up with some numbers
> they can back up.

Regardless of those costs, your proposal would cost more and still not
solve the problem.

> Otherwise we should just all join FB and just use that for communication and
> ditch mail altogether.

We have the stats on what percentage of our users receiving mail mark
messages as spam or not spam.  Its tiny.  For most people, they don't
see the spam, and maybe they don't see enough to actually check their
spam label, but its just not an issue.

As to where the kids are going these days, who knows.  Email is
certainly not the only game in town.

Brandon

---

From: Jan Kundrát <[email-masked]>
Date: Thu, 23 Feb 2012 15:51:21 +0100

> some fucked up redaction of the original message.

GMail still breaks [1] on such messages -- the web interface shows the
message source and the IMAP server returns invalid BODYSTRUCTURE which
doesn't allow accessing the attached message.

Cheers,
Jan

[1] http://permalink.gmane.org/gmane.mail.imap.general/2637

---

From: Tony Hansen <[email-masked]>
Date: Thu, 23 Feb 2012 10:43:56 -0500

Totally off topic: I wish I knew how to get them to fix that. I get bit 
by it regularly.

On topic: because gmail messes up in this area doesn't mean that we 
should also mess things up in the same way.

I am a firm believer in the use of message/rfc822 attachments.

     Tony Hansen

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 23 Feb 2012 16:58:28 +0100

> I am a firm believer in the use of message/rfc822 attachments.

Be a firm believer in having few features, each of which is used often.

Arnt

---

From: Mark Crispin <[email-masked]>
Date: Thu, 23 Feb 2012 09:00:57 -0800 (PST)

On Thu, 23 Feb 2012, Tony Hansen wrote:
> On topic: because gmail messes up in this area doesn't mean that we should
> also mess things up in the same way.

This is where we have the disconnect with the kids. The kids claim that
"it doesn't work in [gmail | hotmail | yahoo | etc.], therefore it's
useless and should be abolished."

In turn, vendors that write this crapware are not made to give a damn.

-- Mark --

http://panda.com/mrc

---

From: Dave Cridland <[email-masked]>
Date: Thu, 23 Feb 2012 23:15:54 +0000

On Thu Feb 23 15:43:56 2012, Tony Hansen wrote:
> Totally off topic: I wish I knew how to get them to fix that.

And now you know...

Dave.

---

From: Brandon Long <[email-masked]>
Date: Thu, 23 Feb 2012 16:09:45 -0800

>> Totally off topic: I wish I knew how to get them to fix that.

I'm sure there are other corner cases as well, just looking through
the code there are all sorts of caveats and such to work around bugs
generated by specific clients.

To bring this back on-topic, if the goal is simpler, is it simpler for
the server to implement parsing or to punt that to the client?

This specific case could have been solved by the client implementing a
viewer for rfc822, though obviously its heavier weight if the embedded
message has many other levels.

OTOH, its simpler on clients if they don't have to implement all of
the work-arounds for broken mail that we do.

Or maybe a much simpler flattened structure for messages as parsed by
the server and you can download the whole thing to parse the original
if you want higher fidelity.

Of course, if we aren't parsing the messages on the server, we have to
ask why we're implementing IMAP5 and not POP4 (still not sure if that
statement will make Marc laugh or cringe).

Brandon

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 24 Feb 2012 10:00:48 +0100

On 02/24/2012 01:09 AM, Brandon Long wrote:
> ask why we're implementing IMAP5 and not POP4

Some time ago you wrote that IMAP is the de facto API for talking to
gmail. I understand that you offer POP too, but IMAP's what people use,
right?

Arnt

---

From: Brandon Long <[email-masked]>
Date: Fri, 24 Feb 2012 11:48:23 -0800

> gmail. I understand that you offer POP too, but IMAP's what people use,

Yes.  For simple backups / migrations, they'll tend to use POP which
is easier to deal with, but for anything that requires actually
writing to the store or getting specific pieces of information,
performing searches, obviously IMAP is the better choice.

I was being mostly facetious about POP4.  There may be an argument for
a simpler protocol which does 2-way syncing in a log-like fashion with
a side-order of mail sending.  That's essentially what the Android
Gmail client uses (our own http based two-way sync based on actual
changes going across).  But, I tend to be a believer in the off-line
syncing clients instead of the always connected ones.

Brandon

---

From: Adrien de Croy <[email-masked]>
Date: Sat, 25 Feb 2012 09:29:04 +1300

The obvious application for server-side parsing is providing the ability 
to search the store without having to download the whole thing first. In 
many cases it's simply not possible otherwise.

I think it's a very compelling feature.  Which for me means server-side 
parsing should stay.

One could argue that to a certain level it could be simplified.  
Providing a complete nested body structure is only really relied on by 
some clients.  Many just download the whole, or parts and then parse 
client-side.

But I don't know to what level, and since we have an existing working 
mechanism.

As for always connected vs offline & connect-as-required.  I think these 
are 2 useful cases.  In-house corporate use can benefit from always 
connected, but it doesn't scale well for huge providers.  Being able to 
re-establish some state without having to go through the whole setup 
phase could be an interesting option to look into, e.g. some reusable 
token (good for a certain amount of time since last use) would allow the 
server to cache the "session" and the client to re-establish quickly.

Adrien

---

From: Brandon Long <[email-masked]>
Date: Fri, 24 Feb 2012 13:23:08 -0800

On Fri, Feb 24, 2012 at 12:29 PM, Adrien de Croy <[email-masked]> wrote:
>
> The obvious application for server-side parsing is providing the ability to
> search the store without having to download the whole thing first. In many
> cases it's simply not possible otherwise.
>
> One could argue that to a certain level it could be simplified.  Providing a
> complete nested body structure is only really relied on by some clients.
>  Many just download the whole, or parts and then parse client-side.
>
> As for always connected vs offline & connect-as-required.  I think these are
> 2 useful cases.  In-house corporate use can benefit from always connected,
> but it doesn't scale well for huge providers.  Being able to re-establish
> some state without having to go through the whole setup phase could be an
> interesting option to look into, e.g. some reusable token (good for a
> certain amount of time since last use) would allow the server to cache the
> "session" and the client to re-establish quickly.

That was in the P-IMAP proposal, which I assume means it was discussed
in lemonade, anyone recall why it didn't go anywhere?

Granted, with qresync/condstore, session maintenance probably isn't as
important.

Brandon

---

From: Dave Cridland <[email-masked]>
Date: Fri, 24 Feb 2012 23:12:40 +0000

> in lemonade, anyone recall why it didn't go anywhere?
It forces scalability to be per-device, rather than per-user.

It generally struck me as fairly fragile, too - typically, the  
server's going to erase older tokens after a while, and limit the  
numbers, and there's no fallback at all.

> Granted, with qresync/condstore, session maintenance probably isn't  

Right, and QRESYNC doesn't require *any* state beyond CONDSTORE's  
64-bits per message and per folder.

Dave.