Subject: [imapext] Fwd: Request to form a new WG: JMAP

From: Alexey Melnikov <[email-masked]>
Date: Mon, 7 Nov 2016 17:31:31 +0000

Hi,

I am forwarding my email here, as I suspect people on these two mailing 
lists might be interested in this.

Please post your comments/questions to [email-masked] or directly to me.

Best Regards,

Alexey

-------- Forwarded Message --------
Subject: 	Request to form a new WG: JMAP
Date: 	Mon, 07 Nov 2016 17:17:59 +0000
From: 	Alexey Melnikov <[email-masked]>
To: 	DISPATCH <[email-masked]>

Hi,
I have received a request to form a new WG: JSON Mail Access Protocol
(JMAP). Draft charter is included below. Please send your comments in
reply to this email or directly to me.

I am also looking for possible chairs for this work, so if you are
interested, please contact me off-list.

Thank you,
Alexey

------------------------------------

Name:    JSON Mail Access Protocol
Acronym: jmap
Area:    Applications and Real-Time Area (art)

Charter for Working Group

Many companies and projects are developing their own JSON based
representations of email which are proprietary, non-standard, and
incompatible with each other. These protocols are proliferating due
to existing standards being insufficient or poorly suited to the
environments they are operating in, particularly mobile and webmail.

The JMAP working group will specify an mechanism to allow clients to
both view and send email from a server over a single stateless HTTPS
channel with minimal round trips. The protocol will support
out-of-band signaling of changes, and will give mobile clients
significant benefits in terms of battery life and network usage.

The use of multiple protocols to perform actions within a single
application creates significant support challenges, as users may get a
variety of partial failure modes (for example, can receive email, but
can not send new messages).  This is further exacerbated if the
different protocols are authenticated separately.

The work of this group is limited to developing a protocol for a client
synchronising data with a server. Any server-to-server issues or
end-to-end encryption of messages are out of scope for this working
group.

The working group will coordinate with the Security Area on credential
management and authentication.

Input to working group discussions shall include:

- Internet Message Format
[RFC 5322]

- CONDSTORE and QRESYNC
[RFC 7162]

- Collection Synchronisation for WebDav
[RFC 6578]

- LEMONADE and experiences from adoption of its output
[https://datatracker.ietf.org/wg/lemonade/charter/]

- SMTP SUBMISSION
[RFC 4409]

- SMTP BURL
[RFC 4468]

The working group will deliver the following:

- A problem statement detailing the deployment environment and
   situations that motivate work on a new protocol for client to
   server email synchronisation.  The working group may choose
   not to publish this as an RFC.

- A document describing an extensible protocol and data structures which
   supports stateless use, flood control and batched operations.

- A document describing how to use the extensible protocol over HTTPS
   with the data structures expressed as JSON.

- A document describing a data model for email viewing, management,
   searching, and submission on top of the extensible protocol.

- An executable test suite and documented test cases to assist
   developers of JMAP servers to ensure they conform to the
   specifications.

---

From: John C Klensin <[email-masked]>
Date: Mon, 07 Nov 2016 12:59:00 -0500

Alexey,

These are just questions, not opposition, but the answers should
probably be clear in any charter or chartering effort.

(1) We've had a long history of variations on The Format to End
All Formats, none of which have lasted a very long time.  I
think JSON is better than most, but, if yet another wonderful
format shows up in a few years and takes over, is it worth
thinking now about what a transition plan would look like?  

(2) Some of us believe, at least on some days, that a key reason
why SMTP, the message formats that started with RFC 822, POP3,
and IMAP have survived against a large number of proprietary and
other SDO alternatives have been precisely because they operate
in plain text, with all but IMAP using very simple
command-argument or name:value syntax.  Are we ready to give
that up?  Do we really need interoperability between a server
and a captive Webmail interface?  Arguably, standards of that
type are part of the province of the IETF only if someone is
contemplating generic webmail clients that can interact with the
servers of more than one organization.

(3) In terms of email models, this would presumably provides an
alternative to IMAP (or maybe POP3) on the mail receiving side.
Would it be equally sensible to define an IMAP extension that
would deliver responses (and maybe accept requests) in a
specified JSON format rather than the LISP-like IMAP one?  Seems
to me that would promote much more compatibility and flexibility.

(4) Similarly, there is lots of practice about use of IMAP to
send.  Is to time to make sure the standards for that
(presumably as an alternative to RFC 6409-style message
submission) and then provide a JSON format IMAP option to
support this form?

Again, just questions, but questions I think we need to ask
carefully before standardizing yet another alternative way to do
much the same thing.

    john

--On Monday, November 07, 2016 17:31 +0000 Alexey Melnikov
<[email-masked]> wrote:

> Hi,
> 
> I am forwarding my email here, as I suspect people on these
> two mailing lists might be interested in this.
>

---

From: Ned Freed <[email-masked]>
Date: Mon, 07 Nov 2016 10:05:06 -0800 (PST)

Without getting into whether or not this is in general a good idea, I want to
note that the proposal as written is self-contradictory, making it very

> environments they are operating in, particularly mobile and webmail.

"Representation of email" would seem to imply that we're talking about a new
message format.

> channel with minimal round trips.

And now we appear to be talking about a replacment for both IMAP and SUBMIT.
This effectively means we're replacing all of the email protocol and format
suite with the exception of SMTP.

> The protocol will support

And now we appear to be talking about server->client synchronization, which
is, in standards terms at least, new.

> different protocols are authenticated separately.

So we're clearly aiming to replace multiple protocols here.

> synchronising data with a server.

Whoops! Now we're talking we're limiting ourselves to synchronization of
the client with the server. Which would seem to exclude SUBMIT, unless
you're doing SUBMIT with magic folders and header fields or something
similar.

> Any server-to-server issues or

If you're talking about a new message format, you're going to have
to deal with end-to-end encryption issues whether you like it or not, either
by changing SMTP as well so that message in this new format can in fact
be transmitted end-to-end, by defining an encapsulation format, by allowing
servers to perform gateways, etc.

> management and authentication.

Again, if you're talking about a new message format, this isn't going
to be remotely adequate.

> - Internet Message Format

This also would seem to imply the message format is in play.

> [RFC 4409]

And SUBMIT is in play.

> [RFC 4468]

Again, I'm not prepared to offer an opinion as to whether or not this whole
thing is a good idea. But I will say that effectively replacing the entire
email protocol suite - which is entirely consistent with what's  been said here
- would at a minimum be a tremendous amount of work requiring a tremendous
amount of justification, and what's in this propsoal doesn't come close.

				Ned

---

From: John C Klensin <[email-masked]>
Date: Mon, 07 Nov 2016 13:46:44 -0500

In addition to my earlier comments, I agree with everything in
Ned's note (the two sets of comments seem to be complementary).
Also, please note:

--On Monday, November 07, 2016 10:05 -0800 Ned Freed
<[email-masked]> wrote:

>...

>> [RFC 4409]

4409 has been obsolete for five years now, implying that the
submitters of this proposal have some reading and other homework
to do.   RFC 6409 should be the reference here.

>...

In addition, I'd add to Ned's comment about end to end
encryption the observation that it is particularly important for
webmail and similar models because users or many such services
may have reason to not trust their email service providers (at
one end or the there) with message content.

     john

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 7 Nov 2016 19:36:33 +0000

My two cents: Interesting in principle but it has to be compatible with 
what gmail is doing. If the world's biggest site implements one protocol 
and an RFC specifies another, the RFC is forgotten.

Remember the old motto, rough consensus and running code.

Arnt

---

From: "Adrien de Croy" <[email-masked]>
Date: Mon, 07 Nov 2016 20:57:01 +0000

I wouldn't necessarily read "representation of email" as meaning message 
format.

It could mean representation of email interface.

As for other things, there are plenty of things which could use 
improvement, and I doubt whether it will ever happen in the current IMAP 
framework due to the problem of there being too many complicated 
combinations of optional extensions.

IMAP5 could do something useful, but the last attempts to kick-start 
that were still-born.

Adrien

------ Original Message ------
From: "Ned Freed" <[email-masked]>
To: "Alexey Melnikov" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>; "[email-masked]'" 
<[email-masked]>
Sent: 8/11/2016 7:05:06 AM
Subject: Re: [imapext] [ietf-smtp] Fwd: Request to form a new WG: JMAP

>_______________________________________________
>imapext mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/imapext

---

From: Julien ÉLIE <[email-masked]>
Date: Mon, 7 Nov 2016 22:56:08 +0100

Hi all,

> servers of more than one organization.

My two cents here:  someone proposed 2 years ago a similar model for 
NNTP.  It was named... JNTP (Json News Transfer Protocol).

Here is its current "specification" (in French):
   http://www.nemoweb.net/?page_id=75

with even an RFC-like format:
   http://www.nemoweb.net/?p=1295

And, as John hinted at, people get then stuck in exchanges between an 
NNTP server and a captive webnews interface like:
   http://news.nemoweb.net/

Since 2014, no much move for JNTP, which is still used only by his creator.
Like SMTP and IMAP, NNTP operates in plain text.  A great advantage over 
other alternatives.

> carefully before standardizing yet another alternative way to do

+1

-- 
Julien ÉLIE

« – Vous ramassez des champignons sans les connaître ?
   – Et alors ? Ce n'est pas pour les manger mais pour les vendre. »

---

From: Doug Royer <[email-masked]>
Date: Tue, 8 Nov 2016 10:15:31 -0700

> (2) Some of us believe, ... because they operate
> in plain text, ...

I thought JSON was 7-bit, and required encoding to use 8-bit data portable.

So, if I am correct, then JSON is not a gain when it comes to non-text data?

> (3) ...  Seems

The complexity of the requests and replies are the issue. If its mostly 
converting IMAP to JSON, the same problems are going to exist.

-- 

Doug Royer - (http://DougRoyer.US)
[email-masked]
714-989-6135

---

From: Ned Freed <[email-masked]>
Date: Wed, 09 Nov 2016 07:15:41 -0800 (PST)

> I thought JSON was 7-bit, and required encoding to use 8-bit data portable.

JSON is almost always UTF-8. So no, it's unabashedly 8-bit. But so is 
the overwhelming major of the existing email infrastructure, so this is 
a wash.

> So, if I am correct, then JSON is not a gain when it comes to non-text data?

This is also a wash.

Binary MIME has been around for almost 25 years. SMTP was extended to
handle it not long after it was defined. And note that it's entirely
possible to use binary MIME only on SUBMIT and not depend on the
rest of the infrastructure handling it.

IMAP has had the ability to transfer parts in binary for a long time. And
IMAP also offers the ability to decode and transfer a encoded part on
the server side.

JSON doesn't support binary, but there are various extensions that do.

> converting IMAP to JSON, the same problems are going to exist.

Yes and no. Part of the problem with IMAP is that it was designed for
a very different sort of client than what we have now. (I note that the
problem SMTP is intended to solve hasn't changed nearly as much.) Getting
rid of historical baggage will definitely simplify things.

IMAP also uses an unnnecessarily large number of format variants.

OTOH, there's absolutely no doubt that there's considerable intrinisic
complexity in the message access space. Especially given the desire to
push more and more of the problem to the server.

On the flip side, basing things on HTTPS adds an enormous amount of inherent
complexity. But most of that complexity is hidden from the majority of
developers, who won't code HTTPS support directly, preferring instead to
use any one of the bazillion libraries out there that does it all for you.

My guestimate is that - and here I'm speaking only of the message access space
- that a HTTPS/JSON approach wins in terms of effective implementation
complexity, although it is far from clear to me by how much.

				Ned

---

From: John C Klensin <[email-masked]>
Date: Wed, 09 Nov 2016 11:42:31 -0500

Ned,

A few questions about some of your comments (again, just
questions);

--On Wednesday, November 09, 2016 07:15 -0800 Ned Freed
<[email-masked]> wrote:

>> The complexity of the requests and replies are the issue. If
>> its mostly converting IMAP to JSON, the same problems are
>> going to exist.

> Yes and no. Part of the problem with IMAP is that it was
> designed for a very different sort of client than what we have
> now.

Which design model are you talking about?   It seems to me that
the original specs (through and including RFC 1064 and IIR 1176)
had only what we now call online mode and was optimized for
"kiosk" uses.  With the exception of the "walk up to someone
else's shared machine" part, that seems very similar to what we
have today with the assumption that everything is on the server,
the client keeps messages only on a transient basis, and one
cannot access messages unless one is online.    It seems to me
that is not very different from where we are headed now, whether
via "dumb MUA" or "MUA implemented on a web interface".   It may
be more complex than needed because it contains some
optimizations are what, by today's standards, are low-bandwidth
environments but some of those features also have security
advantages (that we haven't talked about very much recently).

Now, when we added offline (simulated POP3 on steroids) and
disconnected modes with IMAP4, things got more complex.  From my
personal perspective, disconnected mode (more or less the
ancient PCMAIL model) is the only thing that makes sense for
people with poor or intermittent connectivity, especially if
they need to work offline sometimes and use multiple machines.
However, it has never been broadly and well-supported in MUAs
(there are, IMO, a few "good enough" MUA implementations but no
really good ones) so maybe the marketplace has told us there
aren't enough users to count.  

Now, I've never been a fan of the LISP-like command and response
syntax, but that is much more a matter of taste than
functionality.

> (I note that the problem SMTP is intended to solve hasn't
> changed nearly as much.)

I agree with you but have heard claims that the multiple relay
and alternate destination (mail exhannge) models have outlived
their usefulness in today's highly-connected Internet.  I don't
see things as that highly connected (even looking to the future,
SMTP would work much better in the presence of a DTN than, e.g.,
HTTP[S} and for much the same reason it worked well for delivery
servers that were only connected a few hours a week) and I think
the robustness those features enable as important.  However,
there is no disputing that mail transport could be considerably
simplified if we allowed only endpoint-to-endpoint connections
with no alternatives.  Such connections would also permit
straightforward feature negotiation.  That would be a big help
for, e.g., EAI/SMTPUTP8 and similar arrangements and might even
help blur the boundary between transport-level and end0t9-end
encryption.

>  Getting rid of historical baggage will definitely simplify
things.

But that is my question.  I don't see IMAP disconnected mode as
historical baggage.  We've ended up with some redundant ways to
do things as a result of newer and more powerful options and
commands and the old ways could be cleared out.   We could
probably make some different choices about data structures and
maybe even standardize some things that today just add to the
complexity of the options.  But I'm not sure how significant
those actually would be -- what are you thinking of?

> IMAP also uses an unnnecessarily large number of format

Yes.  See above.
 
> OTOH, there's absolutely no doubt that there's considerable
> intrinisic complexity in the message access space. Especially
> given the desire to push more and more of the problem to the
> server.

Exactly.
 
>,,,
> My guestimate is that - and here I'm speaking only of the
> message access space - that a HTTPS/JSON approach wins in
> terms of effective implementation complexity, although it is
> far from clear to me by how much.

But, unless we can get rid of IMAP (and some of the
functionality, like disconnected mode, that can't obviously be
supported over a purely HTTPS/JSON interface) entirely, it seems
to me that an HTTPS/JOSN approach is additive, requiring the
mail environment to support both it and IMAP for (at least) a
very long time.  So, while I agree with your guess, I don't see
the need to support an additional as a net gain, especially in
the short to medium term.

   john

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 9 Nov 2016 18:22:50 +0000

John C Klensin writes:
> very long time.

Not at all. There are many, many clients that serve a very small audience 
and need only one protocol.

It has been mentioned on this list that a two-digit number of clients 
account for >99% of people's usage, and at another time that around a 
thousand different clients used a particular IMAP extension on a particular 
day at gmail. That's quite a difference, and my guess is that a big part of 
that is programs that implement someone's business logic and act on at most 
a handful of mail accounts. Those programs need to be simple to write.

Arnt

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 9 Nov 2016 18:42:22 +0000

On 09/11/2016 18:22, Arnt Gulbrandsen wrote:

> Not at all. There are many, many clients that serve a very small 
> audience and need only one protocol.
>
> thousand different clients used a particular IMAP extension on a 
> particular day at gmail. That's quite a difference, and my guess is 
> that a big part of that is programs that implement someone's business 
> logic and act on at most a handful of mail accounts. Those programs 
> need to be simple to write.
Agreed. I think there are applications which might still be easier to 
write using IMAP. Some applications might find JMAP to be easier.

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 9 Nov 2016 19:03:36 +0000

Hi Ned,

If this work goes forward, there is definitely some wordsmithing to do 
to address your comments. Quick answers to some of them below:

> "Representation of email" would seem to imply that we're talking about 
No, the email format is unchanged.
> suite with the exception of SMTP.
Yes, at least this is the proposal.
> is, in standards terms at least, new.
This might be web push, but I agree that this is not clear.
> So we're clearly aiming to replace multiple protocols here.
Yes.
> the client with the server.
Yes, basically the IMAP side of things.
> Which would seem to exclude SUBMIT, unless
> you're doing SUBMIT with magic folders
Most likely, yes.
> and header fields or something
> similar.
>
No, see above.

---

From: Doug Royer <[email-masked]>
Date: Wed, 9 Nov 2016 12:56:26 -0700

On 11/09/2016 08:15 AM, Ned Freed wrote:
>
> ... so this is a wash. ...
> ... This is also a wash. ...

Good :-)

> handle it not long after itKaia FIT Boise was defined. And note that it's entirely

Yes, MIME is awesome. However it is now possible to send, transfer, and 
fetch full binary data. The encoding/decoding cycle now seems to be more 
of a habit, than a need. Time to reverse this process and only encode 
when requested?

> JSON doesn't support binary, but there are various extensions that do.

Yes, that is my point earlier.

>> ...The complexity of the requests and replies are the issue. If its mostly

Yes, it was mostly used by command line tools, that allowed you to save 
attachments as files and to decide what part of email you wanted to view.

> IMAP also uses an unnecessarily large number of format variants.

YES!

> OTOH, there's absolutely no doubt that there's considerable intrinsic

Problems that would need to be solved first:

What does a modern MUA's need?

I am not talking about what it takes to make an IMAP aware MUA work. I 
mean what functional needs do they have? (Fetch, fetch body part, 
notifications, get summary information, ...). If the goal is to make 
this new protocol so much like IMAP as to make the coding converting an 
IMAP MUA to a new-protocol faster? Or is the goal to do whatever the 
correct new thing is?

Most of the IMAP command were designed to make command line tools work. 
Those limitations no longer exist. You would get a summary of headers, 
pick which ones you wanted to read. Then perhaps download the attachment 
and save it to a file.

I think the new message fetch protocol could be much simpler.

Polling for email was needed years ago because most computers (or the 
programmers) had difficulty performing asynchronous notifications, or 
multi channel I/O.

Does the new protocol have the server parse the headers and hand them 
over in a more simple-client manner? (You have 10 headers, here they are 
(left side/right side). Or does it send a blob of headers and the MUA is 
responsible for the parsing? Same with body parts.

Does the new protocol help with securing the content in any way? If the 
can is opened, should this be in the that mix of what's changing? How 
can it not be a major issue?

What is needed at the functional level?

> complexity, although it is far from clear to me by how much.

I am not a HTTP(s) solves every protocol problem person.

If the WEB MUA had different needs, what exactly are they?
Do they want the HTML formatting done on the server side? Some of it? 
One of the many SMTP, IMAP, and POP ports are already opened on any 
firewall that needs email.

One problem with putting email on port 80/443 is that some companies 
need to control customer private data. And they would now have to block 
by content, not by port. Much is more difficult.

What functional differences are their between WEB email and application 
clients? What differences are their between mobile device needs and desktop?

Doing JSON because its the new thing as the reason, I would think is an 
insufficient reason to disrupt the email process.

This process is going to be a nightmare if the functional needs are not 
nailed down first.

--

---

From: Ned Freed <[email-masked]>
Date: Wed, 09 Nov 2016 19:57:23 -0800 (PST)

> to address your comments. Quick answers to some of them below:

Sounds good.

> No, the email format is unchanged.

I note in passing that this is actually a departure from some of the
existing APIs, where messages are mapped into and out of JSON at least
some of the time.

> Most likely, yes.

And as I indicated previously, that's a problem, because magic folder are a
poor means of doing message submission, mostly because of their lousy error
handling.

I'll also note that none of the existing APIs I know of use magic folders. For
example, the Gmail API has two methods to send mail - message.send and
drafts.send, which return the error information you need. And the Outlook API
is very similar except that HTTP POST is used

But this really isn't the time to get into details. My point here is that if
you're going to do message submission, you need to get it right - something
essentially none of the current specifications do -  and that's going to
require an expansion of the model being proposed here.

				Ned

---

From: John C Klensin <[email-masked]>
Date: Thu, 10 Nov 2016 09:28:50 -0500

> And as I indicated previously, that's a problem, because magic
> folder are a poor means of doing message submission, mostly
> because of their lousy error handling.

It is also worth noting that magic folders have been repeatedly
proposed for IMAP and rejected every time, always for reasons
that included the above.  If you decide to charter this, please
let's be sure that there are built-in mechanisms for adequate
review that considers those prior discussions.  In principle,
I'm not opposed to a carefully-considered decision that reaches
a different conclusion, but "carefully-considered" --and not
just by advocates of this work -- is key. 

>From my point of view, we went to the trouble to separate
message submission from SMTP for only two important reasons: to
make it clear what things a submission server could reasonably
do (and be expected to do) that an SMTP relay could not and to
permit (and encourage) a better error-handling model among
cooperating parties.  At least for me, that created a message
creation and submission model that was a partnership between the
creating MUA and a first-hop server, paralleling what we've
often called a "split MUA" model between an IMAP (or POP) server
and client at the receiving end.  Again, if a different model is
really needed --and, so far, I'm not convinced -- those
advocating it should be required to demonstrate to the community
that they understand all the implications of discarding or
bypassing the older one.

best,
     john

> require an expansion of the model being proposed here.

best,
   john

---

From: Neil Jenkins <[email-masked]>
Date: Fri, 11 Nov 2016 14:57:45 +1100

Hi all,

Rather than reply to individual emails, I'm going to try to put across
answers to the queries and concerns related so far in this thread. I'm
the author of the draft specification we're putting forward with this
charter; happy to answer any further questions.

# What is JMAP replacing?

JMAP is intended to be a new standard for email clients to connect to
mail stores. It therefore replaces IMAP + SMTP submission. It is also
designed to be more generic such that it can be extended with contacts,
calendars in the future (replacing CardDAV/CalDAV). It does not replace
MTA-to-MTA SMTP transmission.

# Why is this needed?

It's too hard to write a good MUA with current standards, which has led
to a stagnation in good email clients and a proliferation of proprietary
protocols.

In addition, IMAP is really not suited to a constrained network
environment, as often found on mobile networks even in developed
countries. The chatty nature of the IMAP protocol is ill-suited to high
latency connections. Things like a folder rename on the server can mean
that all the downloaded and cached email for that folder is invalidated
(because you can't tell for sure over the protocol that they're the
exact same messages), wasting huge amounts of bandwidth. Stateful
connections make it harder to deal with intermittent network outages.

We're attempting to standardise a new protocol because lots of people
are writing proprietary alternatives to deal with the same deficiencies
with the current standards. Some of these deficiencies could be fixed by
adding things to IMAP (for example persistent IDs on folders and
messages for better caching when another client copies, moves or
renames), but others are structural like the need to calculate a MSGNO
<=> UID mapping even if the client doesn't want it, or the need to find
endpoints and authenticate separately to multiple different protocols to
do related tasks.

Because it's so hard to write a good email client that works with any
IMAP provider, these days many new clients are just for Gmail (with
possibly support for a few other big mailbox providers). It's easy to
see why: IMAP is either the woefully inadequate original RFC3501, or
it's a messy set of incomplete implementations of some of the
extensions. Even CONDSTORE (without QRESYNC) is pushing it; you can't
rely on it to be there in most cases.

As a result of this, proprietary protocols have been popping up as
alternatives to IMAP. Here are a few examples, the latter two being from
whole companies that formed just to try to help people not have to deal
with IMAP:

Gmail: https://developers.google.com/gmail/api/v1/reference/
Outlook:
https://msdn.microsoft.com/office/office365/APi/mail-rest-operations
Nylas: https://nylas.com/cloud/docs
Context.io: https://context.io/docs/lite

In addition, we're seeing most new mobile email clients proxy everything
via their own server rather than talking directly to the user's mail
store, for much the same reasons. Examples include Mailbox (now
defunct), Alto, Outlook and Newton. This is bad for security and
privacy, and also bad for the client authors as they have to run server
infrastructure in addition to just building their clients.

Despite not only being proprietary but patented (and expensive!),
ActiveSync has seen a big increase in adoption, and not just with
Microsoft servers, due to its better support for mobile environments and
ease of setup (one login for mail receive/send, contacts and calendars).

# Why is JMAP better than IMAP?

JMAP not a conversion of IMAP to JSON; it is a new protocol. It was
designed to be make much more efficient use of network resources, be
easier for developers to work with and hopefully make the best
protocol for email an open standard once more. It's based on years of
experience and real-world experimentation at FastMail, and talking to
other major MUA/MTA developers to make sure we understand the common
needs of the industry.

Some important attributes that help achieve these goals:

* The protocol is stateless. It doesn't need a persistent connection,
  which is better for mobile use which may have intermittent network
  access and wants to conserve battery life by turning the radio off
  whenever possible.

* Ids are immutable and not intended to be user visible. So folder
  naming becomes less messy - more like NFS or filesystems with inodes
  rather than a name-based hierarchy, and renaming is easy to detect and
  cheap to sync.

* It has a flexible set of commands, which can be batched in arbitrary
  ways.  You can batch or pipeline single JMAP operations over a stream
  protocol easily enough if you wanted to, but we're mostly envisaging
  it being used for stateless batch operations to make disconnection
  less painful.

With IMAP you can set two messages to both have the same flag (. STORE
1,2 +FLAGS (aflag)) but you can't store two different flags to two
different messages in the same action. JMAP allows multiple create,
update and destroy actions on different messages in a single setMessages
command. Pipelining also has the problem that if the connection drops at
just the wrong moment you can wind up applying the first change but not
the second.

You can use backreferences to other objects created in the same batch -
allowing you to, for example, create a folder tree by referencing
previous parents created in the same request.

* Clients can efficiently fetch updates from their current state a-la
  QRESYNC. This can be implemented effectively using the MODSEQ data
  already in modern IMAP servers, or by using a transaction log data
  structure. The server can always indicate to the client if it
  cannot calculate updates from a particular client state (e.g.
  because it is too old).

* Flood control. The client can always restrict how much data the server
  should send. For example, a command might return a "tooManyUpdates"
  error if it exceeds the client's limit, rather than returning a
  million "* 1 EXPUNGED" lines as can happen in IMAP. Sometimes it's
  just more efficient to throw away cached data and refetch, especially
  if you're a mobile/webmail interface with only a partial cache of the
  server's data.

* It doesn't require a custom parser. I've got a longer explanation to
  the HTTPS/JSON question lower down, but having an encoding format that
  is well understood and has widespread support among all programming
  languages makes it far easier for developers to get started,
  especially if they don't want to build a whole MUA but just integrate
  something with email.

* The data model is backwards compatible with both IMAP folders and
gmail-
  style labels. Servers that implement JMAP are likely to want to
  support IMAP as well for the foreseeable future, so it's important to
  be able to have data structures that support both. Messages are
  similarly immutable other than flags/mailboxes.

* Email can be sent using the same protocol, reducing confusing failure
  modes for users (again I talk more about this below). We also have
  pretty-
  much complete specs for calendaring and contacts via JMAP, but we're
  not pushing for them to be standard yet because the object format is
  still undergoing a lot of work in the CalConnect group. We think a
  single consistent protocol for all of these has a lot of advantages
  though, and we hope to get there in the future.

# Why use HTTPS/JSON?

The short answer is it's good enough, widely understood and it's by far
the easiest thing for developers to adopt. There's support in basically
all OSes and programming languages. It's easy to read and debug.

HTTP doesn't tend to run into firewall issues, and is so commonly used
it has integrations which can help with optimisation (for example, iOS
has built-in support for optimising radio usage by batching HTTP calls
from different apps where possible, which their mail team have told us
they would like to be able to use). This isn't an innate advantage of
HTTP, but rather an advantage of its ubiquity.

With GZIP, JSON data is reasonably compact and fast enough to
serialise/parse. However, the encoding/transport part of JMAP is not
core to its operation, so future specifications could easily add
alternatives (e.g. WebSocket (https://tools.ietf.org/html/rfc6455)
instead of HTTPS, CBOR (http://tools.ietf.org/html/rfc7049) instead of
JSON). For the initial version though, HTTPS+JSON makes the most sense.

# Binary data

Binary data is not transported in the JSON (and indeed, as has been
pointed out, can't be without base64 encoding or similar, which is
inefficient). Instead, attachments are referenced by a blobId, and
uploaded/downloaded separately via HTTPS. Clients can reference the
blobId elsewhere to, for example, attach the same file to a new message
without having to download and reupload it again, a big win on slower
internet connections.

This also means that regularly saving drafts (a common client behaviour)
does not mean sending the same full multi-megabyte attachments over the
network every 60s or so.

As it's out-of-band with the API calls, uploading/downloading files can
easily be parallelised and doesn't block other API operations.

# Representation of email

JMAP defines a JSON structure that represents in a consistent and
structured way all the information that the vast majority of
clients need from an RFC5322 message. The server deals with the
complexities of MIME, encoding issues, parsing headers etc. The
intention is that the server will still operate with RFC5322
messages for storage and certainly transmission; the JSON
representation is not intended to replace RFC5322, just relieve
client authors from having to deal with it.

Clients that want to or need to (for example those doing PGP in the
client) can still fetch the RFC5322 if needed. The message is
represented by a blobId, and the raw bytes can be fetched using the same
binary download mechanism as mentioned above.

# Message submission

Message submission is via means of an "outbox" folder. Messages are
moved there to send. This was chosen over a separate "send" command for
a few reasons. Firstly it's most consist with the rest of the API,
making it easier for clients to implement offline support
(synchronisation is the same as other changes you might make to
messages). Secondly, clients/servers can support delayed send
(particularly useful for "undo send"), by simply setting the date on the
message in the future and only sending from the outbox when this date is
reached. Until then the message is just sitting in the outbox like any
other mailbox, allowing client to list and revoke without needing custom
API commands.

Error handling is flexible enough to return a full range of errors when
you try to move a message to this folder, just as you would to a
separate "send" method.

Clients can use the same JSON structure for sending messages as they get
from the server for received messages, allowing the server to deal with
MIME encoding. This allows clients to be much simpler and easier to
write. (They can also upload a raw RFC5322 message if they want instead
of course.)

Having the same protocol for message sync and submission is a huge win
for usability; we see a lot of support tickets where users can receive
but not send, or vice versa, because one of these is misconfigured. This
is always very confusing for regular users.

# Push mechanism

Immediate updates is an important feature to many users. IMAP IDLE has
two big problems: firstly it only notifies of changes in one folder, so
doesn't inform you of all changes unless you open a connection for every
folder, and secondly it requires a persistent network connection which
is bad for mobile (and not even allowed on iOS).

JMAP defines two push mechanisms to support the two common use cases. In
both cases the data transferred is simply an edge trigger: a new state
string letting the client know something has changed within a particular
datatype. The client then fetches the new data using the standard
synchronisation methods.

For desktop clients and webmail, there's an event source interface
(https://html.spec.whatwg.org/multipage/comms.html#the-eventsource-interface)
. This requires a persistent HTTP connection.

For mobile, and web integrations, you can set a callback handler. This
makes the mail store server do a callback to a server defined by the
client when something changes; the client's server can then send out-of-
band push events using the native push mechanism of the mobile client.
JMAP itself doesn't require any particular mobile push technology.

# End-to-end encryption

A lot of the optimisations for efficient client-server sync require the
server to be able to read the message. If everything were encrypted, the
server would basically be a dumb blob store, unless you have some clever
partial metadata search capability. This is particularly bad for mobile,
where you only want to sync partial information. Users expect to be able
to search their whole archive, so either you need all the data in the
client, or the server needs to have access to the data.

JMAP is therefore not making any new measures to address end-to-end
encryption. The best advice is probably to run your own "JMAP SERVER" on
trusted hardware; otherwise you need to sync the entire multi-gigabyte
mail spool to all your devices. JMAP is also simple enough that you
could run the server on multiple machines with an underlying replication
protocol over encrypted links and have that do your smarts.

# Draft proposals and implementations

As Arnt pointed out, "rough consensus and running code" are key here,
and the current draft of the JMAP spec is being implemented in Cyrus
IMAPd and Dovecot (the two largest open-source IMAP servers), as well as
other open source projects like Apache James and Linagora. Roundcube
have stated they plan to build their next-generation client on JMAP.

On the proprietary side, Atmail have a JMAP proxy and webmail client
they are releasing to production very soon, and we at FastMail have a
version of our client that talks JMAP too.

There is interest among other large mailbox providers/mail client
authors, but they don't tend to be early adopters as much.

The current specification drafts can be found at:

https://datatracker.ietf.org/doc/draft-jenkins-jmap/ (The generic JMAP
protocol)
https://datatracker.ietf.org/doc/draft-jenkins-jmapmail/ (Mail over
JMAP)

We have also developed an open-source JMAP proxy, which you can connect
to an IMAP server to try out the protocol and see what happens over the
wire. There's a hosted version at http://proxy.jmap.io/, or you can grab
the code from https://github.com/jmapio/jmap-perl

Cheers,

Neil.

---

From: Tony Finch <[email-masked]>
Date: Fri, 11 Nov 2016 11:47:35 +0000

> # Message submission

A few vague thoughts...

One of the problems with separating message submission and storage is that
there are a lot of tricky error handling and state transition issues as
the message moves from Drafts -> Outbox -> Sent, rolling back properly if
sending failed, partial failures for a subset of recipients, etc. It's not
uncommon for users who have been having problems to say something like
"but the message is in my Sent folder, it must have been sent!"

Also, my experience of MUAs reporting SMTP errors to users doesn't fill me
with confidence. The servers I used to run went to some effort to avoid
SMTP-time errors as much as possible, preferring to accept and bounce.
There's lots of scope to do much better in this area.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/  -  I xn--zr8h punycode
Trafalgar: North or northwest 4 or 5. Moderate, occasionally rough for a time
in north. Showers. Good.

---

From: Ned Freed <[email-masked]>
Date: Fri, 11 Nov 2016 06:34:55 -0800 (PST)

> "but the message is in my Sent folder, it must have been sent!"

Problems which are exacerbated by using magic folders.

> There's lots of scope to do much better in this area.

Actually, this varies enormously depending on where you are and what sorts of
clients you use. We offer both "accept and report errors later" and "report as
much as possible immediately" modes and we find our servers are overwhelmingly
configured to do the latter.
 
The problem with "accept and report errors later" is the lack of immediate
feedback. This, coupled with poor/nonexistent support for correlating DSNs with
sent messages, works so poorly that a client has to be truly terrible at
reporting errors at submit time for it to be viable.

But there are some truly terrible clients out there.

				Ned

---

From: Ned Freed <[email-masked]>
Date: Wed, 09 Nov 2016 10:27:32 -0800 (PST)

Apologies for the late response on this one.

> advantages (that we haven't talked about very much recently).

Pretty much the entire design model is canted away from what I see being needed
by modern clients.

One assumption the old model makes is that bandwidth is cheap enough that
there's no need to highly optimize metadata transfer. And to some extent this
also applies to message data, e.g., the lack of support for message summaries
and thumbnails, but that's a more complex topic.

This matters because with mobile client view timliness is critical, but not at
the expense of bandwidth. And push notifications only get you half of the way
there - once notified you have to connect and update.

Sit down with a mobile client designer sometime, and you'll quickly see that in
order to build the sorts of message displays they want without transferring far
more data than they need or can afford, you need to engage a large number of
IMAP extensions, and even then you're unlikely to achieve the exact result they
would prefer.

And then there's search. A powerful search capability is now essential, but
IMAP search assumes a particular set of semantics that are simultaneously not
what a mobile client needs (nobody is going to spend time entering complex
search terms on their phone) as well as being misaligned with modern text
search engines, making it difficult to implement on the back end. (What
typically happens is implementations vary substantially from what the
specifications call for, and vary in different ways.)

Then there's the stuff that nobody cares about that you have to support, like
subcriptions.

I could go on and list more stuff, but I think this makes the point that
there's a mismatch. And really, given how the landscape has changed and how old
some of these specifications are, I see no reason for surprise.

> aren't enough users to count.

Offline support is another thing that's increasingly superfluous now. Mobile
devices are expected to always be online and to present an up-to-date view of
the world at all times.

Of course the sync capbilities designed to support offline mode can also be
used to minimize data transfer for a connected client, but even so, it's not
quite the same problem.

> syntax, but that is much more a matter of taste than

Unfortunately, syntax matters more than it really should. I previously pointed
out that the complexity of HTTPS in this context is irrelevant because "there's
a library for that". The same is true for JSON - the automapping mapping of
JSON to and from data structures that many languages offer is very convenient.

> their usefulness in today's highly-connected Internet.

Even if it's true, this is all external to SMTP itself, and I don't think
it's relevant here.

> I don't
> with no alternatives.

Uh, no, not really. To the extent that implementation of message transfer is a
complex problem, it's mostly in the splitting of messages and the handling of
errors. These problems don't go away when you limit things to one hop. In fact
one of the nice things about SMTP is that you don't know and don't need to know
how many hops there are ahead of you.

The complexity of transitions through multiple ADMDs grots up the security and
spam situation, but those are for the most part externalities to the SMTP
implementation.

Additionally, this notion of getting rid of multiple hops across the Internet
ignores the present day reality that as the size of a "typical" mail system
deployment has increased, so has the number of hops messages often take inside
the deployment.

> Such connections would also permit

Perhaps, but again, I don't see the relevance to the matter at hand.

> >  Getting rid of historical baggage will definitely simplify things.

> those actually would be -- what are you thinking of?

IMAP disconnected mode support isn't baggage because it provides capabilities
that are useful to mobile connected clients, albeit in a somewhat suboptimal
fashion.

That said, there absolutely is substantial baggage in IMAP.

> the short to medium term.

Just a guess on my part, but I rather suspect that disconnected mode is
easier to do on top of a protocol designed to optimize mobile sync than
the other way around.

Of course this ignores the cost of conversion. But I don't think you can
necessarily make the case based on protocol features alone.

				Ned

---

From: Ted Lemon <[email-masked]>
Date: Tue, 15 Nov 2016 06:49:10 +0900

I agree with almost everything you said here, Ned, except for the idea that
offline isn't important. For those of us who travel internationally, being
able to look up travel info on your phone before you have your
international SIM is crucial, and this is just one example of a situation
where this is important.

On Nov 15, 2016 4:49 AM, "Ned Freed" <[email-masked]> wrote:

> ietf-smtp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf-smtp
>

---

From: Doug Royer <[email-masked]>
Date: Mon, 14 Nov 2016 17:23:48 -0700

> (lots of stuff ...)

Ned: YES - YES.

I have felt that way for years.

In response to to Ted's concerns about off-line.

I use SeaMonkey (a Thunderbird spin) for my email. When I setup a mail 
account with it, one of the options is for it to cache the email account 
for offline usage.

I would think this is mostly a client issue. In a future protocol, when 
checking for new message data, pull all of the summary data, and 
optionally all of the body parts. Its a client issue. Save the data, or 
not. If you pull it all, then the client is free to save it, or not.

--

---

From: Ted Lemon <[email-masked]>
Date: Tue, 15 Nov 2016 09:30:14 +0900

Making offline support a "client issue" means that every client will
do it differently, and there will be no hooks in the protocol to make
it easy/consistent.   Why _wouldn't_ you want to document offline
support?

On Tue, Nov 15, 2016 at 9:23 AM, Doug Royer <[email-masked]> wrote:
> account with it, one of the options is for it to cache the email account for
> offline usage.
>
> checking for new message data, pull all of the summary data, and optionally
> all of the body parts. Its a client issue. Save the data, or not. If you
> pull it all, then the client is free to save it, or not.
>
>
> --
>

---

From: Bron Gondwana <[email-masked]>
Date: Tue, 15 Nov 2016 12:14:12 +1100

http://jmap.io/client.html

Keeping a full copy of mail in sync

JMAP can also be used to efficiently download and keep in sync the entire set of a user’s mail. [...]

FastMail is planning to implement offline support in our own apps based on JMAP early next year.  We're evaluating which web storage technologies are going to be most robust, because we'll still be basing our own app on an embedded browser.  One of our major goals with JMAP is to be more offline-friendly than our current API.

And I'd be very happy to support any suggestions to make JMAP more offline friendly during the standardisation process.  Offline is really important to me as well.

Bron.

-- 
  Bron Gondwana
  [email-masked]

---

From: Ned Freed <[email-masked]>
Date: Mon, 14 Nov 2016 22:38:53 -0800 (PST)

> international SIM is crucial, and this is just one example of a situation

There's a difference between being able to operate offline and a protocol
specifically designed to support offline operation.

As you note, the former is still useful, but is, as Doug notes, mostly a matter
of client support. (Unfortunately, I suspect that as a practical matter you're
going to have ever-increasing difficulty getting client developers to
prioritize it.)

To the extent it's a protocol issue, it's mostly a subset of the mobile
sync problem.

Or, to put it another way, it seems the mobile sync problem has more
requirements and is more constrained than the offline sync problem. And for
better or worse, the design focus of IMAP was and is on the offline sync
problem.

All that said, it's not entirely clear to me that the design focus of the
current crop of JMAP proposals is geared towards the mobile client sync problem
either. At least some of what I've seen seems more oriented towards support of
business workflows, which is a very different (and IMO much simpler) problem.

				Ned

---

From: Ted Lemon <[email-masked]>
Date: Tue, 15 Nov 2016 16:30:14 +0900

Yup.   The reason I keyed in on what you said is that I've had the
same thought about IMAP for a long time--I would like to have a much
less chatty sync process.

---

From: Tony Finch <[email-masked]>
Date: Tue, 15 Nov 2016 12:10:19 +0000

Is everyone here using the term "offline mode" to mean the same thing?

RFC 1733 defines "offline mode" to mean POP3-style delete-from-server.
"Disconnected mode" is the term for a client that has a local cache
or mirror of what is on the server.

Approxomately no-one wants RFC 1733 offline mode. (Bah, the number of
times I have had to recover mail for someone who accidentally pointed an
offline-mode client at their account.)

How much do the necessary protocol features differ between a full-fat
disconnected mode client (a mirror on the client of the entire
account on the server) vs a poorly-connected mobile client which needs to
sync a smaller window on the account? Concurrent access to multiple
mailboxes? what else?

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/  -  I xn--zr8h punycode
Trafalgar: Easterly or northeasterly 5 or 6, occasionally 4 at first, but 6 to
gale 8 in southeast. Moderate or rough. Fair. Good.

---

From: Doug Royer <[email-masked]>
Date: Tue, 15 Nov 2016 12:53:06 -0700

Documentation:

	Keep copies of the email if you want.

	Outbound is easy, queue it until you connect, then send.

	If the server has emails deleted, that the client still has,
	delete them from your client if you want.

Documentation done? :-)

I do not understand why its a protocol issue. What is needed? What would 
make it easy/consistent?

--

---

From: Bron Gondwana <[email-masked]>
Date: Wed, 16 Nov 2016 12:03:01 +1100

> specifically designed to support offline operation.

There are two really nice things in JMAP for offline operation:

1) client-generated IDs and server-generated IDs are different.  offlineimap does some magic with negative values for UIDs because it can't generate server-side IDs.  This is similar, but
2) you can refer to client-side IDs during a batch command, so you can backreference a mailbox you just created when adding messages to it, so you don't need to try to generate IDs locally and either have to use UUIDs for everything or risk clashes.

And of course everything is by unique ID rather than names, so you don't get problems like the a folder being renamed to something else and then recreated with a new UIDVALIDITY while you were offline, which breaks offlineimap completely, and probably causes mail loss in other clients.  With JMAP and a server which retained ID during renames, the added or changed messages would still apply into the new folder when the updates were applied.

JMAP is quite friendly to the idea of just storing JMAP setFoos commands in a log and then playing them against the server on reconnect, as well as the more clever approach of fetching everything from the server first and then resolving your changes into a minimal set of patches.

Bron.
-- 
  Bron Gondwana
  [email-masked]

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 16 Nov 2016 10:27:15 +0900

Hi,

I've just asked IETF Secretariat to create a new mailing list: [email-masked]. Please subscribe if you are interested (or even curious) about JMAP. You can visit
 https://www.ietf.org/mailman/listinfo/jmap
or send "subscribe" email to [email-masked].

Can I please ask further discussions of JMAP and JMAP Chartering to be done there.

Thank you,
Alexey

---

From: Doug Royer <[email-masked]>
Date: Tue, 15 Nov 2016 21:18:08 -0700

I have started replying to  [email-masked] only for the JMAP issues.

--

---

From: Jacob Palme <[email-masked]>
Date: Thu, 17 Nov 2016 18:47:07 +0100

The MHTML format may be of interest as input to this group. It compiles encapsulation of various parts in MIME-formatted e-mail.

See:
https://en.wikipedia.org/wiki/MHTML
https://tools.ietf.org/html/rfc2557
— 
Jacob Palme professor emeritus Stockholm University
for more info see https://people.dsv.su.se/~jpalme/

> On 07 Nov 2016, at 19:05, Ned Freed <[email-masked]> wrote:
>

---

From: Brandon Long <[email-masked]>
Date: Mon, 21 Nov 2016 21:38:11 -0800

I've said this to some client developers directly, but it's good to share
the concept more widely:

There should be no "full mirror" clients, only disconnected state clients
which make different choices of window size.  There are Gmail accounts
today which would exceed the ssd in many laptops, or the monthly bandwidth
allotment of some broadband customers.  A full sync is useful for backups,
but a regular client would be wise to make full sync a special case of
"reasonable cache space on this device/connection happens to be larger than
the user's account".

Granted, I would also argue against connected clients in general, since the
next billion users typically have poor/expensive connections, and many
users are on mobile networks where the quality can vary greatly from block
to block.  Network usage is also more likely to be a WAN these days, with
the higher latency that implies.  Also, storage size is mostly to the point
where a local cache can be fairly large compared to most content... OTOH,
shared computers may make local sync or install less desirable, but I think
most of those cases tend to web clients these days.

Brandon