Subject: Fwd: Re: [Jmap] WG Review: JSON Mail Access Protocol (jmap) - reducing configuration complexity

From: Gren Elliot <[email-masked]>
Date: Tue, 7 Feb 2017 07:38:26 -0800

> Trying to setup a new device for use with open protocols is a complete
> nightmare - you end up needing to configure 2 things for mail (IMAP
> and SMTP)

and now, because we will have to be backward compatible, it will be
three. so that third had best be a really massive win. and i am not
seeing it. what i am seeing is second system syndrome in a wrapper of
whatever today <http://airmail.calendar/2017-02-07%2012:00:00%20GMT>'s
wrapping fashion is.

randy

Why?  You will either be using IMAP/SMTP to access your mail/submit your
messages or you will be using JMAP.  If you have the option of the latter,
you’ve just halved the number of things that need configuring.

Regards,
Gren Elliot
(forgot to include the mailing lists in reply to Randy earlier)

---

From: Randy Bush <[email-masked]>
Date: Wed, 08 Feb 2017 05:03:42 +0900

>> Trying to setup a new device for use with open protocols is a
>> complete nightmare - you end up needing to configure 2 things for
>> mail (IMAP and SMTP)
> 
> Why?  You will either be using IMAP/SMTP to access your mail/submit
> your messages or you will be using JMAP.  If you have the option of
> the latter, you’ve just halved the number of things that need
> configuring.

how does the server forward the submitted mail to the global internet?

how does the server migrate 200 or 2,000 imap users at human speed,
i.e. over months?

yet another protocol that requires a flag day.  that has worked out
so well for ipv6.

randy

---

From: Dave Crocker <[email-masked]>
Date: Tue, 7 Feb 2017 12:12:31 -0800

On 2/7/2017 7:38 AM, Gren Elliot wrote:
> You will either be using IMAP/SMTP to access your mail/submit your
> messages or you will be using JMAP.  If you have the option of the
> latter, you’ve just halved the number of things that need configuring.

The primary argument being put forward here is simplification of 
end-user configuration effort.

While that has an intuitive appeal, a basic question for any effort to 
replace and existing technology is how big the benefit will be and for 
how much of the market?

In this case, for most people, the configuration effort is quite rare. 
So while it might be a bit of a hassle, it has no effect on daily life.

Well, ok, there are some folk who have to make this change more often, 
due to Draconian and misguided local policies -- the current advice 
about password changes, from the UX community, is not to make changes 
often, since this becomes a serious attack surface.

But how large a community is this and why is this problem not mitigated 
simply by having the user interface take one password specification and 
map it to the two, underlying (and existing) protocols?

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: John C Klensin <[email-masked]>
Date: Tue, 07 Feb 2017 15:54:09 -0500

I agree with most of Dave Crocker's comments and don't have much
time to spend on a proposal that, if approved, I believe (for
reasons that overlap those he cites) will, like many of its
predecessors, die a quiet death after wasting a lot of time.
However, the exchange surrounding one of Randy's comments
prompts a further remark or two...

--On Wednesday, February 8, 2017 05:03 +0900 Randy Bush
<[email-masked]> wrote:

>> Why?  You will either be using IMAP/SMTP to access your
>> mail/submit your messages or you will be using JMAP.  If you
>> have the option of the latter, youʼve just halved the number
>> of things that need configuring.
> 
> how does the server migrate 200 or 2,000 imap users at human
> speed, i.e. over months?

So, suppose I have a mail server that, because it is configured
in the real world, has to support SMTP and IMAP (and maybe POP3)
for existing users while trying to support new MUAs that use
this JMAP stuff.   Randy suggests that requires configuring
three kinds of systems.  I suggest that he underestimates the
problem.

First, there is not just "configuring" but "supporting".
Working with configurations may be an infrequent activity.
Providing support is a continuing one and takes time, even if
one's idea of support is saying "what do you want for nothing,
figure it out yourself" in a firm tone and at frequent
intervals.  A change from an IMAP/SMTP base to supporting a new
model or representation and systems that provide it vastly
increases that job, and increases it by at least a factor of
three (old one, new one, people considering or trying to make
the transition).

Second, there is an issue that Randy and Dave both hinted at in
different ways.  Unless the proposal is to use Jmap end to end,
replacing SMTP, conversion issues have to be worked out.  The
community made a series of decisions around the time MIME was
adopted that essentially require that the SMTP payload be either
unstructured messages (essentially conformant to 822) or MIME.
The previous (albeit controversial) idea that, other than trace
fields inserted by SMTP MTAs, SMTP could treat the message as
header-free and opaque and carry material that was not at least
822-conformant is history.  That means one needs to configure,
maintain, support, and debug:

	SMTP
	IMAP
	JMAP
	SMTP/Standard Message Headers -> JMAP for inbound
	     messages
	JMAP -> Message Submission for outbound messages.
	
That is a big deal.  There better be a lot of added value to
justify it or it will go nowhere (a slightly different negative
analogy to the rapid and overwhelming success of IPv6 is perhaps
obvious), just as a variety of grand solutions to real or
imagined problems with email have gone nowhere.    With this
proposal, I see some small incremental improvements for systems
that are working in JSON environments anyway, but nothing that
would justify widespread implementation and deployment.

The answer might be different for a mail system that assumed
that all users would be using the same mail store and either
webmail or JSON-based clients/MUAs.  But such mail systems exist
only in proprietary walled gardens and should not be IETF topics
except possibly for mail gateways to and from them.  And those
gateway raise most of the issues outlined above.

TL;DR summary: Just say 'no'.

    john

---

From: "Adrien de Croy" <[email-masked]>
Date: Tue, 07 Feb 2017 21:42:27 +0000

I've dealt with a lot of customer support issues relating to inability 
to send mail where reading was available.

the first point I'd make is that most users wouldn't imagine in a 
million years that you would have a different system for sending mail vs 
reading it, and they are often incredulous when we point that out.  I 
actually have a lot of sympathy for that POV.

Apart from that, the main issues we see people battling are, in no 
particular order

* issues with ISPs.  Common to block port 25, many admins don't know 
about Submit port (587)
* issues with firewalls, need to open / map additional ports
* issues with incorrect user data entry due to double the configuration
* various client software issues exacerbated by this

We (since our SMTP + IMAP is integrated) don't see so many issues 
relating to dealing with 2 different products (or more) to support mail 
(e.g. separate vendor for SMTP vs IMAP), but I would expect those would 
come with admin problems as well, especially if they use locked-in user 
databases that other products can't access, different support for 
different auth methods, different support for different crypto 
algorithms (recent changes to OpenSSL to deprecate RC4 and 3-DES 
highlighted some of these).

In short, managing 1 port and endpoint requires work.  Managing 2 
requires more.

I don't know how many people get SMTP and IMAP from the same place, but 
I would expect it to be the majority of users.  It would be interesting 
if someone could get some answers on that.

Adrien

------ Original Message ------
From: "Dave Crocker" <[email-masked]>
To: "Gren Elliot" <[email-masked]>; "[email-masked]" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>; "[email-masked]" <[email-masked]>
Sent: 8/02/2017 9:12:31 AM
Subject: Re: [Jmap] Fwd: Re: WG Review: JSON Mail Access Protocol (jmap) 
- reducing configuration complexity

>_______________________________________________
>Jmap mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/jmap

---

From: Neil Jhaveri <[email-masked]>
Date: Tue, 7 Feb 2017 21:49:48 +0000

> On Feb 7, 2017, at 12:12 PM, Dave Crocker <[email-masked]> wrote:
> 
> But how large a community is this and why is this problem not mitigated simply by having the user interface take one password specification and map it to the two, underlying (and existing) protocols?

I agree that this can go a long way.  On iOS and macOS, we've done this to a limited degree, by building a central database of accounts that each app uses. “Branded partner” accounts like Google/Yahoo!/Aol/etc have the credentials centrally stored, and child accounts (IMAP, SMTP, etc) inherit them, with a single UI framework to re-authenticate.

Unfortunately, regular accounts still don’t have this treatment, and it still drives a lot of calls to AppleCare. There are a bunch of reasons why, but one is that it’s tricky to be sure that the various protocols/hostnames have been appropriately bundled together into what is truly the same service, where the same username/password truly applies, and there isn’t a security issue from taking your user-updated IMAP password and sending it off to a CalDAV server with a different hostname. This does seem like an incrementally solvable problem, though, with some of the other work being done in the account configuration space.

However, there are still some operational issues that arise, even with this kind of UI, for example:
(a) Two different protocols/servers often lead to very confusing partial failures (I can check for my Mail, but I can’t send any).
(b) Port fire-walling. It was a more common support issue for us 8-10 years ago, but has subsided in recent years.

That all being said, I think there are some very compelling aspects here, particularly the stateless nature of the protocol, and using HTTP underneath. From a client’s perspective, I can say that this would lead to a far simpler implementation, with pre-existing EWS/EAS implementations being the backing of that statement.

---

From: Gren Elliot <[email-masked]>
Date: Tue, 7 Feb 2017 23:02:53 +0000

In answer to the UI question below, for the simple reason that you don't
know whether the same password applies, because the standards have
deliberately separated them and there is no standard provided which can
convey the information that in this case, they're the same. You could
perhaps just try the same credentials for each service but that would be a
clear leak of private information if they weren't the same.

On 7 Feb 2017 20:12, "Dave Crocker" <[email-masked]> wrote:

But how large a community is this and why is this problem not mitigated
simply by having the user interface take one password specification and map
it to the two, underlying (and existing) protocols?

---

From: Joe Hildebrand <[email-masked]>
Date: Tue, 7 Feb 2017 18:47:45 -0700

> On Feb 7, 2017, at 1:54 PM, John C Klensin <[email-masked]> wrote:
> 

No hats, personal opinion only, and in response to several of the messages in this thread, not directly to John.

Will it ever be possible for new work to be proposed at the IETF without drowning out a chorus of nitpickers?  You may not want to help standardize it.  You may not want to implement it.  You may not want to deploy it.  You may not want to use it.

So don't.

This work won't hurt the Internet, particularly if done with productive input from a couple of people with experience in the existing system, and some cross-area review.

You may not like that all application developers want to use HTTPS (and derivatives like QUIC), WebSockets, JSON, etc.  However, forcing everyone who wants to standardize a modern protocol to justify their desire to use a set of modern building blocks has gotten tiring, and makes people avoid bringing good technical work to the IETF that would benefit from cross-area review.

Every time we make this sort of work difficult at the IETF, we reinforce the message that the IETF only exists to pay homage to the protocols of our forefathers.  When it comes to email in particular, the system that accreted over the years is baroque at best.  We should not stand in the way of any attempt to simplify the steaming pile of cruft we preside over because the work might be hard.

Several of the concerns that have been raised in this thread are interesting, useful, and might make the work better over time.  Had they not been phrased with an implied "so therefore your whole idea is dumb and you should go away", I probably would have been able to keep myself from writing this note.

TL;DR summary: Just say 'yes'.

— 
Joe Hildebrand

---

From: Ted Lemon <[email-masked]>
Date: Tue, 7 Feb 2017 21:47:39 -0500

> TL;DR summary: Just say 'yes'.

+1

If this winds up having a clean design, it will probably be enough of an improvement to be well worth deploying.   That's what it takes for a protocol to be successful.   I am not yet convinced that it will be a clean design, but if we don't try to design it, it won't matter.

---

From: Dave Crocker <[email-masked]>
Date: Tue, 7 Feb 2017 18:58:25 -0800

On 2/7/2017 5:47 PM, Joe Hildebrand wrote:
> No hats, personal opinion only, and in response to several of the
> messages in this thread, not directly to John.
>
>
> Will it ever be possible for new work to be proposed at the IETF
> without drowning out a chorus of nitpickers?  You may not want to
> help standardize it.  You may not want to implement it.  You may not
> want to deploy it.  You may not want to use it.

Joe,

Sorry, no.

Basic questions about basic utility are not nitpicking.  Basic questions 
about the justifications for basic technical choices are not nitpicking.

And authorizing an IETF working group is not free.  Even the most modest 
IETF working group standards effort had an aggregate cost of around 
US$1M, back when I did a calculation more than 20 years ago, and I would 
love to see someone formulate a claim that it has gone down since then...

So here's an entirely un-novel suggestion:  get the organizations who 
are expecting to implement and deploy this to say so.  From that we can 
get a measure of likely uptake.  Get them also to explain what benefits 
they expect to accrue from this work.

Standards work cannot be a random walk through arbitrary features, no 
matter how intuitively appealing those features might be  There needs to 
be clear and legitimate justifications for each effort and a clear 
indication that there is a market out there wanting the work.

A mild sense that something is benign is not nearly good enough.  Given 
the actual costs of such work.

> This work won't hurt the Internet, particularly if done with

A statement like that ignores issues such as opportunity costs.

d/
-- 

   Dave Crocker

-- 

   Dave Crocker

---

From: Neil Jenkins <[email-masked]>
Date: Wed, 08 Feb 2017 14:15:04 +1100

On Wed, 8 Feb 2017, at 01:58 PM, Dave Crocker wrote:

> are expecting to implement and deploy this to say so.

Sure! So, FastMail[1], Atmail[2], Linagora[3] are all developing
clients and server based around the current draft proposed to the IETF.
There is implementation work happening in both Cyrus[4] and Dovecot[5]
the two largest open-source IMAP servers. So far in this thread we've
had representatives from Apple and Zimbra both expressing positive
interest in JMAP.

The other side of this, which I mentioned earlier, is the many
proprietary protocols that are popping up, invariably HTTP/JSON based,
to replace the need for IMAP/SMTP (submission). Things like Nylas[6],
Context.io[7] and the Gmail API[8].

There are many benefits to the work we're proposing over IMAP/SMTP
submission; the current thread seems to have got a bit hung up on the
configuration bit, but while this is certainly an improvement it's just
one of many[9].

You may think IMAP/SMTP are great solutions to this problem space, but
it would appear industry disagrees.

Neil.

Links:

  1. https://www.fastmail.com/
  2. https://www.atmail.com/
  3. https://www.linagora.com/
  4. https://cyrusimap.org/
  5. https://www.dovecot.org/
  6. https://nylas.com/cloud/docs
  7. Context.iohttps://context.io/docs/lite
  8. https://developers.google.com/gmail/api/v1/reference/
  9. http://jmap.io/#why-is-jmap-better-than-imap?

---

From: Yoav Nir <[email-masked]>
Date: Wed, 8 Feb 2017 09:26:59 +0200

> On 8 Feb 2017, at 5:15, Neil Jenkins <[email-masked]> wrote:
> 
> Sure! So, FastMail <https://www.fastmail.com/>, Atmail <https://www.atmail.com/>, Linagora <https://www.linagora.com/> are all developing clients and server based around the current draft proposed to the IETF. There is implementation work happening in both Cyrus <https://cyrusimap.org/> and Dovecot <https://www.dovecot.org/> the two largest open-source IMAP servers. So far in this thread we've had representatives from Apple and Zimbra both expressing positive interest in JMAP.

That’s good to know. I’m wondering if implementing a generic email client (rather than a specific “gmail” or “live” client) isn’t becoming ever harder. With so many older and newer services, a client has to support pop3, imap, smtp, EWS, ActiveSync, and maybe a few of the others you mentioned. This effort proposes to add yet another one.

Same for a server. A server (even a corporate server, but also public ones like gmail) exposes all kinds of protocols: SMTP, IMAP, POP3 and some proprietary ones. This is yet another one, and it comes with the additional support headache. What do you do if the JMAP clients get disconnected but the IMAP works fine and we can’t get that one POP3 guy on the phone to say if that one is working. And you can’t just “pick one” unless it’s pop3, because of older clients out there.

Not saying that this is not a worthy effort, but it has costs not just for the people doing the actual work.

Yoav

---

From: Dave Cridland <[email-masked]>
Date: Wed, 8 Feb 2017 12:31:02 +0000

On 8 February 2017 at 07:26, Yoav Nir <[email-masked]> wrote:
> That’s good to know. I’m wondering if implementing a generic email client
> (rather than a specific “gmail” or “live” client) isn’t becoming ever
> harder. With so many older and newer services, a client has to support pop3,
> imap, smtp, EWS, ActiveSync, and maybe a few of the others you mentioned.
> This effort proposes to add yet another one.
>

It does; but as someone who has had to implement IMAP from both sides,
I think it's worthwhile to revisit IMAP's space and look at a
green-field rewrite.

JMAP works well particularly for web-based, client-centric
development; that's a space I've seen a lot of, and particularly a
considerable amount of more innovative deployments of existing
protocol work (such as XMPP, which is also very web-friendly).

I would suspect that if we can pull JMAP together and see reasonable
deployment, we'll see an increasing demand for its use for all manner
of things we have yet to think of.

It seems to be well-thought through for mobile, too.

While I agree, with feeling, that writing a generic desktop MUA has
become increasingly hard in recent years, I also note that this isn't
what consumers seem to want either. Consumers respond much better to
Web and/or Mobile - and JMAP makes it considerably simpler to write
either.

TL;DR: Yes.

Dave.

---

From: Ted Lemon <[email-masked]>
Date: Wed, 8 Feb 2017 10:52:44 -0500

> That’s good to know. I’m wondering if implementing a generic email client (rather than a specific “gmail” or “live” client) isn’t becoming ever harder. With so many older and newer services, a clie…

The protocol and database backend parts of implementing a mail client are only difficult in the sense that ActiveSync is not a standard.   IMAP, SMTP and POP protocol implementations are readily available.   The problem is not that they are hard to implement.   It's that they don't work very well.   SMTP works great, don't get me wrong, but IMAP and POP both use the wrong data abstraction, and consequently are very hard to get right, and generally aren't gotten right.

So a new protocol that has the data abstraction right is actually a substantial improvement.   I don't know if that's the case with JMAP—if it's the same data abstraction as IMAP, it's not worth bothering with, but that's something that at least in theory can be discussed.

What JMAP does afford is the opportunity to have an embedded web client that doesn't suck, and a way to escape it if you want something more stateful.   Doing IMAP in a web browser is painful.   I've seen it done.   It wasn't pretty, and the company that did it corkscrewed in and left a crater.   So practically speaking, if you want to do email that way, you are already implementing something like JMAP, but you are doing it by yourself, and it's not standard, so your customers are stuck with what you did.

Even if the data abstraction is wrong in JMAP, you still get a substantial win from that one improvement.

---

From: Pete Resnick <[email-masked]>
Date: Wed, 8 Feb 2017 15:09:30 -0600

On 8 Feb 2017, at 9:52, Ted Lemon wrote:

> IMAP and POP both use the wrong data abstraction, and consequently are 
> very hard to get right, and generally aren't gotten right.
>
> So a new protocol that has the data abstraction right is actually a 
> substantial improvement.   I don't know if that's the case with 
> JMAP—if it's the same data abstraction as IMAP, it's not worth 
> bothering with, but that's something that at least in theory can be 
> discussed.
>
> What JMAP does afford is the opportunity to have an embedded web 
> client that doesn't suck, and a way to escape it if you want something 
> more stateful.

Yep, this is the bit that convinces me that this effort should go 
forward. IMAP's problem has always been its combination of semantic 
bottleneck and leakage: It constrained the semantics that you could 
communicate across the pipe in some ways (hence the myriad of extensions 
required to have a workable IMAP implementation), but leaked a bunch of 
server-specific semantics (e.g., the storage model, the constraints on 
network resources) through to the client instead of having a clean 
protocol. Putting a rich and leak-free protocol in front of IMAP is an 
improvement, and being able to remove IMAP from the equation eventually 
is a laudable goal.

Yes, there are transition and conversion issues. But I haven't heard 
anyone come up with a solution to the problems inherent in IMAP that 
doesn't involve transition pain. The answer can't simply be, "Never 
attempt to solve the IMAP implementation problems" or "Only solve IMAP 
implementation problems within IMAP". Neither one of those answers leads 
to a solution.

(BTW: The submission issue seems a rather boring one to me. There's 
never been good deployment of BURL and the other LEMONADE mechanisms to 
make it happen. If this work just creates a decent protocol interface to 
those submission mechanisms, more power to them. I don't think there's 
anything magical here.)

I have no objection to clarifying the charter in many of the ways 
suggested to make clear what the motivations are and what the 
constraints are on the work, but overall I think this effort should go 
forward.

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
Qualcomm Technologies, Inc. - +1 (858)651-4478