Subject: [Extra] IMAP RFC 6855 (IMAP EAI) implementation experience: input to IMAP4rev2

From: "Chris Newman" <[email-masked]>
Date: Tue, 20 Nov 2018 15:28:02 -0800

Sometimes the philosophy behind a standard and what happens when it 
encounters the real-world result in implementations that willfully 
ignore some aspects of the standard. Since I wrote the first draft of 
RFC 6855, any such differences between the spec and real life are mostly 
my fault.

For better or worse, RFC 6530 took a very different approach from past 
IETF specifications when it comes to backwards compatibility. It choose 
not to introduce yet-another-ascii-encoding for email local parts and 
takes a "MUST bounce" position when a UTF-8 message encounters a 
5321/5322-only system. From a standards-purist viewpoint, this preserves 
interoperability but is a deployment deterrent to the new format. 
However, from an implementer viewpoint, I believe this is mostly a 
non-starter. Although we documented downgrading options (RFC 6857,6858) 
these options end up having poor cost/benefit in the real world. Ned's 
implementation experience from RFC 6531 was that the real world just 
can't usefully work the way the specification describes. That experience 
informed my work implementing RFC 6855.

My experience with RFC 6855 implementation is as follows:

1. Support to accept and validate UTF-8 in quoted-strings was 
straightforward to implement. I made the support unconditional as that 
was much simpler than adding ENABLE-conditional code branches that would 
require extra testing.

2. Support to append UTF-8 messages was medium complexity to implement 
because I decided to validate the header UTF-8 syntax and push any 
garbage-8-bit headers into the message body. I ended up just ignoring 
the <"UTF8" SP> protocol syntax when present and processing the message 
the same way regardless.

3. I choose not to implement downgrading or UTF-8-message hiding. If 
UTF-8 messages become common and legacy clients start choking on them, I 
might add a downgrading option, but I suspect it will be much easier for 
clients to just deal with UTF-8 where it wasn't previously allowed than 
to do anything useful with a downgraded UTF-8 email message or even 
fully implement RFC 6855 (which I suspect is far simpler for clients 
than for servers).

4. With my implementer hat on, I willfully ignored this requirement:
   The IMAP server MUST NOT send UTF-8 in quoted-strings to the client 
unless the client
   has indicated support for that syntax by using the "ENABLE 
UTF8=ACCEPT" command.
It's interesting that my implementer hat and standards hat result in 
diametrically opposed positions on this issue, but so be it. If there's 
a message with UTF-8 headers in the message store, IMAP clients will see 
it uncorrupted regardless of whether the client can handle it.

5. The requirement to eliminate modified-UTF-7 in mailbox names was 
medium complexity due to some past implementation choices. I was 
comfortable with the cost/benefit of the effort. MUTF-7 is ugly; good 
riddance. This code has to be conditional based on use of the ENABLE 
command -- however, if the implementation sees a client-provided UTF-8 
mailbox name, it treats that one name as if ENABLE had been set. It's 
only 7-bit mailboxes containing ampersand that have a behavior 
difference based on whether ENABLE was used.

6. The most difficult part of the project is supporting UTF-8 usernames 
throughout the server. IDN is actually the hardest part of this since 
it's necessary to support both A-labels and U-labels for the domain 
portion of a user name and treat them as synonyms. Luckily RFC 6855 does 
not mandate support for UTF-8 usernames so this doesn't impact the 
previous decision of whether to include RFC 6855 in IMAP4rev2. The 
U-label/A-label issue is also a problem for SMTP servers, milters, and 
Sieve.

Based on 1-5, I believe the text in the current IMAP4rev2 draft is 
correct, appropriate, and implementable.

Based on 6, we might want to consider some additional text with respect 
to UTF-8 usernames & passwords.

		- Chris

---

From: "Jiankang Yao" <[email-masked]>
Date: Wed, 21 Nov 2018 20:51:25 +0800

From: Chris Newman
Date: 2018-11-21 07:28
To: extra
Subject: [Extra] IMAP RFC 6855 (IMAP EAI) implementation experience: input to IMAP4rev2
> Ned's 

ICANN UASG EAI WG is pushing the deployment of EAI. John Levine has run an analysis of the EAI Readiness of the Internet’s mail servers.
His DRAFT report is at https://docs.google.com/document/d/18W-dAr1Nhe_gVlFKDQZqsqB4bf5N86wryEcALnyUPTc/edit#heading=h.dba7p1kazxqa
From this report, the current EAI deployment  is amazing.

You may also be interested in this list discussion here https://mm.icann.org/mailman/listinfo/ua-eai 

Best Regards
Jiankang Yao

---

From: Ned Freed <[email-masked]>
Date: Thu, 22 Nov 2018 07:00:55 -0800 (PST)

> ICANN UASG EAI WG is pushing the deployment of EAI. John Levine has run an
> analysis of the EAI Readiness of the Internet’s mail servers.

> From this report, the current EAI deployment  is amazing.

I would use a very different term to describe the degree of what the report
calls deployment, but that's really beside the point.

The fact is that since these numbers count services like gmail as a successful
deployment of EAI, they are irrelevant to point Chris is making.

Why? Because gmail doesn't implement EAI acccording to the standards - and
Google makes no secret of this. The last I checked gmail will happily send an
EAI message to server that doesn't support EAI - which is the only sensible
thing to do, gmail doesn't implement the user agent side of EAI at all, and
most importantly, gmail only supports other people's EAI addresses; not its
own.

Indeed, the fact that Google saw fit to "implement" EAI this way only serves
to support the point Chris was making: EAI as specified has some serious
issues, and sooner or later the IETF is gong to have to deal with this.

> You may also be interested in this list discussion here
> https://mm.icann.org/mailman/listinfo/ua-eai

It's a private list; the archives of which are not public.

				Ned

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 30 Nov 2018 13:00:05 +0100

Chris Newman wrote:
> It's interesting that my implementer hat and standards hat 
> result in diametrically opposed positions on this issue, but so 
> be it.

I think this is because RFCs are obliged to respect other RFCs more than 
implementers are. You, as implementer, may reasonably look at what hurts 
most in practice in your part of the world; the set of RFCs ought to be 
consistent if at all possible.

BTW, this is why RFC 6858 exists in the first place. Enabling reading using 
6857 is IMO too much work compared to just-send-8, which is IMO tantamount 
to encouraging violating 3501's rules. The simplest possible conformant 
implementation shouldn't be hard.

You should find 6858 easy to implement well enough that mail can be read 
without violating any rule in 3501. If not, I failed. Almost as an aside, 
doing so should minimise the risk of provoking a bug related to sending 
mail, by making replying and composing fail in a safe/predictable manner. 
But not implementing it is IMO also entirely reasonable, even if an RFC 
should not say so.

Arnt