Subject: [Extra] Email header / address parsing

From: Timo Sirainen <[email-masked]>
Date: Tue, 1 Sep 2020 09:53:09 +0200

Hi,

I was reading https://www.usenix.org/system/files/sec20-chen-jianjun.pdf and started wondering if IMAP should be handling some of this better. Especially for generating ENVELOPE. We could even still have time to add recommendations to IMAP4rev2?

For example:

1) Errors while parsing the address syntax: I was wondering if it would be a good idea to indicate this by appending .INVALID suffix to domains of all of the addresses in the ENVELOPE reply? That way it would be at least clear that something is wrong. Otherwise different implementations could always do something slightly different when encountering errors, leading to trust problems. For example could be done with these kind of addresses:

 - From: [email-masked] <[email-masked]>
   --> either: [email-masked]
   --> or: [email-masked]
 - From: <[email-masked], <[email-masked]>
   --> either: [email-masked]
   --> or: [email-masked]
   --> or: [email-masked], [email-masked]

2a) Space preceding the first header name
2b) Space after From header: Again
2c) Folding space before ":"
Not sure what exactly to do about all these. Don't parse these as valid headers? Or parse but append .INVALID for ENVELOPE? But that would work only for ENVELOPE headers.

3) Multiple From lines: Add all of these to ENVELOPE, as if they were all in a single From line.

---

From: "John Levine" <[email-masked]>
Date: 1 Sep 2020 13:41:51 -0400

In article <[email-masked]> you write:
>Hi,
>
>I was reading https://www.usenix.org/system/files/sec20-chen-jianjun.pdf and started wondering if IMAP should be
>handling some of this better. Especially for generating ENVELOPE. We could even still have time to add
>recommendations to IMAP4rev2?
>
>For example:

But the former is completely valid and occasionally useful, and the
latter is a syntax error. Don't we already have a "don't do that" rule
for invalid syntax?

>2a) Space preceding the first header name

That's a continuation line.

>2b) Space after From header: Again

Not sure what you mean but that's probably valid.

>2c) Folding space before ":"

Not valid but I don't think I've ever seen it.

Over in DMARC land people have been arguing for years about what to do
about misleading display names, and getting nowhere. 

While I certainly believe that it's possible to do spam filtering
looking for patterns that are likely to be phishes, I really don't
think that the IMAP address parser is the place to do it.

R's,
John

---

From: Timo Sirainen <[email-masked]>
Date: Wed, 2 Sep 2020 00:15:01 +0200

On 1. Sep 2020, at 19.41, John Levine <[email-masked]> wrote:
> latter is a syntax error.

My understanding is that the first is supposed to be:

   name-addr       =   [display-name] angle-addr
   display-name    =   phrase
   phrase          =   1*word / obs-phrase
   word            =   atom / quoted-string

And '@' is not in atom. So it's not a valid address.

> Don't we already have a "don't do that" rule
> for invalid syntax?

Sure, but that's causing security problems nowadays. Would be nice to try to prevent those.

> That's a continuation line.

Not when it's the first header. There is nothing to continue.

> Not sure what you mean but that's probably valid.

Sorry, messed up writing this. I mean when the header is "From : [email-masked] <mailto:[email-masked]>" - some parsers treat it as "From" header and some as "From " with a space.

> about misleading display names, and getting nowhere. 

These aren't about display-names though, which the client could handle how it wishes, but the actual email addresses visible to client via ENVELOPE.

> think that the IMAP address parser is the place to do it.

But IMAP server is the one parsing the addresses into ENVELOPE, which the MUAs can use. We have the same problem that mail could look like it's fully DMARC-validated (and/or whatever other validation method is being used), but due to different parsers the From address could end up looking like it's fully valid to client even though it's not what was actually validated due to parsing differences. The only way for client to avoid this issue now would be to not use ENVELOPE at all and parse the headers itself.

---

From: "John R Levine" <[email-masked]>
Date: 1 Sep 2020 18:35:09 -0400

> And '@' is not in atom. So it's not a valid address.

Oh, right, it has to be quoted.
>> Don't we already have a "don't do that" rule for invalid syntax?
>

I meant that if the header is invalid, don't try to parse it into an 
envelope entry.  That seems conservative and safe.  There are certainly 
parsers that will accept invalid syntax, which is where don't do that 
applies.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Timo Sirainen <[email-masked]>
Date: Wed, 2 Sep 2020 02:52:00 +0200

On 2. Sep 2020, at 0.35, John R Levine <[email-masked]> wrote:
> I meant that if the header is invalid, don't try to parse it into an envelope entry.  That seems conservative and safe.  There are certainly parsers that will accept invalid syntax, which is where don't do that applies.

It does seem a bit harsh though. There are real emails that attempted to be correct, but failed due to some bugs. Especially bad are the emails where there are multiple addresses and some of the comma-separated addresses are invalid and some of them are correct. Should ENVELOPE then show the entire header as if it didn't exist, without any clue that it was invalid? I was thinking .INVALID or some such similar way of marking them might be more nicer.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 2 Sep 2020 11:04:30 +0200

We've discussed this sort of this before, and I don't see what's different 
this time. If the server can parse it well enough to present the correct 
domain with ".invalid" appended, why not present the domain without 
.invalid? If the server cannot parse it that well, why even try to include 
that address in the list?

IMO, the best way to handle this is to say: "If a header contains syntax 
errors, whether major or minor, the server MAY present a best-effort 
ENVELOPE or BODY. This response may be be either incomplete (the server 
drops the addresses that could not be parsed and presents as many as it can 
of the others) or unreliable (if an address is invalid, there may not be a 
single obvious way to render it valid)."

This is not a compatibility break, doesn't add new requirements, does 
suggest a way to handle some syntax errors, and it doesn't expose clients 
to new expectations: Four out of four, say I.

Arnt