Subject: Re: [Jmap] Review of draft-ietf-jmap-mail-05

From: Neil Jhaveri <[email-masked]>
Date: Mon, 16 Jul 2018 01:37:11 -0700

Thanks for taking the review feedback, and sorry I didn’t get around to replying to this sooner!

> On Jun 8, 2018, at 11:45 PM, Neil Jenkins <[email-masked]> wrote:
> 
> On Thu, 31 May 2018, at 4:25 PM, Neil Jhaveri wrote:
>>> Types signatures are given for all JSON objects in this document…
>> Since this document is presumably dependent upon JMAP Core, can we reference it and avoid repeating the same 4 conventions?  Something like "Type signatures are given for all JSON objects in this document. The conventions follow those in Section 1 of [RFC XXXX]”.
> 
> Sure, sounds reasonable. I'll do that.
> 
>>> 2. Mailboxes
>>> …
>>> Servers *SHOULD* forbid sibling Mailboxes with the same name.
>> Should this be be a MUST? I thought this is expressly forbidden in IMAP - RFC 3501 6.3.3. says "It is an error to attempt to create INBOX or a mailbox with a name that refers to an extant mailbox”.
> 
> For compatibility with IMAP it needs to be a MUST – so yes, I'll change it.
> 
>> The current definition of EmailAddress object doesn’t seem sufficient to cover an RFC 5322 Group Address? e.g.:
>>> To: A Group:Ed Jones <[email-masked]>,[email-masked],John <[email-masked]>;
> 
> The current mapping would be:
> 
> [
>     { "name": "A Group", email: null },
>     { "name": "Ed Jones", email: "[email-masked]" },
>     { "name": "John", email: "[email-masked]" }
> ]
> 
> This gives the group name, but doesn't give the position of the end of the group. I believe the thinking was groups are normally used for the whole set, so the end wasn't important. However, we could add an end marker (name=null and email=null?)
> 
> I would really prefer this stayed a flat list rather than a hierarchical structure though (and remember you are not allowed to have nested groups).
> 
> Thoughts?

I agree with trying to preserve a flat last as a good goal.

What about an optional property “groupName”?

[
  {“name”: “Neil Jhaveri”, email: “[email-masked] <mailto:[email-masked]>”, groupName: “The Neils”},
  {“name”: “Neil Jenkins”, email: “[email-masked] <mailto:[email-masked]>”, groupName: “The Neils”},
  {“name”: “Another Person”, email: “[email-masked] <mailto:[email-masked]>”}
]

This assumes that you don’t have two sequential but different groups with the same name, which I think is legal in RFC 5322 syntax. If that’s important, we could include yet another optional groupIndex property to cover that case (or, a Group property itself having name & index(optional) properties). I lean towards adding the complexity up-front to cover legal syntax.

Thoughts?

> 
>>> 4.1.2. Header fields
>>> …
>>>    Any syntactically correct [RFC2047] encoded sections with a known encoding MUST be decoded, following the same rules as for the _Text_ form...
>> It looks to me like the indentation of this entire section is 1 level too low?  The same issue applies to the comment right after the *MessageIds* property definition. Ditto for *URLs*.
> 
> Yeh, I got the markdown indentation wrong. Will fix.
> 
>>> 4.1.2. Header fields   
>>> ...
>>> In addition, the client may request/send properties representing individual header fields of the form:
>>> 
>>>                         header:{header-field-name}
>>> 
>>> Where "{header-field-name}" means any series of one or more printable ASCII characters (i.e. characters that have values between 33 and 126, inclusive), except colon. 
>> I am not really sure what the conventions are... would this be better expressed with ABNF, or is this textual explanation sufficient?
> 
> I don't know; I'll see what more experienced editors say. My view is as long as it's clear to the average interested person reading the document, it's OK.
> 
>> When I first read this, I was a little confused by the text “including sub parts”, and thought maybe it meant that the tree of parts was flattened into a single array. If others also found this confusing, maybe it could be clarified into something like this:
>>> This is the full MIME structure of the message body, represented as an array of the message’s top-level MIME parts, without recursing into “message/rfc822” or “message/global” parts. Note that EmailBodyParts may have subParts if they are type “multipart/*”.
> 
> Sounds good, will do.
> 
>>>    o  *bodyValues*: "String[*BodyValue*]" (immutable) This is a map of...
>> Should this be EmailBodyValue instead of BodyValue?
> 
> Yes, will fix.
> 
>> 4.1.3. Body parts
>>> ...
>>> MIME structures are arbitrary nested trees of documents, but the majority of email clients present a model of an email body...
>> This entire section is a solid in-depth explanation, but it comes after the property definitions, so it read a little bit confusing to me, and felt like the properties were being re-defined. I think it might be worthwhile considering an edit where the high-level explanation of MIME-tree-flattening comes before the list of Email properties, and then in each property, add more detail as to what each one is, so that it reads a bit more linearly and each property definition is a bit more self-contained. Thoughts?
> 
> Yes, I think it needs to be moved to the very top of the Email description, providing a non-normative overview of the structure before the actual specification. I'll do this.
> 
>> I personally think it may be worthwhile considering consolidating back down to a single attachedFiles property rather than having to have two properties and merge them. Thoughts?
> 
> Yes, I agree. Looking at it again, this is simpler to understand and gives you the full ordering. I'll do this (and name the property just "attachments").
> 
>>> 4.2. Email/get
>>> …
>>>    The following properties are expected to be fast to fetch in a quality implementation:
>> Resent-XX are commonly shown by 
> 
> … by what?! And importantly, are they commonly shown in the mailbox listing or when you open the message (because the latter will be fetched with the body so doesn't have to be fast)?

Oops, I meant to delete that comment :), hence it being unfinished. I don’t know of any clients that show it in the message list, its generally shown along with the message body.

> 
>>> 4.7. Email/import
>>> ...
>>> The server MAY forbid two email objects with the same exact [RFC5322] content, or even just with the same [RFC5322] Message-ID, to coexist within an account.  In this case, it MUST reject attempts to import an email considered a duplicate with an "alreadyExists" SetError.  
>> Allowing a rejection purely based on a Message-ID match seems a little bit restrictive to me — it’s author-generated, so I have seen the global-uniqueness requirement violated by naively written scripts. 
> 
> This text is there because I believe Gmail does this (but please correct me if I'm wrong; maybe it's only on delivery, not append?). It's a MAY, so it's up to the server to define what it wants to do, but the spec needs to be compatible as much as possible with existing systems.

I see. Yeah, Gmail does require unique Message-IDs — at least on append, they are dropped, even if the content is different. This is fine, then.

> 
>>> 5. Email Submission
>>> …
>>>    o  *identityId*: "String" (immutable) The id of the identity to associate with this submission.
>> It’s a little bit confusing that Identity is used here before it’s defined in section 6. Maybe a slight note? 
> 
> I think I'll just reorder it so the Identity section comes before EmailSubmission. That should read better.
> 
>>> 5. Email Submission
>>>   *  *mailFrom*: The email in the _Sender_ header, if present, otherwise the _From_ header, if present, and no parameters.  If multiple addresses are present in one of these headers, or there is more than one _Sender_/_From_ header, the server SHOULD reject the email as invalid but otherwise MUST take the first address in the last _Sender_/_From_ header in the [RFC5322] version of the message.  *If the address found from this is not allowed by the identity associated with this submission, the _email_ property from the identity MUST be used instead*.
>> Instead of silently swapping the user’s requested _From_ to the identity address, I'd prefer returning a "forbiddenFrom” error and give the user a chance to correct their mistake. Thoughts?
> 
> This is the RFC5321 SMTP envelope From remember, not the RFC5322 message From. This passage of spec is also only referring to what to do when the client doesn't explicitly set an envelope. I think therefore this passage is OK, but you raise a good point that we probably need a separate error for a forbidden RFC5321 From in the envelope. I'll add:
> 
> - `forbiddenMailFrom` – The server does not permit the user to send an email
>   with the [@!RFC5321] envelope From.
> 
> which is in addition to:
> 
> - `forbiddenFrom` – The server does not permit the user to send an email
>   with the [@!RFC5322] From header of the email to be sent.

Ah, right! Agreed.

> 
>>> 5. Email Submission
>>> … 
>>>          For emails relayed via an alternative to SMTP, the server MAY
>>>          generate a synthetic string representing the status instead.
>>>          If it does this, the string MUST be of the following form:
>>> 
>>>          +  A 3-digit SMTP reply code, as defined in [
>>> RFC5321], section
>>>          +  Then an SMTP Enhanced Mail System Status Code as defined in [RFC3463], with a registry defined in [RFC5248].
>>>          +  Then a single space character.
>>>          +  Then an implementation-specific information string with a human readable explanation of the response.
>> Not very familiar with the conventions here, but would ABNF be more appropriate to specify this?
> 
> Again, happy to take advice from more experienced editors here.
> 
>>> 5.5 EmailSubmission/set
>>>    Standard _/set_ method, with the following two extra arguments:
>>>    o  *onSuccessUpdateEmail*:...
>> I think a lot of clients will want to move the message from Draft to Sent using this block. It might be worth explicitly calling that out, or even giving a full example.  Thoughts?
> 
> Yes, that's exactly what it's for. I'll add an example.
> 
>>> 7. Search Snippets
>>> …
>>>    o  *attachments*: "String|null" If text from the filter matches the text extracted from an attachment, this is the relevant section of the attachment (converted to plain text), with matching words/phrases wrapped in "<mark></mark>" tags.  It MUST NOT be bigger than 255 octets in size.  If it does not match, this is "null".
>> I could imagine a MUA wanting to present to the user exactly which attachment the search string was found in, so it might be worth including an attachment ID property. Thoughts? 
> 
> That does sound like a useful feature to me. It will probably have to be optional, any suggestions on the best way to represent this? Could make the property a map of partId to matching section, but then what to do if the server doesn't know which part had the match?

Yeah, I was thinking a map of partId to matching section, too. But if that mapping is worrisome for a server implementor, perhaps representing it as an array of Attachment snippet objects, with snippet and partId properties. 

But now the more I think about it, rather than making it optional, maybe this is just over-complicated and the way you had it is best for now.

> 
>>> 8. Vacation Response
>>> The *VacationResponse* object represents the state of vacation-response related settings for an account.  It has the following properties:
>> I’m not too knowledgeable about prior work here, but:
>> 1. Exchange Server offers the ability to configure a separate “Internal" and “External" vacation response — which is useful if you work in a large organization and want to have your internal reply include things like “you can contact my manager at XXX”, but you want your external client-facing response to expose less information.
>> 2. I also think that there may need to be some mention of loop prevention — so that two accounts with vacation responses don’t continuously reply to each other. 
> 
> The spec here was not intended to provide a complete consideration of everything that should be considered when building a vacation response system (which is already covered in previous RFCs), merely a mechanism by which it can be configured by the client. The functionality was chosen as a common subset supported by most major systems, to make adoption feasible. Individual systems (including Exchange, Gmail, FastMail etc.) will have extended options for vacation responses, but then which options do you support in the spec, and how do you make it so other systems can be compliant? Lots of optional features are a pain for everyone. The real question is whether being able to configure the common subset from arbitrary 3rd party clients is useful or not?
> 
>> I am starting to think this should be deferred to a JMAP extension, so that prior work in this area can be more carefully considered (such as RFC 5230). Thoughts?
> 
> So, I agree the vacation response could be neatly separated into a separate specification rather than part of the core mail. The main reason to include it here is to make sure there is concrete extra functionality that an existing IMAP client implementing JMAP would be able to provide that it couldn't with just IMAP. This (sadly) can make it an easier sell to management than just "it will make everything work faster, more efficiently and more reliably than before". However, I'm very interested to hear if you don't think this is likely to be so useful a hook based on your previous employment experience.

Neil & I chatted a bit off-list on this. At least in my experience, I don’t think it will make a material difference in terms of being able to sell it to management — performance & responsiveness tend to be important for mail clients in general, and I certainly spent a lot of my team’s time at Apple on that. I still lean towards separating this into an extension so that it can be made a little bit more sophisticated.

> 
> Neil._______________________________________________
> Jmap mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/jmap

---

From: Neil Jenkins <[email-masked]>
Date: Mon, 16 Jul 2018 05:56:29 -0400

On Mon, 16 Jul 2018, at 6:37 PM, Neil Jhaveri wrote:
>   {“name”: “Neil Jhaveri”, email: “[email-masked]”, groupName: “The Neils”},
>   {“name”: “Neil Jenkins”, email: “[email-masked]”, groupName: “The Neils”},
>   {“name”: “Another Person”, email: “[email-masked]”}
> ]

The main trouble with this is that the most common use of the group name is for something like this:

To: undisclosed-recipients:;

There are no mailboxes here, so there's nothing to add a groupName property too. I think therefore there have to be start/end group objects.

Neil.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 16 Jul 2018 12:14:47 +0200

On Monday 16 July 2018 11:56:29 CEST, Neil Jenkins wrote:
> There are no mailboxes here, so there's nothing to add a 
> groupName property too. I think therefore there have to be 
> start/end group objects.

The problem with that is that it matches the RFC but not the modern world. 
In the modern world, only the example you show is common enough to be seen 
if you test with, say, 10000 messages random messages from the past years.

Syntax that won't be tested is a trouble magnet.

Here's a strawman alternative:

1. Empty groups are handled as a special kind of address. JMAP's 
EmailAddress is extended with an "unreplyable" element, and either "email" 
or "unreplyable" MUST be present, but never both. The latter is 1*CHAR if 
set. The server SHOULD specify an address as "email" if it's possible to 
send to that address, and MUST specify it as "unreplyable" if not.

2. Nonempty groups are handled by adding each address and suppressing the 
group name.

3. The test suite gets a message for each case.

4. Add "someone shows current use of nonempty groups" as a requirement for 
revisiting.

Arnt

---

From: Neil Jenkins <[email-masked]>
Date: Mon, 16 Jul 2018 06:18:56 -0400

> if you test with, say, 10000 messages random messages from the past years.

I agree, which is why the original spec had an object for the start of the group (name=group name; email=null), but nothing for the end of the group—because if it's empty and there are no other addresses (the 99.9% case) it's obvious, and even if there *are* other addresses, probably you still don't really care where the end of the group is.

I'd be happy to go back to this if there's general consensus that the end of the group is not important in the real world.

Neil.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 16 Jul 2018 12:33:25 +0200

On Monday 16 July 2018 12:18:56 CEST, Neil Jenkins wrote:
> I'd be happy to go back to this if there's general consensus 
> that the end of the group is not important in the real world.

I feel pedantic today. I seem to have grown less pedantic over the years, 
but this morning I'm young again.

The JMAP core RFC ought to either say "JMAP explicitly only targets 
singlepart messages and MIME messages" and then questions such as about 
nonempty  groups become irrelevant.

Or it ought to say that e.g. 20-30 year old archived mail is explicitly 
in-scope, in which case someone should go and find some old cruft to test, 
such that it can be specified based on actual testing. Uuencoded 
attachments, undeclared 8-bit in different charsets (.no used three at the 
time), nonempty groups, whatever other cruft that was common once and died 
out 15-20 years ago. Supporting one kind of cruft and not even testing 
enough to enumerate the other kinds would be doubleplusungood.

Arnt

---

From: JFCM <[email-masked]>
Date: Mon, 16 Jul 2018 12:49:17 +0200

At 12:33 16/07/2018, Arnt Gulbrandsen wrote:
>Or it ought to say that e.g. 20-30 year old archived mail is 
>explicitly in-scope,

Amen ++
jefsey

>  in which case someone should go and find some old cruft to test, 
> attachments, undeclared 8-bit in different charsets (.no used three 
> at the time), nonempty groups, whatever other cruft that was common 
> once and died out 15-20 years ago. Supporting one kind of cruft and 
> not even testing enough to enumerate the other kinds would be doubleplusungood.
>
>Arnt
>
>_______________________________________________
>Jmap mailing list
>[email-masked]

---

From: Robert Mueller <[email-masked]>
Date: Mon, 16 Jul 2018 21:08:58 +1000

> I agree, which is why the original spec had an object for the start of
> the group (name=group name; email=null), but nothing for the end of
> the group—because if it's empty and there are no other addresses (the
> 99.9% case) it's obvious, and even if there *are* other addresses,
> probably you still don't really care where the end of the group is.> 
> I'd be happy to go back to this if there's general consensus that the
> end of the group is not important in the real world.
I'd prefer not to end up in a case where you can't represent real world
data accurately as defined by the latest email RFC spec!
I think you should just follow similar to the IMAP RFC.

         [RFC-2822] group syntax is indicated by a special form of
         address structure in which the host name field is NIL.  If the
         mailbox name field is also NIL, this is an end of group marker
         (semi-colon in RFC 822[1] syntax).  If the mailbox name field
         is non-NIL, this is a start of group marker, and the mailbox
         name field holds the group name phrase.

Yeah well, IMAP splits out mailbox name and hostname, so we can't quite
use that, but I think having a null email address to mean "start of
group" and null email address and null name to mean "end of group" is
fairly close to IMAP (so it's historically understandable to email
client authors), and also should be reasonably straightforward for email
server writers to implement (I believe cyrus does it already, because it
only has to slightly tweak the existing IMAP envelope data).
Rob Mueller
[email-masked]

Links:

  1. https://tools.ietf.org/html/rfc822

---

From: Bron Gondwana <[email-masked]>
Date: Mon, 16 Jul 2018 21:59:43 +1000

On Mon, Jul 16, 2018, at 21:08, Robert Mueller wrote:
> Yeah well, IMAP splits out mailbox name and hostname, so we can't
> quite use that, but I think having a null email address to mean
> "start of group" and null email address and null name to mean "end of
> group" is fairly close to IMAP (so it's historically understandable
> to email client authors), and also should be reasonably
> straightforward for email server writers to implement (I believe
> cyrus does it already, because it only has to slightly tweak the
> existing IMAP envelope data).
It was also quite trivial to add to the Perl proxy, which has the native
parsed format as a list of structures, each of which contains a group
name (may be undefined) and a list of addresses within that group.
    if (defined $group) {
      push @res, {
        name => asText($group),
        email => undef,
      };
    }
    foreach my $addr (@$list) {
      my $name = $addr->phrase();
      my $email = $addr->address();
      push @res, {
        name => asText($name),
        email => $email,
      };
    }
    if (defined $group) {
      push @res, {
        name => undef,
        email => undef,
      };
    }

--
  Bron Gondwana, CEO, FastMail Pty Ltd
  [email-masked]

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 16 Jul 2018 14:08:50 +0200

On Monday 16 July 2018 13:08:58 CEST, Robert Mueller wrote:
> I'd prefer not to end up in a case where you can't represent 
> real world data accurately as defined by the latest email RFC 
> spec!

Are there any real world senders that send real world mail to real world 
recipients and expect nonempty groups to be displayed with correct 
membership?

If yes, then the test suite needs some examples of that.

If no, then I suggest that group membership is not required for accurate 
representation, and ought to be discarded. Storing it on spinning rust is 
okay, serving that detail via the API puts an unnecessary burden on client 
developers. JMAP discards many other details, e.g. the server does RFC 2047 
decoding and discards the charset defined in the encoded-word.

BTW. If someone would like to do half the work, I'd be willing to trawl 
github until I find ten python/ruby IMAP clients I've never heard of 
before, so we can assess how buggy or bug-free the support for nonempty 
groups actually is. (I name those languages because the search should be 
simple and I can't think of any reason why their support should be 
better/worse than in other languages.)

Arnt

---

From: Ned Freed <[email-masked]>
Date: Mon, 16 Jul 2018 07:10:11 -0700 (PDT)

> recipients and expect nonempty groups to be displayed with correct

Given how much siloed usage of email there is in the world, do you think the
tiny membership of this group is in a position to make this determination?

I don't. I've lost track of how many times I've been surprised by the use
cases customers present to our support people.

> better/worse than in other languages.)

python/ruby are so last year. Why are we worrying about stuff written in those
obsolete languages when the world is moving to node.js?

(In case it isn't obvious, that was sarcasm.)

Anyway, it took me about two minutes to find an node.js email address parser
that supports nonempty groups as first class objects:

  https://github.com/emailjs/emailjs-addressparser

The package's unit tests include quite a few group constructs, some of which
are actually pretty nasty. (The author has some ideas about how to handle
invalid syntax that make me wonder what is generating some of the email address
lists this package has to hande, but valid syntax handling appears to me to be
correct.)

In fact this was the first node.js email address parser I found. It looks like
it's used by a fair amount of other node-ware, including the associated IMAP
client:

  https://github.com/emailjs/emailjs-imap-client

The IMAP client was last updated five hours ago. The address parser was last
updated 3 days ago, and has 9 forks. So it looks like this code is actively
used and maintained.

I trust I don't need to explain how the node.js ecosystem works and how
packages like this are picked up and used.

				Ned

---

From: Bron Gondwana <[email-masked]>
Date: Tue, 17 Jul 2018 01:07:21 +1000

On Tue, Jul 17, 2018, at 00:10, Ned Freed wrote:
> python/ruby are so last year. Why are we worrying about stuff written
> in those> obsolete languages when the world is moving to node.js?

The better Perl libraries certainly support it :)  For some degree of
"better".  There's plenty of worser Perl libraries.
So if we are supporting Group syntax, are we happy with the
presentation:
{
  name: "GroupName",
  email: null,
},
{
  name: "Member",
  email: "[email-masked]",
},
{ ... },
{
  name: null,
  email: null,
}

It's certainly easy enough to generate on the server.

Bron.

---

From: Neil Jenkins <[email-masked]>
Date: Mon, 16 Jul 2018 14:37:53 -0400

I'm happy with the null/null representation, although I agree it's a hack. The main alternative in my mind is something like this:

{

name: "GroupName",

group: "start",

},

{

name: "Member",

},

{ ... },

{

group: "end"

}

This is more explicit. The main disadvantage is that you have an array of objects that are different shapes then, which might be a pain for some statically typed languages to deal with.

Neil.

---

From: Neil Jhaveri <[email-masked]>
Date: Mon, 16 Jul 2018 12:03:32 -0700

> On Jul 16, 2018, at 2:56 AM, Neil Jenkins <[email-masked]> wrote:
> 

What about  {"name": null, "email": null, "groupName": "undisclosed-recipients" to handle this case? 

A normal address:
{"name": "Neil Jhaveri", email: "[email-masked] <mailto:[email-masked]>"}

An empty group:
{"name": null, "email": null, "groupName": "undisclosed-recipients"}

A nonempty group:
{"name": "Group Member1", email: "[email-masked] <mailto:[email-masked]>", "groupName": "Group"} 

(The only downside I see to this is if there are two sequential groups with identical names but different membership.)

---

From: Neil Jenkins <[email-masked]>
Date: Mon, 16 Jul 2018 15:05:28 -0400

> What about  {"name": null, "email": null, "groupName": "undisclosed-recipients" to handle this case? 

I don't like this, because the non-empty group case is different to the empty group case (which is the only time you would get a null email!). More cases to test, more chance of implementations messing up.

Neil.