Subject: [Jmap] New JSON message representation

From: Neil Jenkins <[email-masked]>
Date: Thu, 08 Feb 2018 17:11:24 +1100

With an eye to closing off all remaining issues before the next IETF
meeting in March, I have been looking at the issues raised with the JSON
representation of an RFC5322 message. This email presents a new proposal
that I think resolves the issues presented. In particular, I aim to
address the following goals:
 * Allow access to the full MIME structure without requiring the client
   to parse the full message itself.
 * Allow access to full header field information if required.
 * Keep a simpler representation available for the common case.
 * Allow the client to only load the data required.Comments, suggestions and other feedback requested. Cheers!

------
Emails

The *Email* object is a representation of an [@!RFC5322] message, which
allows clients to avoid the complexities of MIME parsing, transport
encoding and character encoding.Properties of the Email object

Broadly, a message consists of two parts: a list of header fields, then
a body. The body is normally a MIME-encoded set of documents in a tree
structure. The JMAP Email object provides a way to access the full
structure, or to use simplified properties and avoid some complexity if
this is sufficient for the client application.Due to the number of properties involved, the set of *Email* properties
is specified over the following three sub-sections.Metadata

These properties represent metadata about the [@!RFC5322] message, and
are not derived from parsing the message itself.

 * *id*: String (immutable; server-set) The id of the Email object.
   Note, this is the JMAP object id, NOT the [@!RFC5322] Message-Id
   header field value.

 * *blobId*: String (immutable; server-set) The id representing the raw
   bytes of the [@!RFC5322] message. This may be used to download the
   raw original message, or to attach it directly to another Email etc.

 * *threadId*: String (immutable; server-set) The id of the Thread to
   which this Email belongs.

 * *mailboxIds*: String[Boolean] The set of mailbox ids this email
   belongs to. An email MUST belong to one or more mailboxes at all
   times (until it is deleted). The set is represented as an object,
   with each key being a *Mailbox id*. The value for each key in the
   object MUST be true.

 * *keywords*: String[Boolean] (default: {}) A set of keywords that
   apply to the email. The set is represented as an object, with the
   keys being the *keywords*. The value for each key in the object
   MUST be true.

Keywords are shared with IMAP. The six system keywords from IMAP are
treated specially. The following four keywords have their first
character changed from \ in IMAP to $ in JMAP and have particular
semantic meaning:

   * $Draft: The email is a draft the user is composing.
   * $Seen: The email has been read.
   * $Flagged: The email has been flagged for urgent/special attention.
   * $Answered: The email has been replied to. The IMAP \Recent keyword
     is not exposed via JMAP. The IMAP \Deleted keyword is also not
     present: IMAP uses a delete+expunge model, which JMAP does not. Any
     message with the \Deleted keyword MUST NOT be visible via JMAP.

Users may add arbitrary keywords to an email. For compatibility with
IMAP, a keyword is a (case-sensitive) string of 1–255 characters in the
ASCII subset %x21–%x7e (excludes control chars and space), and MUST NOT
include any of these characters: ( ) { ] % * " \

The IANA Keyword Registry[1] as established in [@!RFC5788] assigns
semantic meaning to some other keywords in common use. New keywords may
be established here in the future. In particular, note:

   * $Forwarded: The email has been forwarded.
   * $Phishing: The email is highly likely to be phishing. Clients
     SHOULD warn users to take care when viewing this email and disable
     links and attachments.
   * $Junk: The email is definitely spam. Clients SHOULD set this
     flag when users report spam to help train automated spam-
     detection systems.
   * $NotJunk: The email is definitely not spam. Clients SHOULD set this
     flag when users indicate an email is legitimate, to help train
     automated spam-detection systems.
 * *size*: Number (immutable; server-set) The size in bytes of the whole
   email as counted by the server towards the user's quota.

 * *receivedAt*: UTCDate (immutable; default: time of creation on
   server) The date the email was received by the message store. This is
   the *internal date* in IMAP.Header fields

These properties are derived from the [@!RFC5322] message header fields.
Header fields may be fetched in a variety of forms:

 * *Decoded* The header field value with:

   1. White space unfolded (as defined in [@!RFC5322] section 2.2.3)
   2. The terminating CRLF at the end of the value removed
   3. Any SP characters at the beginning of the value removed
   4. Any [@!RFC2047] encoded sections decoded
   5. Any [@!RFC6532] UTF-8 values decoded If any decodings fail, the
      parser SHOULD insert a unicode replacement character (U+FFFD) and
      attempt to continue as much as possible.

 * *Raw Base64* The raw bytes of the header field value from the first
   byte following the header field name terminating colon, up to but
   excluding the header field terminating CRLF, encoded as Base64
   ([@!RFC4648]). (Note, this is required as the value may not be valid
   UTF-8 if 8-bit values are present.)

 * *Addresses* The header is parsed as an address-list value, as
   specified in [@!RFC5322] section 3.4, into the EmailAddress[] type.
   The *EmailAddress* object has the following properties:

   * *name*: String|null The *display name* of the [@!RFC5322] *mailbox*
     or *group*, or null if none.
   * *email*: String|null The *addr-spec* of the [@!RFC5322] *mailbox*,
     or null if a *group*. Parsing SHOULD be best-effort in the face of
     invalid structure to accommodate invalid messages and semi-complete
     drafts. EmailAddress objects MAY have an *email* property that does
     not conform to the *addr-spec* form (for example, may not contain
     an @ symbol).

 * *Date* The header is parsed as a date-time value, as specified in
   [@!RFC5322] section 3.3, into the Date type.The following *Email* properties are specified for complete access to
the header data of the message. Where {header-field-name} is specified
within a property name, this means any series of one or more printable
US-ASCII characters (i.e. characters that have values between 33 and
126, inclusive), except colon:

 * *headers*: EmailHeader[] (immutable) This is a list of all
   [@!RFC5322] header fields, in the same order they appear in the
   message. The values are in *Decoded* form.
 * *headers:asRaw*: EmailHeader[] (immutable) This is a list of all
   [@!RFC5322] header fields, in the same order they appear in the
   message. The values are in *Raw Base64* form.
 * *header:{header-field-name}*: String|null (immutable) The value of
   the *last* header field in the message with the given header field
   name (matched case-insensitively), or null if none. The value is in
   *Decoded* form.
 * *header:{header-field-name}:asRaw*: String|null (immutable) The value
   of the *last* header field in the message with the given header field
   name (matched case-insensitively), or null if none. The value is in
   *Raw Base64* form.
 * *header:{header-field-name}:asAddresses*: EmailAddress[]|null
   (immutable) The value of the *last* header field in the message with
   the given header field name (matched case-insensitively), or null if
   none. The value is in *Addresses* form.
 * *header:{header-field-name}:asDate*: Date|null (immutable) The value
   of the *last* header field in the message with the given header field
   name (matched case-insensitively), or null if none. The value is in
   *Date* form.An *EmailHeader* object has the following properties:

 * *name*: String The header *field name* as defined in RFC5322, with
   the same capitalization that it has in the message.
 * *value*: String The header *field value* as defined in RFC5322, in
   either *Decoded* or *Raw Base64* form as determined by the property
   being fetched.The following convenience properties are also specified for the
*Email* object:

 * *sender*: EmailAddress[]|null (immutable) The value is identical to
   the value of *header:Sender:asAddresses*.
 * *from*: EmailAddress[]|null (immutable) The value is identical to the
   value of *header:From:asAddresses*.
 * *to*: EmailAddress[]|null (immutable) The value is identical to the
   value of *header:To:asAddresses*.
 * *cc*: EmailAddress[]|null (immutable) The value is identical to the
   value of *header:Cc:asAddresses*.
 * *bcc*: EmailAddress[]|null (immutable) The value is identical to the
   value of *header:Bcc:asAddresses*.
 * *replyTo*: EmailAddress[]|null (immutable) The value is identical to
   the value of *header:Reply-To:asAddresses*.
 * *subject*: String|null (immutable) The value is identical to the
   value of *header:Subject*.
 * *sentAt*: Date (immutable; default: time of creation on server) The
   value is identical to the value of *header:Date:asDate*.Body parts

These properties are derived from the [@!RFC5322] message body and its
[@RFC2045] MIME entities.A *EmailBodyPart* object has the following properties:

 * *partId*: String Identifies this part uniquely within the Email. This
   is scoped to the *emailId* and has no meaning outside of the JMAP
   Email object representation.
 * *blobId*: String The id references the raw bytes of the contents of
   the part after decoding any *Content-Transfer* encoding (as defined
   in [@!RFC2045]). Note, two parts may be transfer-encoded differently
   but have same the same blob id if their decoded bytes are identical
   and the server is using a secure hash of the data for the blob id.
 * *size*: Number The size, in bytes, of the raw data after content-
   transfer decoding (as refrenced by the *blobId*, i.e. the number of
   bytes in the file the user would download).
 * *headers*: EmailHeader[] (immutable) This is a list of all header
   fields in the part, in the order they appear. The values are in
   *Decoded* form.
 * *headers:asRaw*: EmailHeader[] (immutable) This is a list of all
   header fields in the part, in the order they appear. The values are
   in *Raw Base64* form.
 * *name*: String|null This is the [@!RFC2231] decoded *filename*
   parameter of the *Content-Disposition* header field, or (for
   compatibility with existing systems) if not present then the
   [@!RFC2047] decoded *name* property of the *Content-Type* header
   field. For compatibility with the widest array of existing systems,
   both forms should be set if creating a message.
 * *type*: String The value of the *Content-Type* header field of the
   part, excluding any parameters.
 * *charset*: String|null The value of the charset parameter of the *Content-
   Type* header field, if present.
 * *disposition*: String|null The value of the *Content-Disposition*
   header field of the part, excluding any parameters, if present.
 * *cid*: String|null The value of the *Content-Id* header field of the
   part, if present. This may be used to reference the content from
   within an html body part using the cid: protocol.
 * *language*: String[]|null The list of language tags, as defined in
   [@!RFC3282], in the *Content-Language* header field of the part,
   if present.
 * *location*: String|null The URI, as defined in [@!RFC2557], in the
   *Content-Location* header field of the part, if present.
 * *width*: Number|null The width (in px) of the image, if the type is
   an image and the server is able to determine this information for
   the image. This MAY not be supported for all or any image types on
   some servers.
 * *height*: Number|null The height (in px) of the image, if the type
   is an image and the server is able to determine this information for
   the image. This MAY not be supported for all or any image types on
   some servers.
 * *subParts*: EmailBodyPart[] (optional) If type is multipart/*, this
   contains the body parts of each child.The following *Email* properties are specified for access to the body
data of the message:

 * *bodyStructure*: EmailBodyPart[] (immutable) This is the full MIME
   structure of the message body, including sub parts but not recursing
   into message/rfc822 parts.

 * *bodyValues*: String[BodyValue] (immutable) This is a map of *partId*
   to an *EmailBodyValue* object. Which parts are included and whether
   the value is truncated is determined by various arguments to
   *Email/get* and *Email/parse*.

An *EmailBodyValue* object has the following properties:

   * *value*: String The value of the body part after decoding *Content-
     Transport* encoding and decoding the *Content-Type* charset.
   * *isTruncated* Boolean This is true if the *value* has been
     truncated.
 * *textBody*: EmailBodyPart[] (immutable) A list of text/plain,
   text/html and image/* parts to display (sequentially) as the message
   body, with a preference for text/plain when alternative versions are
   available.

 * *htmlBody*: EmailBodyPart[] (immutable) A list of text/plain,
   text/html and image/* parts to display (sequentially) as the message
   body, with a preference for text/html when alternative versions are
   available.

 * *attachedEmails*: EmailBodyPart[] (immutable) A list of all parts of
   type message/rfc822. Note, this *does not* recurse, so the parts
   within these are not included. The attached message may be fetched
   using the Email/parse method and the blobId.

 * *attachedFiles*: EmailBodyPart[] (immutable) A list of all parts in
   *bodyStructure*, traversing depth-first, not of type multipart/*, and
   not included in any of the *textBody*, *htmlBody* or *attachedEmails*
   properties. Note, an HTML body part may reference image parts in
   attachedFiles using cid: links to reference the *Content-Id* or by
   referencing the *Content-Location*.

 * *hasAttachment*: Boolean (immutable; server-set) This is true if
   there are one or more parts in the message that a client UI should
   offer as downloadable. A server SHOULD set hasAttachment if either:

   * The *attachedEmails* list contains at least one item.
   * The *attachedFiles* list contains at least one item that does not
     have Content-Disposition: attachment. The server MAY ignore parts
     in this list that are processed automatically in some way, or are
     referenced as embedded images in one of the text/html parts of the
     message. The server MAY set hasAttachment based on implementation-
     defined or site configurable heuristics.

 * *preview*: String (immutable; server-set) Up to 256 characters of the
   beginning of a plain text version of the message body. This is
   intended to be shown as a preview line on a mailbox listing, and the
   server may choose to skip quoted sections or salutations to return a
   more useful preview.MIME structures are arbitrary nested trees of documents, but the
majority of email clients present a model of an email body (normally
plain text or HTML), with a set of attachments. Interpreting the MIME
structure to form this flat model represents considerable difficulty and
causes inconsistency between clients. Therefore in addition to the
*bodyStructure* property, which gives the full tree, the Email object
contains 4 alternate properties with flat lists of body parts:

 * *textBody*/*htmlBody*: These provide a list of parts that should be
   rendered as the "body" of the message. This is a list rather than a
   single part as messages may have headers and/or footers
   appended/prepended as separate parts as they are transmitted, and
   some clients send text and images intended to be displayed inline in
   the body as multiple parts rather than a single HTML part with
   referenced images.

Because MIME allows for multiple representations of the same data (using
multipart/alternative), there is a textBody property (which prefers a
plain text representation) and an htmlBody property (which prefers an
HTML representation) to accommodate the two most common client
requirements. The same part may appear in both lists where there is no
alternative between the two.

 * *attachedEmails*/*attachedFiles*: These provide a list of parts that
   should be presented as "attachments" to the message. Emails are
   presented in a separate list so their contents may be easily fetched
   via a back-reference with the Email/parse method in the same request,
   if the client wishes to. Some images in attachedFiles may be solely
   there for embedding within an HTML body part; clients may wish to not
   present these as attachments in the user interface if they are
   displaying the HTML with the embedded images directly.The *bodyValues* property allows for clients to fetch the value of text
parts directly without having to do a second request for the blob, and
have the server handle decoding the charset into unicode. This data is
in a separate property rather than on the EmailBodyPart object to avoid
duplication of large amounts of data, as the same part may be included
twice if the client fetches more than one of bodyStructure, textBody
and htmlBody.The exact algorithm for decomposing bodyStructure into textBody,
htmlBody, attachedEmails and attachedFiles part lists is not mandated,
as this is a quality-of-service implementation issue and likely to
require workarounds for malformed content discovered over time.However, we will define a suggested algorithm for the RFC to help
implementors. TODO.Email/get

Standard */get* method, with the following additional arguments:

 * *bodyProperties*: String[] (optional) A list of properties to fetch
   for each EmailBodyPart returned. If omitted, this defaults to:

[ "partId", "blobId", "size", "name", "type", "charset", "disposition",
cid", "language", "location", "width", "height" ]

 * *fetchTextBodyValues*: Boolean (default: false) If true, the
   *bodyValues* property includes any text/* part in the textBody
   property.

 * *fetchHTMLBodyValues*: Boolean (default: false) If true, the
   *bodyValues* property includes any text/* part in the htmlBody
   property.

 * *fetchAllBodyValues*: Boolean (default: false) If true, the
   *bodyValues* property includes any text/* part in the
   bodyStructure property.

 * *maxBodyValueBytes*: Number (optional) If supplied by the client, the
   value MUST be a positive integer greater than 0. If a value outside
   of this range is given, the server MUST reject the call with an
   invalidArguments error. When given, the *value* property of any
   EmailBodyValue object returned in *bodyValues* MUST be truncated if
   necessary so it does not exceed this number of bytes in size. The
   server MUST ensure the truncation results in valid UTF-8 and does not
   occur mid-codepoint. If the part is of type text/html, the server
   SHOULD NOT truncate inside an HTML tag.If the standard *properties* argument is omitted or null, the following
default MUST be used instead of "all" properties:

[ "id", "blobId", "threadId", "mailboxIds", "keywords", "size",
"receivedAt", "sender", "from", "to", "cc", "bcc", "replyTo", "subject",
"sentAt", "hasAttachment", "preview", "bodyValues", "textBody",
"htmlBody", "attachedFiles", "attachedEmails" ]The following properties are expected to be fast to fetch in a quality
implementation:

 * id
 * blobId
 * threadId
 * mailboxIds
 * keywords
 * size
 * receivedAt
 * sender
 * from
 * to
 * cc
 * bcc
 * replyTo
 * subject
 * sentAt
 * hasAttachment
 * previewClients SHOULD take care when fetching any other properties, as there
may be significantly longer latency in fetching and returning the data.Example

Request:

["Email/get", { "ids": [ "f123u456", "f123u457" ], "properties": [
"threadId", "mailboxIds", "from", "subject", "receivedAt", "htmlBody",
"bodyValues" ], "bodyProperties": [ "partId", "blobId", "size", "type"
], "fetchHTMLBodyValues": true, "maxBodyValueBytes": 256 }, "#1"]and response:

["Email/get", { "accountId": "abc", "state": "41234123231", "list": [ {
"id": "f123u457", "threadId": "ef1314a", "mailboxIds": { "f123": true },
"from": [{name: "Joe Bloggs", email: "[email-masked]"}], "subject":
"Dinner on Thursday?", "receivedAt": "2013-10-13T14:12:00Z", "htmlBody":
[{ "partId": "1", "blobId": "841623871", "size": 283331, "type":
"text/html" }, { "partId": "2", "blobId": "319437193", "size": 10343,
"type": "text/plain" }], "bodyValues": { "1": { "isTruncated": true,
"value": "<html><body><p>Hello ..." }, "2": { "isTruncated": false,
"value": "-- \nSent by your friendly mailing list ..." } } } ],
notFound: [ "f123u456" ] }, "#1"]

Links:

  1. https://www.iana.org/assignments/imap-keywords/imap-keywords.xhtml

---

From: Alexey Melnikov <[email-masked]>
Date: Fri, 9 Feb 2018 11:41:27 +0000

Hi Neal,

I haven't digested your whole email yet, but a very quick comment about 
keywords:

On 08/02/2018 06:11, Neil Jenkins wrote:

>  *
>
>     *keywords*: |String[Boolean]| (default: |{}|) A set of keywords
>     that apply to the email. The set is represented as an object, with
>     the keys being the /keywords/. The value for each key in the
>     object MUST be |true|.
>
>     Keywords are shared with IMAP. The six system keywords from IMAP
>     are treated specially. The following four keywords have their
>     first character changed from |\| in IMAP to |$| in JMAP and have
>     particular semantic meaning:
>
>       o |$Draft|: The email is a draft the user is composing.
>       o |$Seen|: The email has been read.
>       o |$Flagged|: The email has been flagged for urgent/special
>         attention.
>       o |$Answered|: The email has been replied to.
>
>     The IMAP |\Recent| keyword is not exposed via JMAP. The IMAP
>     |\Deleted| keyword is also not present: IMAP uses a delete+expunge
>     model, which JMAP does not. Any message with the |\Deleted|
>     keyword MUST NOT be visible via JMAP.
>
>     Users may add arbitrary keywords to an email. For compatibility
>     with IMAP, a keyword is a (case-sensitive)
>
IMAP keywords are case insensitive. So $MDNSent and $mdnsent are the same.
>
>  *
>
>     string of 1–255 characters in the ASCII subset %x21–%x7e (excludes
>     control chars and space), and MUST NOT include any of these
>     characters: |( ) { ] % * " \|
>
|Probably not including } and [ as well. But I need to double check.

Best Regards,
Alexey

|

---

From: Barry Leiba <[email-masked]>
Date: Fri, 9 Feb 2018 12:48:04 -0500

>> string of 1–255 characters in the ASCII subset %x21–%x7e (excludes control
>> chars and space), and MUST NOT include any of these characters:
>>   ( ) { ] % * " \
>
> Probably not including } and [ as well. But I need to double check.

According to the IMAP grammar, that's the complete list: } and [ are
not forbidden in IMAP.

b

---

From: "Chris Newman" <[email-masked]>
Date: Fri, 09 Feb 2018 11:08:22 -0800

Overall I'm supportive of this approach. Detailed comments:

I concur with Alexey's comment about keywords being case-insensitive.

>    email as counted by the server towards the user's quota.

The problem with a quota-based size measurement is "0" is a reasonable 
answer for single-copy message stores that allow multiple Email objects 
that refer to the same message (particularly if a reference to the 
message in a trash folder is not the only reference). Also, some number 
substantially less than what the user might want to see in a UI is 
possible in a single-attachment-copy mail store where each attachment is 
charged only once against the user's quota.

One option is to define *size* as the RFC822 size of the message (same 
definition as IMAP RFC 3501). Another option is that *size* is a number 
that would be appropriate to display to a user representing the size of 
the message content and attachments. This allows use of RFC 5322 size, 
RFC 6532 size or a binary MIME variation of either.

>    the *internal date* in IMAP.Header fields

I note the IMAP APPEND command allows this to be client-specified. This 
is very useful in practice when migrating mail between mail stores. The 
date the user really wants is when the mail first entered a mailstore 
owned by the user (or operated on behalf of the user). The date when the 
mail first entered this particular mailstore is less useful to the end 
user.

> These properties are derived from the [@!RFC5322] message header 

It's also fine to allow [@!RFC6532] message header fields, I believe.

>    4. Any [@!RFC2047] encoded sections decoded

This is a security design error. I suggest:

Any syntactically correct [@!RFC2047] encoded sections with a known 
character set are decoded and re-encoded as a UTF-8 string with proper 
quoting and escaping for the active RFC 5322 syntactic context 
appropriate for the specific header field. Any [@!RFC2047] encoded NUL 
octets or control characters are dropped from the decoded value. Any 
text that looks like [@!RFC2047] syntax but violates [@!RFC2047] 
placement or whitespace rules MUST NOT be decoded. Any [@!RFC2047] 
content in an unrecognized header MUST NOT be decoded. Same rules apply 
to [@!RFC2231] decoding.

The alternative is to not do 2047/2231 decoding for *decoded* form, or 
only do it for unstructured headers like Subject & Content-Description.

>       attempt to continue as much as possible.

We might also want to add 6. require the server to convert denormalized 
Unicode to NFC. This is easy enough for a server to do and makes it 
easier for clients to comply with RFC 5198.

>    UTF-8 if 8-bit values are present.)

Are non-UTF-8 headers so prevalent that it merits corruption of JMAP? 
There has never been a standard allowing non-UTF-8 content with the 
high-bit set in email headers. I think it's fine to drop (or use Unicode 
replacement character) for any octet or octet run with the high bit set 
that violates UTF-8 syntax. Avoiding the unnecessary base64 step will 
simplify clients. NUL octets should also be dropped, although most mail 
stores will drop them on ingress since they became illegal in generated 
email on April 2001.

>      or *group*, or null if none.

It's important for 2047 decoding (as described previously) to be applied 
to this when possible.

>      not conform to the *addr-spec* form (for example, may not contain

Should this allow non-punycode IDN? Should this allow RFC 6532?

>    both forms should be set if creating a message.

I think this is the wrong place to include rules for generating email. I 
note that setting a "name" parameter on a MIME content type which 
doesn't explicitly register a "name" parameter is a violation of the 
MIME model. Advice to violate the model may be fine as an implementation 
suggestion, but I wouldn't make it a standards-level recommendation.

>    part, excluding any parameters.

Should this have CFWS removed? I think it should.
>    into message/rfc822 parts.

make that "message/rfc822 or message/global parts."

>      Transport* encoding and decoding the *Content-Type* charset.

I think you mean *Content-Transfer-Encoding*, and decoding it is fine. 
Decoding the charset unconditionally in this way violates the MIME model 
as the charset parameter has no strictly defined meaning outside the 
top-level text media-type.

I think there are three design options here:
1. CTE only
2. CTE, plus charset decoding of text/ and other recognized media types 
the server knows have a registered and defined meaning for charset.
3. allow the server to be helpful and briefly describe what it did.

If you choose option 3, I'd add a third "decodingKind" field to 
EmailBodyValue that explains the value. It could be:

*decodingKind*

This is an enumerated type indicating what decoding the server applied 
to the content. I suggest the following options:
*text*: The server decoded any content-transfer-encoding, recognized the 
declared media type as a textual type (regardless of whether it's a 
"text/" or "application/" media type), correctly interpreted the charset 
according to that media type's documented rules and converted it to 
UTF-8. No NUL code points were found in the content.
*UTF8*: The server decoded any content-transfer-encoding and determined 
that the content follows UTF-8 syntax and contains no NUL octets. This 
is never used on standards-compliant body parts with a top-level text 
type. However, if a body part with a top-level text type contains 
non-ASCII UTF-8 content and has no charset parameter (a violation of RFC 
2046), this may be used.
*base64*: The server decoded any content-transfer-encoding but the 
resulting content contains a NUL octet or does not meet UTF-8 syntax 
rules, the value will be base64-encoded without additional modification.
*heuristic*: The server applied some heuristic to convert the content 
charset to UTF-8 text (and no NUL code points were found). For example, 
the server looked at the magic number in an application/octet-stream and 
identified the content as a structured textual type with a known charset 
that could be converted to UTF-8. This may also be used, for example, if 
the charset media type parameter does not match a charset attribute in 
HTML/XML but the content matches the latter parameter.

>    type message/rfc822. Note, this *does not* recurse, so the parts

or type message/global.

>      have Content-Disposition: attachment. The server MAY ignore parts

I think you mean "Content-Disposition: inline".

		- Chris

---

From: Bron Gondwana <[email-masked]>
Date: Mon, 12 Feb 2018 15:08:26 +1100

On Thu, 8 Feb 2018, at 17:11, Neil Jenkins wrote:
> meeting in March, I have been looking at the issues raised with the
> JSON representation of an RFC5322 message. This email presents a new
> proposal that I think resolves the issues presented. In particular, I
> aim to address the following goals:
>  * Allow access to the full MIME structure without requiring the
>    client to parse the full message itself.
>  * Allow the client to only load the data required.> Comments, suggestions and other feedback requested. Cheers!

Sorry, saw this on my phone for the first read-through, so I didn't
reply.  I see Chris already caught some of this, but I'll do my read-
through again and reply as well.

> The *Email* object is a representation of an [@!RFC5322] message,
> which allows clients to avoid the complexities of MIME parsing,
> transport encoding and character encoding.> Properties of the Email object

> Broadly, a message consists of two parts: a list of header fields,
> then a body. The body is normally a MIME-encoded set of documents in a
> tree structure. The JMAP Email object provides a way to access the
> full structure, or to use simplified properties and avoid some
> complexity if this is sufficient for the client application.> Due to the number of properties involved, the set of *Email*
> properties is specified over the following three sub-sections.> Metadata

>  * *blobId*: String (immutable; server-set) The id representing the
>    raw bytes of the [@!RFC5322] message. This may be used to download
>    the raw original message, or to attach it directly to another
>    Email etc.

>    * $Answered: The email has been replied to. The IMAP \Recent
>      keyword is not exposed via JMAP. The IMAP \Deleted keyword is
>      also not present: IMAP uses a delete+expunge model, which JMAP
>      does not. Any message with the \Deleted keyword MUST NOT be
>      visible via JMAP.

> IMAP, a keyword is a (case-sensitive) string of 1–255 characters in
> the ASCII subset %x21–%x7e (excludes control chars and space), and
> MUST NOT include any of these characters: ( ) { ] % * " \

> semantic meaning to some other keywords in common use. New keywords
> may be established here in the future. In particular, note:

>      SHOULD warn users to take care when viewing this email and
>      disable links and attachments.
>    * $Junk: The email is definitely spam. Clients SHOULD set this flag
>      when users report spam to help train automated spam-detection
>      systems.
>    * $NotJunk: The email is definitely not spam. Clients SHOULD set
>      this flag when users indicate an email is legitimate, to help
>      train automated spam-detection systems.
did we want to talk about $HasAttachment and friend as well?

>  * *size*: Number (immutable; server-set) The size in bytes of the
>    whole email as counted by the server towards the user's quota.
I noticed this inconsistency as well - matching blobId size is by far
the most useful thing to do here.
>    server) The date the email was received by the message store. This
>    is the *internal date* in IMAP.> Header fields

> These properties are derived from the [@!RFC5322] message header
> fields. Header fields may be fetched in a variety of forms:

>       parser SHOULD insert a unicode replacement character (U+FFFD)
>       and attempt to continue as much as possible.

>    ([@!RFC4648]). (Note, this is required as the value may not be
>    valid UTF-8 if 8-bit values are present.)

>    * *name*: String|null The *display name* of the [@!RFC5322]
>      *mailbox* or *group*, or null if none.
>    * *email*: String|null The *addr-spec* of the [@!RFC5322]
>      *mailbox*, or null if a *group*.
I don't know enough about the group syntax, but I thought it was
nestable in a way that makes it hard to flatten.  Maybe just a null
group is enough, but the example in RFC5322 shows a group with 3
addresses in it:
Appendix A.1.3[2].  Group Addresses  ---- From: Pete
<[email-masked]> To: A Group:Ed Jones <[email-masked]>,[email-masked],John
<[email-masked]>; Cc: Undisclosed recipients:; Date: Thu, 13 Feb 1969
23:32:54 -0330 Message-ID: <[email-masked]>  Testing. ----
In this message, the "To:" field has a single group recipient named "A
Group", which contains 3 addresses, and a "Cc:" field with an empty
group recipient named Undisclosed recipients.And I have a horrible sinking feeling that the group could contain
nested groups in the full RFC5322 syntax, meaning you could have an
arbitrarily deep tree of addresses.
>  * Parsing SHOULD be best-effort in the face of invalid structure to
>    accommodate invalid messages and semi-complete drafts. EmailAddress
>    objects MAY have an *email* property that does not conform to the
>    *addr-spec* form (for example, may not contain an @ symbol).
>    [@!RFC5322] section 3.3, into the Date type.> The following *Email* properties are specified for complete access to

>    name (matched case-insensitively), or null if none. The value is in
This does seem to mean that it's impossible to ask for all instances of
a particular named header.  You either get the whole lot, or you get
just the last of a particular named one.  I don't know that it's likely
to be a big problem, but I do note the limitation.
>  * *header:{header-field-name}:asRaw*: String|null (immutable) The
>    value of the *last* header field in the message with the given
>    header field name (matched case-insensitively), or null if none.
>    The value is in *Raw Base64* form.
>    (immutable) The value of the *last* header field in the message
>    with the given header field name (matched case-insensitively), or
>    null if none. The value is in *Addresses* form.
>  * *header:{header-field-name}:asDate*: Date|null (immutable) The
>    value of the *last* header field in the message with the given
>    header field name (matched case-insensitively), or null if none.
>    The value is in *Date* form.> An *EmailHeader* object has the following properties:

>    being fetched.> The following convenience properties are also specified for the
> *Email* object:

>  * *from*: EmailAddress[]|null (immutable) The value is identical to
>    the value of *header:From:asAddresses*.
>  * *bcc*: EmailAddress[]|null (immutable) The value is identical to
>    the value of *header:Bcc:asAddresses*.
>  * *replyTo*: EmailAddress[]|null (immutable) The value is identical
>    to the value of *header:Reply-To:asAddresses*.
>    value is identical to the value of *header:Date:asDate*.> Body parts

> [@RFC2045] MIME entities.> A *EmailBodyPart* object has the following properties:

>  * *partId*: String Identifies this part uniquely within the Email.
>    This is scoped to the *emailId* and has no meaning outside of the
>    JMAP Email object representation.
>    in [@!RFC2045]). Note, two parts may be transfer-encoded
>    differently but have same the same blob id if their decoded bytes
>    are identical and the server is using a secure hash of the data for
>    the blob id.
>  * *charset*: String|null The value of the charset parameter of the
>    *Content-Type* header field, if present.
>  * *cid*: String|null The value of the *Content-Id* header field of
>    the part, if present. This may be used to reference the content
>    from within an html body part using the cid: protocol.
>    [@!RFC3282], in the *Content-Language* header field of the part, if
>    present.
>    is an image and the server is able to determine this information
>    for the image. This MAY not be supported for all or any image types
>    on some servers.
>    contains the body parts of each child.> The following *Email* properties are specified for access to the body
> data of the message:

>    structure of the message body, including sub parts but not
>    recursing into message/rfc822 parts.

>  * *bodyValues*: String[BodyValue] (immutable) This is a map of
>    *partId* to an *EmailBodyValue* object. Which parts are included
>    and whether the value is truncated is determined by various
>    arguments to *Email/get* and *Email/parse*.

>    text/html and image/* parts to display (sequentially) as the
>    message body, with a preference for text/plain when alternative
>    versions are available.

>    text/html and image/* parts to display (sequentially) as the
>    message body, with a preference for text/html when alternative
>    versions are available.
There was some discussion about how to handle RFC 8255
multipart/multilingual or other things here - whether we add something
along the lines of "the server will choose which representation to
include based on the original language plus what it knows about the
user's preferred language" - plus of course the full information is
available in the bodyStructure if the client wants to be smart about it.
This is also a good place for a JMAP extension which could give a more
powerful query language for choosing specific parts in multipart.
>  * *attachedEmails*: EmailBodyPart[] (immutable) A list of all parts
>    of type message/rfc822. Note, this *does not* recurse, so the parts
Might be worth having an example of this - it's going to be one of the
tricky bits for people to understand.
>    *bodyStructure*, traversing depth-first, not of type multipart/*,
>    and not included in any of the *textBody*, *htmlBody* or
>    *attachedEmails* properties. Note, an HTML body part may reference
>    image parts in attachedFiles using cid: links to reference the *Content-
>    Id* or by referencing the *Content-Location*.
This could do with being a little clearer that "even if referenced by an
HTML body part, attached files will be included in this list" - or
something like that.

>      referenced as embedded images in one of the text/html parts of
>      the message.
It's already been noted that the logic is reversed here, should be "does
have" or maybe even "does not have Content-Disposition: inline" instead.
>  * The server MAY set hasAttachment based on implementation-defined or
>    site configurable heuristics.

>  * *preview*: String (immutable; server-set) Up to 256 characters of
>    the beginning of a plain text version of the message body. This is
>    intended to be shown as a preview line on a mailbox listing, and
>    the server may choose to skip quoted sections or salutations to
>    return a more useful preview.
I wouldn't even say "of the beginning" - it's up to 256 characters
that provide a good summary of the content of the message.  I would
say something like: This is intended to be shown as a preview line on
a mailbox listing, and not all characters may be shown.  The server
may choose which part of the message to include in the preview, for
example skipping quoted sections and salutations can result in a more
useful preview.
> MIME structures are arbitrary nested trees of documents, but the
> structure to form this flat model represents considerable difficulty
> and causes inconsistency between clients. Therefore in addition to the

>    some clients send text and images intended to be displayed inline
>    in the body as multiple parts rather than a single HTML part with
>    referenced images.

> Because MIME allows for multiple representations of the same data
> (using multipart/alternative), there is a textBody property (which
> prefers a plain text representation) and an htmlBody property (which
> prefers an HTML representation) to accommodate the two most common
> client requirements. The same part may appear in both lists where
> there is no alternative between the two.

>  * *attachedEmails*/*attachedFiles*: These provide a list of parts
>    that should be presented as "attachments" to the message. Emails
>    are presented in a separate list so their contents may be easily
>    fetched via a back-reference with the Email/parse method in the
>    same request, if the client wishes to. Some images in attachedFiles
>    may be solely there for embedding within an HTML body part; clients
>    may wish to not present these as attachments in the user interface
>    if they are displaying the HTML with the embedded images directly.> The *bodyValues* property allows for clients to fetch the value of
> text parts directly without having to do a second request for the
> blob, and have the server handle decoding the charset into unicode.
> This data is in a separate property rather than on the EmailBodyPart
> object to avoid duplication of large amounts of data, as the same part
> may be included twice if the client fetches more than one of
> bodyStructure, textBody and htmlBody.> The exact algorithm for decomposing bodyStructure into textBody,
> require workarounds for malformed content discovered over time.> However, we will define a suggested algorithm for the RFC to help
> implementors. TODO.> Email/get

> [ "partId", "blobId", "size", "name", "type", "charset",
> "disposition", cid", "language", "location", "width", "height" ]

>    *bodyValues* property includes any text/* part in the bodyStructure
>    property.

>  * *maxBodyValueBytes*: Number (optional) If supplied by the client,
>    the value MUST be a positive integer greater than 0. If a value
>    outside of this range is given, the server MUST reject the call
>    with an invalidArguments error. When given, the *value* property of
>    any EmailBodyValue object returned in *bodyValues* MUST be
>    truncated if necessary so it does not exceed this number of bytes
>    in size. The server MUST ensure the truncation results in valid UTF-
>    8 and does not occur mid-codepoint. If the part is of type
>    text/html, the server SHOULD NOT truncate inside an HTML tag.> If the standard *properties* argument is omitted or null, the
> following default MUST be used instead of "all" properties:

> "receivedAt", "sender", "from", "to", "cc", "bcc", "replyTo",
> "subject", "sentAt", "hasAttachment", "preview", "bodyValues",
> "textBody", "htmlBody", "attachedFiles", "attachedEmails" ]
>> The following properties are expected to be fast to fetch in a quality
> implementation:

>  * id
>  * blobId
>  * threadId
>  * mailboxIds
>  * keywords
>  * size
>  * receivedAt
>  * sender
>  * from
>  * to
>  * cc
>  * bcc
>  * replyTo
>  * subject
>  * sentAt
>  * hasAttachment
>  * preview> Clients SHOULD take care when fetching any other properties, as
> there may be significantly longer latency in fetching and returning
> the data.> Example

> ["Email/get", { "accountId": "abc", "state": "41234123231", "list": [
> { "id": "f123u457", "threadId": "ef1314a", "mailboxIds": { "f123":
> true }, "from": [{name: "Joe Bloggs", email: "[email-masked]"}],
> "subject": "Dinner on Thursday?", "receivedAt": "2013-10-
> 13T14:12:00Z", "htmlBody": [{ "partId": "1", "blobId": "841623871",
> "size": 283331, "type": "text/html" }, { "partId": "2", "blobId":
> "319437193", "size": 10343, "type": "text/plain" }], "bodyValues": {
> "1": { "isTruncated": true, "value": "<html><body><p>Hello ..." },
> "2": { "isTruncated": false, "value": "-- \nSent by your friendly
> mailing list ..." } } } ], notFound: [ "f123u456" ] }, "#1"]
>> 
> _________________________________________________
> Jmap mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/jmap

--
  Bron Gondwana, CEO, FastMail Pty Ltd
  [email-masked]

Links:

  1. https://www.iana.org/assignments/imap-keywords/imap-keywords.xhtml
  2. https://tools.ietf.org/html/rfc5322#appendix-A.1.3

---

From: Neil Jenkins <[email-masked]>
Date: Mon, 12 Feb 2018 18:05:54 +1100

Thanks for the great feedback!

> I concur with Alexey's comment about keywords being case-insensitive.
I've changed it so they are case-insensitive but the server MUST
return them lower-cased (as JSON is case-sensitive and it's a pain to
have to iterate through keys in an object rather than be able to do a
direct lookup).
>> * **size**: Number (immutable; server-set) The size in bytes of the
>> whole
> The problem with a quota-based size measurement is "0" is a reasonable> answer for single-copy message stores that allow multiple Email
> objects> that refer to the same message (particularly if a reference to the
> message in a trash folder is not the only reference).

Yep, this is also inconsistent with everywhere else there are
blobId/size properties on an object. I've changed it so it's the size
of the file you get if you download the blob (i.e. the size of the
RFC5322 message), which is then consistent with the rest of the spec
and with IMAP.
>> * **receivedAt**: UTCDate
> I note the IMAP APPEND command allows this to be client-specified.

This is also already client-specifiable in JMAP (see the Email/import
method, which is the equivalent of APPEND). It's just the *default*
that's the time the message is inserted in the mail store if no other
date is supplied.
>> fields.
Yep, I'll note this in the spec.

>> * **Decoded** The header field value with:
> character set are decoded and re-encoded as a UTF-8 string with proper> quoting and escaping for the active RFC 5322 syntactic context
> appropriate for the specific header field. Any [@!RFC2047] encoded NUL> octets or control characters are dropped from the decoded value. Any
> content in an unrecognized header MUST NOT be decoded. Same
> rules apply> to [@!RFC2231] decoding.

Yes, looking at this again we have come to the conclusion that allowing
clients to request arbitrary headers be parsed in arbitrary forms is
allowing them to shoot themselves in the foot too much, and going to
lead to security and/or interoperability errors.
My new suggestion:
 * Raw form doesn't base64 (as you say, any standards-compliant email
   should be valid ASCII or UTF-8), and becomes the default, and the
   only form in which you can request any header.
 * A number of parsed forms are defined. These may only be fetched/set
   for appropriate header fields from RFC5322 or fields not defined in
   this spec (which allows for more future proofing, if someone defines
   a new header field that contains a date for example).I'll send another email with the full revised spec shortly.

> We might also want to add 6. require the server to convert
> denormalized> Unicode to NFC. This is easy enough for a server to do and makes it

Seems reasonable.

>> * **Addresses** The header is parsed as an address-list value, as
>>   specified in [@!RFC5322] section 3.4, into the EmailAddress[] type.>>   The **EmailAddress** object has the following properties:
>> 
>> 
>>   * **name**: String|null The *display name* of the [@!RFC5322]
>> **mailbox**
>>     or **group**, or null if none.
> 
> It's important for 2047 decoding (as described previously) to
> be applied> to this when possible.

Yes, this was meant to be the case, looks like I missed specifying that.
>> EmailAddress objects MAY have an **email** property that
>> does not conform to the **addr-spec** form (for example, may not
>> contain an @ symbol).> Should this allow non-punycode IDN? 

I think the value there should be automatically converted to or from
punycode, but other than that I don't think we should be rejecting
anything, just doing best-effort parsing into a form that's easier for
the client to handle. The client can convert or analyse for phishing
issues with the domain.
> Should this allow RFC 6532?

Yes, I've now noted this.

>> * **name**: String|null This is the [@!RFC2231] decoded **filename**>>   parameter of the **Content-Disposition** header field, or (for
>>   [@!RFC2047] decoded **name** property of the **Content-Type**
>>   header>>   field. For compatibility with the widest array of existing systems,>>   both forms should be set if creating a message.
> 
> I think this is the wrong place to include rules for generating
> email. I> note that setting a "name" parameter on a MIME content type which
> MIME model. Advice to violate the model may be fine as an
> implementation> suggestion, but I wouldn't make it a standards-level recommendation.

OK. I'll remove the last sentence and add a note to the server
implementation guide[1].
>> * **type**: String The value of the **Content-Type** header field
>>   of the>>   part, excluding any parameters.

Yes.

>> * **bodyStructure**: EmailBodyPart[] (immutable) This is the
>>   full MIME>>   structure of the message body, including sub parts but not
>> recursing

Yep.

>>   * **value**: String The value of the body part after decoding
>> *Content-
>>     Transport* encoding and decoding the **Content-Type** charset.
> 
> I think you mean **Content-Transfer-Encoding**, and decoding it
> is fine.> Decoding the charset unconditionally in this way violates the
> MIME model> as the charset parameter has no strictly defined meaning outside the
> 2. CTE, plus charset decoding of text/ and other recognized media
>    types> the server knows have a registered and defined meaning for charset.

The bodyValues property is *only* used to return text/* parts.
(Anything else, the client has to download the blob, which does CTE
decoding only).
> EmailBodyValue that explains the value.

I feel this is overkill. The main purpose of this property is to let
clients fetch the message text/html body without having to deal with
charset decoding. I think what we want here are:
1. A best-effort decoded value.
2. A flag to tell the client if the charset was unknown, or invalid data
   was found for the charset. The client can then choose what to do
   (download the blob and try to decode it itself, or more likely show a
   warning to the user that the message may display incorrectly, or even
   just not show the message at all and  say it could not be
   understood).
With that in mind, here's a new suggestion:

 * *value*: String The value of the body part after decoding *Content-Transport-
   Encoding* and decoding the *Content-Type* charset, if known to the
   server. The server MAY use heuristics to determine the charset to use
   for decoding if the charset is unknown, or if no charset is given, or
   if it believes the charset given is incorrect. Decoding is best-
   effort and SHOULD insert the unicode replacement character (U+FFFD)
   and continue when a malformed section is encountered.
 * *isEncodingProblem*: Boolean This is true if malformed sections were
   found while decoding the charset, or the charset was unknown.
 * *isTruncated*: Boolean This is true if the *value* has been
   truncated.
>> * **attachedEmails**: EmailBodyPart[] (immutable) A list of all parts>> of  type message/rfc822. Note, this *does not* recurse, so the parts> or type message/global.

Yes

>>   * The **attachedFiles** list contains at least one item that does
>>     not>>     have Content-Disposition: attachment. The server MAY ignore parts> I think you mean "Content-Disposition: inline".

Oops, yes.

Neil.

Links:

  1. http://jmap.io/server.html

---

From: Neil Jenkins <[email-masked]>
Date: Mon, 12 Feb 2018 18:15:08 +1100

> did we want to talk about $HasAttachment and friend as well?

I believe it is worthwhile having a hasAttachment property in JMAP
despite the existence of this keyword, because the keyword is not
guaranteed to be set when appropriate, so the client cannot rely on it,
and it's an important feature for building a good client.
> nestable in a way that makes it hard to flatten.
No, if you look at the syntax[1], you can't nest groups.

> This does seem to mean that it's impossible to ask for all instances
> of a particular named header.  You either get the whole lot, or you
> get just the last of a particular named one.  I don't know that it's
> likely to be a big problem, but I do note the limitation.
I think this is still a reasonable balance between power and complexity.
>>    of type message/rfc822. Note, this *does not* recurse, so the
>>    parts within these are not included. The attached message may be
>>    fetched using the Email/parse method and the blobId.> 
Yeh, we need a lot more examples in general.

>>  * *attachedFiles*: EmailBodyPart[] (immutable) A list of all parts
>>    in *bodyStructure*, traversing depth-first, not of type
>>    multipart/*, and not included in any of the *textBody*, *htmlBody*
>>    or *attachedEmails* properties. Note, an HTML body part may
>>    reference image parts in attachedFiles using cid: links to
>>    reference the *Content-Id* or by referencing the *Content-
>>    Location*.> 
> This could do with being a little clearer that "even if referenced by
> an HTML body part, attached files will be included in this list" - or
> something like that.
I'm not sure I think that's clearer than the current description. 

>>    return a more useful preview.> 
Yep, good point, I've updated this.

Neil.

Links:

  1. https://tools.ietf.org/html/rfc5322#section-3.4

---

From: Neil Jenkins <[email-masked]>
Date: Mon, 12 Feb 2018 18:26:47 +1100

Revised text below for further review/comment. (You can also look at the
emailformat branch commits on GitHub[1] for diffs).
Cheers,
Neil.

-------

 * *id*: String (immutable; server-set) The id of the Email object.
   Note, this is the JMAP object id, NOT the [@!RFC5322] Message-ID
   header field value.

 * *blobId*: String (immutable; server-set) The id representing the raw
   octets of the [@!RFC5322] message. This may be used to download the
   raw original message, or to attach it directly to another Email etc.

   * $draft: The email is a draft the user is composing.
   * $seen: The email has been read.
   * $flagged: The email has been flagged for urgent/special attention.
   * $answered: The email has been replied to. The IMAP \Recent keyword
     is not exposed via JMAP. The IMAP \Deleted keyword is also not
     present: IMAP uses a delete+expunge model, which JMAP does not. Any
     message with the \Deleted keyword MUST NOT be visible via JMAP.

Users may add arbitrary keywords to an email. For compatibility with
IMAP, a keyword is a case-insensitive string of 1–255 characters in the
ASCII subset %x21–%x7e (excludes control chars and space), and MUST NOT
include any of these characters: ( ) { ] % * " \

Because JSON is case-sensitive, servers MUST return keywords in
lower-case.

The IANA Keyword Registry[2] as established in [@!RFC5788] assigns
semantic meaning to some other keywords in common use. New keywords may
be established here in the future. In particular, note:

   * $forwarded: The email has been forwarded.
   * $phishing: The email is highly likely to be phishing. Clients
     SHOULD warn users to take care when viewing this email and disable
     links and attachments.
   * $junk: The email is definitely spam. Clients SHOULD set this
     flag when users report spam to help train automated spam-
     detection systems.
   * $notjunk: The email is definitely not spam. Clients SHOULD set this
     flag when users indicate an email is legitimate, to help train
     automated spam-detection systems.
 * *size*: Number (immutable; server-set) The size, in octets, of the
   raw data for the RFC5322 message (as referenced by the *blobId*, i.e.
   the number of octets in the file the user would download).

These properties are derived from the [@!RFC5322] and [@!RFC6532]
message header fields. All header fields may be fetched in a raw
form. Some headers may also be fetched in a parsed form. The
structured form that may be fetched depends on the header. The
following forms are defined:

 * *Raw* The raw octets of the header field value from the first octet
   following the header field name terminating colon, up to but
   excluding the header field terminating CRLF. Any standards-
   compliant message MUST be either ASCII (RFC5322) or UTF-8
   (RFC6532), however other encodings exist in the wild. A server MAY
   use heuristics to determine a charset and decode the octets, or MAY
   replace any octet or octet run with the high bit set that violates
   UTF-8 syntax with the unicode replacement character (U+FFFD). Any
   NUL octet MUST be dropped.

 * *Text* The header field value with:

   1. White space unfolded (as defined in [@!RFC5322] section 2.2.3)
   2. The terminating CRLF at the end of the value removed
   3. Any SP characters at the beginning of the value removed
   4. Any syntactically correct [@!RFC2047] encoded sections with a
      known character set are decoded. Any [@!RFC2047] encoded NUL
      octets or control characters are dropped from the decoded value.
      Any text that looks like [@!RFC2047] syntax but violates
      [@!RFC2047] placement or whitespace rules MUST NOT be decoded.
   5. Any [@!RFC6532] UTF-8 values decoded.
   6. The resulting unicode is converted to NFC form. If any decodings
      fail, the parser SHOULD insert a unicode replacement character
      (U+FFFD) and attempt to continue as much as possible.

To prevent obviously nonsense behaviour, which can lead to
interoperability issues, this form may only be fetched or set for the
following header fields:

   * Subject
   * Comment
   * Any header not defined in [@!RFC5322] or [@!RFC2369]
 * *Addresses* The header is parsed as an address-list value, as
   specified in [@!RFC5322] section 3.4, into the EmailAddress[] type.
   The *EmailAddress* object has the following properties:

   * *name*: String|null The *display name* of the [@!RFC5322] *mailbox*
     or *group*, or null if none.
   * *email*: String|null The *addr-spec* of the [@!RFC5322] *mailbox*,
     or null if a *group*. Any syntactically correct [@!RFC2047]
     encoded sections with a known encoding MUST be decoded, following
     the same rules as for the *Text* form. Any [@!RFC6532] UTF-8
     values MUST be decoded.

Parsing SHOULD be best-effort in the face of invalid structure to
accommodate invalid messages and semi-complete drafts. EmailAddress
objects MAY have an *email* property that does not conform to the *addr-
spec* form (for example, may not contain an @ symbol).

To prevent obviously nonsense behaviour, which can lead to
interoperability issues, this form may only be fetched or set for the
following header fields:

   * From
   * Sender
   * Reply-To
   * To
   * Cc
   * Bcc
   * Resent-From
   * Resent-Sender
   * Resent-Reply-To
   * Resent-To
   * Resent-Cc
   * Resent-Bcc
   * Any header not defined in [@!RFC5322] or [@!RFC2369]
 * *MessageIds* The header is parsed as a list of msg-id values, as
   specified in [@!RFC5322] section 3.6.5, into the String[] type.

To prevent obviously nonsense behaviour, which can lead to
interoperability issues, this form may only be fetched or set for the
following header fields:

   * Message-ID
   * In-Reply-To
   * References
   * Resent-Message-ID
   * Any header not defined in [@!RFC5322] or [@!RFC2369]
 * *Date* The header is parsed as a date-time value, as specified in
   [@!RFC5322] section 3.3, into the Date type.

To prevent obviously nonsense behaviour, which can lead to
interoperability issues, this form may only be fetched or set for the
following header fields:

   * Date
   * Resent-Date
   * Any header not defined in [@!RFC5322] or [@!RFC2369]
 * *URL* The header is parsed as a URL, as described in [@!RFC2369].
   This does not include the surrounding angle brackets or any comments
   in the header with the URL.

To prevent obviously nonsense behaviour, which can lead to
interoperability issues, this form may only be fetched or set for the
following header fields:

   * List-Help
   * List-Unsubscribe
   * List-Subscribe
   * List-Post
   * List-Owner
   * List-Archive
   * Any header not defined in [@!RFC5322] or [@!RFC2369]The following low-level *Email* properties are specified for complete
access to the header data of the message. Where {header-field-name} is
specified within a property name, this means any series of one or more
printable ASCII characters (i.e. characters that have values between 33
and 126, inclusive), except colon. Where {header-form} is specified
within a property name, this means one of the parsed-form name
specified above.

 * *headers*: EmailHeader[] (immutable) This is a list of all
   [@!RFC5322] header fields, in the same order they appear in the
   message. An *EmailHeader* object has the following properties:

     *Raw* form.
   *Raw* form.

 * *header:{header-field-name}:as{header-form}*: String|null (immutable)
   The value of the *last* header field in the message with the given
   header field name (matched case-insensitively), or null if none,
   decoded according to the parsing algorithm given.The following convenience properties are also specified for the
*Email* object:

 * *messageId*: MsgIds[] (immutable) The value is identical to the value
   of *header:Message-ID:asMessageIds*.
 * *inReplyTo*: MsgIds[] (immutable) The value is identical to the value
   of *header:In-Reply-To:asMessageIds*.
 * *references*: MsgIds[] (immutable) The value is identical to the
   value of *header:References:asMessageIds*.
 * *sender*: EmailAddress[]|null (immutable) The value is identical to
   the value of *header:Sender:asAddresses*.
 * *from*: EmailAddress[]|null (immutable) The value is identical to the
   value of *header:From:asAddresses*.
 * *to*: EmailAddress[]|null (immutable) The value is identical to the
   value of *header:To:asAddresses*.
 * *cc*: EmailAddress[]|null (immutable) The value is identical to the
   value of *header:Cc:asAddresses*.
 * *bcc*: EmailAddress[]|null (immutable) The value is identical to the
   value of *header:Bcc:asAddresses*.
 * *replyTo*: EmailAddress[]|null (immutable) The value is identical to
   the value of *header:Reply-To:asAddresses*.
 * *subject*: String|null (immutable) The value is identical to the
   value of *header:Subject:asText*.
 * *sentAt*: Date (immutable; default on creation: current time on
   server) The value is identical to the value of *header:Date:asDate*.Body parts

 * *partId*: String Identifies this part uniquely within the Email. This
   is scoped to the *emailId* and has no meaning outside of the JMAP
   Email object representation.
 * *blobId*: String The id references the raw octets of the contents of
   the part after decoding any *Content-Transfer* encoding (as defined
   in [@!RFC2045]). Note, two parts may be transfer-encoded differently
   but have same the same blob id if their decoded octets are identical
   and the server is using a secure hash of the data for the blob id.
 * *size*: Number The size, in octets, of the raw data after content-
   transfer decoding (as referenced by the *blobId*, i.e. the number of
   octets in the file the user would download).
 * *headers*: EmailHeader[] (immutable) This is a list of all header
   fields in the part, in the order they appear. The values are in
   *Raw* form.
 * *name*: String|null This is the [@!RFC2231] decoded *filename*
   parameter of the *Content-Disposition* header field, or (for
   compatibility with existing systems) if not present then the
   [@!RFC2047] decoded *name* parameter of the *Content-Type*
   header field.
 * *type*: String The value of the *Content-Type* header field of the
   part, if present, otherwise null. CFWS is removed and any parameters
   are stripped.
 * *charset*: String|null The value of the charset parameter of the *Content-
   Type* header field, if present.
 * *disposition*: String|null The value of the *Content-Disposition*
   header field of the part, if present, otherwise null. CFWS is removed
   and any parameters are stripped.
 * *cid*: String|null The value of the *Content-Id* header field of the
   part, if present, otherwise null. CFWS is removed. This may be used
   to reference the content from within an html body part using the
   cid: protocol.
 * *language*: String[]|null The list of language tags, as defined in
   [@!RFC3282], in the *Content-Language* header field of the part,
   if present.
 * *location*: String|null The URI, as defined in [@!RFC2557], in the
   *Content-Location* header field of the part, if present.
 * *width*: Number|null The width (in px) of the image, if the type is
   an image and the server is able to determine this information for
   the image. This MAY not be supported for all or any image types on
   some servers.
 * *height*: Number|null The height (in px) of the image, if the type
   is an image and the server is able to determine this information for
   the image. This MAY not be supported for all or any image types on
   some servers.
 * *subParts*: EmailBodyPart[] (optional) If type is multipart/*, this
   contains the body parts of each child.The following *Email* properties are specified for access to the body
data of the message:

 * *bodyStructure*: EmailBodyPart[] (immutable) This is the full MIME
   structure of the message body, including sub parts but not recursing
   into message/rfc822 or message/global parts.

 * *bodyValues*: String[BodyValue] (immutable) This is a map of *partId*
   to an *EmailBodyValue* object for none, some or all text/* parts.
   Which parts are included and whether the value is truncated is
   determined by various arguments to *Email/get* and *Email/parse*.

   * *value*: String The value of the body part after decoding *Content-Transport-
     Encoding* and decoding the *Content-Type* charset, if known to the
     server. The server MAY use heuristics to determine the charset to
     use for decoding if the charset is unknown, or if no charset is
     given, or if it believes the charset given is incorrect. Decoding
     is best-effort and SHOULD insert the unicode replacement character
     (U+FFFD) and continue when a malformed section is encountered.
   * *isEncodingProblem*: Boolean This is true if malformed sections
     were found while decoding the charset, or the charset was unknown.
   * *isTruncated*: Boolean This is true if the *value* has been
     truncated.
 * *textBody*: EmailBodyPart[] (immutable) A list of text/plain,
   text/html and image/* parts to display (sequentially) as the message
   body, with a preference for text/plain when alternative versions are
   available.

 * *attachedEmails*: EmailBodyPart[] (immutable) A list of all parts of
   type message/rfc822 or message/global. Note, this *does not* recurse,
   so the parts within these are not included. The attached message may
   be fetched using the Email/parse method and the blobId.

   * The *attachedEmails* list contains at least one item.
   * The *attachedFiles* list contains at least one item that does not
     have Content-Disposition: inline. The server MAY ignore parts in
     this list that are processed automatically in some way, or are
     referenced as embedded images in one of the text/html parts of the
     message. The server MAY set hasAttachment based on implementation-
     defined or site configurable heuristics.

 * *preview*: String (immutable; server-set) Up to 256 characters of
   plain text, summarising the message body. This is intended to be
   shown as a preview line on a mailbox listing, and may be
   truncated when shown. The server may choose which part of the
   message to include in the preview, for example skipping quoted
   sections and salutations and collapsing white-space can result in
   a more useful preview.MIME structures are arbitrary nested trees of documents, but the
majority of email clients present a model of an email body (normally
plain text or HTML), with a set of attachments. Interpreting the MIME
structure to form this flat model represents considerable difficulty and
causes inconsistency between clients. Therefore in addition to the
*bodyStructure* property, which gives the full tree, the Email object
contains 4 alternate properties with flat lists of body parts:

   necessary so it does not exceed this number of octets in size. The

[ "id", "blobId", "threadId", "mailboxIds", "keywords", "size",
"receivedAt", "messageId", "inReplyTo", "references", "sender", "from",
"to", "cc", "bcc", "replyTo", "subject", "sentAt", "hasAttachment",
"preview", "bodyValues", "textBody", "htmlBody", "attachedFiles",
"attachedEmails" ]The following properties are expected to be fast to fetch in a quality
implementation:

 * id
 * blobId
 * threadId
 * mailboxIds
 * keywords
 * size
 * receivedAt
 * messageId
 * inReplyTo
 * sender
 * from
 * to
 * cc
 * bcc
 * replyTo
 * subject
 * sentAt
 * hasAttachment
 * previewClients SHOULD take care when fetching any other properties, as there
may be significantly longer latency in fetching and returning the data.As specified above, parsed forms of headers may only be used on
appropriate header fields. Attempting to fetch a form that is forbidden
(e.g. header:From:asDate) MUST result in the method call being rejected
with an invalidArguments error.Where a specific header is requested as a property, the capitalization
of the property name in the response MUST be identical to that used in
the request.Example

Request:

["Email/get", { "ids": [ "f123u456", "f123u457" ], "properties": [
"threadId", "mailboxIds", "from", "subject", "receivedAt", "header:List-
POST:asURL" "htmlBody", "bodyValues" ], "bodyProperties": [ "partId",
"blobId", "size", "type" ], "fetchHTMLBodyValues": true,
"maxBodyValueBytes": 256 }, "#1"]and response:

["Email/get", { "accountId": "abc", "state": "41234123231", "list":
[ { "id": "f123u457", "threadId": "ef1314a", "mailboxIds": { "f123":
true }, "from": [{name: "Joe Bloggs", email: "[email-masked]"}],
"subject": "Dinner on Thursday?", "receivedAt": "2013-10-
13T14:12:00Z", "header:List-POST:asURL":
"mailto:[email-masked]", "htmlBody": [{ "partId": "1",
"blobId": "841623871", "size": 283331, "type": "text/html" }, {
"partId": "2", "blobId": "319437193", "size": 10343, "type":
"text/plain" }], "bodyValues": { "1": { "isEncodingProblem": false,
"isTruncated": true, "value": "<html><body><p>Hello ..." }, "2": {
"isEncodingProblem": false, "isTruncated": false, "value": "--
\nSent by your friendly mailing list ..." } } } ], notFound: [
"f123u456" ] }, "#1"]

Links:

  1. https://github.com/jmapio/jmap/commits/emailformat
  2. https://www.iana.org/assignments/imap-keywords/imap-keywords.xhtml

---

From: "Chris Newman" <[email-masked]>
Date: Mon, 12 Feb 2018 11:57:25 -0800

>      (U+FFFD) and continue when a malformed section is encountered.

I'm fine with this approach, but this does have security considerations. 
I suggest adding to this section:
"See the security considerations section for issues related to heuristic 
determination of content-type and charset."

and here's suggested text for the security considerations:
---
Service providers typically perform security filtering on incoming email 
and it's important the detection of content-type and charset for the 
security filter aligns with the heuristics performed by JMAP servers. 
Servers that apply heuristics to determine the content-type or charset 
for *EmailBodyValue* SHOULD document the heuristics and provide a 
mechanism to turn them off in the event they are misaligned with the 
security filter used at a particular site. Automatic conversion of 
charsets that allow hidden channels for ASCII text, such as UTF-7, have 
been problematic for security filters in the past so server 
implementations can mitigate this risk by having such conversions 
off-by-default and/or separately configurable.
---

		- Chris

---

From: Neil Jenkins <[email-masked]>
Date: Tue, 13 Feb 2018 10:19:27 +1100

> I'm fine with this approach, but this does have security
Agreed. I've already added a security consideration in relation to the
truncation of the value, I'll add in your text regarding heuristic
determination of content-type and charset.
I've created a pull request[1] with the changes. Further comments
welcome either there or on the list. I'd like to merge this in the next
week or two.
Neil.

Links:

  1. https://github.com/jmapio/jmap/pull/175

---

From: JFCM <[email-masked]>
Date: Tue, 13 Feb 2018 09:18:00 +0100

On 20:08 09/02/2018, Chris Newman said:
>This is very useful in practice when migrating mail between mail 
>stores. The date the user really wants is when the mail first 
>entered a mailstore owned by the user (or operated on behalf of the 
>user). The date when the mail first entered this particular 
>mailstore is less useful to the end user.

I also have difficulty in corellating this with RFC 2244 modtime 
requirements. I call "neb" the "nebularity" of a user autoritative 
digital resources. The "rgnr" is the immutable "nebReceivedAt" (down 
to the receiving machine cpu cycle). It turns to be terse enough when 
expressed in alphadigital. This was first introduced to time sort 
domain name registration requests (after synchronization in case of 
parallel or back-up registry servers).
jfcm

---

From: JFCM <[email-masked]>
Date: Tue, 13 Feb 2018 08:34:04 +0100

At 12:41 09/02/2018, Alexey Melnikov wrote:
>I haven't digested your whole email yet, but a very quick comment 
>about keywords:
>
>>    * string of 1–255 characters in the ASCII subset %x21ââ€"%x7e 
>> (excludes control chars and space), and MUST NOT include any of 
>> these characters: ( ) { ] % * " \

I concur for another reason: internationalization. If we name 
"polynyms" strict computed semantic synonym strings in different 
character sets that support langague diversity. ASCII lowercases turn 
out being a common subset for universal reference (hence English 
being used by the IETF, cf. RFC 3935), through internationalization 
standardization. The main barrier to polynymy is upper cases that 
many languages/charsets do not have. I also note that most of the 
world's users are accustomed to Windows paltforms which are case insensitive.
jfcm