Subject: [Uta] Updated TLSRPT

From: "Brotman, Alexander" <[email-masked]>
Date: Fri, 26 Jan 2018 14:41:35 +0000

Hello,

We uploaded a new version of the TLSRPT a few moments ago.  The majority of the changes are meant to address comments submitted by Viktor Dukhovni (https://www.ietf.org/mail-archive/web/uta/current/msg02388.html).  A rough list of changes:

- Update to mime type
- Add dnssec-required
- TLSA record change
- Change scale phrasing
- Make source-of-truth language more clear
- Change label vs labels
- Add IP to report

Please let us know if you have any further comments.  Thank you.

--
Alex Brotman
Sr. Engineer, Anti-Abuse
Comcast

---

From: Viktor Dukhovni <[email-masked]>
Date: Mon, 29 Jan 2018 00:16:16 -0500

> On Jan 26, 2018, at 9:41 AM, Brotman, Alexander <[email-masked]> wrote:
> 

Thanks.  Progress on most of the issues, but some nits remain or were
introduced (especially the MIME changes are not right yet):

Section 4.4, page 11: example policy string:

   o  "policy-string": A string representation of the policy, whether
      TLSA record ([RFC6698] section 2.3) or MTA-STS policy.  Examples:
      TLSA: ""_25._tcp.mx.example.com. 3 0 1 \ 1F850A337E6DB9C609C522D13
      6A475638CC43E1ED424F8EEC8513D747D1D085D)"" MTA-STS: ""version:
      STSv1\nmode: report\nmx: mx1.example.com\nmx: \
      mx2.example.com\nmx: mx.backup-example.com\nmax_age: 12345678""

This was incompletely updated and the TLSA example is rather mangled.
This element lacks a proper grammar.  The intention seems to be a string
of newline separated elements (I'd prefer a JSON array of strings).
Please provide a better example, and ideally some text describing the
syntax in a bit more detail.

Section 5.1, page 13: file format recommendation:

This section provides an ABNF grammar for an element of the report
that is merely a convenience for when reports arrive in mailbox
read by a human postmaster who saves the attached reports manually.

Such a specific prescription is unnecessary.  Better text would just
indicate that each report's filename should ideally identify the
origin of the report and the policy domain plus a unique report id.

The section ends with:

   For example, this
   is a possible filename for the gzip file of a report to the Policy
   Domain "example.net" from the Sending MTA "mail.sender.example.com":

   "mail.sender.example.com!example.net!1470013207!1470186007!001.json"

But, if the file is indeed compressed, a more friendly file extension
would be ".json.gz" not ".json".

The new version then confuses the MIME contexts of HTTPS and Email.
It is in HTTPS where "Content-Encoding" is supported, and so no
separate MIME type or MIME type parameter is needed to signal
compression.  While Email has no support for "Content-Encoding"
and so it is there a "conversions = gzip" parameter could be
added to the Content-Type.

In either case, there'd be no need for the "application/tlsrpt+gzip"
media type defined in Section 6.4.

Bottom line it looks like the MIME changes were made in haste, and
are incomplete.  Please take a closer look at these and decide whether
there's to be a single media type with compression signalled separately,
via Content-Encoding for HTTPS and a new "conversions" parameter (clearly
defined with the proper context) for email.

Please be less prescriptive about the email filename, it is not essential
to interoperability.  If the draft is much less prescriptive about the
format of the message subject or attachment filename, there's probably
no need for section 5.6, as there is then no possibility to specify
conflicting metadata in what are essentially free-form courtesy fields
for human consumption.

-- 
	Viktor.

---

From: "Valery Smyslov" <[email-masked]>
Date: Mon, 29 Jan 2018 10:35:25 +0300

Hi Viktor,

many thanks to you for the deep review and thanks to authors for the update.

> > We uploaded a new version of the TLSRPT a few moments ago.  The majority of the changes are meant to
> address comments submitted by Viktor Dukhovni (https://www.ietf.org/mail-
> archive/web/uta/current/msg02388.html).  A rough list of changes:
> >
> > - Update to mime type
> > - Add dnssec-required
> > - TLSA record change
> > - Change scale phrasing

We are on the late stage of document processing, so, please 
provide a concrete wording that you are happy with.

> origin of the report and the policy domain plus a unique report id.

Again, your suggestion? Remove the ABNF? Or just change RECOMMENDED to MAY 
(or to plain "may")? Please, be more specific.

> would be ".json.gz" not ".json".

I tend to agree with you here. What authors think?

> defined with the proper context) for email.

Again, please provide the text. Otherwise the iterations 
update-review may last forever.

Regards,
Valery.

> _______________________________________________
> Uta mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/uta

---

From: Viktor Dukhovni <[email-masked]>
Date: Mon, 29 Jan 2018 10:46:17 -0500

> On Jan 29, 2018, at 2:35 AM, Valery Smyslov <[email-masked]> wrote:
> 

It is unclear from the current state of the document whether
my suggestion to have just one MIME type with compression
signalled separately was deemed acceptable or not.

Unless others feel strongly otherwise, I think the document
should have just one MIME type, with compression indicated
via "Content-Encoding" for HTTP and:

  Content-Type: application/tlsrpt+json; conversions=gzip

for email.  With "conversions" defined in the document in
section 6.3, optional parameters.  And section 6.4 and
all reference to the additional gzip-specific MIME type
deleted.

I am not a fan of a separate MIME type, just for compression,
when there is just a single underlying entity type.

I'll have to leave fixing the MIME text to the document editors.
Too cycle-starved.  Sorry about that.

-- 
	Viktor.

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 29 Jan 2018 15:51:33 +0000

Viktor,

On 29/01/2018 15:46, Viktor Dukhovni wrote:
> for email.
What you propose above doesn't work for email, because this is no longer 
JSON, so can't be decoded by a generic JSON parser. What you are 
proposing is a new Content-Transfer-Encoding, which has absolutely 0 
chance of being deployed in email.
> With "conversions" defined in the document in
While I have some sympathy to your argument, it is not going to work for 
email (It could have worked for HTTP).

Best Regards,
Alexey

---

From: Viktor Dukhovni <[email-masked]>
Date: Mon, 29 Jan 2018 11:17:21 -0500

> On Jan 29, 2018, at 10:51 AM, Alexey Melnikov <[email-masked]> wrote:
> 
> JSON, so can't be decoded by a generic JSON parser. What you are proposing
> is a new Content-Transfer-Encoding, which has absolutely 0 chance of being
> deployed in email.

Note, compression in this context is NOT a transfer encoding, it cannot be
added by MTAs en route (as is the case with Base64 or Quoted-Printable)
nor is it an encoding applied at submission time by the MUA.  This is
the original Content-Encoding associated with the payload.  So I am NOT
proposing a new Content-Transfer-Encoding.  Rather, I am proposing a way
of tagging a new MIME type as having its content optionally compressed
prior to (not during) transmission.

Are you sure your objection is a barrier to adoption? The proposed MIME
type defines an optional "conversions" parameter, which takes the value
"gzip" to mean that the content is compressed.  Any implementation of
this MIME type would need to decompress the payload before reading it as
JSON, just as it would if the content type were "application/tlsrpt+gzip"
from section 6.4.  That too can't be read (directly) by a JSON parser,
without first decompressing the content.

I don't yet agree with your objection.  Why have two essentially identical
MIME types, when one will do?  What's wrong with using a "conversions"
parameter to indicate compression?  (There's precedent for this, as
indicated upthread in my message from Dec 7th, quoted below my signature)

Either way, the present draft version has partially adopted my suggestion.
It should either be fully adopted, or the relevant changes reverted.

-- 
	Viktor.

Quoting my post of Dec 7th:

The issue is perhaps that while "HTTP" has "Content-Encoding",
MIME as defined in RFC-2045 does not:

   https://www.ucolick.org/~sla/fits/mime/inetstds.html

      MIME media types and the WWW
      ...
      Section 14.11 gives the proper means of describing the HTTP
      transmission of a file which is already compressed

      The Content-Encoding entity-header field is used as a modifier
      to the media-type. When present, its value indicates what
      additional content codings have been applied to the entity-body,
      and thus what decoding mechanisms must be applied in order to
      obtain the media-type referenced by the Content-Type header field.
      Content-Encoding is primarily used to allow a document to be
      compressed without losing the identity of its underlying media
      type. Unfortunately it seems reasonably clear that Content-Encoding
      cannot be applied as if it were a MIME "Optional parameter" with the
      expectation that it could be used when transferring compressed files
      via e-mail. This observation is based upon section 19.4.4. It appears
      that a complete description of compressed files transmitted via e-mail
      requires a new RFC enhancing the MIME standard.

   https://www.w3.org/Protocols/rfc2616/rfc2616-sec19.html#sec19.4.4

       19.4.4 Introduction of Content-Encoding

       RFC 2045 does not include any concept equivalent to HTTP/1.1's
       Content-Encoding header field. Since this acts as a modifier on
       the media type, proxies and gateways from HTTP to MIME-compliant
       protocols MUST either change the value of the Content-Type header
       field or decode the entity-body before forwarding the message.
       (Some experimental applications of Content-Type for Internet mail
       have used a media-type parameter of ";conversions=<content-coding>"
       to perform a function equivalent to Content-Encoding. However, this
       parameter is not part of RFC 2045.)

Thus, at the very least for HTTP, there should be just one media type,
and Content-Encoding should be used to signal compression.  For email,
since we're introducing a new "multipart/report" subtype, and "Content-Encoding"
is not a standard MIME header, an appropriate media-type parameter (what
I earlier referred to as MIME "attribute") should be defined for
"application/tlsrpt".  It could be the above "; conversions=gzip" or
something similar.  The "application/tlsrpt" being a new media type,
you're free to define appropriate additional metadata.

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 29 Jan 2018 16:36:29 +0000

> prior to (not during) transmission.
What you are proposing is not a part of MIME. You can't introduce 
Content-Encoding header field in email and remaing backward compatible 
with existing code. The time for this proposal was unfortunately 20 
years ago.
> "gzip" to mean that the content is compressed.
MIME handling need to operate only on MIME type without optional 
parameters being available. This is limitation of some existing systems, 
for example Windows.
>    Any implementation of
You are proposing non backward compatible change. An application that 
doesn't understand your "conversions" parameters wouldn't be able to 
decode and process application/tlsrpt+json. "+json" suffix means that 
the payload is JSON and not some other format.

The only way to make it work for email is to introduce a new MIME suffix 
which will mean "this is either JSON or gzip of JSON" and let 
applications guess whether something is gzip or not. For 
example"+jgzip". This suffix will need to be registered with IANA and 
has to be reviewed by Ned Freed (as the Designated Expert).
> MIME types, when one will do?
Because your proposal doesn't work.
> What's wrong with using a "conversions"
Best Regards,
Alexey

---

From: Viktor Dukhovni <[email-masked]>
Date: Mon, 29 Jan 2018 12:01:23 -0500

> On Jan 29, 2018, at 11:36 AM, Alexey Melnikov <[email-masked]> wrote:
> 
> What you are proposing is not a part of MIME. You can't introduce Content-Encoding header field in email and remaing backward compatible with existing code. The time for this proposal was unfortunately 20 years ago.
> MIME handling need to operate only on MIME type without optional parameters being
> available.

In practice, or in principle based on the standard?  Surely MIME content-type
parameters are there for a reason, and can and do affect the interpretation of
various MIME types.  For example "text/plain" has an optional "format = flowed".

> This is limitation of some existing systems, for example Windows.

This is not actually relevant to Windows, since compressed TLSRPT attachments
will have a ".json.gz" suffix, and when they are saved as files (no MUA will
display compressed JSON inline for the user to read).  Once stored, the MIME
type will be inferred by Windows from the file extension, and everything will
work as expected.

This MIME type is primarily for machine processing with the aggregated results
from multiple sources available to users to be viewed via a suitable UI.  Even
if it sometimes fails to display natively in email, it is not material, so long
as the content remains machine readable.  Email messages with TLSRPT payloads
will be read primarily by bots, not people, and the bots could/would handle
a "conversions" parameter just fine.

> You are proposing non backward compatible change.

No, this is a new MIME type, there is no legacy.  The proposed parameter
applies *only* to this type.  I am not proposing any sort of universal
retrofit of gzip encoding for legacy MIME types.

> An application that doesn't understand your "conversions" parameters
> wouldn't be able to decode and process application/tlsrpt+json. "+json"
> suffix means that the payload is JSON and not some other format.

No such applications presently exist.  This is a new media type.
If this specification requires support for "conversions", then
the applications that parse this new media type will have to
support "conversions".

> which will mean "this is either JSON or gzip of JSON" and let applications
> guess whether something is gzip or not. For example"+jgzip". This suffix
> will need to be registered with IANA and has to be reviewed by Ned Freed
> (as the Designated Expert).

I'd agree with you, if I were trying to retrofit "conversions" on existing
media types.  But that's not the case.  I'd be curious to hear what Ned or
has to say on the subject.

> Because your proposal doesn't work.

I don't think the object is valid.  The proposal does not change
email MIME in an incompatible way.  Just another new media type, with
a new optional parameter.  If media types are not able to add parameters,
then why bother with parameters?

RFC2045 says:

   Parameters are modifiers of the media subtype, and as such do not
   fundamentally affect the nature of the content.  The set of
   meaningful parameters depends on the media type and subtype.  Most
   parameters are associated with a single specific subtype.  However, a
   given top-level media type may define parameters which are applicable
   to any subtype of that type.  Parameters may be required by their
   defining content type or subtype or they may be optional. MIME
   implementations must ignore any parameters whose names they do not
   recognize.

I think that matches this situation quite well, the content is a
JSON TLSRPT, that may be compressed if the conversions=gzip parameter
is present.

All this said, I will accept two media types if need be, the important
thing is that draft now needs some editing to make it conform to one
view or the other, not a chimera.

-- 
	Viktor.

---

From: Keith Moore <[email-masked]>
Date: Wed, 31 Jan 2018 05:46:01 -0500

On Jan 29, 2018, at 10:51 AM, Alexey Melnikov wrote:

> What you propose above doesn't work for email, because this is no longer JSON, so can't be decoded by a generic JSON parser. What you are proposing is a new Content-Transfer-Encoding, which has absolutely 0 chance of being deployed in email.

Alexey is 100% correct.

The whole point of the "+json" convention is to let email processors know that they can parse the attachment as JSON without having to know more about the rest of the content.   So existing mail processors (including spam and virus filters) might well try to parse the gzipped content as JSON and fail because it's not JSON, and either delete the attachment, bounce the message, or declare it unreadable.

Even if you omit the "+json" suffix from the content-type name, most mail readers essentially ignore the other parameters to the content-type header when deciding how to interpret the attachment (except in a small number of special cases), and don't pass those parameters to whatever application interprets the content.   This is unfortunate and contrary to the design of MIME, but it's also been the behavior of the vast majority of implementations practice for the entire history of MIME deployment.   It was relatively easy for mail readers to lookup content-type in a table and determine what application to launch to pass the content to, but there was no obvious mechanism that could be used to pass the content-type parameters to that application that didn't also allow the sender of a message to invoke existing applications in dangerous ways.

Keith

---

From: Viktor Dukhovni <[email-masked]>
Date: Wed, 31 Jan 2018 08:43:24 -0500

> On Jan 31, 2018, at 5:46 AM, Keith Moore <[email-masked]> wrote:
> 

Fine, but could you explain, so I understand going forward, what
existing mail processors you have in mind that understand the
application/tlsrpt+json media type, or even expect it to be JSON?

Or generically understand a "+json" suffix (there's no special
semantics for the "+" character and such a media type suffix in
RFC2045).

What useful processing can they do with such JSON, without
understanding the schema?

If, as I expect, processing this media type in any meaningful
way requires new code, surely said new code can equally make
note of an optional "conversions" parameter associated with the media
type.

I accept that I am not "winning" on this point, but I am entirely
mystified as to why.  There is no legacy here, and nothing useful
for general-purpose MUAs to display to users.  The payload is a
file attachment that will be filed away for machine processing.
The media type could be:

	application/tlsrpt+json [ ; conversions=gzip ]

or it could be

	application/tlsrpt+{json,gzip}

And, I rather expect that in order to avoid problems with
line breaks and the like in email transport, the attachment
will *always* be compressed for transmission in email.  Indeed
that should be strongly recommended if not outright required
by this specification.

-- 
	Viktor.

---

From: James Cloos <[email-masked]>
Date: Wed, 31 Jan 2018 14:25:57 -0500

FWIW, I tried sending myself a gzip(1)ed json file using:

It worked fine.  And gnus even gunzip(1)ed it for me when
I asked it to inline the attachment.

This was sent to another system which has a .forward file
pointing to my main address.  So it went through at least
two instances of virus/trojan scanning w/o issue.

(The file in question was not a tlsrpt, but it /was/ well-
formed json, and manually compressed with gzip(1).)

So Viktor's suggestion works fine here, at least.

Perhaps a wider test would be in order?

-JimC
-- 
James Cloos <[email-masked]>         OpenPGP: 0x997A9F17ED7DAEA6

---

From: Keith Moore <[email-masked]>
Date: Wed, 31 Jan 2018 14:44:25 -0500

No.  You don't violate 27 years of practice based on any number of tests that this WG is capable of doing, just to avoid defining two content-types.

Sent from my iPhone

> On Jan 31, 2018, at 2:25 PM, James Cloos <[email-masked]> wrote:
>

---

From: Viktor Dukhovni <[email-masked]>
Date: Wed, 31 Jan 2018 14:58:55 -0500

> On Jan 31, 2018, at 2:44 PM, Keith Moore <[email-masked]> wrote:
> 
> You don't violate 27 years of practice based on any number of tests that this WG is capable of doing, just to avoid defining two content-types.

I remain quite puzzled as to the nature of the violation.
All I see is a new media type that takes an optional parameter
which specifies that the content is gzip compressed.

The new media type has no legacy consumers, and is not suitable
for inline display by MUAs.  The data is intended for aggregation
and machine analysis alongside reports from other senders.

I fail to see any interoperability concern.  Perhaps, I'm missing
something.  Please explain.

-- 
-- 
	Viktor.

---

From: [email-masked]
Date: Wed, 31 Jan 2018 12:00:54 -0800 (PST)

> No.  You don't violate 27 years of practice based on any number of tests that
> this WG is capable of doing, just to avoid defining two content-types.

Exactly right. We couldn't possibly perform a meaningful test of this sort.

And even if we could, and if the results showed there were no problems
currently, nothing prevents someone from writing new software based on
guarantees the standards provide in regards to the meaning of media type
suffixes.

				Ned

---

From: [email-masked]
Date: Wed, 31 Jan 2018 12:12:41 -0800 (PST)

> something.  Please explain.

The +json suffix is defined to mean that the content consists of JSON,
and can be parsed as such, independent of any understanding of the
rest of the media type name or its parameters.

So unless you have a time machine and can go back and change the rules defining
+json, structured syntax suffixes, content type parameters, etc. this is a
nonstarter.

				Ned

---

From: Viktor Dukhovni <[email-masked]>
Date: Wed, 31 Jan 2018 15:52:10 -0500

> On Jan 31, 2018, at 3:12 PM, [email-masked] wrote:
> 

Ah finally, the right keywords to help me find the relevant document:

   https://tools.ietf.org/html/rfc6838#section-4.2.8
   https://tools.ietf.org/html/rfc6839#section-3.1

I have never before run into RFC6839.  So "+json" is special after all.

I don't see a "+gzip" in the relevant IANA registry:

  https://www.iana.org/assignments/media-type-structured-suffix/media-type-structured-suffix.xhtml

It seems that is that's to be used, then this draft must also register "+gzip"
with IANA.

The alternative is to forgo the explicit "+json" and "+gzip" structured syntax
suffixes entirely and stick with application/tlsrpt (with optional "conversions"),
because, for example, the compressed document is actually still ultimately JSON,
so it is not clear why we elide that fact when compressing, if it is useful when
not compressing?

It does not look like multiple suffixes are allowed, such as:

	application/tlsrpt+json+gzip

And since the content should almost always be compressed in email
(to make sure the contend is not broken by in-transit line folding,
or is too large), the "+json" tag is will be rarely present, while
"+gzip" adds little information.

Indeed tlsrpt already implies "JSON" anyway.  So "+json" is redundant.

Bottom line, I guess I don't care anymore.  The editors can just
fix the draft to remove the partial use of the "conversions" approach.

-- 
	Viktor.

---

From: Keith Moore <[email-masked]>
Date: Wed, 31 Jan 2018 17:24:34 -0500

IMO if you want to define application/tlsrpt in such a way that readers/processors can reliably distinguish between clear text JSON and gzipped JSON , say by looking at the first few bytes of the content for the header magic number indicating gzip compression, you're free to do so.  You just can't name the content-type something+gzip, nor expect readers to parse a parameter to tell them how to interpret it.

Sent from my iPhone

> On Jan 31, 2018, at 3:52 PM, Viktor Dukhovni <[email-masked]> wrote:
> 
> 
>

---

From: Viktor Dukhovni <[email-masked]>
Date: Wed, 31 Jan 2018 17:44:00 -0500

> On Jan 31, 2018, at 5:24 PM, Keith Moore <[email-masked]> wrote:
> 
> You just can't name the content-type something+gzip...

Not sure I understood the above correctly.  That's exactly what this draft is doing:

  https://tools.ietf.org/html/draft-ietf-uta-smtp-tlsrpt-13#section-6.4

-- 
	Viktor.