Subject: Re: [Anima] Content-Transfer-Encoding and HTTP 1.x in ANIMA BRSKI

From: Michael Richardson <[email-masked]>
Date: Wed, 12 Jun 2019 11:01:58 -0400

{resending, now that I'm subscribed to ietf-http-wg.  If you aren't
on that list, the IETF global white list won't help you}

RFC7030 (Enrollment over Secure Transport) includes language like (section
4.1.3): https://tools.ietf.org/html/rfc7030#section-4.1.3

   A successful response MUST be a certs-only CMC Simple PKI Response,
   as defined in [RFC5272], containing the certificates described in the
   following paragraph.  The HTTP content-type of
   "application/pkcs7-mime" is used.  The Simple PKI Response is sent
   with a Content-Transfer-Encoding of "base64" [RFC2045].

draft-ietf-anima-bootstrap-keyinfra (now in IETF Last Call), extends EST.
It creates a few more end points, and transfers RFC8366 format artifacts
over those end points.  RFC8366 defines them to be CMS signed objects,
in DER (not PEM) format. Another document in ANIMA models the artifacts
as COSE signed CBOR (also CMS signed CBOR).  All binary objects.

In doing interop testing we had some surprises about whether we should see
base64 encoding of objects "on-the-wire".

Some implementations have what I consider to be typical HTTP client and
server code.  The application sticks binary in, an appropriate
Content-Transfer-Encoding is added, and the binary is adapted.  On the
client, if there is an encoding, it is removed, and the client sees binary
plus a Content-Type.

Other implementations were doing something more optimal: observing the
base64, but noticing that there is only one possible Content-Type, and
the Content-Transfer-Encoding is implicit, and so emitting base64 with an
implicit text/plain content type.

In addition, people make mistakes, and the desire to write test cases with
curl --data (vs --data-binary) easily has led some of us astray at times.

While we think that constrained devices should speak the constrained protocol
(see below), in some cases code-constrained devices speak HTTPS, and
wish to do away with any base64 layer of encoding, as a naive use of it
can come with unknown memory requirements.

Some questions:

1) Is Content-Transfer-Encoding even valid in HTTP1.x?
   RFC2616 and RFC7230
   speak about Transfer-Encoding, and this relates to Chunked or not.
   https://tools.ietf.org/html/rfc2616#section-14.41

   https://tools.ietf.org/html/rfc2616#section-19.4.5 says:
   HTTP does not use the Content-Transfer-Encoding (CTE) field of RFC
   2045. Proxies and gateways from MIME-compliant protocols to HTTP MUST
   remove any non-identity CTE ("quoted-printable" or "base64") encoding
   prior to delivering the response message to an HTTP client.

   RFC7230 does not include the above text making CTE unwanted.
   This made it rather hard to track down the truth :-)

2) Assuming the answer to (1) is no, what should we make of RFC7030
   that says to use it, and to base64 binary objects?

   Would it be reasonable to assume that this is an error, to
   permit an absent (or CTE: Binary) to mean binary for RFC7030?
   There is clearly an interoperability issue here if existing
   implementations do not understand this.

   Did we miss a cross-area review, or did we do this on purpose?
   RFC7030 is terribly repetitive about CTE suggesting it needed
   to hit people over the head with a hammer.

3) What should draft-ietf-anima-bootstrapping-keyinfra do going forward?
   We make use of RFC7030 functionality, and after bootstrap, we can't
   be sure that the EST server we use for certificate renewal is the
   same (brand of) EST server as before, so just because we did BRSKI,
   doesn't mean we can assume a binary version of 7030.

   Should BRSKI end-points:
    a) omit CTE, and assume binary.
    b) permit CTE, suggest it say, "binary"
    c) permit CTE, support "binary" or "base64"
    d) permit CTE, support "base64" only (like RFC7030).

Some implementors received pressure within to reduce the number of bytes on
the wire significantly.  The use of implicit Content-Type and CTE was
a result, rather than saving 25% by going base64->binary.

Constrained users are directed to draft-ietf-ace-est-coaps (replaces 7030),
and draft-ietf-anima-constrained-voucher (replaces 8366, and BRSKI),
which uses CBOR, COSE and COAP.

Note that the constrained-voucher version of BRSKI uses the standard
BRSKI-MASA protocol between registrar and MASA.  That path has no "legacy"
RFC7030 end points, and so should be modern HTTPS.  Because it transfers
both RFC8366 and the three formats of constrained voucher (all binary
artifacts), it relies on Content-Type: and Accept: in both directions to
signal the format present, and the desired format response.

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Julian Reschke <[email-masked]>
Date: Wed, 12 Jun 2019 19:26:49 +0200

>     This made it rather hard to track down the truth :-)

There is no Content-Transfer-Encoding header field in HTTP. It is simply
not needed.

>     that says to use it, and to base64 binary objects?

Raise an erratum :-).

>     to hit people over the head with a hammer.

I'm sure I haven't looked at that spec. Anybody from this WG should have
spotted this.

>      a) omit CTE, and assume binary.

Yes.

> ...

Best regards, Julian

---

From: Michael Richardson <[email-masked]>
Date: Wed, 12 Jun 2019 18:14:17 -0400

> Just as a matter of curiosity, what happened in HTTP1.1 to the fragment in
    > RFC2616 that says (under Content-MD5):

    > "The entity-body for composite
    > types MAY contain many body-parts, each with its own MIME and HTTP
    > headers (including Content-MD5, Content-Transfer-Encoding, and
    > Content-Encoding headers)."

yes, I was going to bring this up next :-)

    > This seems to be a source of confusion, e.g.
    > https://stackoverflow.com/questions/5169434/content-transfer-encoding-in-file-uploading-request
    > . RFC7030 uses a content type of application/pkcs7-mime. So is it
    > allowed to specify a MIME header?

I also found that while googling.

I went through the effort of doing a multipart *reply* in HTTP for
draft-ietf-anima-constrained-voucher, and I really found it hard to determine
what the MIME rules for *HTTP* were...

---

From: Julian Reschke <[email-masked]>
Date: Thu, 13 Jun 2019 08:19:09 +0200

> yes, I was going to bring this up next :-)

That's about header fields in payloads of type multipart/*. The text is
gone in later specs, because Content-MD5 is.

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 13 Jun 2019 08:27:55 +0200

>>    that says to use it, and to base64 binary objects?

Not sure about the smiley here.  This is an erratum.
Now how to resolve this depends on what option causes the smallest damage.

RFC 7030 is supposed to be a widely deployed protocol, so it should not be hard to find out what people out there are actually doing.

Grüße, Carsten

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 13 Jun 2019 19:38:20 +1200

On 13/06/19 6:27 pm, Carsten Bormann wrote:
> On Jun 12, 2019, at 19:26, Julian Reschke wrote:
>>

IME the desired behaviour is for the agent mapping into HTTP(S) to
decide between three actions, in order of preference:

 1) remove the encoding and send binary of the resources native
Content-Type.

 2) if all agents end-to-end are wanted to retain the same encoding -
AND the final recipient is known able to decode; then C-T-E can be
mapped to HTTP's Content-Encoding header and the body sent as-is.

 3) if the next hop (only) is expected to be able to decode, but other
hops uncertain; then C-T-E can be mapped to HTTP Transfer-Encoding
header and the body sent as-is.
 * T-E is a mechanism without widespread support. So this option usually
turns out to be not available at the HTTP next-hop.

Amos

---

From: Eliot Lear <[email-masked]>
Date: Thu, 13 Jun 2019 13:20:12 +0200

I am looking at libest and it certainly generates the header.

> On 13 Jun 2019, at 08:27, Carsten Bormann <[email-masked]> wrote:
> 
> _______________________________________________
> Anima mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/anima

---

From: Michael Richardson <[email-masked]>
Date: Thu, 13 Jun 2019 10:29:08 -0400

> I am looking at libest and it certainly generates the header.

How does it react to the absense of the header?
(or the header containing "binary")
Does it process binary directly in that case?

---

From: Michael Richardson <[email-masked]>
Date: Mon, 17 Jun 2019 12:20:17 -0400

> Now, I don’t know how other EST clients would act. There are many out
> there by now that we can’t safely tell if they would act up.
> The commercial and enterprise CAs I tested with interoped fine with
> the libest client and they were not all sending the CTE field. They
> payload was base64 though.

I didn't read this well enough before.

So effectively, the CTE header has effectively been dropped, but the payload
is now assumed to be base64, regardless.

This suggests that we can not use the CTE header as a signal.
One has to assume base64 encoded values for the RFC7030 end-points.

---

From: Julian Reschke <[email-masked]>
Date: Tue, 18 Jun 2019 06:45:53 +0200

On 17.06.2019 22:44, Brian E Carpenter wrote:
> On 18-Jun-19 05:18, Panos Kampanakis (pkampana) wrote:
>>> So effectively, the CTE header has effectively been dropped, but the payload is now assumed to be base64, regardless.
> I went and looked at RFC4648 for my own education, and then spent a few minutes
> trying to design a Turing machine that can distinguish a binary bit string from
> a base64 bit string. Fail. You can determine that a bit string is definitely
> not base64 if it contains at least one character outside the base64 alphabet,
> but not the converse. So it needs a signal. Not having a signal would be wide
> open to malicious misuse, IMHO. Indicating the length of the payload would be
> enough, I think.

I agree that using heuristics should be avoided. But how does
Content-Length help here?

Best regards, Julian

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 13 Jun 2019 20:33:59 +0000

On 14-Jun-19 05:18, Panos Kampanakis (pkampana) wrote:
> The libest server or proxy will generate the CTE header as specified in RFC7030. The libest client will parse it, but it will not reject the response if the header is not there. It expects base64 encoded PKCS#7, not binary though. Note that in _https://datatracker.ietf.org/doc/draft-ietf-ace-coap-est/_ we assume all cert payloads are binary.
>  
> Now, I don’t know how other EST clients would act. There are many out there by now that we can’t safely tell if they would act up.
>  
> The commercial and enterprise CAs I tested with interoped fine with the libest client and they were not all sending the CTE field. They payload was base64 though.
>  
> To address the erratum, I would lean towards a recommendation against using the CTE header based on the referenced standards and state that base64 encoding is implied.

https://tools.ietf.org/html/rfc7231#appendix-A from June 2014 makes it all very plain. However, there is a small problem of running code.

There's already an erratum: https://www.rfc-editor.org/errata/eid5107

For whatever reason, it is sitting in state "reported" since 2017.

   Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 12 Jun 2019 20:50:28 +0000

Hi Julian,

This seems to be a source of confusion, e.g.
https://stackoverflow.com/questions/5169434/content-transfer-encoding-in-file-uploading-request <https://stackoverflow.com/questions/5169434/content-transfer-encoding-in-file-uploading-request> . RFC7030 uses a content type of application/pkcs7-mime. So is it allowed to specify a MIME header?

   Brian

> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/anima <https://www.ietf.org/mailman/listinfo/anima>

---

From: Michael Richardson <[email-masked]>
Date: Tue, 11 Jun 2019 17:47:37 +0000

{a little bit surprised that the IETF HTTP list is on w3c. Makes sense
From a cross-effort point of view.  I guess the IETF global white list might
not work for me}

Some questions:

  Should BRSKI end-points:

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 17 Jun 2019 20:45:19 +0000

Brian

> 
> I am not sure I can speak about all implementations out there, but that is what I saw in my interop testing.
> 
> In all tests I did yes.
> -----Original Message-----
> From: Michael Richardson <[email-masked]>
> Sent: Monday, June 17, 2019 12:20 PM
> To: Panos Kampanakis (pkampana) <[email-masked]>
> Cc: Eliot Lear <[email-masked]>; Carsten Bormann <[email-masked]>; Julian Reschke <[email-masked]>; [email-masked]; [email-masked]; Anima WG <[email-masked]>
> Subject: Re: [Anima] Content-Transfer-Encoding and HTTP 1.x in ANIMA BRSKI
> 
> * PGP Signed by an unknown key
> 
> 
> Michael Richardson <[email-masked]>, Sandelman Software Works  -= IPv6 IoT consulting =-
> 
> 
> 
> 
> * Unknown Key
> * 0xFDFC4290