Subject: [Acme] draft-ietf-acme-acme-06: Don't call it PEM Certificate Chain

From: Sean Leonard <[email-masked]>
Date: Wed, 29 Mar 2017 14:32:17 -0500

Hello,

I reviewed draft-ietf-acme-acme-06. As the co-author of RFC 7468, I have concerns.

First of all as an editorial matter there is no such thing as “PEM encoding”--that term is not defined in RFC 7468. The correct term is “textual encoding”. PEM is dead: just because some popular tools (i.e., OpenSSL) call it PEM doesn’t mean that it’s PEM, which is a mail format from RFC 1421. Therefore the text in Section 9.1.: “File should contain one or more certificates encoded as PEM according to RFC 7468.” is wrong because RFC 7468 doesn’t define “encoded as PEM”.

Second of all, I see negative value in transmitting certificates in the proposed “PEM Certificate Chain” format (Section 7.4.2., Section 9.1). Again, I understand that some popular tools (i.e., OpenSSL) dump out this format by default, but one command-line switch or pipe can change it to a much better format. Don’t be lazy.

A bunch of textually encoded certificates are *text* data, and text is in a charset, so something like text/plain; charset=us-ascii is appropriate. Registering application/pem-certificate-chain is counterproductive because it omits the charset (and if you really want to omit the charset, then Section 9.1 is wrong because it needs to say “US-ASCII ONLY” or some such).

A bunch of certificates as text data can include arbitrary text in between the encapsulation boundaries, such as printouts of the certificates (also OpenSSL options). This is superfluous and in my opinion dangerous. If you want such arbitrary text as a design matter, maybe for debugging purposes, that’s fine but that’s all the more reason why the contents should be text/* contents rather than application/* contents.

It’s a lot cleaner to emit certificates in a blob as application/pkcs7-mime; smime-type=certs-only. This is also a trivial OpenSSL option. If you want to ASCII-armor it, you can put it in a -----BEGIN CMS-----, -----BEGIN PKCS7----- , or if you really want, -----BEGIN CERTIFICATE CHAIN----- (see Section 8 of RFC 7468). But you gain 37% size reduction by letting it be binary CMS/binary PKCS #7 over HTTP, and the certificate processing libraries can deal with the sorting issues. If it’s so important to put the subject certificate first, there are several other format choices to deal with that (including a multipart response).

I don’t see value in specifying an order to the certificates in the text: “Each following certificate SHOULD directly certify one preceding it.” Again I know what the tools out there do, but a robust receiver still needs to handle out-of-order certificates, as well as certificates that end up not being part of the chain. CMS/PKCS #7 deals with this by just supplying an arbitrary set of certificates.

Thank you,

Sean

---

From: Ilari Liusvaara <[email-masked]>
Date: Wed, 29 Mar 2017 22:54:27 +0300

On Wed, Mar 29, 2017 at 02:32:17PM -0500, Sean Leonard wrote:
> Hello,
> 
> Second of all, I see negative value in transmitting certificates in
> the proposed “PEM Certificate Chain” format (Section 7.4.2., Section
> 9.1). Again, I understand that some popular tools (i.e., OpenSSL)
> dump out this format by default, but one command-line switch or
> pipe can change it to a much better format. Don’t be lazy.

AFAIK, the PEM format is intended for transporting the End-Entity
and intermediate certificates at once, as incomplete chains are
extremely common.

> I don’t see value in specifying an order to the certificates
> in the text: “Each following certificate SHOULD directly certify
> one preceding it.” Again I know what the tools out there do,
> but a robust receiver still needs to handle out-of-order
> certificates, as well as certificates that end up not being
> part of the chain. CMS/PKCS #7 deals with this by just supplying
> an arbitrary set of certificates.

TLS 1.2 has a MUST for ordering certificates.

And nothing that doesn't build paths itself (and thus can fetch
certiifcates itself) handles CMS/PKCS#7. This is way too much logic
to expect from ACME client.

And handling unordered certificates is quite a bit of extra attack
surface. I took not just one but two vulns on that (one being auth
bypass, which barring type safety bugs are just about the worst that
can happen).

-Ilari

---

From: Sean Leonard <[email-masked]>
Date: Wed, 29 Mar 2017 15:48:02 -0500

> On Mar 29, 2017, at 2:54 PM, Ilari Liusvaara <[email-masked]> wrote:
> 
[…]
> TLS 1.2 has a MUST for ordering certificates.

Yes, TLS expects ordering. But this statement is inconsistent with the statement above: “transporting the End-Entity and intermediate certificates at once, as incomplete chains are extremely common.”

My point is that if you have a valid chain of 5 certificates:

SUBJECT
 -> intermediate A
  -> intermediate B
   -> intermediate C
    -> ROOT CA

The first statement, “incomplete chains are extremely common”, means that you are expecting a transmission that is missing, e.g., intermediate B, therefore:
SUBJECT
intermediate A
intermediate C
ROOT CA

This example is not “ordered” according to TLS 1.2, because RFC 5246 says: “Each following certificate MUST directly certify the one preceding it.”

My point is that even though {SUBJECT, A, C, ROOT} is in an order, it is not TLS 1.2-ordered, therefore it is “unordered”. So, a receiver needs to deal with this unordered situation under the current draft.

Unless ACME expects all senders to transmit TLS 1.2-ordered certificates in this part of the protocol? If so, the text needs to say that, as in “MUST”, not “SHOULD”.

> to expect from ACME client.

Too many double-negatives for me to parse. Please clarify, thanks.

> bypass, which barring type safety bugs are just about the worst that

If you are saying that the receiver is only expected to handle TLS 1.2-ordered certificates: “Each following certificate MUST directly certify the one preceding it” (MUST, not SHOULD) then we have a different situation and PKCS #7/CMS certs-only may not be appropriate. But the text does not currently say that, so I need clarification before suggesting a better data format for the certificate chain.

Regards,

Sean

---

From: Jacob Hoffman-Andrews <[email-masked]>
Date: Wed, 29 Mar 2017 23:07:04 -0700

On 03/29/2017 01:48 PM, Sean Leonard wrote:
> If you are saying that the receiver is only expected to handle TLS
> 1.2-ordered certificates: “Each following certificate MUST directly
> certify the one preceding it” (MUST, not SHOULD) then we have a
> different situation and PKCS #7/CMS certs-only may not be appropriate.
> But the text does not currently say that, so I need clarification
> before suggesting a better data format for the certificate chain.
In general, the expectation is that the ACME client passes the
certificate chain straight to a web server or other TLS server.
Generally speaking, the TLS server will then pass along the chain in the
same order to the TLS client during the handshake. So I assume by
"receiver" you mean TLS client, right?

My understanding of TLS deployment is that, even though TLS 1.2, 1.1,
and 1.0 required strict ordering of certificates in the chain, deployed
clients were actually tolerant of out-of-order certificates. I haven't
followed TLS 1.3 standardization closely, but my guess is that that's
the reason the ordering requirement was relaxed.

So I think the answer to your question is: The receiver (TLS client) is
expected to handle TLS 1.3-ordered certificates, and our belief is that
that's reasonable given today's deployments.

---

From: Sean Leonard <[email-masked]>
Date: Thu, 30 Mar 2017 10:27:37 -0500

> On Mar 30, 2017, at 1:07 AM, Jacob Hoffman-Andrews <[email-masked]> wrote:
> 

Thank you. In the message above, the first sentence’s “receiver” was meant to be the receiver that is the HTTP client in the ACME protocol that issues the GET /acme/cert request, and receives back the certificate chain. Therefore the “receiver” is the ACME client, which also is the web/TLS server that incorporates the chain into its operations.

I spoke with Rich Salz in person about this yesterday, sorry that I probably won’t be at the ACME meeting today due to a scheduling conflict. But if your purpose is to communicate an ordered list of certificates in TLS 1.2 order (according to Section 7.4.2 of RFC 5246), I have a simple solution for you:

application/tls-certificate-list [or tls-cert-list]

Contents: DER-encoded certificates, concatenated in RFC 5246 Section 7.4.2 order

The end.

This is very simple and makes it easy for both sides to deal with the certificates. The first certificate is the end-entity/TLS subject certificate. The remaining certificates are supposed to chain to some root (use the appropriate RFC 5246 language). It is safe to concatenate binary (X.690/BER/DER) certificates because the first TLV indicates the full length of the certificate; therefore they are self-delimiting.

This is easier and better than stringing the textual encoded versions because you don’t have to deal with text metadata issues, charsets, line endings, and extra processing. Specifically, an ACME server is going to have to take the certificate bytes and base64-encode each one, followed by ASCII armoring. Then the ACME client is going to have to reverse this. No point and confusing.

For those who don’t like binary, ACME is JSON-based, so you can just have an application/json response that is just an array:

[“MIICAS...base64 of end-entity cert”,“MIICAS…base64 of second cert…”…to root]

This way you don’t have to register a media type and you can ingest the response using the same JSON stack. Also, you can encapsulate this very simple JSON structure in more articulated JSON entities if you want to include other metadata or whatever down the line.

Regards,

Sean

---

From: Jacob Hoffman-Andrews <[email-masked]>
Date: Thu, 30 Mar 2017 08:47:01 -0700

> Therefore the “receiver” is the ACME client, which also is the web/TLS
server that incorporates the chain into its operations.

Note that in common deployment today, the ACME client is separate from
the web server.

> Contents: DER-encoded certificates, concatenated in RFC 5246 Section
Is there a straightforward OpenSSL command to convert this entity into a
PEM file containing multiple certificates? I don't think there is but I
could be wrong.
> This is easier and better than stringing the textual encoded versions because you don’t have to deal with text metadata issues, charsets, line endings, and extra processing. Specifically, an ACME s…
Our expectation is that many ACME clients will *not* reverse the base64
and ASCII armoring, but will write out the entire contents to a file for
ingestion by a web server. The two most popular web servers, Apache and
Nginx, consume PEM-formatted certificate chains.

You're right that an ACME client would have to decode this format if
they wanted to change it to something else, but what we are seeing in
practice is clients that simply convert the end-entity certificate from
DER to PEM and don't bother with the intermediates, resulting in
end-user pain.

> [“MIICAS...base64 of end-entity cert”,“MIICAS…base64 of second cert…”…to root]
For me, "don't like binary" isn't the issue. If CMS were the most common
certificate chain input format for web servers, I would be fine with
returning binary CMS. Unfortunately, it is not.

---

From: Sean Leonard <[email-masked]>
Date: Thu, 30 Mar 2017 11:04:51 -0500

> On Mar 30, 2017, at 10:47 AM, Jacob Hoffman-Andrews <[email-masked]> wrote:
> 

Ok.

> PEM file containing multiple certificates? I don't think there is but I

It’s straightforward. I’ll research it later…if it’s not “straightforward enough” I can just write a script for you all to take.

> Nginx, consume PEM-formatted certificate chains.

Yeah, that’s a problem, you need to be #@($#@ sure that you are validating the response from the ACME client.

RFC 7468 discusses implementations considerations with multiple textually encoded crypto items (Section 2, Section 14). If you are ingesting a stream that has textually encoded items, you need to “hunt-and-pick” exactly what you are looking for, and validate the contents.

IN PARTICULAR: both Apache and Ngnix may be subject to a private key substitution attack with naive passing of the ACME response to the web server! See:
http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_certificate <http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_certificate>
http://httpd.apache.org/docs/current/mod/mod_ssl.html#sslcertificatefile <http://httpd.apache.org/docs/current/mod/mod_ssl.html#sslcertificatefile>

The SSL Certificate option includes the option of including the private key in the same input: “A secret key in the PEM format may be placed in the same file.” 

What this means is that a malicious ACME server can serve a certificate per the client’s request, but substitute the server’s specified private key with the ACME server’s own choice of private key by including -----BEGIN RSA PRIVATE KEY----- in the response. Then when put into production, the ACME server operator will be able to decrypt all of the traffic. WHOOPS. (Obviously the ACME server can impersonate the web/TLS server, since it’s a CA component, but this is not what you want.)

Different implementations may behave differently: the moral lesson is to validate your inputs, and in this case, that means to deal with the certificate chain response with a lot more finesse than is contemplated with “application/pem-certificate-chain”.

> DER to PEM and don't bother with the intermediates, resulting in

Check out the above. :-)

> returning binary CMS. Unfortunately, it is not.

Right, but “PEM” is not the format that you are actually looking for in this case, despite its apparent ease of use. What this draft is calling application/pem-certificate-chain boils down to text/plain, and at least text/plain makes the security issues more obvious (because the ACME client is ingesting arbitrary inputs).

Let’s address the issue raised above first…

Sean

---

From: Jacob Hoffman-Andrews <[email-masked]>
Date: Thu, 30 Mar 2017 10:25:24 -0700

On 03/30/2017 09:04 AM, Sean Leonard wrote:
> IN PARTICULAR: both Apache and Ngnix may be subject to a private key
> substitution attack with naive passing of the ACME response to the web
> server! See:
> http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_certificate
> http://httpd.apache.org/docs/current/mod/mod_ssl.html#sslcertificatefile
>
> The SSL Certificate option includes the option of including the
> private key in the same input: “A secret key in the PEM format may be
> placed in the same file.” 
>
> What this means is that a malicious ACME server can serve a
> certificate per the client’s request, but substitute the server’s
> specified private key with the ACME server’s own choice of private key
> by including -----BEGIN RSA PRIVATE KEY----- in the response. Then
> when put into production, the ACME server operator will be able to
> decrypt all of the traffic. WHOOPS. (Obviously the ACME server can
> impersonate the web/TLS server, since it’s a CA component, but this is
> not what you want.)
This is a good point, and makes me more inclined towards the
"concatenated DER" approach you suggested, if it looks like it would be
straightforward for clients to implement. Specifically because any
client transforming the DER into PEM would be responsible for adding the
"-----BEGIN CERTIFICATE-----" delimiters, and would be very unlikely to
output private key delimiters instead.