Subject: [Acme] Retrying failed authorizations

From: Hugo Landau <[email-masked]>
Date: Sat, 25 Mar 2017 16:40:24 +0000

Currently, when an order is created, authorizations are created for it,
and challenges are created for each of those authorizations.

If a challenge is failed once, the challenge comes to have status
"invalid", as does the authorization containing it. The order containing
the authorization becomes unfulfillable.

This is highly undesirable because it is not necessarily possible to
discern ahead of time which challenges will succeed and which will fail
for a given hostname. Moreover, random failures or failures due to
differing viewpoints (e.g. DNS propagation for DNS challenges) increase
the likelihood that a given challenge will fail. This is a particular
problem for DNS challenges, where it can be unclear when propagation is
complete, but also affects other challenge types.

Where there is doubt as to the challenges which will be successful for a
given hostname, different methods may be attempted by a client in
sequence. If the failure of a challenge results in the failure of an
entire order, this results in an excessively large number of failed and
wasted order, authorization and challenge objects.

For example, if a CA which allows up to 100 SANs in each certificate
supports three challenge methods (e.g. HTTP, TLS-SNI, DNS), successful
completion of the order could involve the creation of about 120,300 objects.
(Assume that every time, only the last challenge method attempted of the
three methods available is successful. For each attempt, 401 objects are
created (1 order, 100 authorizations and 300 challenges). 300 attempts
are required: 3 for each authorization.)

Although this may be avoidable using new-authz, the intention is clearly
to prefer the new-order workflow, support for the new-authz endpoint is
not mandatory, and the new-authz endpoint may not be usable for some use
cases, such as wildcard domains.

The following resolutions are possible:

1. Allow challenges to be retried. Optionally, this could include the
   removal of the "invalid" status for challenges; there would be only
   "pending" and "valid". If audit trails are desired, challenge
   failures would be indicated by appending error objects to a
   "failedValidations" array, not by changing a status field. Challenge
   failures during polling would be detected by observing increases in
   the length of this array.

2. Allow challenges to be regenerated via request, in a fresh, pending
   state. The challenge URL is changed. The old challenge remains
   accessible at the old URL for auditing purposes, but the
   authorization is updated to reference the new challenge URL where
   previously it updated the old one.

3. Allow authorizations to be regenerated via request. Like challenge
   regeneration, the authorization URL in the order object would be
   replaced, and the old authorization would remain accessible at the
   old URL.

4. Don't change anything about the current specification. This is highly
   inefficient, insofar that it inflates the number of database objects
   which must be generated and stored by a CA. It also increases the
   time that the issuance process will take very substantially.

I'd prefer option 1. Thoughts?

Hugo Landau

---

From: Jacob Hoffman-Andrews <[email-masked]>
Date: Tue, 28 Mar 2017 15:48:51 -0700

> the authorization becomes unfulfillable.
I agree that this is an issue; You and I talked about it previously in
the thread "New application flow and retries," but didn't get much
feedback from the rest of the group.

Note that authz's are recyclable, so the impact is not quite so bad as
you describe. But even if you just consider the CSR, each new-order is
fairly expensive. A minimum of 1k of storage for a CSR with a 2048-bit
RSA key. Under the current model, a client that is having validation
problems and retries repeatedly will consume at least 1k of database
space per try. As a result, CAs will likely need to limit new-order
creations rather strictly, which makes life harder for clients.

I think the way CAs traditionally handle this is as you suggest: by
making challenges retryable. Note that this implies that orders
typically never get invalidated, they just expire after some period of
time (or error out if there's a problem during issuance, like CAA). This
also implies that clients would potentially need to keep state across
invocations. Specifically, clients would need to store the private keys
used for extant orders, so if a subsequent run completes an order, the
client is able to use the resulting certificate.

What's your thought on the additional burden on clients?

What is the rest of the group's thoughts on whether we'd have to restart
WGLC for this?

---

From: Hugo Landau <[email-masked]>
Date: Thu, 30 Mar 2017 04:31:16 +0100

> What's your thought on the additional burden on clients?
I don't think the additional burden is a problem.

> What is the rest of the group's thoughts on whether we'd have to restart
I'd hope that if the 'can retry individual challenges' option is chosen
this is considered a minor enough change to avoid restarting WGLC.

However, it also seems pointless to finalize the ACME specification as
an RFC when we have the opportunity to wait until Let's Encrypt obtains
operational experience in production with the revised specification.
It's a little bizarre that a version of the specification that isn't
going to be standardized as an RFC has a lot more deployment experience
attached to it than a substantially different version with no current
deployment experience.

It would be a different story if this WG thought it was holding anything
up by virtue of deferring standarization; if another WG was waiting on
an RFC to cite, or so on. But the evidence is that the fact that ACME is
still an I-D hasn't stopped real-world deployment of either CAs (Let's
Encrypt) or a multitude of clients.

In that regard, even though I can usually be as weary as anyone at how
long it can take standards organizations to finalize things (C++
concepts comes to mind), in this case, I think it would make sense to
postpone finalization of the ACME specification until such time as Let's
Encrypt moves the new specification into production (or, alternatively,
until another CA enters on the scene and implements ACME, and gains
experience with it). I'm not sure what position this WG would take on
tying a neutral standards process to the implementation timeline of a
particular organization; potentially it is objectionable. At the same
time, the fact that ACME has been field-tested before finalization has
been valuable.

---

From: Jacob Hoffman-Andrews <[email-masked]>
Date: Thu, 30 Mar 2017 00:58:26 -0700

> Encrypt) or a multitude of clients.
Thanks for expressing this; I tend to agree, but that makes sense given
that I work on Let's Encrypt.

The main countervailing pressure, in my mind, is that other CAs are
starting to show interest in ACME, but of course the main reason to use
ACME is interop with existing tools. So we want to have something
reasonably final (and in production), so that new integrations are built
against the final-ish version rather than Boulder's outdated variant as
much as possible. However, I think the main gating factor there is, as
you say, getting the latest spec into production in Boulder. Which we're
working on. :-)