Subject: [Acme] Revisiting Proactive Issuance & new-order CSR

From: Daniel McCarney <[email-masked]>
Date: Tue, 19 Sep 2017 15:26:00 -0400

Hi folks,

Just over a year ago in a thread titled "Proactive vs On-Finalization
Certificate Issuance"[0] we reached the consensus on the question of
whether to
issue a certificate "on-finalization" or "proactively" with the conclusion
to
standardize on proactive issuance.

Implementation experience shows that proactive issuance is slow in
practice. An
order can be associated with many authorizations and an authorization can be
associated with many orders. This interacts poorly with optimizations Let's
Encrypt has developed from in-the-field experience with ACME and it is
difficult
to rate-limit effectively. I propose that we return to the on-finalization
style
approach and that we provide the CSR during finalization..

## Issues with proactive issuance

### Authorization Reuse

Proactive issuance requires that when an authorization is switched to a
valid
state by the result of a challenge update that we find the associated
orders and
iterate each order's authorizations to see if all of the authorizations are
valid and issuance can occur.

Let's Encrypt reuses existing authorizations[1][2] whenever possible, as
encouraged by ACME. This has been of huge value in reducing resource
consumption
(both in terms of database storage and challenge/API requests). Many clients
create pending authorizations they do not finalize and similarly re-create
pending authorizations for identifiers that the account already has valid
authorizations for. Unchecked this leads to significant database growth and
resource consumption, even with aggressive rate-limits[3].

The combination of authorization reuse and proactive issuance means a given
authorization can be associated with many orders. This necessitates an
expensive
many-to-many query at each authorization update to see if any associated
orders are now complete.

One possible workaround is a rate-limit that would constrain the number of
orders an authorization can be associated with. However establishing the
value
of the limit to both reduce server-side load & also effectively support
Let's
Encrypt's large-volume integrators and their issuance patterns is
challenging.

### CSR-first new-order flow

To support proactive issuance the CA also needs to have the CSR for the
order
at-hand when checking each authorization in case issuance can proceed. The
`new-order` request that creates the order therefore carries the full CSR
that
was previously delivered by the "certificate finalize" message.

We see users create many more authorizations than they are able to finalize
successfully. With the legacy "new-cert" flow our authorization reuse
dramatically helped reduce the DB space used up by authorizations that will
never lead to issuance because no CSR is stored until all identifiers are
validated.

With the new-order flow we have to accept and store a full CSR before
handing
back authorizations to the client. Reuse of authorizations does not prevent
the
DB bloat that will occur from users submitting new order requests and not
fulfilling them. The CSR contains a full public key and is one of the single
largest contributors to DB disk space usage. Addressing this will require
another strict rate limit on outstanding new-orders and is again difficult
to
“dial-in” in to both prevent excessive resource consumption and also
maintain
the ability for big integrators to process large-scale issuance using our
API.

# Proposed Change

I believe we should change the `new-order` endpoint to not accept a full
CSR but
to instead accept a list of identifiers that the ACME client wishes to
authenticate & issue for. Authorizations can be created and returned for
those
domains as happens today.

The order resource should be changed to have a “finalizeURL” field which
provides the client a URL to POST a CSR to indicate that the CA should
issue the
certificate. Proactive issuance should be removed outright.

This will address both the problem of scanning existing many-to-many orders
& their authorizations as well as the db bloat that will occur from
front-loading the delivery of the CSR.

It may be tempting to leave proactive issuance as an optional feature but
this
will require maintaining the CSR in the new-order and I also believe that
having
two very distinct methods of issuance will complicate the client ecosystem
and
returns us to the question of how to determine which will occur that
spawned the
original "Proactive vs On-finalization Certificate Issuance"
thread[0].

[0] https://mailarchive.ietf.org/arch/msg/acme/0lVmGl8e-rmSH0x7ycDW5dj6GAY
[1] https://community.letsencrypt.org/t/upcoming-api-changes/17947
[2]
https://community.letsencrypt.org/t/automatic-recycling-of-pending-authorizations/41321
[3] https://letsencrypt.org/docs/rate-limits/

---

From: Logan Widick <[email-masked]>
Date: Tue, 19 Sep 2017 15:34:50 -0500

Would it be possible to extract the key and identifiers from the CSR, add
the key  to the database if it doesn't already exist, find or create the
authorizations for the identifiers, not store the CSR, and then assemble
the certificate from the (valid) authorizations and key later?

Alternatively, could one get rid of the CSRs altogether, and have the
new-order be a nested JWS? The inner JWS would be signed by the key that
goes on the certificate and would contain everything required to generate
the certificate (identifiers, requested dates if applicable, etc.). The
outer JWS would be signed by the ACME account key and link the inner JWS to
the ACME account.

What goes in a CSR or certificate that can't be reconstructed later?

Logan

On Sep 19, 2017 2:26 PM, "Daniel McCarney" <[email-masked]> wrote:

Hi folks,

### Authorization Reuse

# Proposed Change

[0] https://mailarchive.ietf.org/arch/msg/acme/0lVmGl8e-rmSH0x7ycDW5dj6GAY
[1] https://community.letsencrypt.org/t/upcoming-api-changes/17947
[2] https://community.letsencrypt.org/t/automatic-recycling-of-p
ending-authorizations/41321
[3] https://letsencrypt.org/docs/rate-limits/

_______________________________________________
Acme mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/acme

---

From: Ilari Liusvaara <[email-masked]>
Date: Tue, 19 Sep 2017 23:51:08 +0300

> the certificate from the (valid) authorizations and key later?

Most CSRs have fresh keys that are never reused (thanks to forced key-
rotation[1] in most ACME clients).

> outer JWS would be signed by the ACME account key and link the inner JWS to

Using the private key to sign a JWS is cryptographically very much non-
kosher.

Also, some systems can emit CSRs for non-exportable key. With those,
you are SoL unless you can send a CSR to the CA.

> What goes in a CSR or certificate that can't be reconstructed later?

For Let's Encrypt, AFAIK the only things in CSR that matter are:

- Requested domains
- Public key
- Signature (to verify proof-of-possession).

[1] Even if the client has mode that does not rotate the keys, that
mode is in most of the clients somehow crippled, so it does not
actually work well.

-Ilari

---

From: Jacob Hoffman-Andrews <[email-masked]>
Date: Wed, 20 Sep 2017 08:43:41 -0700

On 09/19/2017 01:34 PM, Logan Widick wrote:
> Would it be possible to extract the key and identifiers from the CSR,
> add the key  to the database if it doesn't already exist, find or
> create the authorizations for the identifiers, not store the CSR, and
> then assemble the certificate from the (valid) authorizations and key
> later?
Note that the size of the CSR for an RSA key is about half from the
signature, and half from the key. So extracting the key would only save
the size of the signature, about half. Also, this wouldn't solve the
more important problem, the overhead of reviewing all authzs on all
related orders on finalizing each authz, when an order may have a
hundred or more authzs.

I support Daniel's proposal.

---

From: Richard Barnes <[email-masked]>
Date: Wed, 20 Sep 2017 13:22:25 -0700

Hey Daniel,

Thanks for the input.  I can see why you might be feeling some pain here,
but I'm not sure the proposed solution is quite right.

Note that ACME already has the "new-authorization" flow, which does pretty
much exactly what you're proposing.  The only difference is that instead of
authorizations being created all at once in response to a "new-cert"
request, they're created individually by the client.  But it's ultimately
the same number of RTTs (authz fetch ~ new-authz) -- or rather, one fewer
in the new-authz case because there's no new-cert request.  To what degree
would your problem be solved simply by having the affected integrators move
over to that flow?

I can sympathize that the new-order flow might not be suitable for all use
cases.  Maybe it would be useful to write down some thoughts on which use
cases are best addressed by which flows.

--Richard

On Tue, Sep 19, 2017 at 12:26 PM, Daniel McCarney <[email-masked]>
wrote:

> back authorizations to the client. Reuse of authorizations does not
> prevent the
> [2] https://community.letsencrypt.org/t/automatic-recycling-of-
> pending-authorizations/41321

---

From: Richard Barnes <[email-masked]>
Date: Thu, 21 Sep 2017 11:40:03 -0700

I talked a bit with Daniel about this off-list. Here's a summary of where
we got to:

1. In the current "new-order" flow, the server doesn't *really* have to be
proactive -- instead of updating an order object when an authz changes
state, it can wait to update/issue until the client polls for the order
object, indicating that it's actually interested in having a certificate.
That doesn't alleviate the pain of storing the CSR, but it at least avoids
the expense of finding affected orders and issuing in cases where the
client abandons the order.  If we think this approach is OK, I would
propose that we revise the document to make clear that it is OK.  What do
folks think?  Daniel noted that there might be some issues with GET
idempotency here, but I don't think this actually makes GET non-idempotent.

2. Some CAs might want to support only the "new-authz" flow, i.e., never
allow a "new-order" request if the relevant authorizations haven't already
been done.  It seems like this could be addressed in a pretty
straightforward manner just by defining a new error code (say
"preauthRequired") that the server can return in this case.

Personally, it seems to me that if we make the clarification (1) and add
the error code (2) described here, that would address the pain points
Daniel raises, without a whole lot of new mechanism.  I'm not really seeing
much additional value to be had from an explicit "ok issue now" request
(vs. the GET poll).

--Richard

On Wed, Sep 20, 2017 at 1:22 PM, Richard Barnes <[email-masked]> wrote:

>> issue a certificate "on-finalization" or "proactively" with the
>> conclusion to
>> to rate-limit effectively. I propose that we return to the
>> on-finalization style
>> another strict rate limit on outstanding new-orders and is again
>> difficult to
>> two very distinct methods of issuance will complicate the client
>> ecosystem and

---

From: "Salz, Rich" <[email-masked]>
Date: Thu, 21 Sep 2017 23:25:41 +0000

If the suggestion is fine with Daniel, I think editorial clarification and a new error code can be added without disrupting the  cycle and I am strongly in favor of that.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 22 Sep 2017 22:55:39 +1000

On Fri, Sep 22, 2017 at 4:40 AM, Richard Barnes <[email-masked]> wrote:
> Daniel noted that there might be some issues with GET idempotency here, but
> I don't think this actually makes GET non-idempotent.

It's still idempotent, because doing the GET twice has the same effect
as doing it once.  You are concerned about the GET no longer being
safe.  That is, it is no longer side-effect free.

It is OK to do things in response to a GET as long as the server owns
the consequences.  Servers frequently log GET requests, which isn't
safe or even idempotent, but servers do it because there is an
advantage to them that exceeds the costs.  As long as clients aren't
held accountable for these side-effects, it's OK.

Or, you could consider the resource to have been updated at some point
in the past as though the authorization triggered proactive issuance,
but it's just that the GET has taken a little while to "find" the
updated state.  Think of it as a lazy load, or maybe an extreme case
of the server cache needing to be populated.

If you like, we could also dive into whether this specific type of
request is safe to send in 0-RTT, but I don't have the brain cells for
that right now.

---

From: Daniel McCarney <[email-masked]>
Date: Fri, 22 Sep 2017 09:42:24 -0400

RE: #1 - this does help, but will require changing the spec to clarify. The
current wording does not support this behaviour and cleary indicates the
server MUST issue for any order that has been satisfied.

RE: #2 - a CA relying on the new-authz flow to avoid the cost of accepting
a CSR up front would never be able to use the new-order flow to give the
client authorizations they didn't ask for or they are back to accepting
CSRs for orders that may never complete. That's why we need a new error to
return to reject new-order requests that aren't fully pre-authorized. If
the largest ACME CAs will not implement new-order as described outside of
pre-authorization why have it at all? Who will implement it?

The primary failing of #2 is that it causes those CAs to lose one
substantial benefit to the new-order flow that I would like to keep:
allowing for policy-based decisions on the CA side to influence which
authorizations are required. Without this property we will further have to
update the specification for wildcard issuance. The current spec doesn't
permit asking for authorizations for wildcard identifiers with new-authz so
you'd need some way of asking the server for a wildcard identifier and have
it respond with an authorization for something completely different (An
authorization for some wildcard resource say with a  DNS-01 challenge for
the base domain).

I agree that this is a late change but we are nearing the last chance we
have to address what I see as a fairly significant point of complication in
the protocol.  I note that no one has proposed any reasons why proactive
issuance must remain since this thread was restarted except that it is the
status-quo and we want to minimize change.

I am not in favour of the suggestions from Richard so far and would like to
hear more about why my proposal is not suitable. Richard's changes are
smaller but leave protocol complication and prevent large CAs from enjoying
the benefits of the new-order flow. My proposal addresses the problems and
leaves a simpler protocol without features that will not be used by large
providers.

We have to change the specification no matter whether my proposals or
Richards are taken so why not make changes that will result in the clearest
protocol with the least amount of cruft few are likely to implement?

- Daniel

On Thu, Sep 21, 2017 at 2:40 PM, Richard Barnes <[email-masked]> wrote:

>> Note that ACME already has the "new-authorization" flow, which does
>> pretty much exactly what you're proposing.  The only difference is that
>> instead of authorizations being created all at once in response to a
>> "new-cert" request, they're created individually by the client.  But it's
>> ultimately the same number of RTTs (authz fetch ~ new-authz) -- or rather,
>> one fewer in the new-authz case because there's no new-cert request.  To
>> what degree would your problem be solved simply by having the affected
>> integrators move over to that flow?
>>
>> I can sympathize that the new-order flow might not be suitable for all
>> use cases.  Maybe it would be useful to write down some thoughts on which
>> use cases are best addressed by which flows.
>>
>> --Richard
>>
>>
>>
>>> `new-order` request that creates the order therefore carries the full
>>> CSR that
>>> the ability for big integrators to process large-scale issuance using
>>> our API.
>>>
>>> # Proposed Change
>>>
>>> It may be tempting to leave proactive issuance as an optional feature
>>> but this
>>> will require maintaining the CSR in the new-order and I also believe
>>> that having

---

From: Roland Bracewell Shoemaker <[email-masked]>
Date: Fri, 22 Sep 2017 16:19:29 -0700

Broad points:

I understand that this group has been moving towards standardization for
a while but I really think that attempting to fix these issues with
creative readings of the existing language and minimal editorial changes
is asking for trouble and return trips to the WG in the future when
these band-aids turn out to not be fully thought out or themselves
introduce new problems.

These issues have come up during, as far as I'm aware, the only full-fat
implementation of the current version of this specification and that
implementation itself is maybe only 40-50% done. Assuming that these
will be the only issues that are discovered seems to be, again, asking
for trouble.

Comments on the proposed fixes:

1. Causing issuance really seems like it should be required to be an
authorized action, allowing this to happen by a GET allows anyone who is
able to determine the order URL to cause issuance to happen.
Implementing this behavior, even if not explicitly defined in the
specification, will almost definitely lead to users relying on it
(Hyrum's Law) which again will be a problem if another user is able to
simply send of a handful of GETs and cause another users certificate to
be issued.

2. Unless another implementer comes along this would mean that the only
existing implementation of the current specification before
standardization would not be implementing a major section of this
document. At this point why not just go back to the -03/-04 draft
new-authz flow. As Daniel has already pointed out this also completely
breaks one of the major reasons to use the new-order flow: policy based
authorization creation. How would a CA indicate that a user _may not_
need an authorization for www.example.com when issuing for example.com
or that they require a specific set of authorizations when issuing for
*.example.com?

Like I said at the start these band-aid editorial fixes seem to just
create another set of problems that cannot themselves be solved with
further minor editorial  changes.

On 09/21/2017 11:40 AM, Richard Barnes wrote:
> I talked a bit with Daniel about this off-list. Here's a summary of
> where we got to:
> 
> 1. In the current "new-order" flow, the server doesn't *really* have to
> be proactive -- instead of updating an order object when an authz
> changes state, it can wait to update/issue until the client polls for
> the order object, indicating that it's actually interested in having a
> certificate.  That doesn't alleviate the pain of storing the CSR, but it
> at least avoids the expense of finding affected orders and issuing in
> cases where the client abandons the order.  If we think this approach is
> OK, I would propose that we revise the document to make clear that it is
> OK.  What do folks think?  Daniel noted that there might be some issues
> with GET idempotency here, but I don't think this actually makes GET
> non-idempotent.
> 
> allow a "new-order" request if the relevant authorizations haven't
> already been done.  It seems like this could be addressed in a pretty
> Daniel raises, without a whole lot of new mechanism.  I'm not really
> seeing much additional value to be had from an explicit "ok issue now"
> request (vs. the GET poll).
> 
> --Richard
> 
> 
> 
> On Wed, Sep 20, 2017 at 1:22 PM, Richard Barnes <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
>     Hey Daniel,
> 
>     Thanks for the input.  I can see why you might be feeling some pain
>     here, but I'm not sure the proposed solution is quite right.
> 
>     pretty much exactly what you're proposing.  The only difference is
>     that instead of authorizations being created all at once in response
>     to a "new-cert" request, they're created individually by the
>     client.  But it's ultimately the same number of RTTs (authz fetch ~
>     new-authz) -- or rather, one fewer in the new-authz case because
>     there's no new-cert request.  To what degree would your problem be
>     solved simply by having the affected integrators move over to that flow?
> 
>     I can sympathize that the new-order flow might not be suitable for
>     all use cases.  Maybe it would be useful to write down some thoughts
>     on which use cases are best addressed by which flows. 
> 
>     --Richard
> 
> 
> 
>     On Tue, Sep 19, 2017 at 12:26 PM, Daniel McCarney
>     <[email-masked] <mailto:[email-masked]>> wrote:
> 
>         Hi folks,
> 
>         Certificate Issuance"[0] we reached the consensus on the
>         question of whether to
>         Implementation experience shows that proactive issuance is slow
>         in practice. An
>         Encrypt has developed from in-the-field experience with ACME and
>         it is difficult
>         Proactive issuance requires that when an authorization is
>         switched to a valid
>         state by the result of a challenge update that we find the
>         associated orders and
>         encouraged by ACME. This has been of huge value in reducing
>         resource consumption
>         authorization can be associated with many orders. This
>         necessitates an expensive
>         orders an authorization can be associated with. However
>         establishing the value
>         of the limit to both reduce server-side load & also effectively
>         support Let's
>         Encrypt's large-volume integrators and their issuance patterns
>         is challenging.
> 
>         To support proactive issuance the CA also needs to have the CSR
>         for the order
>         `new-order` request that creates the order therefore carries the
>         full CSR that
>         With the new-order flow we have to accept and store a full CSR
>         before handing
>         back authorizations to the client. Reuse of authorizations does
>         not prevent the
>         “dial-in” in to both prevent excessive resource consumption and
>         also maintain
>         the ability for big integrators to process large-scale issuance
>         using our API.
> 
>         # Proposed Change
> 
>         I believe we should change the `new-order` endpoint to not
>         accept a full CSR but
>         authenticate & issue for. Authorizations can be created and
>         returned for those
>         provides the client a URL to POST a CSR to indicate that the CA
>         should issue the
>         It may be tempting to leave proactive issuance as an optional
>         feature but this
>         will require maintaining the CSR in the new-order and I also
>         believe that having
>         returns us to the question of how to determine which will occur
>         that spawned the
>         [0]
>         https://mailarchive.ietf.org/arch/msg/acme/0lVmGl8e-rmSH0x7ycDW5dj6GAY
>         <https://mailarchive.ietf.org/arch/msg/acme/0lVmGl8e-rmSH0x7ycDW5dj6GAY>
>         [1]
>         https://community.letsencrypt.org/t/upcoming-api-changes/17947
>         <https://community.letsencrypt.org/t/upcoming-api-changes/17947>
>         [2]
>         <https://community.letsencrypt.org/t/automatic-recycling-of-pending-authorizations/41321>
>         <https://letsencrypt.org/docs/rate-limits/>
> 
> 
>         [email-masked] <mailto:[email-masked]>
>         <https://www.ietf.org/mailman/listinfo/acme>
> 
> 
> 
> 
>

---

From: Daniel McCarney <[email-masked]>
Date: Tue, 26 Sep 2017 10:14:00 -0400

I've proposed a first cut at implementing the changes I described in my
original post as an ACME PR https://github.com/ietf-wg-acme/acme/pull/342

On Fri, Sep 22, 2017 at 7:19 PM, Roland Bracewell Shoemaker <
[email-masked]> wrote:

> >         <https://community.letsencrypt.org/t/automatic-
> recycling-of-pending-authorizations/41321>

---

From: Andrew Ayer <[email-masked]>
Date: Tue, 26 Sep 2017 10:34:36 -0700

FWIW, I previously expressed support for proactive issuance because 
I thought it would lay the groundwork for a better renewal process,
especially for short-lived certificates.  However, this idea never
gained traction and the other necessary bits weren't added.  Instead,
STAR seems to be handling this use case.

Therefore, I no longer see a compelling reason for proactive issuance.
If it's causing problems it should be removed.  I also agree with Roland
that issuing upon an unauthenticated GET request is a bad idea.

Regards,
Andrew

---

From: Daniel McCarney <[email-masked]>
Date: Wed, 27 Sep 2017 09:24:10 -0400

Hi again,

Richard: Do you have any subsequent thoughts now that Roland and myself
have given feedback on your proposed changes and why we still prefer the
suggestions I made initially?

Andrew's original support of proactive issuance was the most constructive
vote for why that behaviour was preferred during the initial discussion
period. Given that his support is now withdrawn and Roland and Jacob have
voiced support for my proposal I feel like we are approaching consensus.
There has not been any arguments made for the benefit of proactive issuance.

Boulder is beginning implementation of my proposed specification update and
I feel the best way to minimize Rich Salz's concerns about cycle disruption
is to resolve this thread quickly.

Thanks all,