Subject: [Acme] Rolling keys and pending validations

From: "Salz, Rich" <[email-masked]>
Date: Mon, 19 Jun 2017 18:08:10 +0000

Speaking purely as an individual.

This is about Section 7.3.3

It can be difficult to change the account credentials, because you have to make sure that nothing is "in transit."  For a large client, or perhaps a reseller type of arrangement, this can be difficult if not impossible.

I would like to see a sentence added to the end of the section that says "rolling keys does not invalidate any challenges currently in progress."

I leave this to the WG as to whether or not this is strictly editorial.  Even if there is no consensus, an explicit statement about the validity should be added.  I think NOT invalidation is better, as the inverse makes changes hard.
--
Senior Architect, Akamai Technologies
Member, OpenSSL Dev Team
IM: [email-masked] Twitter: RichSalz

---

From: Richard Barnes <[email-masked]>
Date: Mon, 19 Jun 2017 14:34:45 -0400

This seems sensible; rolling keys shouldn't invalidate things in transit
any more than changing your Gmail password should delete your drafts folder.

I would have a little bit of a hard time calling this "purely editorial",
since it specifies server behavior.  But it seems like you're just
codifying an expectation that at least I already had (TBH, I would not have
thought to build a server otherwise), so I would be inclined to go ahead
and merge it if at least one or two other people chime in.

Here's a PR:  https://github.com/ietf-wg-acme/acme/pull/323

--Richard

On Mon, Jun 19, 2017 at 2:08 PM, Salz, Rich <[email-masked]> wrote:

> It can be difficult to change the account credentials, because you have to
> make sure that nothing is “in transit.”  For a large client, or perhaps a
> reseller type of arrangement, this can be difficult if not impossible.
>
>
>
> I would like to see a sentence added to the end of the section that says
> “rolling keys does not invalidate any challenges currently in progress.”
>
>
>
> I leave this to the WG as to whether or not this is strictly editorial.
> Even if there is no consensus, an explicit statement about the validity
> should be added.  I think NOT invalidation is better, as the inverse makes
> changes hard.
>
> --
>
> _______________________________________________
> Acme mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/acme
>
>

---

From: Ilari Liusvaara <[email-masked]>
Date: Mon, 19 Jun 2017 21:44:23 +0300

> Here's a PR:  https://github.com/ietf-wg-acme/acme/pull/323

If there is pending validation over key change, which key hash should
the validation use when it is resolved? The old one? The new one?

-Ilari

---

From: "Dunning, John" <[email-masked]>
Date: Tue, 20 Jun 2017 22:07:04 +0000

I would advocate for the new one.  IOW, I would (expect to) interpret the semantics as being

1. The challenge token was created with respect to an account
2. An attribute of the account is the credential (whichever credential is current)
3. Therefore the credential authorizes access to anything associated with the account.

If the concensus here is that a server needs to associate the credential which was in force at creation time with a challenge token, in other words that the token is tied to the credential rather than the account, I believe my use case would be ok with that, as long as the challenge tokens do not go invalid after the credential change.

I would expect that latter semantics would be hairier for server implementors, because they’d need to keep multiple credentials alive and associated with the account for the maximum lifetime of any extant tokens.

On 6/19/17, 2:44 PM, "Ilari Liusvaara" <[email-masked]> wrote:

    -Ilari

---

From: Richard Barnes <[email-masked]>
Date: Wed, 21 Jun 2017 12:57:38 -0400

It seems like the challenge update and validation process should follow
natural patterns for checking the key.

For the TLS-SNI and DNS challenges, the only time in the validation when
the server checks the key is when the client responds with a
keyAuthorization.  Further processing is based on the verified
keyAuthorization.  So if the key change happens before the client responds,
then the server will use the new key; if it happens after the response,
then it will use the old key.

For the HTTP challenge as written right now, there's an additional problem
that the server does another key check after it gets a keyAuthorization in
the validation query, rather than checking against the keyAuthorization in
the challenge resource (updated post-response).  If you have the following
sequence of events, you'll have a problem:

- Client provisions HTTP resource with old account key
- Client sends response with keyAuthorization over old account key
- Client changes key
- Server sends validation HTTP request
- Client response with keyAuthorization over the old account key
- Server compares to new account key; fails.

ISTM that the best way to fix this is simply to align the HTTP challenge to
the other challenges, and have the server check the keyAuthorization in the
response against the keyAuthorization in the challenge, rather than
checking directly for the current account key.

===

This proposed solution framework will result in the old key being used
sometimes.  That's a little unfortunate (e.g., if you're rolling due to
compromise), but the window of vulnerability is fairly small, and you can
always revoke certificates or invalidate authorizations.

I have updated the PR.  Note that I have not updated the challenge type,
since this seems like a minor change with small compatibility risk:

https://github.com/ietf-wg-acme/acme/pull/323/files

--Richard

On Tue, Jun 20, 2017 at 6:07 PM, Dunning, John <[email-masked]> wrote:

> If the concensus here is that a server needs to associate the credential
> which was in force at creation time with a challenge token, in other words
> that the token is tied to the credential rather than the account, I believe
> my use case would be ok with that, as long as the challenge tokens do not
> go invalid after the credential change.
>
> I would expect that latter semantics would be hairier for server
> implementors, because they’d need to keep multiple credentials alive and
> associated with the account for the maximum lifetime of any extant tokens.
>
>

---

From: "Salz, Rich" <[email-masked]>
Date: Wed, 21 Jun 2017 18:07:35 +0000

So far the only “objection” about this has been that it defines server behavior and is therefore not strictly editorial.  But that comment didn’t disagree with the actual change.

So please go ahead.  If anyone disagrees, start discussions now.

---

From: Jacob Hoffman-Andrews <[email-masked]>
Date: Mon, 26 Jun 2017 12:35:57 -0700

> If the concensus here is that a server needs to associate the credential which was in force at creation time with a challenge token, in other words that the token is tied to the credential rather t…
This seems different from the behavior I thought you were looking for,
based on previous conversations. This may be a difference in
interpretation of "challenge token" vs "key authorization." Every
challenge has a random "token" field, but this is only a part of what
gets provisioned to a web server for HTTP-01. For an HTTP-01 challenge,
the ACME client calculates a "key authorization" as "<token>.<account
key thumbprint>". The client then POSTs a signed request to the
challenge URL, containing a "keyAuthorization" field containing this key
authorization. The ACME server then requests
"/.well-known/acme-validation/<token>" from the validation domain, and
expects "<token>.<account key thumbprint>" (that is, the key
authorization object) in response.

I think a more customary way of talking would be to call this "key
authorization" object that is provisioned on the validation domain a
"challenge token." And in fact, in an earlier (pre-draft-01) version of
the ACME spec, the "token" field of a challenge was provisioned on the
validation domain and requested by the ACME server. However, some people
on the list pointed out that this would encourage development of "naive"
challenge solvers that would just echo the token part of the URL. So we
wound up in this awkward state where what most people would consider a
"challenge token" is, in ACME, considered a "key authorization."

As I understand the situation we're trying to solve, it's something like
this:

1. Monday: an ACME client operated by a hosting provider receives a
request for a certificate for `www.example.com`. This ACME client
requests an authorization object from the server, and receives one. The
authorization object contains a challenge of type "HTTP-01":

{
  "type": "http-01",
  "status": "pending",
  "uri":
"https://ca.example.net/acme/challenge/l-2zFCdEQT3H5adBnO_WFuTAA3t4BV4hnFqWwtlgnpo/8543",
  "token": "LC2eEY_9lRLgxUfCME6TMB4bS0wslm2PDLTC2-JU-YY",
}

The client's current account key has a thumbprint of
"MKgAB8ljZpQv0hxuipA70L0SZkmXIUhbO61fR3vKvj8", so it calculates a key
authorization by concatenating that with the challenge token field:
"LC2eEY_9lRLgxUfCME6TMB4bS0wslm2PDLTC2-JU-YY.MKgAB8ljZpQv0hxuipA70L0SZkmXIUhbO61fR3vKvj8".
It then tells the owner of `example.com` to put a file containing that
string at
"/.well-known/acme-challenge/LC2eEY_9lRLgxUfCME6TMB4bS0wslm2PDLTC2-JU-YY".
The owners of `www.example.com` say "Okay, but that will take a few days.

2. Tuesday: The ACME client rotates its account key as part of regular
maintenance. The new account key thumbprint is
"bJcgrK6oTvjgvkuMOOkpdS8BdOotqf_z9T8zw-eFWOE".

3. Wednesday: `www.example.com` says "Okay, I've created a file at
"/.well-known/acme-challenge/LC2eEY_9lRLgxUfCME6TMB4bS0wslm2PDLTC2-JU-YY",
and it contains
"LC2eEY_9lRLgxUfCME6TMB4bS0wslm2PDLTC2-JU-YY.MKgAB8ljZpQv0hxuipA70L0SZkmXIUhbO61fR3vKvj8".
The account key thumbprint in this file (the part after the ".") is out
of date, because the ACME client rotated its account key.

3. Wednesday: The ACME client POSTs to the existing challenge URL with a
key authorization based on its current account key:

POST /acme/challenge/l-2zFCdEQT3H5adBnO_WFuTAA3t4BV4hnFqWwtlgnpo/8543

... note: JWK wrapping elided ...
{
  "keyAuthorization":
"LC2eEY_9lRLgxUfCME6TMB4bS0wslm2PDLTC2-JU-YY.bJcgrK6oTvjgvkuMOOkpdS8BdOotqf_z9T8zw-eFWOE"
}
...

Note: The key authorization in this POST contains the current account
key thumbprint. If it contained the old account key thumbprint, the ACME
server would reject it.

4. Wednesday: The ACME server accepts the challenge update, and makes a
validation request to
http://www.example.com/.well-known/acme-challenge/LC2eEY_9lRLgxUfCME6TMB4bS0wslm2PDLTC2-JU-YY.
It receives a response
"LC2eEY_9lRLgxUfCME6TMB4bS0wslm2PDLTC2-JU-YY.MKgAB8ljZpQv0hxuipA70L0SZkmXIUhbO61fR3vKvj8".
However, the ACME server was expecting
"LC2eEY_9lRLgxUfCME6TMB4bS0wslm2PDLTC2-JU-YY.bJcgrK6oTvjgvkuMOOkpdS8BdOotqf_z9T8zw-eFWOE",
because that's what was POSTed as part of the challenge update. So the
validation request fails, and `www.example.com` has to start the whole
issuance process all over again.

There's one additional piece of context here: We assume that the hosting
provider operating the ACME client doesn't yet control
`www.example.com`. This can happen, for instance, when a hosting
provider is trying to onboard a customer who already uses HTTPS. The
hosting provider needs to have a valid certificate in place *before* the
customer CNAMEs their domain name the the hosting provider.

The "clarify impact of roll-over" PR
(https://github.com/ietf-wg-acme/acme/pull/323/files) is a nice
clarification, but doesn't actually address this use case. The issue
here is what happens during the period between creation of a challenge
and the ACME client requesting validation of that challenge, which in
current implementations can be up to 7 days.

There are two main approaches I see:

A) We could change the spec to set the keyAuthorization field when a
challenge is created, so that it would have the current account key
thumbprint baked in. This would allow hosting providers to treat the key
authorization more like an opaque string, and calculate it at challenge
creation time rather than at challenge update time.

B) We could encourage implementers to shorten the time between challenge
creation and validation by doing delegation. That is, instead of giving
`www.example.com` a token, and waiting for them to deploy it, the
hosting provider could say "Please delegate issuance authority to us,
either by redirecting all of /.well-known/acme-validation/* to our
validation server, or by CNAMEing _acme-challenge.www.example.com to our
validation server." The main problems with this: the owners of
`www.example.com` may not be familiar enough with redirects (although
CNAMEs should be easy); the owners of `www.example.com` may have a
concurrently running ACME client of their own that needs to renew during
the time window they would be serving a redirect or CNAME.

---

From: "Dunning, John" <[email-masked]>
Date: Fri, 30 Jun 2017 16:54:39 +0000

Sorry about the delayed response.

I probably was using imprecise terminology in my earlier mail on this topic.  Based on re-reading bits of the spec, I believe you’re right about my confusion between challenge token and key authorization.

Very high level, the use case I’m after is to allow for a challenge “process” to start prior to a credential rotation, and finish afterwards, without needing to be restarted.  Sorry, I’m making up more terminology there ☺  I liked Richard’s analogy earlier:  If I start a draft in gmail, then change my password, I can continue with that draft afterwards.

Based on your description below, I think door A makes more sense to me.  My paraphrase of it is that key authorizations get bound at creation time, and thus get “grandfathered” in after a credential rotation.

Door B would be problematic for my users.  For reasons which I don’t really want to elaborate too much, the relationships between my customers, their customers, and their customers’ hosting providers, is often tenuous and problematic.

I hope that helps.

I will be on vacation the next couple of weeks, so further responses on this thread will also be delayed.

Thanks . . .

From: Jacob Hoffman-Andrews <[email-masked]>
Date: Monday, June 26, 2017 at 3:35 PM
To: "Dunning, John" <[email-masked]>, Ilari Liusvaara <[email-masked]>, Richard Barnes <[email-masked]>
Cc: "Salz, Rich" <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: [Acme] Rolling keys and pending validations

If the concensus here is that a server needs to associate the credential which was in force at creation time with a challenge token, in other words that the token is tied to the credential rather than the account, I believe my use case would be ok with that, as long as the challenge tokens do not go invalid after the credential change.
This seems different from the behavior I thought you were looking for, based on previous conversations. This may be a difference in interpretation of "challenge token" vs "key authorization." Every challenge has a random "token" field, but this is only a part of what gets provisioned to a web server for HTTP-01. For an HTTP-01 challenge, the ACME client calculates a "key authorization" as "<token>.<account key thumbprint>". The client then POSTs a signed request to the challenge URL, containing a "keyAuthorization" field containing this key authorization. The ACME server then requests "/.well-known/acme-validation/<token>" from the validation domain, and expects "<token>.<account key thumbprint>" (that is, the key authorization object) in response.

I think a more customary way of talking would be to call this "key authorization" object that is provisioned on the validation domain a "challenge token." And in fact, in an earlier (pre-draft-01) version of the ACME spec, the "token" field of a challenge was provisioned on the validation domain and requested by the ACME server. However, some people on the list pointed out that this would encourage development of "naive" challenge solvers that would just echo the token part of the URL. So we wound up in this awkward state where what most people would consider a "challenge token" is, in ACME, considered a "key authorization."

As I understand the situation we're trying to solve, it's something like this:

1. Monday: an ACME client operated by a hosting provider receives a request for a certificate for `www.example.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.example.com&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=gXwrr-LtE9-WtzMFkuPVVQ&m=Kf71ta8IvyHrmRBLyoftWxjzSsiAc4tAN7OdanLQods&s=BWdYUEKrwtj9CJCO_xPHluLnoRWsgHzPCuNl4Kn88oY&e=>`. This ACME client requests an authorization object from the server, and receives one. The authorization object contains a challenge of type "HTTP-01":

{
  "type": "http-01",
  "status": "pending",
  "uri": "https://ca.example.net/acme/challenge/l-2zFCdEQT3H5adBnO_WFuTAA3t4BV4hnFqWwtlgnpo/8543"<https://urldefense.proofpoint.com/v2/url?u=https-3A__ca.example.net_acme_challenge_l-2D2zFCdEQT3H5adBnO-5FWFuTAA3t4BV4hnFqWwtlgnpo_8543&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=gXwrr-LtE9-WtzMFkuPVVQ&m=Kf71ta8IvyHrmRBLyoftWxjzSsiAc4tAN7OdanLQods&s=nzdWow0Gyf08idX9pkss225_1ViJdNlE1uY7ohmhnaU&e=>,
  "token": "LC2eEY_9lRLgxUfCME6TMB4bS0wslm2PDLTC2-JU-YY",
}

The client's current account key has a thumbprint of "MKgAB8ljZpQv0hxuipA70L0SZkmXIUhbO61fR3vKvj8", so it calculates a key authorization by concatenating that with the challenge token field: "LC2eEY_9lRLgxUfCME6TMB4bS0wslm2PDLTC2-JU-YY.MKgAB8ljZpQv0hxuipA70L0SZkmXIUhbO61fR3vKvj8". It then tells the owner of `example.com` to put a file containing that string at "/.well-known/acme-challenge/LC2eEY_9lRLgxUfCME6TMB4bS0wslm2PDLTC2-JU-YY". The owners of `www.example.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.example.com&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=gXwrr-LtE9-WtzMFkuPVVQ&m=Kf71ta8IvyHrmRBLyoftWxjzSsiAc4tAN7OdanLQods&s=BWdYUEKrwtj9CJCO_xPHluLnoRWsgHzPCuNl4Kn88oY&e=>` say "Okay, but that will take a few days.

2. Tuesday: The ACME client rotates its account key as part of regular maintenance. The new account key thumbprint is "bJcgrK6oTvjgvkuMOOkpdS8BdOotqf_z9T8zw-eFWOE".

3. Wednesday: `www.example.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.example.com&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=gXwrr-LtE9-WtzMFkuPVVQ&m=Kf71ta8IvyHrmRBLyoftWxjzSsiAc4tAN7OdanLQods&s=BWdYUEKrwtj9CJCO_xPHluLnoRWsgHzPCuNl4Kn88oY&e=>` says "Okay, I've created a file at "/.well-known/acme-challenge/LC2eEY_9lRLgxUfCME6TMB4bS0wslm2PDLTC2-JU-YY", and it contains "LC2eEY_9lRLgxUfCME6TMB4bS0wslm2PDLTC2-JU-YY.MKgAB8ljZpQv0hxuipA70L0SZkmXIUhbO61fR3vKvj8". The account key thumbprint in this file (the part after the ".") is out of date, because the ACME client rotated its account key.

3. Wednesday: The ACME client POSTs to the existing challenge URL with a key authorization based on its current account key:

... note: JWK wrapping elided ...
{
  "keyAuthorization": "LC2eEY_9lRLgxUfCME6TMB4bS0wslm2PDLTC2-JU-YY.bJcgrK6oTvjgvkuMOOkpdS8BdOotqf_z9T8zw-eFWOE"
}
...

Note: The key authorization in this POST contains the current account key thumbprint. If it contained the old account key thumbprint, the ACME server would reject it.

4. Wednesday: The ACME server accepts the challenge update, and makes a validation request to http://www.example.com/.well-known/acme-challenge/LC2eEY_9lRLgxUfCME6TMB4bS0wslm2PDLTC2-JU-YY<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.example.com_.well-2Dknown_acme-2Dchallenge_LC2eEY-5F9lRLgxUfCME6TMB4bS0wslm2PDLTC2-2DJU-2DYY&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=gXwrr-LtE9-WtzMFkuPVVQ&m=Kf71ta8IvyHrmRBLyoftWxjzSsiAc4tAN7OdanLQods&s=zk_hyfZnih6WypgNCWZghVBhfSJtaKuy6V18jqhIRsI&e=>. It receives a response "LC2eEY_9lRLgxUfCME6TMB4bS0wslm2PDLTC2-JU-YY.MKgAB8ljZpQv0hxuipA70L0SZkmXIUhbO61fR3vKvj8". However, the ACME server was expecting "LC2eEY_9lRLgxUfCME6TMB4bS0wslm2PDLTC2-JU-YY.bJcgrK6oTvjgvkuMOOkpdS8BdOotqf_z9T8zw-eFWOE", because that's what was POSTed as part of the challenge update. So the validation request fails, and `www.example.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.example.com&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=gXwrr-LtE9-WtzMFkuPVVQ&m=Kf71ta8IvyHrmRBLyoftWxjzSsiAc4tAN7OdanLQods&s=BWdYUEKrwtj9CJCO_xPHluLnoRWsgHzPCuNl4Kn88oY&e=>` has to start the whole issuance process all over again.

There's one additional piece of context here: We assume that the hosting provider operating the ACME client doesn't yet control `www.example.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.example.com&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=gXwrr-LtE9-WtzMFkuPVVQ&m=Kf71ta8IvyHrmRBLyoftWxjzSsiAc4tAN7OdanLQods&s=BWdYUEKrwtj9CJCO_xPHluLnoRWsgHzPCuNl4Kn88oY&e=>`. This can happen, for instance, when a hosting provider is trying to onboard a customer who already uses HTTPS. The hosting provider needs to have a valid certificate in place *before* the customer CNAMEs their domain name the the hosting provider.

The "clarify impact of roll-over" PR (https://github.com/ietf-wg-acme/acme/pull/323/files<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ietf-2Dwg-2Dacme_acme_pull_323_files&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=gXwrr-LtE9-WtzMFkuPVVQ&m=Kf71ta8IvyHrmRBLyoftWxjzSsiAc4tAN7OdanLQods&s=ssQvg07YjwC47XPynTnrpQMWwcOpkBIHFvmvgW7a07o&e=>) is a nice clarification, but doesn't actually address this use case. The issue here is what happens during the period between creation of a challenge and the ACME client requesting validation of that challenge, which in current implementations can be up to 7 days.

A) We could change the spec to set the keyAuthorization field when a challenge is created, so that it would have the current account key thumbprint baked in. This would allow hosting providers to treat the key authorization more like an opaque string, and calculate it at challenge creation time rather than at challenge update time.

B) We could encourage implementers to shorten the time between challenge creation and validation by doing delegation. That is, instead of giving `www.example.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.example.com&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=gXwrr-LtE9-WtzMFkuPVVQ&m=Kf71ta8IvyHrmRBLyoftWxjzSsiAc4tAN7OdanLQods&s=BWdYUEKrwtj9CJCO_xPHluLnoRWsgHzPCuNl4Kn88oY&e=>` a token, and waiting for them to deploy it, the hosting provider could say "Please delegate issuance authority to us, either by redirecting all of /.well-known/acme-validation/* to our validation server, or by CNAMEing _acme-challenge.www.example.com to our validation server." The main problems with this: the owners of `www.example.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.example.com&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=gXwrr-LtE9-WtzMFkuPVVQ&m=Kf71ta8IvyHrmRBLyoftWxjzSsiAc4tAN7OdanLQods&s=BWdYUEKrwtj9CJCO_xPHluLnoRWsgHzPCuNl4Kn88oY&e=>` may not be familiar enough with redirects (although CNAMEs should be easy); the owners of `www.example.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.example.com&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=gXwrr-LtE9-WtzMFkuPVVQ&m=Kf71ta8IvyHrmRBLyoftWxjzSsiAc4tAN7OdanLQods&s=BWdYUEKrwtj9CJCO_xPHluLnoRWsgHzPCuNl4Kn88oY&e=>` may have a concurrently running ACME client of their own that needs to renew during the time window they would be serving a redirect or CNAME.