Subject: Re: [Acme] Support for domains with redundant but not immediately synchronized servers

From: Jonas Wielicki <[email-masked]>
Date: Thu, 3 Dec 2015 21:56:57 +0100

On 01.12.2015 02:36, Hugo Landau wrote:
>> Is such a thing planned? Are there security reasons against
>> doing this? Are there security reasons against doing this on a
>> DNSSEC signed domain (which klausurschokola.de is)?
> 
> Personally, I wouldn't think it unreasonable to allow an ACME
> client to request that a specific IP be used for the purposes of a
> challenge. The server would then verify that that IP is one of the
> candidate IPs, rather than selecting one at random. I don't see
> that this causes any loss of security, so it seems like a sensible
> inclusion.

That is great to hear! What is the preferred course of action now?
Should I open an issue on the protocol draft repository? (Which I
assume is at [1])

best regards,
Jonas

   [1]: https://github.com/letsencrypt/acme-spec

---

From: "Salz, Rich" <[email-masked]>
Date: Thu, 3 Dec 2015 21:08:20 +0000

> Should I open an issue on the protocol draft repository? (Which I assume is at [1])

No!  Please use the IETF WG repo, at https://github.com/ietf-wg-acme/acme/issues

---

From: Jacob Hoffman-Andrews <[email-masked]>
Date: Thu, 3 Dec 2015 15:26:42 -0800

I think there are a few possible approaches to domain validation for
hostnames that resolve to multiple IP addresses:

1) It is sufficient to deploy a challenge response on any IP address.
2) The challenge response must be deployed on all IP addresses.
3) The server tries one deterministic IP address and succeeds or fails
based on whether the challenge is deployed there.

Let's Encrypt currently does (3), but it's probably the worst of both
worlds and we should fix it.

Arguably (2) is the most secure, since a single-machine compromise isn't
sufficient to issue a certificate.

And (1) is the easiest and most user-friendly. I think it's also similar
to existing practice in the wild.

Jonas, security-wise your proposal is equivalent to (1). Rather than
adding a preferred IP address to the spec, why not specif that ACME
servers SHOULD implement (1)? Alternately we could state in the spec
that the choice of IP is a policy choice of the CA, and lay out the two
most reasonable options (1 and 2).

On 11/30/2015 09:17 AM, Jonas Wielicki wrote:
> Hi list,
>
> I have asked this in the IRC and was pointed to this mailing list. I
> tried to get a certificate for klausurschokola.de via Let’s Encrypt
> during the currently running limited beta (we have the domain
> whitelisted). The name has the following address records:
>
> 1800     IN     A    176.9.101.187
> 1800     IN     A     217.115.12.71
>
> (in addition, there is one AAAA record for each of the machines
> addressed by the A records)
>
> As you can see, two different machines are addressed. Those are
> physically separated machines with different main administrators.
> Both are pulling their web content from the same source, but it is not
> supposed to be dynamic, so there is no "fast" (order of seconds) way
> to mirror the content.
>
> Our wish would be to be able to use different private keys and
> certificates for both hosts, and renew these independently from the
> other host. We thought that this would be possible using Let’s Encrypt.
>
> The problem is that currently, the Let’s Encrypt server sometimes
> chooses the wrong of the two IPs to ask for the file in
> /.well-known/acme-challenge. Ideally, it would use the IP of the
> requester (of course only after it has verified that the IP is in the
> DNS) or allow the requester to specify a preferred IP.
>
> For example, on 176.9.101.187:
>
> # letsencrypt certonly -c ~/schoko.ini -d klausurschokola.de -d
> www.klausurschokola.de
>
> [… curses …]
>
> Failed authorization procedure. klausurschokola.de (http-01):
> unauthorized :: The client lacks sufficient authorization :: Invalid
> response from
> http://klausurschokola.de/.well-known/acme-challenge/c5HJrtp8t8JhfNgTXVC
> 8N7OsCrguAWGw-JTIJxCFeIQ
> [217.115.12.71]: 404
>
>
> Is such a thing planned? Are there security reasons against doing
> this? Are there security reasons against doing this on a DNSSEC signed
> domain (which klausurschokola.de is)?
>
> best regards,
> Jonas
> > _______________________________________________ > Acme mailing list >
[email-masked] > https://www.ietf.org/mailman/listinfo/acme

---

From: Martin Thomson <[email-masked]>
Date: Fri, 4 Dec 2015 19:23:54 +1100

This seems to be a common problem, so I opened a PR that someone on
that project can merge.

On 4 December 2015 at 08:08, Salz, Rich <[email-masked]> wrote:
> _______________________________________________
> Acme mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/acme

---

From: Peter Eckersley <[email-masked]>
Date: Fri, 4 Dec 2015 00:46:01 -0800

There's a fairly good solution available with the current protocol,
which is to serve a (long lived) redirect from
/.well-known/acme-challenge/ on all of the servers to a different URL
that is always answered by the machine you run an ACME client on.

Are there any cases where that is sufficiently unworkable to warrant a
protocol change?

On Mon, Nov 30, 2015 at 06:17:21PM +0100, Jonas Wielicki wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
> 
> Hi list,
> 
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2
> 
> iQIcBAEBCgAGBQJWXIShAAoJEMBiAyWXYliKJ1wP/iGVeGRxnAkrAstfjeGLvLeC
> TXnF76X/8xC3s4dd/UR0DE2n9Pdn0FYCK+6jRTn+Xpa0MvrA2ME20AZMh070Ghy0
> JRbdTWqjQTHzvjXYQHjSkW24pyZNgdfnmwd0HiAhn1mANv3dhVTnHR4hibZww+Su
> ty3XzsyZYjrfQ3K5/bTb/jz+QZUoZ/fJJuNlyMsVInF3rzagj34WWR4sYbAIwKEF
> CTvBFxINY04pUeemYlywPYrUOmcJTOK/wVi1ya2BgLgTqNJP5FJOX5jCHHr8m5ej
> A7G/nGWFSybOG1GkjMOdST3uMeL7HlpqhUnuNzsiC3ZAfmgVwceLsG3bTCAxcrgB
> 7XiSs3MrURuEk17w2QB0Oyt487DrmftzFo3vzvCrrl42au9JV69Y14/0W3z5piYM
> DIGpd/KNSL2m6xvzoJHoi+o5lTl9GiP6KQKlJiIUtn2cz8Ro6CiwXkhD0FmG8sP7
> 4wqg+vnpcTdhrzsWuAPrpGej+GT1LlWOLERnyPOfVhQ8EUPanwgUbGo1uTfHB2mj
> T2CdCCZhcmJFurvz+7FVI1WaVgGR/rdZbu4ueC+0YNZEOICXE0pIJEw8rKWJbqe3
> lKchgpR6jR3TKHHwNFDIZj049TBiEGxMXsdEaGlLOHdnr4ZlIDgfycumhYVTNJUi
> IDHRifjFUchCynluOhZi
> =3akD
> -----END PGP SIGNATURE-----
> 

-- 
Peter Eckersley                            [email-masked]
Chief Computer Scientist          Tel  +1 415 436 9333 x131
Electronic Frontier Foundation    Fax  +1 415 436 9993

---

From: Ryan Pendleton <[email-masked]>
Date: Fri, 4 Dec 2015 02:52:54 -0700

Personally, I think that's a more appropriate approach.

Even if a protocol change was made that allowed an ACME client to pin
the challenge to a certain IP address, the requested IP may not always
be returned by the authoritative DNS server. Any type of latency, geo or
weighted routing algorithm could potentially get in the way.

---

From: Yoav Nir <[email-masked]>
Date: Fri, 4 Dec 2015 14:48:54 +0200

+1

> On 4 Dec 2015, at 11:52 AM, Ryan Pendleton <[email-masked]> wrote:
>

---

From: Ted Hardie <[email-masked]>
Date: Fri, 4 Dec 2015 11:35:04 -0800

> Are there any cases where that is sufficiently unworkable to warrant a
​So, the initial use case Jonas laid out was for a service where they
wanted two different certs associated with the domain, which could be
maintained independently.  The draft currently says:

The path at which the resource is provisioned is comprised of the fixed
> prefix ".well-known/acme-challenge/", followed by the "token" value in the
> challenge. The value of the resource MUST be the ASCII representation of
> the key authorization.
>
> .well-known/acme-challenge/evaGxfADs6pSRb2LAv9IZf17Dt3juxGJ-PCt92wr-oA
>
>
> The client's response to this challenge indicates its agreement to this
> challenge by sending the server the key authorization covering the
> challenge's token and the client's account key:
>
​If I understand your suggestion, to get what he wants, whoever is running
the machine on which the ACME client is run would have to generate two URIs
of the form

http://{domain}/.well-known/acme-challenge/{token}

do the signing with the account for the bodies created for each, and so
on.   That seems a little different from the work flow he had in mind, but
it is probably doable if we have the CAs recognize that having more than
one token outstanding may match this workflow rather than a replacement
workflow.

regards,

Ted

---

From: "Manger, James" <[email-masked]>
Date: Mon, 7 Dec 2015 11:32:15 +1100

>> Ideally, it [Let's Encrypt] would use the IP of the 

This is quite a sensible feature request from Jonas. It supports multiple servers for a domain while encouraging keys that are tied to a single piece of hardware, without adding extra coordination requirements. It doesn't feel too onerous for CAs to implement.

> There's a fairly good solution available with the current protocol, which is to serve a (long lived) redirect from /.well-known/acme-challenge/ on all of the servers to a different URL that is always answered by the machine you run an ACME client on.

This redirect-based workaround feels far from ideal. It assumes 1 server does all the ACME bits, which discourages per-hardware keys. It requires more coordination between servers (1 is different; others need its IP; need some extra mechanism to distribute key+cert once issued). Section 9.2 "Integrity of Authorizations" [draft-ietf-acme-acme-01] warns that an ACME CA following redirects can be risky as a web site might have a catch-all redirect rule. Even worse, the spec hints that CAs might not (or must not?) follow redirects at all due to the risk when it says: "suppose an ACME server follows HTTP redirects in Simple HTTP validation…".

App-level redirects might be necessary to cope with some load balancing schemes, but specifying that an ACME server must prefer the requester's IP or a specified IP from the A/AAAA records DNS returns would still be useful.

--
James Manger

-----Original Message-----
From: Acme [mailto:[email-masked]] On Behalf Of Ryan Pendleton
Sent: Friday, 4 December 2015 8:53 PM

Even if a protocol change was made that allowed an ACME client to pin the challenge to a certain IP address, the requested IP may not always be returned by the authoritative DNS server. Any type of latency, geo or weighted routing algorithm could potentially get in the way.

-----Original Message-----
From: Acme [mailto:[email-masked]] On Behalf Of Peter Eckersley
Sent: Friday, 4 December 2015 7:46 PM
To: Jonas Wielicki <[email-masked]>
Cc: [email-masked]
Subject: Re: [Acme] Support for domains with redundant but not immediately synchronized servers

There's a fairly good solution available with the current protocol, which is to serve a (long lived) redirect from /.well-known/acme-challenge/ on all of the servers to a different URL that is always answered by the machine you run an ACME client on.

Are there any cases where that is sufficiently unworkable to warrant a protocol change?

-----Original Message-----
From: Acme [mailto:[email-masked]] On Behalf Of Jacob Hoffman-Andrews
Sent: Friday, 4 December 2015 10:27 AM

I think there are a few possible approaches to domain validation for hostnames that resolve to multiple IP addresses:

1) It is sufficient to deploy a challenge response on any IP address.
2) The challenge response must be deployed on all IP addresses.
3) The server tries one deterministic IP address and succeeds or fails based on whether the challenge is deployed there.

Let's Encrypt currently does (3), but it's probably the worst of both worlds and we should fix it.

Arguably (2) is the most secure, since a single-machine compromise isn't sufficient to issue a certificate.

And (1) is the easiest and most user-friendly. I think it's also similar to existing practice in the wild.

Jonas, security-wise your proposal is equivalent to (1). Rather than adding a preferred IP address to the spec, why not specif that ACME servers SHOULD implement (1)? Alternately we could state in the spec that the choice of IP is a policy choice of the CA, and lay out the two most reasonable options (1 and 2).