Subject: [DNSOP] DNS cookies and multi-vendor anycast incompatibility

From: Petr Špaček <[email-masked]>
Date: Wed, 20 Jun 2018 10:40:28 +0200

Hello dnsop,

it seems that current specification of DNS cookies in RFC 7873 is not 
detailed enough to allow deployment of DNS cookies in multi-vendor 
anycast setup, i.e. a setup where one IP address is backed by multiple 
DNS servers.

The problem is lack of standardized algorithm to generate server cookie 
from a shared secret. In practice, even if users manually configure the 
same shared secret, Knot DNS and BIND will use diffrent algorithm to 
generate server cookie and as consequence these two cannot reliably back 
the same IP address and have DNS cookies enabled.

One of root server operators told me that they are not going to enable 
DNS cookies until it can work with multi-vendor anycast, and I think 
this is very reasonable position.

So, vendors, would you be willing to standardize on small number of 
server cookie algorithms to enable multi-vendor deployments?

-- 
Petr Špaček  @  CZ.NIC

---

From: Paul Wouters <[email-masked]>
Date: Wed, 20 Jun 2018 10:10:03 -0400 (EDT)

On Wed, 20 Jun 2018, Petr Špaček wrote:

> detailed enough to allow deployment of DNS cookies in multi-vendor anycast 
> setup, i.e. a setup where one IP address is backed by multiple DNS servers.
>
> The problem is lack of standardized algorithm to generate server cookie from 
> a shared secret. In practice, even if users manually configure the same 
> shared secret, Knot DNS and BIND will use diffrent algorithm to generate 
> server cookie and as consequence these two cannot reliably back the same IP 
> address and have DNS cookies enabled.
>
> One of root server operators told me that they are not going to enable DNS 
> cookies until it can work with multi-vendor anycast, and I think this is very 
> reasonable position.
>
> So, vendors, would you be willing to standardize on small number of server 
> cookie algorithms to enable multi-vendor deployments?

I think this is a good idea but there are already two examples in RFC
7873 for cookie generation. Is there a problem with those examples, or
is there only a lack of options in the implementation to configure
these? If the latter, than no new IETF work would be needed.

Paul

---

From: Petr Špaček <[email-masked]>
Date: Wed, 20 Jun 2018 16:25:56 +0200

On 20.6.2018 16:10, Paul Wouters wrote:
>> The problem is lack of standardized algorithm to generate server
>> cookie from a shared secret. In practice, even if users manually
>> configure the same shared secret, Knot DNS and BIND will use diffrent
>> algorithm to generate server cookie and as consequence these two
>> cannot reliably back the same IP address and have DNS cookies enabled.
>>

These are mere examples and not specifications with all the details
necessary for reliable interoperability.
E.g. when a cookie is "old" according to B.2.?
E.g. are there privacy considerations with plain HMAC vs. encryption?

Besides this, BIND defaults to AES-based algorithm which is not
specified in the RFC and Knot DNS has its own because developers
considered the BIND's approch overkill.

If we decide to standardize we need to find a reasonable algorihm and
standardize all its variables to make it work without run-time
synchronization (posssibly except key rotation but it can be done
avoided as well).

This message is for other DNS vendors to see if there is an interest in
standardizing something we can all share and operators use in practice.

-- 
Petr Špaček  @  CZ.NIC

---

From: Mark Andrews <[email-masked]>
Date: Thu, 21 Jun 2018 07:01:52 +1000

> On 21 Jun 2018, at 12:25 am, Petr Špaček <[email-masked]> wrote:
> 

The server cookie examples have all the details required to build a interoperable
implementation.  i.e. with the same inputs you will get the same outputs.

> _______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742              INTERNET: [email-masked]

---

From: Petr Špaček <[email-masked]>
Date: Thu, 21 Jun 2018 09:24:31 +0200

> implementation.  i.e. with the same inputs you will get the same outputs.

Sorry, I still do not think it is sufficient for interoperable
implementations for multiple reasons:

- Let's assume that serverA has time limit for cookie freshness "1 hour"
and the serverB in the same anycast cluster "2 hours". This will produce
unnecessary BADCOOKIE errors from time to time.

- It is not specified how "Time" sub-field in B.2. is encoded on the
wire so it is not guaranteed that two different implementations will
decode the value in the same way.
(Is it in integer seconds? Unix timestamp? Signed or unsigned? How do we
handle rollover over 0xFFFFFFFF? ...)

- Besides data format we need to explicitly state if sub-fields are big
endinan (I guess) or not, etc.

Also, let me ask why BIND decided to use AES by default instead of
HMAC-SHA256-64 described in the RFC? For me it is appealing to hide
content of the cookie from attacker eyes but I'm not willing to
implement something without proper description and understanding
(cargo-cult algorithms instead of spec, anyone? :-).

So let me ask again:
Are other vendors willing to work on sufficiently detailed
specification? If not just say it!

In that case I will be happy to reply to our friendly root operator that
it is not going to happen, and consequently throw away code for cookies
from Knot Resolver. I do not see value in it without interoperable spec,
just maintenance costs.

Petr Špaček  @  CZ.NIC

---

From: Ondřej Surý <[email-masked]>
Date: Thu, 21 Jun 2018 09:53:48 +0200

> On 21 Jun 2018, at 09:24, Petr Špaček <[email-masked]> wrote:
> So let me ask again:

+1 from ISC. I believe that we need to improve interoperability between the
implementation or people will not be willing to deploy DNS cookies at all.

Ondrej
--
Ondřej Surý
[email-masked]

---

From: Donald Eastlake <[email-masked]>
Date: Thu, 21 Jun 2018 08:38:49 -0400

Hi,

As the first author of the DNS Cookies RFC, I would be happy to generate a
draft to standardize this to improve inter vendor interoperability for
anycast servers.

Thanks,
Donald

-- 
Sent from Gmail Mobile

---

From: Petr Špaček <[email-masked]>
Date: Thu, 21 Jun 2018 15:13:31 +0200

On 21.6.2018 14:38, Donald Eastlake wrote:
> Hi,
> 
> As the first author of the DNS Cookies RFC, I would be happy to generate
> a draft to standardize this to improve inter vendor interoperability for
> anycast servers.

Good! Where do we start?

Right now I'm aware of couple requirements:
1. it has to be easy to configure shared parameters (obviously)
2. code to generate server side cookies must not slow down servers in
benchmarks
3. bonus points if we can standardize standardized key derivation
function so all servers can be configured using the same "master key"
and derive/rotate keys in lockstep without operator intervention and
explicit synchronization
[i.e. something along lines deriv_func(master_key, day_in_GMT) or so]

Besides these I'm eager to hear reasoning why BIND selected AES/SHA-HMAC
and other requirements which were considered during the original
implementation.

On Knot DNS side the main criterion was basically speed, that's why
Daniel Salzman ended up using SipHash algorithm.

Petr Špaček  @  CZ.NIC

> Thanks,
> Donald 
> 
> On Thu, Jun 21, 2018 at 03:54 Ondřej Surý <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
>     > On 21 Jun 2018, at 09:24, Petr Špaček <[email-masked]
>     <mailto:[email-masked]>> wrote:
>     > So let me ask again:

---

From: Mark Andrews <[email-masked]>
Date: Fri, 22 Jun 2018 00:36:30 +1000

> E.g. when a cookie is "old" according to B.2.?

I really depends on the algorithm being used.

> E.g. are there privacy considerations with plain HMAC vs. encryption?

Not the way AES is being used. 

> considered the BIND's approch overkill.

What does it matter what the default algorithm is.  Unless you are
running a anycast cluster IT DOES NOT MATTER.  If you are running
a anycast cluster you just needs to make sure all the server are set
up the same way.

> synchronization (posssibly except key rotation but it can be done

BIND implemented 3 algorithms.  All three were similar.  The only
difference was how the hash is generated.  The hmac-sha1 and hmac-sha256-8
take identical inputs.  AES is slightly different because it is AES.

---

From: Mark Andrews <[email-masked]>
Date: Fri, 22 Jun 2018 01:01:58 +1000

> On 21 Jun 2018, at 5:24 pm, Petr Špaček <[email-masked]> wrote:
> 

Your worried about a client that hasn’t talked to the anycast cluster for
a hour getting a BADCOOKIE?  Knit picking to the extreme. 

> handle rollover over 0xFFFFFFFF? …)

It is the seconds since Jan 1 1970 ignoring leap seconds in named.  Serial
arithmetic will be fine for the rollover in 2038 or you can just use a
straight unsigned modulus value and have a extra BADCOOKIE per client
during the first hour.

> endinan (I guess) or not, etc.

We used network byte order like every other field in the DNS.

> HMAC-SHA256-64 described in the RFC?

Because it is faster and unless you are running a anycast cluster it
doesn’t matter what algorithm is being used as all you need to be able
to do is handle your own output being returned to you.

BIND implements 2 or 3 hashes.  How many depends on what version of OpenSSL
it is linked against does it have AES support or not. AES is used to generate
a HASH which is actually documented in RFC 7873.  There is no “hiding” the
cookie from from the attacker.

> For me it is appealing to hide

---

From: Mark Andrews <[email-masked]>
Date: Fri, 22 Jun 2018 01:09:14 +1000

> On 21 Jun 2018, at 5:21 pm, Daniel Salzman <[email-masked]> wrote:
> 
> Hello Mark,
> 
> On 06/20/2018 11:01 PM, Mark Andrews wrote:
>> 
> So how should the DNS cookies be implemented? IMHO if one server uses https://tools.ietf.org/html/rfc7873#appendix-B.1
> and another server uses https://tools.ietf.org/html/rfc7873#appendix-B.2, then it's not interoperable.
> Actually the upcoming Knot DNS 2.7 implemented "B.1" using Siphash instead of FNV. Bind probably implemented B.2.
> Are both implementations correct?

Well you are free to inspect the code to ensure that it behaves the way
we said it did.  The code is in lib/ns/client.c:compute_cookie.  Older
branches that is bin/named/client.c:compute_cookie

> Daniel

---

From: Warren Kumari <[email-masked]>
Date: Thu, 21 Jun 2018 11:19:55 -0400

> a anycast cluster you just needs to make sure all the server are set
​There are a number of anycast clusters which run different implementations
on the same IP.

As an example: https://www.ripe.net/analyse/dns/k-root
"A K-root node consists of one or more servers running BIND, Knot or NSD."​

​Unless I'm missing something obvious, it would be good having cookies be
valid when you e.g happen to hit a node in Lyon then a node in Geneva
​(because of routing changes).

​W​

-- 
I don't think the execution is relevant when it was obviously a bad idea in
the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair of
pants.
   ---maf

---

From: Mukund Sivaraman <[email-masked]>
Date: Thu, 21 Jun 2018 21:18:31 +0530

> branches that is bin/named/client.c:compute_cookie

Mark, can you check the code in the compute_cookie() vs. RFC 7873 section B.2?

RFC 7873 says:

 hash =
           HMAC-SHA256-64( Server Secret,
               (Client Cookie | Nonce | Time | Client IP Address) )

compute_cookie() seems to perform:

           HMAC-SHA256-64( Server Secret,
               (Client Cookie | Nonce | Time | Client IP Address | Client Cookie) )

Both process_cookie() and ns_client_addopt() use the same
compute_cookie() to generate and verify, so it would be hard to spot
this.

		Mukund

---

From: Mark Andrews <[email-masked]>
Date: Fri, 22 Jun 2018 09:43:54 +1000

> On 22 Jun 2018, at 1:48 am, Mukund Sivaraman <[email-masked]> wrote:
> 
>               (Client Cookie | Nonce | Time | Client IP Address |Client Cookie) )
> 

Looks like a bug.  Will remove the second Client Cookie.

---

From: Mark Andrews <[email-masked]>
Date: Fri, 22 Jun 2018 10:00:08 +1000

> On 22 Jun 2018, at 1:19 am, Warren Kumari <[email-masked]> wrote:
> 
> 
> 
> ​There are a number of anycast clusters which run different implementations on the same IP.
> 
> ​Unless I'm missing something obvious, it would be good having cookies be valid when you e.g happen to hit a node in Lyon then a node in Geneva ​(because of routing changes).
> 
> ​W​

Having different cookies return was a EXPECTED failure mode and the
client has code to handle that if they have followed RFC 7873.  These
servers all handle TCP clients.  They should be stable enough to not
trigger the fallback to TCP.

   If the extended RCODE in the reply is BADCOOKIE and the Client Cookie
   in the reply matches what was sent, it means that the server was
   unwilling to process the request because it did not have the correct
   Server Cookie in it.  The client SHOULD retry the request using the
   new Server Cookie from the response.  Repeated BADCOOKIE responses to
   requests that use the Server Cookie provided in the previous response
   may be an indication that either the shared secrets or the method for
   generating secrets in an anycast cluster of servers is inconsistent.
   If the reply to a retried request with a fresh Server Cookie is
   BADCOOKIE, the client SHOULD retry using TCP as the transport, since
   the server will likely process the request normally based on the
   security provided by TCP (see Section 5.2.3).

> I don't think the execution is relevant when it was obviously a bad idea in the first place.
> This is like putting rabid weasels in your pants, and later expressing regret at having chosen those particular rabid weasels and that pair of pants.
>    ---maf

---

From: Daniel Salzman <[email-masked]>
Date: Thu, 21 Jun 2018 09:21:58 +0200

Hello Mark,

Thanks,
Daniel

---

From: Daniel Salzman <[email-masked]>
Date: Thu, 21 Jun 2018 17:20:36 +0200

> branches that is bin/named/client.c:compute_cookie

I mean if one implementation computes cookies based on "Client IP Address | Client Cookie | Server Secret",
then it cannot be interoperable with another implementation which is based on "Server Secret, (Client Cookie | Nonce | Time | Client IP Addres)",
no matter which hashing algorithm is used. And still both implementations comply with the RFC (I think).

---

From: Evan Hunt <[email-masked]>
Date: Sat, 23 Jun 2018 00:29:16 +0000

On Thu, Jun 21, 2018 at 11:19:55AM -0400, Warren Kumari wrote:
> There are a number of anycast clusters which run different
> implementations on the same IP.

Sure, but as long as the algorithm is settable for each server in the
anycast so that all of them can match, then I don't think it matters
if the different implementations have different defaults, does it?

-- 
Evan Hunt -- [email-masked]
Internet Systems Consortium, Inc.

---

From: Warren Kumari <[email-masked]>
Date: Fri, 22 Jun 2018 22:26:55 -0400

> if the different implementations have different defaults, does it?

​Yup, as long as they can be set to be the same algorithm, and same inputs,
you are correct. ​

> a anycast cluster you just needs to make sure all the server are set
​

I have not tried configuring cookie on Knot, but looking
in alg_containers.c, I can configure:
{ 0, "FNV-64" },
{ 1, "HMAC-SHA256-64" }

Under BIND:
cookie-algorithm:
Set the algorithm to be used when generating the server cookie. One of
"aes", "sha1"
or "sha256". The default is "aes" if supported by the cryptographic library
or otherwise
"sha256".

So, if I set both to use their (non-default) of SHA256 (and set the same
secret:-)) do they actually generate compatible cookies?
I'd guess / assume so, but I haven't tested this...
W

---

From: Mukund Sivaraman <[email-masked]>
Date: Sat, 23 Jun 2018 08:18:40 +0530

> I'd guess / assume so, but I haven't tested this...

BTW, speaking of secret, section 7.1 (Client and Server Secret Rollover)
of RFC 7873 says this:

   The longer a secret is used, the higher the probability that it has
   been compromised.  Thus, clients and servers are configured with a
   lifetime setting for their secret, and they roll over to a new secret
   when that lifetime expires, or earlier due to deliberate jitter as
   described below.  The default lifetime is one day, and the maximum
   permitted is one month.

I don't think BIND has any support for automatic secret
rollovers. Re-configuring the secret manually per-day is not realistic
for a nameserver cluster.. at least it would be inconvenient and require
external tooling.

There should be a specification of KDF usage for this to work across
implementations.

		Mukund

---

From: Evan Hunt <[email-masked]>
Date: Sat, 23 Jun 2018 17:09:21 +0000

> I'd guess / assume so, but I haven't tested this...

That's the intention.  Mukund recently pointed out a bug in the hash inputs
BIND is using, so it might not work right now.

We really should have a COOKIE bakeoff (worth doing for the pun alone) to
check for interoperability issues.  Montreal would seem like a good time
and place for it, but I'm not going to be able to attend this time, so I
can't volunteer to run it. If someone else wants to step in, that'd be
great.

---

From: Petr Špaček <[email-masked]>
Date: Mon, 25 Jun 2018 12:15:44 +0200

> BIND is using, so it might not work right now.

There are differences in *inputs* to the hash function in BIND and Knot
DNS as well, so it will not work anyway.

> can't volunteer to run it. If someone else wants to step in, that'd be

I'm not doing Montreal myself so maybe Bangkok could be the place to
test a new spec?

-- 
Petr Špaček  @  CZ.NIC