Subject: [DNSOP] ECDSA woes

From: Mikael Abrahamsson <[email-masked]>
Date: Sat, 15 Oct 2016 08:22:43 +0200 (CEST)

Hi,

we have a deployment of home gateways, based on OpenWrt BB that uses 
dnsmasq v2.71 as resolver, with DNSSEC validation turned on. It seems some

Dnsmasq v2.71 does not support ECDSA. A rather large CDN uses ECDSA only. 
I also found bug reports for Debian with same problem, because they also 
used dnsmasq.

Breakage occured, for instance www.ietf.org was not resolvable.

Our plan is now to disable DNSSEC validation on all of these HGWs.

So I read some documents:

https://tools.ietf.org/html/draft-wouters-sury-dnsop-algorithm-update-02
https://tools.ietf.org/html/draft-york-dnsop-deploying-dnssec-crypto-algs-01

I via these found RFC4035:

"If the resolver does not support any of the algorithms listed in an
    authenticated DS RRset, then the resolver will not be able to verify
    the authentication path to the child zone.  In this case, the
    resolver SHOULD treat the child zone as if it were unsigned."

So obviously dnsmasq doesn't implement this SHOULD, because it treats 
these zones as bogus and doesn't respond back to the client.

(btw, what happens if the entire child zone and all its RRs are signed 
with an unknown algoritm, is that even covered in the above paragraph?)

It took us a while to figure out why things didn't work. We even fault 
reported this to the CDN who never at any time (during their prompt and 
friendly communication) indicated that they had any knowledge of resolvers 
that didn't support their chosen algorithm, or pointed me in that 
direction.

So... my question to you fine people is:

Is there any (existing and freely available) testing suite I can run 
against my chosen resolver that tests all the SHOULDs and MUSTs regarding 
DNSSEC validation, including future proofing for new algorithms?

If not, I would like to call upon for instance ccTLD registrys, ISOC and 
others, to develop a test suite for this, maintain it over time, and make 
it freely available.

I like DNSSEC and want to see it widely deployed. It's an important part 
of Internet plumbing. These kinds of problems that I've had last weeks 
mean people who oppose it with FUD actually have concrete breakage to 
point at that means it's not "Uncertain" anymore.

Thanks.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Ray Bellis <[email-masked]>
Date: Sat, 15 Oct 2016 09:30:43 -0500

On 15/10/2016 01:22, Mikael Abrahamsson wrote:

> against my chosen resolver that tests all the SHOULDs and MUSTs
> regarding DNSSEC validation, including future proofing for new algorithms?
> 
> others, to develop a test suite for this, maintain it over time, and
> make it freely available.
> 

It's not exactly what you've asked for, but I have an iOS app under
development that can test your phone's configured resolvers for various
DNS protocol conformance issues.

At the moment the extent of the DNSSEC specific tests is to check that a
query for the root zone's SOA with the +CD flag returns the expected RRSIGs.

I hadn't considered algorithm-specific tests, but the app could in
theory include tests for whether zones known to be signed with specific
algorithms can be correctly resolved with +AD returned.

Ray

---

From: Roy Arends <[email-masked]>
Date: Sat, 15 Oct 2016 15:31:08 +0100

> On 15 Oct 2016, at 07:22, Mikael Abrahamsson <[email-masked]> wrote:
> 
> 
> Hi,
> 
> we have a deployment of home gateways, based on OpenWrt BB that uses dnsmasq v2.71 as resolver, with DNSSEC validation turned on. It seems some
> 
> Dnsmasq v2.71 does not support ECDSA. A rather large CDN uses ECDSA only. I also found bug reports for Debian with same problem, because they also used dnsmasq.

Dnsmasq v2.73 does support ECDSA. Is there a possibility to get these home gateways on the newer dnsmasq?

Roy

---

From: Mikael Abrahamsson <[email-masked]>
Date: Sat, 15 Oct 2016 17:22:43 +0200 (CEST)

> algorithms can be correctly resolved with +AD returned.

What I would like to see are tests like:

set up a domain with a algorithm ID nobody will ever implement (reserve it 
if need be), and check that this domain returns as unvalidated (as per 
SHOULD in the RFC). Put in a MUST in relevant standards that 
implementation must not treat this identifier as anything but "I don't 
know anything about this" (ie don't implement specific tests for this 
"algorithm" and treat it differently from any other algorithm ID that is 
unknown).

These kinds of migration scenarios to newer algorithms MUST be hashed out, 
because otherwise we're never going to be able to deploy new algorithms 
(and per previous experience, it seems we want to change them every 5-10 
years).

---

From: "Ralf Weber" <[email-masked]>
Date: Sat, 15 Oct 2016 10:42:05 -0500

Moin!

On 15 Oct 2016, at 10:22, Mikael Abrahamsson wrote:

> set up a domain with a algorithm ID nobody will ever implement 
> (reserve it if need be), and check that this domain returns as 
> unvalidated (as per SHOULD in the RFC).
Geoff Houston did some research here some years ago and just did an 
update to his findings. You might want to look at:
	http://www.potaroo.net/ispcol/2016-10/ecdsa-v2.html

> Put in a MUST in relevant standards that implementation must not treat 
> this identifier as anything but "I don't know anything about this" (ie 
> don't implement specific tests for this "algorithm" and treat it 
> differently from any other algorithm ID that is unknown).
I'm not sure a change in the standards will be possible as if remember 
correct some people think that the fallback to insecure is a not a good 
thing. So am not sure if we could achieve consensus on that. I think the 
current RFC are clear enough and later version of dnsmasq have corrected 
the problem.

> These kinds of migration scenarios to newer algorithms MUST be hashed 
> out, because otherwise we're never going to be able to deploy new 
> algorithms (and per previous experience, it seems we want to change 
> them every 5-10 years).
Yes and there is some work in the TLD space. You might want to listen to 
Ondreys talk at DNS-OARC:
	https://indico.dns-oarc.net/event/25/session/2/contribution/2

There are always issues rolling out new stuff, the sooner we encounter 
and fix them the better. So thanks for finding and pointing it out.

So long
-Ralf

---

From: Mikael Abrahamsson <[email-masked]>
Date: Sat, 15 Oct 2016 17:53:40 +0200 (CEST)

On Sat, 15 Oct 2016, Ralf Weber wrote:

> Geoff Houston did some research here some years ago and just did an update to 
> his findings. You might want to look at:

Do we know how many experiments failed because the resolver erroneously 
reported error for ECDSA signed domains?

>From reading Geoffs text, it's not obvious to me that this error case is 
caught by his tests?

---

From: Marek Vavruša <[email-masked]>
Date: Sat, 15 Oct 2016 11:44:00 -0700

Hi,

not sure if it's exactly what you're looking for, but there's
https://github.com/CZ-NIC/deckard for (generic) resolver testing.
It mocks the environment for the tested binary, so you'll have to
provide a configuration template for dnsmasq.

Marek

On Fri, Oct 14, 2016 at 11:22 PM, Mikael Abrahamsson <[email-masked]> wrote:
>
> Hi,
>
> we have a deployment of home gateways, based on OpenWrt BB that uses dnsmasq
> v2.71 as resolver, with DNSSEC validation turned on. It seems some
>
> Dnsmasq v2.71 does not support ECDSA. A rather large CDN uses ECDSA only. I
> also found bug reports for Debian with same problem, because they also used
> dnsmasq.
>
> So obviously dnsmasq doesn't implement this SHOULD, because it treats these
> zones as bogus and doesn't respond back to the client.
>
> (btw, what happens if the entire child zone and all its RRs are signed with
> an unknown algoritm, is that even covered in the above paragraph?)
>
> that didn't support their chosen algorithm, or pointed me in that direction.
>
> Is there any (existing and freely available) testing suite I can run against
> my chosen resolver that tests all the SHOULDs and MUSTs regarding DNSSEC
> validation, including future proofing for new algorithms?
>
> others, to develop a test suite for this, maintain it over time, and make it
> freely available.
>
> I like DNSSEC and want to see it widely deployed. It's an important part of
> Internet plumbing. These kinds of problems that I've had last weeks mean
> people who oppose it with FUD actually have concrete breakage to point at
> that means it's not "Uncertain" anymore.
>
> Thanks.
>
> --
> _______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop

---

From: Geoff Huston <[email-masked]>
Date: Sun, 16 Oct 2016 08:23:35 +1100

> On 16 Oct. 2016, at 2:53 am, Mikael Abrahamsson <[email-masked]> wrote:
> 
>> Geoff Houston did some research here some years ago and just did an update to his findings. You might want to look at:
> Do we know how many experiments failed because the resolver erroneously reported error for ECDSA signed domains?
> 

so I have three tests:

A: a validly-signed ECDSA P-256 domain

B: an invalidly-signed ECDSA P-256 domain

C: an unsigned control

now if the resolver does NOT recognise ECDSA we should see a fetch for A, B and C  (as they treat both A and B as if they were unsigned)

if the resolver recognises ECDSA we will see a fetch of A and C but not B

and if the resolver incorrectly returns SERVFAIL when it sees ECDSA (which I presume is what DNSMASQ 2.71 is doing) then we should see only C and not A or B

The report generated uses these definitions to determine if a user is passing their queries to ECDSA-aware resolvers

So thats the long answer to yes, this error is caught by these tests, and the error is put into the “does not do ECDSA” bucket.

thanks,

   Geoff

---

From: Ólafur Guðmundsson <[email-masked]>
Date: Sat, 15 Oct 2016 21:15:51 -0400

I have domains signed by all combinations of signing algorithms and DS
digests as well as Nsec variants
Ds-n.alg-m-nsec.dnssec-test.org

Replace n with 1..4
M with 1..14
Nsec is one of Nsec nsec3 none

Ólafur

sent from phone

On Oct 15, 2016 17:29, "Geoff Huston" <[email-masked]> wrote:

> now if the resolver does NOT recognise ECDSA we should see a fetch for A,
> B and C  (as they treat both A and B as if they were unsigned)
>
> and if the resolver incorrectly returns SERVFAIL when it sees ECDSA (which
> I presume is what DNSMASQ 2.71 is doing) then we should see only C and not
> A or B
>
> The report generated uses these definitions to determine if a user is
> passing their queries to ECDSA-aware resolvers
>
> So thats the long answer to yes, this error is caught by these tests, and
> the error is put into the “does not do ECDSA” bucket.
>
> thanks,
>
>    Geoff
>
>
>
>

---

From: Mikael Abrahamsson <[email-masked]>
Date: Sun, 16 Oct 2016 10:14:06 +0200 (CEST)

> So thats the long answer to yes, this error is caught by these tests, and the error is put into the “does not do ECDSA” bucket.

Ok, thanks! It seems to me that anyone not loading A or B, but loads C, is 
of high interest. They're obviously behind a broken resolver.

Could you please try to create two buckets out of the "does not do ECDSA", 
one being "doesn't understand ECDSA, loads invalid resources" and "doesn't 
load ECDSA resources regardless of valid or not".

I'm very interested in the last of these two, because they're hindering 
rollout of new algorithms. I'd like to understand how big this breakage 
is.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Sun, 16 Oct 2016 10:16:19 +0200 (CEST)

> Nsec is one of Nsec nsec3 none

I'd be veryinterested if you could create an algorithm called "99" (or 
something), and we could test that. Anyone not loading the "99" resource 
is violating the "SHOULD", even if they understand ECDSA.

This would investigate ratio of problems when we want to introduce a new 
algorithm in the future.

---

From: Ólafur Guðmundsson <[email-masked]>
Date: Sun, 16 Oct 2016 12:29:04 -0400

I will be happy to do that,  stay tuned as I need to create a special
signer for it :-)

Olafur

On Sun, Oct 16, 2016 at 4:16 AM, Mikael Abrahamsson <[email-masked]>
wrote:

> something), and we could test that. Anyone not loading the "99" resource is
> violating the "SHOULD", even if they understand ECDSA.
>

---

From: Mark Andrews <[email-masked]>
Date: Mon, 17 Oct 2016 09:31:09 +1100

> I will be happy to do that,  stay tuned as I need to create a special

dnssec-signzone + awk + dnssec-dsfromkey works well.

e.g.
awk '$4 == "RRSIG" && $6 == 8 { $6 = 99 }
     $4 == "DNSKEY" && $7 == 8 { $7 = 99}
     { print }'

Mark

> > On Sat, 15 Oct 2016, =C3=93lafur Gu=C3=B0mundsson wrote:
> >
> > something), and we could test that. Anyone not loading the "99" resource =
> is
> --94eb2c0cd28c3de9dd053efdf57f
> Content-Type: text/html; charset=UTF-8
> Content-Transfer-Encoding: quoted-printable
> 
> <div dir=3D"ltr">I will be happy to do that, =C2=A0stay tuned as I need to =
> create a special signer for it :-)=C2=A0<div><br></div><div>Olafur</div><di=
> v><br></div></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote"=
> >On Sun, Oct 16, 2016 at 4:16 AM, Mikael Abrahamsson <span dir=3D"ltr">&lt;=
> <a href=3D"mailto:[email-masked]" target=3D"_blank">[email-masked]</a>&=
> gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 =
> 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=3D"">On Sat=
> , 15 Oct 2016, =C3=93lafur Gu=C3=B0mundsson wrote:<br>
> <br>
> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
> x #ccc solid;padding-left:1ex">
> I have domains signed by all combinations of signing algorithms and DS<br>
> digests as well as Nsec variants<br>
> <a href=3D"http://Ds-n.alg-m-nsec.dnssec-test.org" rel=3D"noreferrer" targe=
> t=3D"_blank">Ds-n.alg-m-nsec.dnssec-test.or<wbr>g</a><br>
> <br>
> Replace n with 1..4<br>
> M with 1..14<br>
> Nsec is one of Nsec nsec3 none<br>
> </blockquote>
> <br></span>
> I&#39;d be veryinterested if you could create an algorithm called &quot;99&=
> quot; (or something), and we could test that. Anyone not loading the &quot;=
> 99&quot; resource is violating the &quot;SHOULD&quot;, even if they underst=
> and ECDSA.<br>
> <br>
> This would investigate ratio of problems when we want to introduce a new al=
> gorithm in the future.<div class=3D"HOEnZb"><div class=3D"h5"><br>
> <br>
> -- <br>
> Mikael Abrahamsson=C2=A0 =C2=A0 email: <a href=3D"mailto:[email-masked]" =
> target=3D"_blank">[email-masked]</a></div></div></blockquote></div><br></=
> div>
> 
> --94eb2c0cd28c3de9dd053efdf57f--
> 
> 
> --===============9042271128241020298==
> Content-Type: text/plain; charset="us-ascii"
> MIME-Version: 1.0
> Content-Transfer-Encoding: 7bit
> Content-Disposition: inline
> 
> --===============9042271128241020298==--
> 
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Mark Andrews <[email-masked]>
Date: Mon, 17 Oct 2016 09:35:44 +1100

>      $4 == "DNSKEY" && $7 == 8 { $7 = 99}

Which by the way is what we do in our system tests for BIND 9.

#
# A zone with a unknown DNSKEY algorithm.
# Algorithm 7 is replaced by 100 in the zone and dsset.
#
zone=dnskey-unknown.example.
infile=dnskey-unknown.example.db.in
zonefile=dnskey-unknown.example.db

keyname=`$KEYGEN -q -r $RANDFILE -a NSEC3RSASHA1 -b 768 -n zone $zone`

cat $infile $keyname.key >$zonefile

$SIGNER -P -3 - -r $RANDFILE -o $zone -O full -f ${zonefile}.tmp $zonefile > /dev/null 2>&1

awk '$4 == "DNSKEY" { $7 = 100; print } $4 == "RRSIG" { $6 = 100; print } { print }' ${zonefile}.tmp > ${zonefile}.signed

$DSFROMKEY -A -f ${zonefile}.signed $zone > dsset-${zone}

---

From: Jan Včelák <[email-masked]>
Date: Mon, 17 Oct 2016 06:50:07 -0500

Hi Mikael,

> an unknown algoritm, is that even covered in the above paragraph?)

On delegation, the resolver checks the DS records in the parent zone. If
the DS record set contains a supported algorithm, the child zone should
be treated as secure. If no algorithm in the set is supported, the zone should
be treated as insecure.

So what happens in this case is that the domain name space below
the delegation with unsupported DS algorithm will be treated as insecure.

Cheers,

Jan

---

From: Dan York <[email-masked]>
Date: Tue, 18 Oct 2016 18:03:45 +0000

Mikael,

On Oct 15, 2016, at 11:22 AM, Mikael Abrahamsson <[email-masked]<mailto:[email-masked]>> wrote:

These kinds of migration scenarios to newer algorithms MUST be hashed out, because otherwise we're never going to be able to deploy new algorithms (and per previous experience, it seems we want to change them every 5-10 years).

Agreed! To capture this kind of information, a group of us wrote a draft in DNSOP about new crypto algorithms:

In section 2.1.1 we mention the situation with resolvers and unknown algorithms. However, we assume compliance with RFC 4035. Your case study here shows that we need to add some text about the challenge that can happen if the resolver does the wrong thing and fails the validation.

I'll add that. Thank you for bringing this case to the list.

It seems to me there is a larger issue of whether a system will "fail insecure" (or "fail open") or "fail secure".

RFC 4035 has the "fail insecure" view where the DNS info is still passed along, thus allowing the deployment of new algorithms to NOT break things, although with a lower level of security until the new algorithms are supported.

It seems the dnsmasq developers chose to "fail secure" thus potentially "protecting" the end user from insecure data, although in this case the data is secure, just not understood to be secure.

This is one of the tougher points of algorithm change, particularly when so many of the resolvers may be in commodity customer-premises equipment (CPE) that may or may not be easily updated or replaced.

Dan

--
Dan York
Senior Content Strategist, Internet Society
[email-masked]<mailto:[email-masked]>   +1-802-735-1624
Jabber: [email-masked]<mailto:[email-masked]>
Skype: danyork   http://twitter.com/danyork

http://www.internetsociety.org/

---

From: Mark Andrews <[email-masked]>
Date: Wed, 19 Oct 2016 08:23:56 +1100

In message <[email-masked]>, Dan York writes:
> Mikael,
>
> On Oct 15, 2016, at 11:22 AM, Mikael Abrahamsson
> <[email-masked]<mailto:[email-masked]>> wrote:
>
> algorithms (and per previous experience, it seems we want to change them
> every 5-10 years).
>
> In section 2.1.1 we mention the situation with resolvers and unknown
> algorithms. However, we assume compliance with RFC 4035. Your case study
> here shows that we need to add some text about the challenge that can
> happen if the resolver does the wrong thing and fails the validation.
>
> It seems to me there is a larger issue of whether a system will "fail
> insecure" (or "fail open") or "fail secure".
>
> RFC 4035 has the "fail insecure" view where the DNS info is still passed
> along, thus allowing the deployment of new algorithms to NOT break
> things, although with a lower level of security until the new algorithms
> are supported.

No.  RFC 4035 is "fail secure".  Not supporting a algorithm is not a
failure.

> It seems the dnsmasq developers chose to "fail secure" thus potentially
> "protecting" the end user from insecure data, although in this case the
> data is secure, just not understood to be secure.

No. The dnsmasq developers just stuffed this up.  No rational
developer of CPE equipement would do this deliberately as there is
no way to upgrade every piece of equipment before deploying the new
code.

> This is one of the tougher points of algorithm change, particularly when
> so many of the resolvers may be in commodity customer-premises equipment
> (CPE) that may or may not be easily updated or replaced.
>
> Dan
>
>
> --
> Dan York