Subject: [DNSOP] public consultation on root zone KSK rollover

From: Joe Abley <[email-masked]>
Date: Wed, 3 Apr 2013 10:11:10 -0400

Hi all,

As advised a month or so ago, the following public comment period is open:

  http://www.icann.org/en/news/public-comment/root-zone-consultation-08mar13-en.htm

We have received a small number of responses which are accessible from that page.

The topic at hand and the specific questions that have been asked as part of the consultation are important ones; the decisions taken will have operational consequences to any user of the Internet who validates DNS responses with DNSSEC.

If you have experience, opinions or expertise to contribute, it would be greatly appreciated. The window for being able to submit comments closes on 12 April 2013 at 23:59 UTC.

Many thanks,

Joe

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 3 Apr 2013 08:00:48 -0700

> We have received a small number of responses which are accessible from that page.

Maybe that should be a strong indication of how little people care about this? Note that none of the responses so far come from administrators of signed TLDs, the folks most directly affected by a roll. It is hard to tell why that might be, but I suspect that it involves trepidation and maybe outright fear. Of course, they cannot voice that publicly.

> If you have experience, opinions or expertise to contribute,

The first and third are way more important that the second, although people with the second can certainly make it sound like they have the third.

--Paul Hoffman

---

From: Joe Abley <[email-masked]>
Date: Wed, 3 Apr 2013 11:03:46 -0400

On 2013-04-03, at 11:00, Paul Hoffman <[email-masked]> wrote:

> Maybe that should be a strong indication of how little people care about this?

Maybe, for sure. But in case it's because people just forgot the question had been asked, I thought I'd send a reminder.

> Note that none of the responses so far come from administrators of signed TLDs, the folks most directly affected by a roll. It is hard to tell why that might be, but I suspect that it involves trepidation and maybe outright fear. Of course, they cannot voice that publicly.

I think there's an argument that the people most directly affected by a roll will be end users to whom DNS responses are being validated. A KSK rollover in the root zone has no impact on signing operations at TLDs, or at any other zone.

> The first and third are way more important that the second, although people with the second can certainly make it sound like they have the third.

People with experience or expertise usually have opinions, in my experience :-)

Joe

---

From: Paul Wouters <[email-masked]>
Date: Wed, 3 Apr 2013 11:11:55 -0400 (EDT)

On Wed, 3 Apr 2013, Paul Hoffman wrote:

> Maybe that should be a strong indication of how little people care about this? Note that none of the responses so far come from administrators of signed TLDs, the folks most directly affected by a roll.

Those are not at all most directly affected. The TLDs dont have to do
anything. It's the vendors of equipment supporting DNSSEC that have
the real issues. If they shipped with a root anchor, and their stuff
is offline for 5 years and turned on, their DNS will be broken and 5011
isn't going to be useful to them.....

I'd say addressing that problem should be done before rolling the root
key.

Paul

---

From: Joe Abley <[email-masked]>
Date: Wed, 3 Apr 2013 11:14:24 -0400

> I'd say addressing that problem should be done before rolling the root

It would be great to hear such opinions expressed as part of the public comment process, so that they can be used to identify the approach that will be followed.

Joe

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 3 Apr 2013 08:15:39 -0700

On Apr 3, 2013, at 8:03 AM, Joe Abley <[email-masked]> wrote:

> I think there's an argument that the people most directly affected by a roll will be end users to whom DNS responses are being validated.

There is such an argument, and there is a counter-argument. If rolling the root key causes visible problems (which I think most of us expect), it will be followed by lots of press articles that say "see, DNSSEC isn't reliable". The diminishing of trust has a greater effect on those who have committed resources to making their zone trustable than to relying parties.

> A KSK rollover in the root zone has no impact on signing operations at TLDs, or at any other zone.

Fully agree. If you focus just on the operations effects, it is easy to say that there will only be good coming from the roll. I prefer to look at the effect on the whole system, including the trust that the system is worthwhile.

--Paul Hoffman (who probably should have sent this and the earlier reply to dnssec-deploy, not dnsop, given that they were about trust and not operations)

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Wed, 3 Apr 2013 17:17:35 +0200

On Wed, Apr 03, 2013 at 08:00:48AM -0700,
 Paul Hoffman <[email-masked]> wrote 
 a message of 15 lines which said:

> Note that none of the responses so far come from administrators of
> signed TLDs, 

OK, signed TLD employee hat on.

> the folks most directly affected by a roll.

IMHO, no, the people most affected will be the manager of the
validating resolvers (and, by extension, of their users). Many have no
easy way to change the root key, or even knowledge they may have to
do so (don't shout "5011". Was RFC 5011 actually tested in a real
rollover with the current resolvers?)

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Wed, 3 Apr 2013 17:20:54 +0200

On Wed, Apr 03, 2013 at 11:14:24AM -0400,
 Joe Abley <[email-masked]> wrote 
 a message of 14 lines which said:

> It would be great to hear such opinions expressed as part of the
> public comment process, 

Is there a way in this process to say "me too"? (A Like button, may be
:-) Because I fully agree with David Burns and do not see the need to
write a long message to just say "+1"

---

From: Evan Hunt <[email-masked]>
Date: Wed, 3 Apr 2013 16:38:26 +0000

On Wed, Apr 03, 2013 at 05:17:35PM +0200, Stephane Bortzmeyer wrote:
> Was RFC 5011 actually tested in a real rollover with the current
> resolvers?)

Depends what you mean by "real".  The BIND implementation has been
tested with real keys, but obviously it's never been confronted with
an actual real-world root-zone rollover.

In principle there's no difference, but in practice I'm less confident:
Rolling the root zone means exercising the RFC 5011 code in *many*
validating resolvers, on different platforms with different configurations,
and with high stakes in the event of failure.  The possibility that we've
overlooked a test scenario and some validators out there will fail to roll
to the new trust anchor correctly is going to give me jitters until we've
done it the first time.

Then there's the issue Paul mentioned -- gear configured with a root KSK
that gets switched off and not rebooted for a few months or years, and then
no longer works and can't recover.

Unfortunately, none of these concerns get smaller if we wait longer.

-- 
Evan Hunt -- [email-masked]
Internet Systems Consortium, Inc.

---

From: Joe Abley <[email-masked]>
Date: Wed, 3 Apr 2013 12:40:43 -0400

> write a long message to just say "+1"

Well, you could always comment and say that :-)

Joe

---

From: Jim Reid <[email-masked]>
Date: Thu, 4 Apr 2013 20:45:38 +0100

On 3 Apr 2013, at 16:11, Paul Wouters <[email-masked]> wrote:

> It's the vendors of equipment supporting DNSSEC that have

Fair enough Paul, but how much of a problem could that realistically be and is it worth bothering about?

I think nothing is needed here except perhaps a statement of the bleeding obvious: "if you miss too many key rollovers, Very Bad Things will happen so make sure you have a foolproof way of recovering from that". eg Have some out of band means of fetching and verifying the current version of the One True Trust Anchor.

---

From: Joe Abley <[email-masked]>
Date: Thu, 4 Apr 2013 15:53:30 -0400

> Fair enough Paul, but how much of a problem could that realistically be and is it worth bothering about?

I tend to think that regardless of whether the operator of a device is technical or not, if you pull a 5-year-old anything out of the cupboard and plug it in only to find it doesn't work, that's almost expected behaviour. The astonishment comes if it *does* work.

Joe

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 4 Apr 2013 13:19:56 -0700

On Apr 4, 2013, at 12:45 PM, Jim Reid <[email-masked]> wrote:

> I think nothing is needed here except perhaps a statement of the bleeding obvious: "if you miss too many key rollovers, Very Bad Things will happen so make sure you have a foolproof way of recovering from that".

We need that statement because it's *not* bleeding obvious. I cannot think of a single thing built into a 2007-era ISO of a Linux distro that would have the property similar to "it will automatically give mysterious results for DNS service". It might have lots of unsafe software turned on, but none that will say "I'll serve you" but then it doesn't.

> eg Have some out of band means of fetching and verifying the current version of the One True Trust Anchor.

And has the IETF supplied anything like that? If not, should ICANN wait for the first roll until we have?

--Paul Hoffman

---

From: Nicholas Weaver <[email-masked]>
Date: Thu, 4 Apr 2013 13:23:31 -0700

> We need that statement because it's *not* bleeding obvious. I cannot think of a single thing built into a 2007-era ISO of a Linux distro that would have the property similar to "it will automatical…

Also, there is a LOT of old, NEVER updated, 5 year old networking kit out there.  Well, fortunately they are often clueless about DNSSEC, but still...

---

From: Mark Andrews <[email-masked]>
Date: Fri, 05 Apr 2013 07:35:14 +1100

In message <[email-masked]>, Nicholas Weaver writes:
> >> I think nothing is needed here except perhaps a statement of the 
> >> bleeding obvious: "if you miss too many key rollovers, Very Bad Things 
> >> will happen so make sure you have a foolproof way of recovering from 
> >> that".
> > 
> > We need that statement because it's *not* bleeding obvious. I cannot 
> > think of a single thing built into a 2007-era ISO of a Linux distro that 
> > would have the property similar to "it will automatically give mysterious 
> > results for DNS service". It might have lots of unsafe software turned 
> > on, but none that will say "I'll serve you" but then it doesn't.
> 
> Also, there is a LOT of old, NEVER updated, 5 year old networking kit out 
> there.  Well, fortunately they are often clueless about DNSSEC, but 
> still...
> 
> _______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop

The real problem here is that 5011 is a hack.  It overloads SEP.
It makes for a bigger DNSKEY rrset.  It doesn't provide for per
zone refresh policy.  It doesn't give guidance for effectivity
interval for DNSKEYs.

Validators need to have end dates for DNSKEYS.  If it starts up
after that date it goes to all insecure.  At the moment a DNSKEY
is valid till the end of time.  You can recover relatively easily
from all insecure.  Recovering from all bogus is much harder.

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Joe Abley <[email-masked]>
Date: Thu, 4 Apr 2013 16:37:01 -0400

> Also, there is a LOT of old, NEVER updated, 5 year old networking kit out there.  Well, fortunately they are often clueless about DNSSEC, but still...

I'm guessing that *all* kit today that has been on the shelf for the past 5 years is clueless about the current root zone trust anchor, considering that we generated it less than 3 years ago :-)

Joe

---

From: Joe Abley <[email-masked]>
Date: Thu, 4 Apr 2013 16:39:30 -0400

> And has the IETF supplied anything like that? If not, should ICANN wait for the first roll until we have?

ICANN has published documentation for exactly this, and efforts to produce equivalent documentation within the IETF (to which, once stable, the published procedures for root zone KSK management can refer to) is ongoing.

You know this, so I presume you meant something oblique by your comment?

Joe

---

From: Joe Abley <[email-masked]>
Date: Thu, 4 Apr 2013 16:46:39 -0400

On 2013-04-04, at 16:35, Mark Andrews <[email-masked]> wrote:

> after that date it goes to all insecure.

http://tools.ietf.org/html/draft-jabley-dnsop-validator-bootstrap-00

was a first attempt to describe how a validator should bootstrap itself.

I got zero feedback that anybody was interested in that problem space, and the draft never went any further. I continue to think that behaviour upon cold boot is important to specify however, and if root zone KSK rollover thoughts have changed peoples minds about its usefulness, I'd gladly pick it up again.

It needs some work (any sentence that includes the word "certificate" is liable to make Mr Hoffman shake his fist in its current form), but I think the basic approach described has merit.

In general, if we acknowledge for the purposes of this discussion that root zone KSK rollovers will happen, and will happen often enough to care about mitigating damage during roll, I think we need a two-pronged approach to this problem space:

1. Use 5011 or some similar mechanism to accommodate key rollovers (for devices that are turned on often enough to be able to do that)

2. Carefully specify bootstrapping behaviour so that any cold-start of a long-dormant validator can be handled in some sane way.

Joe

---

From: Paul Wouters <[email-masked]>
Date: Thu, 4 Apr 2013 16:46:47 -0400 (EDT)

> And has the IETF supplied anything like that? If not, should ICANN wait for the first roll until we have?

The only thing we have is the ICANN PEM bundle, and unbound-anchor like
mechanisms where we try to get the root key verified by the signed
certificate of ICANN. But it all depends on the security of the CA's,
and an ever lasting URL at the icann site for the bundle and the
continued publication of the certs and root keys there. None of which
I believe is codified in an RFC.

Paul

---

From: Paul Wouters <[email-masked]>
Date: Thu, 4 Apr 2013 16:49:55 -0400 (EDT)

> I'm guessing that *all* kit today that has been on the shelf for the past 5 years is clueless about the current root zone trust anchor, considering that we generated it less than 3 years ago :-)

But we designed DNSSEC in such a way that the 5 year old kit will still
work as (insecurely) as it was designed 5 years ago. That's not true if
we find root key copies in firmware now.

But we all had these discussions before, we had a draft for a special
zone that would contain historic root keys, etc. We just never found
something that would properly work (I believe the thread was entitled
"big router vendor" or something along those lines)

Paul

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 4 Apr 2013 14:00:43 -0700

> You know this, so I presume you meant something oblique by your comment?

You must have misread the verb tense in my message. :-) "has supplied" is quite different than "efforts is ongoing".

This is a serious question: would it be reasonable for ICANN to do a key roll when there is no IETF documentation on how to recover from missed rolls? Some might say "yes", but you can tell I would not.

While I appreciate that ICANN wrote a document, unless you want to say "this is the the ICANN document you should use to deal with the operation we are about to perform", it needs to have an RFC stamp on it.

--Paul Hoffman

---

From: Jim Reid <[email-masked]>
Date: Thu, 4 Apr 2013 22:16:41 +0100

On 4 Apr 2013, at 21:19, Paul Hoffman <[email-masked]> wrote:

> We need that statement because it's *not* bleeding obvious.

Words fail me. If this really does need to be written down somewhere, then let's do it. I'll be happy to help write such a document. And add a section on what bears do in the woods. :-) Though I understand ICANN already has published something which covers this. Well OK, the key rollover bit...

> I cannot think of a single thing built into a 2007-era ISO of a Linux distro that would have the property similar to "it will automatically give mysterious results for DNS service". It might have lots of unsafe software turned on, but none that will say "I'll serve you" but then it doesn't.

There's no need to discuss what things 2007 vintage Linux gets wrong or right. I doubt if any production software from that time did anything remotely useful with DNSSEC validation, but let that pass.

The issue should be making sure current and future platforms that come with DNSSEC validation enabled by default have got some sort of safety net to recover from missing a key rollover. I would hope/expect anyone shipping such stuff today or in the future would have the clue to do that without needing to be told. It seems improbable anyone would hard-wire any fundamental configuration information when it's already known that data MUST change from time to time. Though if someone does do that, fine: it's their look-out.
 
> And has the IETF supplied anything like that?

Not yet but ICANN has done this IIRC.

> If not, should ICANN wait for the first roll until we have?

No. A certain level of clue is needed to do DNSSEC validation and get it right. This *has* to include providing a reliable way to recover from losing the fundamental trust anchor. YMMV. At present the validating DNSSEC installed base is fairly small and doesn't yet have to rely on Standards-Track RFCs (or whatever) to tell them what to do. It would of course be nice to have these RFCs. However IMO their absence is not a showstopper for rolling the root key. DNSSEC validation is still something for consenting adults who (should) know what they are doing.

---

From: Joe Abley <[email-masked]>
Date: Thu, 4 Apr 2013 17:20:02 -0400

> You must have misread the verb tense in my message. :-) "has supplied" is quite different than "efforts is ongoing".

Well, what we have is a design that was put together by ICANN, Verisign and NTIA (the three organisations that maintain and publish the root zone) that was widely discussed by e-mail and in person across five continents prior to July 2010. We heard no dissent or suggestions to change, so that's what we went with. It's entirely understood that lack of feedback doesn't mean improvements can't be made; my point is the origins of the design work, and the venues in which it was widely circulated for comment.

There is no IETF-produced documentation for any operational aspect of the system. It was all designed and documented by the root zone partners. The I-D referred to earlier (as a successor to the TA retrieval document we published in July 2010) was not intended to be an IETF work product; it was instead intended to be an individual submission documenting an operational system of relevance to the IETF, published in the RFC series for ease of citation.

> This is a serious question: would it be reasonable for ICANN to do a key roll when there is no IETF documentation on how to recover from missed rolls? Some might say "yes", but you can tell I would…

Ah, now you're talking about something different -- you're talking about IETF guidance to the relying parties of the system rather than IETF guidance on how the system should be operated.

> While I appreciate that ICANN wrote a document, unless you want to say "this is the the ICANN document you should use to deal with the operation we are about to perform", it needs to have an RFC st…

That sounds like exactly what we will say, with the minor correction that the root zone partners are working together on this; it's not all ICANN's work.

If there was relevant IETF work that we could incorporate into the system, surely we would do so (as we did with 4033-5 and 5011). But the design of the system and the operational procedures associated with it are not products of the IETF, and given the contractual framework within which the work takes place it's not obvious how they could be.

Joe

---

From: Jim Reid <[email-masked]>
Date: Thu, 4 Apr 2013 22:25:48 +0100

On 4 Apr 2013, at 21:46, Joe Abley <[email-masked]> wrote:

> I think we need a two-pronged approach to this problem space:
> 

+1

An enumeration of the solutions of this problem space doesn't need to be done before the first live root zone key rollover.* IMO, there's little that could break and if there was any collateral damage it would happen in places where there's enough clue to put things right.

*Did anything significant break when the rollover from the DURZ key happened? OK, that was moving from a state where nothing validated to where things did validate, but still...

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 4 Apr 2013 14:32:08 -0700

> That sounds like exactly what we will say, with the minor correction that the root zone partners are working together on this; it's not all ICANN's work.

That sounds fine to me, then. It's different than what Jim said at the beginning of this thread ("it's obvious"), and it gives a way forwards.

FWIW, I don't think any of this needs to be in the IETF; it just needs to be in a stable place that ICANN points to, and that is maintained.

--Paul Hoffman

---

From: Mark Andrews <[email-masked]>
Date: Fri, 05 Apr 2013 13:11:56 +1100

> In message <[email-masked]>, Jim Reid 
> writes:
> *Did anything significant break when the rollover from the DURZ key 
> happened? OK, that was moving from a state where nothing validated to 
> where things did validate, but still...

This not the same.  This is from "usable to usable" not
"unusable to usable".

---

From: Tony Finch <[email-masked]>
Date: Sat, 6 Apr 2013 09:55:14 +0100

> no longer works and can't recover.

Validator vendors have to provide an out-of-band trust anchor update mechanism to cope with this. It needs to be coded and included in long-term support releases of validators and operating systems before rollover, I think.

I am not sure if ICANN intend their trust anchor download server to be used for this purpose or if vendors are expected to provision their own mirrors. I also don't know how to assess the trustworthiness of ICANN's signatures on the trust anchor.

Tony.
--
f.anthony.n.finch  <[email-masked]>  http://dotat.at/

---

From: Joe Abley <[email-masked]>
Date: Sat, 6 Apr 2013 17:04:09 +0800

> Validator vendors have to provide an out-of-band trust anchor update mechanism to cope with this. It needs to be coded and included in long-term support releases of validators and operating systems…

draft-jabley-dnsop-validator-bootstrap.

> I am not sure if ICANN intend their trust anchor download server to be used for this purpose or if vendors are expected to provision their own mirrors.

Our server is fine. Others' servers are also fine, although we would
likely prefer to wrap some small process around contact info,
notifications when there is new content, etc.

> I also don't know how to assess the trustworthiness of ICANN's signatures on the trust anchor.

draft-jabley-dnsop-validator-bootstrap. There is some work required on
the details, but the intended direction should be clear.

Joe

---

From: Tony Finch <[email-masked]>
Date: Sat, 6 Apr 2013 10:04:43 +0100

> isn't going to be useful to them.....

The real problem occurs when the latest release of the validator software was published before the rollover, and you install it after the rollover. It is perfectly reasonable to install software that is a few months old.

Tony.

---

From: Tony Finch <[email-masked]>
Date: Sat, 6 Apr 2013 10:22:24 +0100

> draft-jabley-dnsop-validator-bootstrap.

Still needs implementation.

My point about trustworthiness is that there is (as far as I know) no documentation of how the private keys are managed for the certificates / signatures on the trust anchor files, which rather undermines the elaborate root KSK management. I am also worried about being vulnerable to a screwup by any number of CAs; it would be good to pin the list of CA certs that might be used to verify the DNS trust anchor signatures.

Tony.

---

From: Joe Abley <[email-masked]>
Date: Sat, 6 Apr 2013 17:28:29 +0800

Hi Tony,

The intention was that those distributing code that relies upon
retrieval of an authentic trust anchor would make arrangements with
ICANN to sign trusted copies of the relevant objects themselves and
have those signatures published alongside the ICANN-generated
signatures.

So, ISC could use the same PGP key as they use to sign BIND9
distributions, Apple could use a key derived from their code-signing
CA, etc. In this way users could have the same trust in the retrieved
trust anchor as they do in the software that has retrieved it.

We have not had significant interest from vendors in developing this
approach, but we remain interested.

Joe

Aue Te Ariki! He toki ki roto taku mahuna!

---

From: Ralf Weber <[email-masked]>
Date: Sat, 6 Apr 2013 09:32:41 +0000

Moin!

> The real problem occurs when the latest release of the validator software was published before the rollover, and you install it after the rollover. It is perfectly reasonable to install software th…
I don't think that this is the real problem. The real problem is when a validator has a history of 5011 keys and gets shut down for a year or a couple of months while the root KSK rolls.

Initially it might be better for validators instead of being shipped with a Key to follow draft-jabley-dnssec-trust-anchor to get the initial root key. There are some implementations out there that already do this.

I think it might be good to extend draft-jabley-dnsop-validator-bootstrap to also cover problems introduced by root KSK rollover in order to give people guidance in case their bootstrap process is stuck.

I'll also add these comments to the ICANN root key roll consultation page later (just returned from vacation).

So long
-Ralf
---
Ralf Weber
Senior Infrastructure Architect
Nominum Inc.
2000 Seaport Blvd. Suite 400 
Redwood City, California 94063
[email-masked]

---

From: Tony Finch <[email-masked]>
Date: Sat, 6 Apr 2013 10:44:32 +0100

> have those signatures published alongside the ICANN-generated

Sounds sensible to me.

Really must press forward with implementing this well before the rollover.

Tony.