Subject: [DNSOP] Problems with DS change in registry/registrar environment

From: Patrik Fältström <[email-masked]>
Date: Tue, 30 Jun 2009 08:36:47 +0200

[On request from Olaf, also dnsop is included:ed]

I think this discussion have derailed a bit, while on the other hand  
explained somewhat to me what things are really creating problems.

We have a problem when "a domain changes hands" and the private DS key  
in some way is changed, should be changed, and sometimes is not  
changed as it should. We all agree that could lead to a blackout, but  
in many cases the blackout is not more serious than a normal blackout  
when NS is changed.

First a description on the issues:

1. We have three different kinds of changes that can happen:

1.1. Change of Registrar
1.2. Change of Registrant
1.3. Change of Domain Manager (DNS Hosting Provider)

2. We have when using epp some limited number of commands available

2.1. Transfer (change of registrar)
2.2. Update of holder in the domain object (change of Registrant)
2.3. Update of host in the domain object (change of nameservers)
2.4. Update of IP address in host object (change of IP address of  
nameservers)
2.5. Update of techc in the domain object (change of Techc) [rarely  
used even in thick registry situations]

First conclusion is that we see that the operations under (1) does not  
match operations under (2), and specifically, there is in (2) not  
anything that maps clearly with (1.3) which might be the key operation  
when we know the DS for the zone should change.

I started the discussion because I have seen problems in .SE in these  
two cases:

A. The domain is transferred (1.1) to a registrar that does NOT handle  
DNSSEC
B. The domain is transferred (1.3) to a Domain Manager that does NOT  
Handle DNSSEC

I also pointed out I have (just like Patrik Wallström) seen cases where:

C. The Registrar and Domain Operator are different entities, and  
although the registrar do handle DNSSEC operations, the Domain  
Operator is not using it.

Conclusion of the main portion of the discussion I thinks about the  
simple cases:

A change like any of the above that implies change of DS is for the  
parties that know what they are doing, "just like" any key rollover.  
Further, the key rollover can, just like change of NS, happen  
regardless of what kind of change we deal with (1.1, 1.2, 1.3) with a  
minimum involvement of the donating registrar and Domain Manager:

   1 New Nameservers are set up
   2 Old zone is served on new nameservers
   3 Old zone is signed on new nameservers
   4 Domain is transferred to new registrar
   5 New Domain Operator is adding new DS to the old DS set
   6 New Domain Operator is adding new NS
   7 New Domain Operator is removing old NS
   8 New Domain Operator is removing old DS

Or similar. The only problem I might see is that in step 2, the old  
zone must be available for the new domain operator (nothing new, this  
has always been a problem). In step 4, a notification is sent to old  
registrar, and if the old registrar is also the old domain manager,  
the zone is often ripped from the old nameservers (without any delay),  
and a blackout is happening.

This is old news for many parties, and only the DS is adding some  
complexity to this, and of course finding that there are some errors  
is a new thing if there is such blackout (but I can tell you you learn  
fast ;-) ). This scenario is also independent of whether the gaining  
Domain Operator is the same or different organisation as the gaining  
Registrar.

The problems I started the discussion with still exists though. And my  
suggestions for fixes I think are still on the table. I have still not  
contacted Patrik Wallström to see how bad the situation is in .SE, but  
we will do that as soon as we can.

To fix A, which I think is serious, because DS operations can _NOT_ be  
made with the registry if the registrar is not handling DNSSEC  
operations with DNSSEC.

I only see three possible solutions:

A.1 Have as a new requirement on accredited registrars that they MUST  
be able to handle DNSSEC operations with epp. In .SE we had as a  
requirement that they MUST be able to remove DS (but not add), but I  
claim that is not enough.

A.2. Transfer the domain (again) during the blackout to a new  
registrar that do handle DNSSEC (hopefully at least one exists...see  
the RSTEP review of .ORG request on using DNSSEC.

A.3. Have the registry remove DS implicitly if domain is transferred  
to registrar that does NOT handle DNSSEC.

My suggestion is that we look carefully on option A.3. This does not  
imply any changes to any pieces of the protocol, deployed operation or  
such. And A.2 is still possible to fix the problem if the registrant  
want DNSSEC on their domain.

To fix B, we have a larger problem from a registry/registrar  
perspective, because it is impossible in epp to know when the domain  
operator is changed. There are some experimentation in some TLDs on  
defining Domain Operator as a special role/contact, but epp already  
today have problems with incompatibe handling of contacts so that  
scares me a bit. I do think we MUST reach some conclusions here. In  
the discussions on the list, people unfortunately (I think) talked too  
much about the special case when one of the Domain Operators  
(specifically the gaining one) is a registrar, but if so, we have a  
"case A". And for me that is not so interesting (it is a different  
problem).

The good thing with case B is that it is still possible for the  
registrant to remove the DS, if not the domain is moved again to some  
Domain Operator that can handle DNSSEC.

My suggestion for fixing B is that (we will test this in .SE) the  
registry is going through all DS in the zone, and detect what DS are  
bad, what are "category B" and then notify the domain holder and  
registrar. We'll see on the details, but "handle just like lame  
delegations" is probably the best option.

To fix C, we need a protocol so that registrars can give a better  
interface than "point and click" for update of the DS. Some registrars  
in Sweden (at least two) are working on ideas on such protocols. Still  
too early for IETF work on this. Until this is fixed, each Domain  
Operator will most certainly either be the same organisation as  
registrars, or be friend with one registrar that have a special API  
that they use.

Is this summary at least partially correct?

     Patrik

---

From: "Antoin Verschuren" <[email-masked]>
Date: Tue, 30 Jun 2009 12:02:49 +0200

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Subject: [DNSOP] Problems with DS change in registry/registrar environment
> 
> 

Partially, yes.
I agree with you that there is no match between DNS operations and the definitions of registry transactions like you describe in 1 and 2.
That's not the problem however, or at least is solvable at a local registry level by defining correct procedures.
So let's not discuss the mixing up of roles like registrar, registrants, dns-operators, etc.
The only reason they matter is because in practice:
-95% of registrar changes INVOLVE a change of DNS operator
-95% of registrant changes INVOLVE a change of DNS operator
-10% of nameserver changes at a registry INVOLVE a change of DNS operator, in 90% of the cases they don't.

The problem is in any transaction that involves a dns-operator change for the zone.
Since this is a DNS discussion, let's not discuss registrars, registrants, etc.
I only want to know how to move a DNSSEC zone from one DNS operator to another one, without outage.

> in many cases the blackout is not more serious than a normal blackout

In normal non-DNSSEC DNS, when an NS set changes, both versions of the truth are still accepted by DNS resolvers.
As long as you make sure the same data is in the old and new zone (except for the NS set), the general public doesn't notice anyting.
In the period between the delegation change at the parent and the expire time of the old NS set, any answer in a resolver works.
There is no blackout DNS-wise if you handle it right.

With DNSSEC, it doesn't. Only one version of the truth works (only one validates), leading to a partial blackout for those resolvers/validators that still have the old DNSKEY in their cache.

>    3 Old zone is signed on new nameservers

With whose key ? DNS-operator A or DNS-operator B ?

>    6 New Domain Operator is adding new NS

Don't get you here. Where does he do this ?

>    8 New Domain Operator is removing old DS

I think this procedure does not work.
When you remove the old DS, the DNSKEY of the old zone is still in caches.
That's the real issue.

I think the best way to get this working is if we can make sure that a validator will go back to the parent to verify if NS and DS records have changed once validation fails. Search for a changed chain of trust.

If we cannot mandate that to resolvers/validators, then the procedures for changing DNS-operator (and thus registrar, registrant, etc) will become very complex, and can only be done with the cooperation of the losing DNS-operator if you don't want outage.
The loosing DNS operator then needs to sign the DNSKEY records from the gaining DNS operator with his private key, and publish those RRsigs.

I would like best not to have complex procedures. So can we mandate resolver/validation behavior ?

Antoin Verschuren

Technical Policy Advisor SIDN
Utrechtseweg 310, PO Box 5022, 6802 EA Arnhem, The Netherlands

P: +31 26 3525500  F: +31 26 3525505  M: +31 6 23368970
mailto:[email-masked]  xmpp:[email-masked]  http://www.sidn.nl/

---

From: Patrik Fältström <[email-masked]>
Date: Tue, 30 Jun 2009 12:43:35 +0200

> The only reason they matter is because in practice:

Where have you got these numbers from?

> -10% of nameserver changes at a registry INVOLVE a change of DNS  
> operator, in 90% of the cases they don't.
>

Agree.

> In normal non-DNSSEC DNS, when an NS set changes, both versions of  
> the truth are still accepted by DNS resolvers.
> As long as you make sure the same data is in the old and new zone  
> (except for the NS set), the general public doesn't notice anyting.
> In the period between the delegation change at the parent and the  
> expire time of the old NS set, any answer in a resolver works.
> With DNSSEC, it doesn't. Only one version of the truth works (only  
> one validates), leading to a partial blackout for those resolvers/ 
> validators that still have the old DNSKEY in their cache.

I am talking about having two DS in the parent zone, and sure, if the  
client have cached the old DS, while following the new NS, then there  
will be a blackout.

> With whose key ? DNS-operator A or DNS-operator B?

Signed with new DS.

So, on the donating DNS operators servers you have the old DS and old  
SIG. On the gaining DNS operator, you have the new DS and new SIG.

In the parent zone, you have both new and old DS.

> Don't get you here. Where does he do this?

Add the new NS to the parent zone.

> When you remove the old DS, the DNSKEY of the old zone is still in  

I agree with this, but is it not the case that you only(!) get this  
problem if the NS and DS are timing out at different points in time?  
And you have a cached DS, while looking up a new NS from parent zone?  
And if you do, do you not get the new DS as well (or both new and old  
according to my scheme)?

I.e. I was thinking that first of all, we have a period where old zone  
content (including NS) will be cached all over the place. During this  
time period, we have both old and new DS at parent, and we keep both  
old and new DS in parent for a while. As long as we think anyone have  
old NS cached.

Problem exists of course if people either only have old DS and follow  
new NS, or new DS and follow old NS.

If the validating resolver then refetch the DS (and get both old and  
new from parent, as a normal key rollover), then we are ok, right?

> I think the best way to get this working is if we can make sure that  
> a validator will go back to the parent to verify if NS and DS  
> records have changed once validation fails. Search for a changed  
> chain of trust.

Agree is this good.

> If we cannot mandate that to resolvers/validators, then the  
> procedures for changing DNS-operator (and thus registrar,  
> registrant, etc) will become very complex, and can only be done with  
> the cooperation of the losing DNS-operator if you don't want outage.
> The loosing DNS operator then needs to sign the DNSKEY records from  
> the gaining DNS operator with his private key, and publish those  
> RRsigs.

Not fun. Not possible I would say.

> resolver/validation behavior ?

Since when? ;-)

I see problems today where someone (me) have A and AAAA for a  
nameserver. If that nameserver is not reachable over IPv6 (I only have  
one such nameserver unfortunately) the resolvers seems to not fall  
back to IPv4 as transport of the DNS queries. I.e. the NS have both A  
and AAAA, but only AAAA is tried. Not fun.

    Patrik

---

From: Peter Koch <[email-masked]>
Date: Tue, 30 Jun 2009 13:31:28 +0200

> [On request from Olaf, also dnsop is included:ed]

<hat "dnsop co-chair">
The discussion and input is very wolcome in DNSOP.  For reasons related
to "Note Well" <http://www.ietf.org/maillist.html> we'll not be able to
routinely approve cross postings.  That said, I'd humbly suggest we move
followups that help framing document updates to dnsop only.
</hat>

> in many cases the blackout is not more serious than a normal blackout  

ack

> 1.3. Change of Domain Manager (DNS Hosting Provider)

Terminology is crucial here and domain manager as well as domain operator
are ambiguous.  The entity changed under 1.3 is probably the one that has
the "hands" on the zone content and essentially holds the key(s).  The
former is often called the "zone maintainer".  Of course there are
popular scenarios more complicated than that, for example where a customer
can maintain - to a certain extent - the content of their zone through some
web or database interface that will then feed the primary master.  I'd assume
that DNSSEC will be transparently added by the service provider here and
it's arguable who best fits the "zone maintainer" role.

> 2.4. Update of IP address in host object (change of IP address of  

Just as a side note, this is another place where the R-R-R model breaks,
because the most authoritative party, the operator of said nameserver
is not - per se - involved in this scenario.  We've discussed that to - almost -
death for root zone delegation changes.

> used even in thick registry situations]

For DE, there exist both "tech-c" and "zone-c" where the latter is supposed to
be the "zone maintainer".

> claim that is not enough.

Well, this is a registry policy issue and hence not necessarily workable
in all environments. But it may be an option.

> imply any changes to any pieces of the protocol, deployed operation or  

I'd support that suggestion.

> scares me a bit. I do think we MUST reach some conclusions here. In  

Well, conclusions maybe, but probably not a consensus.  This situation
(name server change to servers that don't or don't all support DNSSEC)
is similar to changing the delegation into a lame delegation. If I understand
your scenario B correctly, sensible pre delegation checks can avoid damage here.

> delegations" is probably the best option.

OK, it helps to read to the end ;-)

> Is this summary at least partially correct?

It may be more than a summary, but I believe it raised all issues.

-Peter

---

From: Masataka Ohta <[email-masked]>
Date: Tue, 30 Jun 2009 21:44:15 +0900

Patrik;

> in some way is changed, should be changed, and sometimes is not  changed 
> as it should.

That is a manifestation of the fundamental problem that DNSSEC
is not end to end.

If DNSSEC were end to end, end users could have full control over
security information that propagation of changes could have been
instantaneous.

However, the reality is that, timely and secure revocation of
security information is possible only if there is secure real
time channel to revoke the security information exists between
peers.

Of course, timely and secure revocation of security information
on the channel needs yet another secure real time channel. :-)

						Masataka Ohta

---

From: Paul Wouters <[email-masked]>
Date: Tue, 30 Jun 2009 10:33:15 -0400 (EDT)

On Tue, 30 Jun 2009, Patrik Fältström wrote:

> A.3. Have the registry remove DS implicitly if domain is transferred to 
> registrar that does NOT handle DNSSEC.
>
> My suggestion is that we look carefully on option A.3. This does not imply 
> any changes to any pieces of the protocol, deployed operation or such. And

Note that if you remove the DS record from the parent, you should then wait
the TTL before allowing the transfer, or else you will still have the domain
go dark for those who validators that have the DS record cached.

Paul

---

From: Mark Andrews <[email-masked]>
Date: Wed, 01 Jul 2009 00:43:59 +1000

In message <[email-masked]>, =?WINDOWS-1252?Q?Patrik_F=E4ltstr=F6m?=
 writes:
> > So let's not discuss the mixing up of roles like registrar, =20
> > registrants, dns-operators, etc.
> > -10% of nameserver changes at a registry INVOLVE a change of DNS =20
> > The problem is in any transaction that involves a dns-operator =20
> > change for the zone.
> > Since this is a DNS discussion, let's not discuss registrars, =20
> > registrants, etc.
> > I only want to know how to move a DNSSEC zone from one DNS operator =20=
> 
> > to another one, without outage.
> 
> Agree.
> 
> > I don=92t agree with you here:
> >
> >> We have a problem when "a domain changes hands" and the private DS =20=
> 
> >> key
> > In normal non-DNSSEC DNS, when an NS set changes, both versions of =20
> > As long as you make sure the same data is in the old and new zone =20
> > In the period between the delegation change at the parent and the =20
> > With DNSSEC, it doesn't. Only one version of the truth works (only =20
> > one validates), leading to a partial blackout for those resolvers/=20
> I am talking about having two DS in the parent zone, and sure, if the =20=
> 
> client have cached the old DS, while following the new NS, then there =20=
> 
> will be a blackout.
> 
> So, on the donating DNS operators servers you have the old DS and old =20=
> 
> > When you remove the old DS, the DNSKEY of the old zone is still in =20
> > caches.
> > That's the real issue.
> 
> I agree with this, but is it not the case that you only(!) get this =20
> problem if the NS and DS are timing out at different points in time? =20
> And you have a cached DS, while looking up a new NS from parent zone? =20=
> 
> And if you do, do you not get the new DS as well (or both new and old =20=
> 
> according to my scheme)?
> 
> I.e. I was thinking that first of all, we have a period where old zone =20=
> 
> content (including NS) will be cached all over the place. During this =20=
> 
> time period, we have both old and new DS at parent, and we keep both =20
> old and new DS in parent for a while. As long as we think anyone have =20=
> 
> old NS cached.
> 
> Problem exists of course if people either only have old DS and follow =20=
> 
> If the validating resolver then refetch the DS (and get both old and =20

	Validators shouldn't have to refetch DS records to work
	around a broken key rollover.

	This is simultaneous roll of KSK and ZSK keys.  You introduce
	the keys the *same* way as you would with a single operator.
	The new operator generates new keys.  The are added to the
	existing DNSKEY RRset and the resulting RRset is signed by
	all the KSKs (old and new).  DS are added to the parent for
	then new KSKs.  Wait for the pre-change DS and DNSKEYs to
	expire from caches.  You use the new ZSK keys to sign the
	zone content.  If you are changing servers as well you make
	the the old servers serve the new zone content.  Once all
	the old zone content has timed out of caches you remove the
	old keys.  If you changed servers this is the time to
	decommision the zone on the old servers.
 
> > I think the best way to get this working is if we can make sure that =20=
> 
> > a validator will go back to the parent to verify if NS and DS =20
> > records have changed once validation fails. Search for a changed =20
> > chain of trust.
> 
> Agree is this good.

Can you say DoS attack?

> > If we cannot mandate that to resolvers/validators, then the =20
> > procedures for changing DNS-operator (and thus registrar, =20
> > registrant, etc) will become very complex, and can only be done with =20=
> > The loosing DNS operator then needs to sign the DNSKEY records from =20=
> > the gaining DNS operator with his private key, and publish those =20
> > RRsigs.
> 

Can we stop giving up before we have tried to make this work properly?
If we say it can't work then no one will try.  If we say it can work 
and expect it to work then it will work.

Publish a BCP on how to do this.  Seller and buyers alike can
offer/request BCP compliance.  Failure to follow BCP is not likely
to a seller if they end up in court for deliberately damaging the
property they are selling.  It's like you trashing a house as you
move out if you fail to follow the BCP.

> > I would like best not to have complex procedures. So can we mandate =20=
> 
> > resolver/validation behavior ?
> 
> Since when? ;-)
> 
> I see problems today where someone (me) have A and AAAA for a =20
> nameserver. If that nameserver is not reachable over IPv6 (I only have =20=
> one such nameserver unfortunately) the resolvers seems to not fall =20
> back to IPv4 as transport of the DNS queries. I.e. the NS have both A =20=
> 

	That's a broken iterative resolver.  AAAA vs A is no different
	to two A's or two AAAA's.  You should try both today as you
	know the will be path failure that make one unreachable but
	not the other.  On the otherhand if you have a bad answer
	returned over IPv6 there is no reason to expect a good
	answer over IPv4 as both nominally refer to the same machine.

	Mark
 
> > mailto:[email-masked]  xmpp:[email-masked]  =
> http://www.sidn.nl/
> >
> >
> >
> 
> 
> #############################################################
> This message is sent to you because you are subscribed to
>   the mailing list <[email-masked]>.
> To unsubscribe, E-mail to: <[email-masked]>
> A public archive is available here: <http://mail.shinkuro.com:8100/List=
> s/dnssec-deployment/>
> and older material is at
> <http://mail.shinkuro.com:8100/Lists/dnssec-deployment-archive/>
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Peter Koch <[email-masked]>
Date: Tue, 30 Jun 2009 18:58:17 +0200

> Add the new NS to the parent zone.

That's a task for the registrar.

> And if you do, do you not get the new DS as well (or both new and old  

Yes, the referral should be accompanied by the DS RR and a new RRSet should
drive the old one out of the cache.  An identical would, but probably should
not, extend the TTL of the cached copy.

> >records have changed once validation fails. Search for a changed  

I do not think NS RRs should ever explicitly asked for.  In addition, I also
don't believe that the resolver or validator should derive any conclusions
from a new or changed NS RRSet - except that new data might replace old data.
However, the parent's servers may not be in sync and I wonder whether this
approach wouldn't lead to a DDoS on the parent's servers without careful
throttling of the re-queries.

> and AAAA, but only AAAA is tried. Not fun.

It depends on how multiple A or AAAA RRs are treated. If a server is dead
(doesn't respond over v6) I see no reason not to try the next server
until you get a response.

-Peter

---

From: [email-masked]
Date: Tue, 30 Jun 2009 17:52:38 +0000

On Tue, Jun 30, 2009 at 10:33:15AM -0400, Paul Wouters wrote:
> On Tue, 30 Jun 2009, Patrik Fdltstrvm wrote:
> 

Troll...

	just a status check here...  the TTL of the RRset or the TTL imbedded
	in the DS record?  or ... the TTL of the DNSKEY RR?

--bill