Subject: [dnsop] ccTLD delegation analysis

From: Mark Foster <[email-masked]>
Date: Tue, 9 Dec 2003 13:53:10 -0800

Hello everyone.

I have recently completed some independent research and analysis in 
the area of DNS delegation of ccTLDs (county code Top Level Domains).
If you would, please take a moment to review the scores and findings.

Here is the link:
http://www.credentia.cc/research/cctlds/report-2003-Oct.html

Thank you.
-- 
Some days it's just not worth chewing through the restraints...
Mark Foster <[email-masked]>  http://mark.foster.cc/

.
dnsop resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/dnsop.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/dnsop/index.html

---

From: Mans Nilsson <[email-masked]>
Date: Wed, 10 Dec 2003 00:30:14 +0100

> http://www.credentia.cc/research/cctlds/report-2003-Oct.html

I appreciate the work, and am pleased to see that many zones I'm
involved in serving are scored high. But:

Allowing zone transfers is, in my humble opinion,  not a problem.
Blocking them is done for two reasons; either performance or paranoia.
Cases where there are performance problems are very rare. Paranoia
is, sadly enough, much more common. This particular paranoia is
more often known (and mildly detested) as "security by obscurity".

While it is mentioned in a column, I'd be more interested in finding
the recursers, because there lies a real problem, mainly through 
cache poisoning attacks and more complex server code being active. 

Keep up the good work! 
-- 
Måns Nilsson         Systems Specialist
+46 70 681 7204         KTHNOC
                        MN1334-RIPE

My face is new, my license is expired, and I'm under a doctor's
care!!!!

---

From: bill <[email-masked]>
Date: Tue, 9 Dec 2003 17:15:29 -0800 (PST)

> the recursers, because there lies a real problem, mainly through=20
> cache poisoning attacks and more complex server code being active.=20

	the TLDs servers have been id'ed for recursion a couple
	of times in the past and I expect it will be regular 
	feature of DNS auditing in future.  Being a bit paranoid,
	I'll be happy to share w/ you the results for the zones
	and/or servers you are admin for.  (although if you are
	admin for servers or zones, I would expect you to know that
	anyway... :)

--bill
.

---

From: Mark Foster <[email-masked]>
Date: Tue, 9 Dec 2003 21:52:28 -0800

> more often known (and mildly detested) as "security by obscurity".

Yes I suppose this could go either way. My concern is for those operators that
are unwittingly allowing them, not folks like yourself.
Also, allowing axfr on a large zone is proportionately more problematic 
with regard to loss of bandwidth in the event of DOS attack.

> cache poisoning attacks and more complex server code being active. 

If you check out http://www.credentia.cc/research/cctlds/ you will see the 
rolling checks done at 2,6,10,14,18,22 Pacific Time. The recursion warnings 
can be more easily found there.
-- 
Some days it's just not worth chewing through the restraints...
Mark Foster <[email-masked]>  http://mark.foster.cc/

.

---

From: Mans Nilsson <[email-masked]>
Date: Wed, 10 Dec 2003 07:25:51 +0100

Subject: Re: [dnsop] ccTLD delegation analysis Date: Tue, Dec 09, 2003 at 05:15:29PM -0800 Quoting bill ([email-masked]):
> 
> 	...  Being a bit paranoid,

Oh, I sort of know that, yes ;-) 

I'm encased in the lining of a pure pork sausage!!

---

From: Alexander Gall <[email-masked]>
Date: 10 Dec 2003 09:51:23 +0100

Hello Mark

> If you would, please take a moment to review the scores and findings.

Thanks for your effort.  I have some remarks concerning the serial
mismatches.  We are running the primary for the ch and li ccTLDs.
Both zones have the same set of name servers and both are reloaded
together 4 times a day.  Yet, the survey reports consistent serial
mismatches for ch but not for li.  I suspect that this could be a sort
of "phasing" effect, because your tests are probably also run at fixed
times.  Also, I think that mismatches within the refresh interval
should not be counted as errors.  Of course, this is difficult to
check if you don't know when the serial changes on the primary.

Regards,
--
Alexander Gall

 ______ SWITCH - The Swiss Education and Research Network _______
 SWITCH NOC,    Limmatquai 138,    CH-8001 Zurich,    Switzerland
 [email-masked]      Tel: +41 1 268 1530       Fax: +41 1 268 1568
 http://www.switch.ch/network/noc/

.

---

From: Jaap Akkerhuis <[email-masked]>
Date: Wed, 10 Dec 2003 11:04:30 +0100

Hello Mark

    > If you would, please take a moment to review the scores and findings.
    
Can you relate your findings to what is displayed at
http://dnscheck.se/zoneinfo.pl?zone=.

Of course, it is a bit of apples and oranges, you have a point
system, while Patrick uses more or less three cases (OK, Warning,
Error). Stil it will be an interesting exercise to see whether the
data somehow matches.

In your report, you mention ns.eu.net specifically (lame for a
couple of zones). That server is switched of beginning November.
Are you planning to generate the report on a regular base?

	jaap
.

---

From: Kim Davies <[email-masked]>
Date: Wed, 10 Dec 2003 11:29:46 +0100

> http://www.credentia.cc/research/cctlds/report-2003-Oct.html

I am not sure how relevant it is to score "Frequent Updates".

The rate of zone updating is usually a matter of administrative policy 
choice rather than a form of technical misstep. Also, many zones (.uk, 
.au etc.) only allow registrations in the third level - so the ccTLD 
zone would remain relatively static.

kim
-- 
Kim Davies, Council of European National TLD Registries
t +43 662 872 563; f +43 662 872 563 17; http://www.centr.org/

.

---

From: Simon Leinen <[email-masked]>
Date: Wed, 10 Dec 2003 13:19:18 +0100

Mark Foster writes:
> Also, allowing axfr on a large zone is proportionately more
> problematic with regard to loss of bandwidth in the event of DOS
> attack.

I wouldn't take that particular problem too seriously (loss of
bandwidth in the event of DoS attack due to AXFRs).

When your servers are under DoS, then the bandwidth expended on AXFRs
will decrease quickly - that's a benefit of using TCP for AXFRs.

Unless you have very many AXFRs going on in parallel - and that in
itself is a more serious risk, because every TCP connection takes up
resources in the server, and at some point it won't be able to handle
AXFRs to/from its actual slaves/masters.

Regards,
-- 
Simon.
.

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Wed, 10 Dec 2003 17:23:04 +0430

On Tue, Dec 09, 2003 at 01:53:10PM -0800,
 Mark Foster <[email-masked]> wrote 

> If you would, please take a moment to review the scores and findings.

How do you measure frequent updates? By a change in the SOA? (The SOA
of the root changes every day, even when there is no change in
content, other ccTLD might do the same.)

NS RR mismatches can be because of IANA slowness to update the root.
.
dnsop resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/dnsop.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/dnsop/index.html

---

From: Mark Foster <[email-masked]>
Date: Wed, 10 Dec 2003 07:40:06 -0800

> zone would remain relatively static.
 
I actually concur with that. In most cases, it means very little or nothing.
The reason I chose to check and score that, is because as a registrant, I 
have concern about how often a typical TLD regenerates their zone, because
that determines how long the wait is between a registration and having it
actually resolve. I now realize that when you add up all the ccTLDs that really 
offer registrations under their second level (uk, au, nz) there are a number
of things about my results that become less interesting and relevant. 

If I were to go back, I would identify those ccTLDs which are actually using
com.tld, net.tld, etc and focus on those relevant SLDs instead. The research
was wide not deep.

.

---

From: Mark Foster <[email-masked]>
Date: Wed, 10 Dec 2003 07:51:04 -0800

> content, other ccTLD might do the same.)
 
Right.
In examining each ccTLD, I looked at the data for a 24-hour period. So there
were 6 distinct serial numbers to look at. I did not examine the zone content.

> NS RR mismatches can be because of IANA slowness to update the root.

I see. 

.

---

From: Mark Foster <[email-masked]>
Date: Wed, 10 Dec 2003 07:25:22 -0800

> http://dnscheck.se/zoneinfo.pl?zone=.

It is the first I've seen of that. The dns.pl script looks to be more
comprehensive than the check-cctld.pl I wrote. There is another called
'doc' which I just recently came to know of as well. Before, all I knew of
was dnswalk, dig & nslookup .

> Error). Stil it will be an interesting exercise to see whether the
There is some overlap with regard to mismatched NS records (in-zone vs.
parent zone), authority check (lameness) and serial consistency. Otherwise,
the dns.pl check seems to focus in on MX/SMTP issues. I like it.

> Are you planning to generate the report on a regular base?
I will not recompile the report anytime soon because it took alot of my (spare)
time. But I will continue to run the automated checks at least once-a-day, 
which can be accessed ongoing at http://www.credentia.cc/research/cctlds/
-- 
Some days it's just not worth chewing through the restraints...
Mark Foster <[email-masked]>  http://mark.foster.cc/

.

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Thu, 11 Dec 2003 09:24:06 +0430

On Wed, Dec 10, 2003 at 07:25:22AM -0800,
 Mark Foster <[email-masked]> wrote 

> was dnswalk, dig & nslookup .

You do not know ZoneCheck? http://www.zonecheck.fr/
 
.

---

From: Bruce Campbell <[email-masked]>
Date: Thu, 11 Dec 2003 11:05:52 +0100 (CET)

> couple of zones). That server is switched of beginning November.

The byline on the report is the 20th of October 2003, at which point
ns.eu.net was still powered on.

I suspect that you'll find that the servers which are lame for notable
numbers of ccTLDs are also secondarying a lot of ccTLDs, and those are
merely the ccTLDs that don't respond to emails of:

	'you still list our server but we can't contact your master
	 server, can you please fix this?'

-- 
                             Bruce Campbell                            RIPE
                   Systems/Network Engineer                             NCC
                 www.ripe.net - PGP562C8B1B             Operations/Security
.
dnsop resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/dnsop.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/dnsop/index.html

---

From: Mark Foster <[email-masked]>
Date: Thu, 11 Dec 2003 07:35:38 -0800

> 	 server, can you please fix this?'

Yes -let me clarify. I began the analysis on (or about) Oct 20, 2003. 
The analysis completed about Dec 8, 2003. The sequence was A-Z.
Also, if queries to a name server timed out consistently, that was designated
lame, although some might argue against that.

If ns.eu.net is permanently shut down, shouldn't IANA remove it from the root
zone regardless of whether the ccTLD operator can be contacted? Or, in other 
words, I believe IANA should have a policy of removing historically lame servers,
to reduce the burden on the roots.
-- 
Some days it's just not worth chewing through the restraints...
Mark Foster <[email-masked]>  http://mark.foster.cc/

.

---

From: Jim Reid <[email-masked]>
Date: Thu, 11 Dec 2003 16:16:02 +0000

>>>>> "Mark" == Mark Foster <[email-masked]> writes:

    Mark> If ns.eu.net is permanently shut down, shouldn't IANA remove
    Mark> it from the root zone regardless of whether the ccTLD
    Mark> operator can be contacted?

No, absolutely not. This MUST ONLY be the responsibility of the domain
owner, not the administrator of its parent zone. In the case of the
root zone, it's even more essential that IANA/ICANN only do what they
are told to do for some delegation by the owner of that TLD. They
should not make value judgements on the state of a delegation and use
those to unilaterally change the TLD's delegation. This could get into
tricky politics and awkward issues of national sovereignty. How a
country manages its national infrastructure is up to them. [If some
TLD *wants* to have a lame or dead name server, that's their decision.
Even if we don't like it.] Taking ns.eu.net as an example, how could
IANA know for sure that this name server has truly gone away forever?
Perhaps one of the ccTLDs that used it is quietly making arrangements
for it to be resurrected? Unless those ccTLD's officially tell IANA
they're done with ns.eu.net, IANA has no way of really knowing that.

IMO, it might be OK for IANA to tell a TLD that they have a problem
with a broken server or to discourage broken servers being added to
the delegation. But that's all. It should be up to the TLD to
decide what to do about the fault.

Would it be right for the .cc NIC decide to remove the delegation for
foster.cc because they think your name servers are lame or have gone
away? And what if their definition of broken name servers doesn't
match yours?

    Mark> Or, in other words, I believe
    Mark> IANA should have a policy of removing historically lame
    Mark> servers, to reduce the burden on the roots.

What burden? A lookup to the root servers will return a referral to
the appropriate TLD, complete with glue. The resolving server doesn't
need to go back to the root servers after that because one of the
TLD's servers is dead. After the query to the root, the resolving
server already has all the info about the TLD delegation.

FYI the overwhelming burden on the root servers is crap that shouldn't
be going to them at all: reverse lookups for RFC1918 nets, recursive
queries, dynamic updates, etc. Garbage like this accounts for ~90-95%
of the traffic that goes to the root servers. It would be nice if we
could do something about that very real problem.
.
dnsop resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/dnsop.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/dnsop/index.html

---

From: Mark Foster <[email-masked]>
Date: Thu, 11 Dec 2003 09:23:10 -0800

> decide what to do about the fault.

OK. However this approach ignores the real-world observations that lame 
delegations produce an inordinate load on the parent name servers,
apparently due to broken resolver implementation(s).
 
I have seen this happen a number of times. The resulting query loop
has comprised up to 80% of the parent server query traffic.
-- 
Some days it's just not worth chewing through the restraints...
Mark Foster <[email-masked]>  http://mark.foster.cc/

.

---

From: Bruce Campbell <[email-masked]>
Date: Thu, 11 Dec 2003 18:48:04 +0100 (CET)

On Thu, 11 Dec 2003, Mark Foster wrote:

> zone regardless of whether the ccTLD operator can be contacted?

No.  IANA/operators of authoritative ne lame nameservers should contact
the ccTLD operators to say that its lame, but arbitarily changing root
delegations is a political powderkeg that you really don't want to touch.

> Or, in other

I've got it on my list of things to do should I ever end up as God-Emperor
of the world.

---

From: bill <[email-masked]>
Date: Thu, 11 Dec 2003 09:33:50 -0800 (PST)

> words, I believe IANA should have a policy of removing historically lame servers,
         ---------

	Ah the bliss!  Hundreds of thousands have expressed similar 
	sentiments over the years.  Simple belief does not make it so.

.

---

From: Jaap Akkerhuis <[email-masked]>
Date: Thu, 11 Dec 2003 23:43:32 +0100

Bruce,

We are in violent agreement.

I noticed that and my remark was meant as a warning for the
people not realising this.

    I suspect that you'll find that the servers which are lame for
    notable numbers of ccTLDs are also secondarying a lot of ccTLDs,
    and those are merely the ccTLDs that don't respond to emails
    of:

Sure, that is my suspicion as well. And I hope by this way some of
these master server operators might get this message, directly or
indirectly.

	jaap
.

---

From: Francisco Arias <[email-masked]>
Date: Fri, 12 Dec 2003 19:54:42 -0600

Mark,

         Finally I just got the time to read your report.  I'm glad to see, 
maybe the first technical ccTLD benchmark.  That's good.

         On the other hand I think this work has at least three points that 
really need to be fixed:

         - Geographic distribution and number of Active servers; if someone 
is using Shared Unicast (aka Anycast) whith the tools you used you were 
unable to see the whole thing.
         - Frequent Updates; as Mr. Davies already pointed out this 
indicator is completely useless for ccTLD's that do not offer registrations 
directly under the TLD; and maybe of little use for those that do.
         - Serial match consistency; as Mr. Gall already pointed out it may 
depend on the way of sampling data.

         As it is the case for the ccTLD I work for, we have our own 
implementation of Shared Unicast, four active servers geographically 
distributed and we do not offer registrations directly under the TLD.  This 
set of characteristics gave our ccTLD a poor qualification on this report 
which I do not think remotely reflects the effort that has been and is 
taking place to improve the ccTLD DNS infrastructure.

>http://www.credentia.cc/research/cctlds/report-2003-Oct.html

Best Regards,

--
         Francisco Javier Arias Correa
         NIC Mexico
         http://www.nic.mx
         E-mail: [email-masked]
         Phone/Fax: +52 81 8387 5346

The content of this data transmission is confidential, therefore, it shall 
not be modified, distributed and/or disclosed through any mean without the 
original sender's previous authorization.  Any information, offer or 
agreement made by the sender of this data transmission, should be consider 
as personal and not binding for NIC Mexico. 

.

---

From: "J-F C. (Jefsey) Morfin" <[email-masked]>
Date: Sat, 13 Dec 2003 13:12:22 +0100

I work on a permanent project oriented towards large users internet 
security. I document a budget for 2 half time persons and a secure mirrored 
system. I document also its deliverables. It sould provide documentation, 
surveys, mirror of the root, alarm, support in case of top level problem, 
management of concerted software developments, etc. The initial concerned 
user basis is significant.

A global TLD control and reporting service like Mark's study is mandatory 
for such a project. As told to Mark, I am interested in discussing the best 
way to build and support a TLD observation and reporting tool. And to help 
developping it when I get the necessary support and resources. I am also 
interested to understand if there could be mathematical correlations 
between some the outputs and any DNS unstability, as a source of warnings.
jfc

At 02:54 13/12/03, Francisco Arias wrote:
>         On the other hand I think this work has at least three points 
> that really need to be fixed:
>
>         - Geographic distribution and number of Active servers; if 
> someone is using Shared Unicast (aka Anycast) whith the tools you used 
> you were unable to see the whole thing.
> indicator is completely useless for ccTLD's that do not offer 
> registrations directly under the TLD; and maybe of little use for those 
> that do.
>         - Serial match consistency; as Mr. Gall already pointed out it 
> may depend on the way of sampling data.
>
> distributed and we do not offer registrations directly under the 
> TLD.  This set of characteristics gave our ccTLD a poor qualification on 
> this report which I do not think remotely reflects the effort that has 
> been and is taking place to improve the ccTLD DNS infrastructure.
>
>..

.

---

From: Jaap Akkerhuis <[email-masked]>
Date: Sat, 13 Dec 2003 16:07:58 +0100

Not to knock down the work Mark did, but there have been actually
more reports. See for example Martin Lindner's talk at the ICANN
conference November 2001 in Marina Del Ray. (I never can find the
link on the icann site, but search for the author get's me there
after a while). There is also contineously measurements gonig on
at dnscheck.se. Other people have also reported (some of) these
things.

	jaap
.

---

From: Jaap Akkerhuis <[email-masked]>
Date: Sat, 13 Dec 2003 15:53:27 +0100

management of concerted software developments, etc.

    The initial concerned user basis is significant.

And that user basis is?

There are quite some tools like that around already.  They are even done
without a budget. Furthermore, there is also https://oarc.isc.org/.

	jaap

.

---

From: Doug Barton <[email-masked]>
Date: Mon, 15 Dec 2003 12:19:19 -0800

> has comprised up to 80% of the parent server query traffic.

There is a serious problem with one very popular vendor's resolving name 
server implementation wherein if _all_ the name servers in the parent's 
delegation are lame, it will enter a loop. If even one of the name 
servers in the delegation respond, the loop is broken.

Doug

.