Subject: Re: gTLD server losses

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 01 May 2001 14:29:33 +0200

At 12:08 17.04.2001 -0400, Tim Seaver wrote:
>I apologize to VeriSign for mis-stating my gTLD performance results. What
>I was actually seeing in my DNS tests was a regular 4% loss from the two
>gTLD servers closest to me, c and g. However, this loss was due to the
>servers being unresponsive for two 30 minute periods per day during zone
>reloads rather than being overloaded. With the move of g on April 4, it
>no longer shows that behavior, and Matt tells me that c will be moving
>to a new configuration shortly to avoid the reload outages. That's
>good news, since it should remove some 20% of the server retries I see.

A random measurement from a site in Trondheim, Norway, April 15 to now:

[hta@dokka dnsstuff]$ tail -5000 log.log | ./buckets
384 samples
Server               Samp      %  Min  Ave  Max   0 1.0 2.0 3.0 4.0 5.0 6.0
A.GTLD-SERVERS.NET    384 100.00  133  189 5150 381   1               2
B.GTLD-SERVERS.NET    363  94.53  194  351 6665 360                   2   1
C.GTLD-SERVERS.NET    382  99.48  132  161 5126 380   1               1
D.GTLD-SERVERS.NET    384 100.00  133  215 5433 379   1               4
E.GTLD-SERVERS.NET    377  98.18  134  185  383 377
F.GTLD-SERVERS.NET    384 100.00  207  235 1584 383   1
G.GTLD-SERVERS.NET    383  99.74  205  231 1548 382   1
I.GTLD-SERVERS.NET    384 100.00   32   91 5079 381                   3
J.GTLD-SERVERS.NET    381  99.22  309  443 6095 372   1               7   1
K.GTLD-SERVERS.NET    385 100.26   53   93 5068 384                   1
L.GTLD-SERVERS.NET    385 100.26  150  189 6504 384                       1
M.GTLD-SERVERS.NET    355  92.45  375  773 5535 328   2              25

My test seems to be doing 1 retry after 5 seconds.
Apart from B and M, I think they are doing very well.

Same measurement for the root servers:

[hta@dokka dnsstuff]$ tail -5000 arpa.log | ./buckets
500 samples
Server               Samp      %  Min  Ave  Max   0 1.0 2.0 3.0 4.0 5.0 6.0 7
A.ROOT-SERVERS.NET    495  99.00  142  280 5160 482      2          11
B.ROOT-SERVERS.NET    490  98.00  207  602 5486 461      2          27
C.ROOT-SERVERS.NET    479  95.80  140  848 8947 424                 53       1
D.ROOT-SERVERS.NET    499  99.80  133  170 5173 496      2           1
E.ROOT-SERVERS.NET    490  98.00  191  226 5209 487      2           1
F.ROOT-SERVERS.NET    497  99.40  194  278 5214 491          1       5
G.ROOT-SERVERS.NET    448  89.60  145  949 7548 385      1          61       1
H.ROOT-SERVERS.NET    379  75.80  173 1652 6962 286      1          90   2
I.ROOT-SERVERS.NET    500 100.00   33   47  250 500

G and H are not terribly useful to me. I is in Sweden, proving how much
speed-of-light matters in these things; its average is a third of the
minimum roundtrip time of the second best root server.
The reason J-M are not included is that the script actually looks for the 
current root servers of in-addr.arpa....

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 01 May 2001 16:30:15 +0200

At 02:20 01.05.2001 -0400, Piet E. Barber wrote:
>Harald,
>I'm pleased to see that the numbers are so high.
>I'm also especially pleased to see that K and L are greater than 100%.  Seems
>like they're going above and beyond the call of duty.   ;)
>
>(I guess it's a bug in your script?)

yes - I did a "tail" without checking where the boundary hit, so a couple 
of packets from the previous run sneaked into the stats - sorry!

>FYI, the people at "G.root" have also just gone through a connectivity 
>upgrade,
>so I don't think the packet loss will be as high as it was before.

Good!

---

From: "Piet E. Barber" <[email-masked]>
Date: Tue, 01 May 2001 02:20:40 -0400

Harald,

> G.ROOT-SERVERS.NET    448  89.60  145  949 7548 385      1          61       1

FYI, the people at "G.root" have also just gone through a connectivity upgrade,
so I don't think the packet loss will be as high as it was before.

--
Piet Barber ([email-masked])
  Verisign Global Registry Services, gTLD team
  http://www.verisign-grs.com

---

From: Simon Leinen <[email-masked]>
Date: 06 May 2001 14:46:55 +0200

>>>>> "ts" == Tim Seaver <[email-masked]> writes:
> [...] However, this loss was due to the servers being unresponsive
> for two 30 minute periods per day during zone reloads rather than
> being overloaded. With the move of g on April 4, it no longer shows
> that behavior, and Matt tells me that c will be moving to a new
> configuration shortly to avoid the reload outages. That's good news,
> since it should remove some 20% of the server retries I see.

How do people solve this problem of authoritative name servers being
unresponsive while they reload big zones (such as TLDs etc.)? I can
think of a few possibilities:

* Have a dual/cluster setup, and while one box is reloading its
  zone(s), the IP address is routed to the other one.  This seems easy
  enough and improves robustness in other areas as well.

* Move to dynamic updates (should also make slaves happier if they can
  use IXFR).  This would be nice because it would also permit
  near-real-time update service for registries, provided IXFR works
  well and all slaves support it.

* Implement authoritative servers as front-ends to the registry
  database.  Seems challenging to implement well with traditional
  slave servers.  And moving to non-traditional slaves (e.g. multiple
  front-ends to (mirrored versions of) a single database) doesn't
  sound trivial either.

* Someone has implemented zone reload in a way such that queries to
  the zone that is being reloaded can still be handled, preferably in
  a consistent manner (i.e. using only old zone contents).
-- 
Simon.

---

From: bert hubert <[email-masked]>
Date: Sun, 6 May 2001 15:32:15 +0200

>   enough and improves robustness in other areas as well.

As the GTLD servers are, reportedly, behind Alteon Acedirectors, this seems
likely. I would be hesistant to assume that the GTLD servers consist of only
two boxes each, though.

>   front-ends to (mirrored versions of) a single database) doesn't

If you have the right software, it is trivial.

Regards,

bert

-- 
http://www.PowerDNS.com      Versatile DNS Services  
Trilab                       The Technology People   
'SYN! .. SYN|ACK! .. ACK!' - the mating call of the internet

---

From: Mans Nilsson <[email-masked]>
Date: Sun, 6 May 2001 16:27:03 +0200

> likely. I would be hesistant to assume that the GTLD servers consist of only

They do. It's a pair of big multiprocessor boxes with oodles of RAM. 
 
Måns, previous local maintainer of "I". 
-- 
Måns Nilsson		Systems Specialist
+46 70 681 7204		KTHNOC
			MN1334-RIPE

I invented skydiving in 1989!