Subject: [DNSOP] Handling of unreachable servers

From: Alexander Gall <[email-masked]>
Date: Fri, 26 Jan 2007 14:12:53 +0100

We (AS 559) provide transit service to a TLD server located at an
exchange point in Zurich, Switzerland.  The server was unreachable for
almost three days in January.  The attached graph shows the relevant
traffic on our peering with GEANT (AS 20965 which we map to AS 8933 in
our accounting system for historical reasons).  All traffic marked as
"other" is essentially DNS (queries inbound and replies outbound).

The interesting thing is that the query rate increased dramatically
while the name server was down (6 unicast and about 25 instances of an
anycast server for the TLD were operational).  I don't expect all
recursive name servers out there to be able to handle unreachable
servers in an intelligent manner, but the magnitude surprises me.  Are
there any surveys of software versions used on DNS caches?  Studies
like <http://dns.measurement-factory.com/surveys/200504.html> are
probably biased towards authoritative servers, because many caches are
hidden behind firewalls.

What strategies to deal with unreachable servers do common
implementations use?  I would have thought that most of them prefer
the servers with the lowest RTT, like BIND does.  Unreachable servers
would have an infinite RTT and should only be queried occasionally to
check whether they have become reachable again.  Either this is not
true and the number of misbehaving servers is large or a relatively
small number of misbehaving servers generates the bulk of these
queries.

We happen to have full NetFlow data for this period of time, but
haven't analyzed it yet.  It would be interesting to see, for example,
how the distribution of queries per IP adddress changed during the
event, which could allow to estimate the fraction of the population of
caches that exhibits this behavior.

--
Alex

_______________________________________________
DNSOP mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/dnsop

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Fri, 26 Jan 2007 14:40:48 +0100

On Fri, Jan 26, 2007 at 02:12:53PM +0100,
 Alexander Gall <[email-masked]> wrote 
 a message of 1149 lines which said:

> haven't analyzed it yet.  It would be interesting to see, for
> example, how the distribution of queries per IP adddress

IMHO, this is the first thing to do. The rate, if I read correctly
your graph, is very low (40 kb/s) and one single broken machine could
generate it.

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Fri, 26 Jan 2007 14:40:48 +0100

---

From: Edward Lewis <[email-masked]>
Date: Fri, 26 Jan 2007 09:20:46 -0500

>small number of misbehaving servers generates the bulk of these

Have you heard of ISC's Operation and Analysis Research Center 
(https://oarc.isc.org/)?  That is one forum for questions like this.

There is no standard defined for the behavior of iterating servers. 
There is also no standard practice.  There was an interesting paper 
by the Cooperative Association for Internet Data Analysis 
(http://www.caida.org/home/) that illustrated this, the paper might 
be a bit dated now.

The paper measured the name server selection of different 
implementations when no servers were reachable.  That sounds like a 
crazy measurement, but it revealed tendencies.  Arguably the most 
logical were results that showed an even distribution among all of 
the unreachable servers, the least logical were servers that would 
try only some.  (You'll have to consult the paper to fill in the 
details.)

I have seen situations where other factors have played a role in the 
behavior of iterating servers, usually involving firewalls, load 
balancers, routing tricks, etc.  When you have two ends of a protocol 
each trying to figure out how to deal with the other by interpreting 
what and how things are being said instead of being open about it, 
well, divorces and alimony are the results.  Ummm, what I mean is, 
the resolvers are trying to sense the best way to deal with the name 
servers.  Name serves are trying to sense what is the best way to 
scale to meet demand.  In all the dancing about the issue, toes get 
stepped on.

-- 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Edward Lewis                                                +1-571-434-5468
NeuStar

Dessert - aka Service Pack 1 for lunch.

---

From: Edward Lewis <[email-masked]>
Date: Fri, 26 Jan 2007 09:20:46 -0500

---

From: Alexander Gall <[email-masked]>
Date: Fri, 26 Jan 2007 15:25:44 +0100

> your graph, is very low (40 kb/s) and one single broken machine could

I should have mentioned that we see the same pattern from all other
peers and our transit providers.  Also, I wouldn't expect to see the
typical daily variations in this case.  The total traffic to this
server is a few hundred kilobits per second which translates into a
couple hundred queries per second (most queries are below 100 bytes).

I should really look at the flows, but that isn't trivial because they
are stored on tape together with about 3 years worth of flows of our
network.  I'll have to talk to the researchers at ETH Zurich who
analyze this data for various projects.

--
Alex

---

From: Alexander Gall <[email-masked]>
Date: Fri, 26 Jan 2007 15:25:44 +0100

--
Alex

---

From: Alexander Gall <[email-masked]>
Date: Fri, 26 Jan 2007 17:23:16 +0100

> (https://oarc.isc.org/)?  That is one forum for questions like this.

Yes.  The reason I chose to post this on DNSOP is that it might make
sense after all to make recommendations for iterative servers with
regard to this.  Yes, it has probably been declared OT ages ago, but
maybe it's time to revisit.  An increase of queries by a factor of 3-4
when a server is dead strikes me as pretty bad.  If it turns out that
many implementations make a bad choice, shouldn't there be some
guidance on how to do it better?

I agree, though, that OARC might be a better place at least as long as
the cause for this effect is unknown.  I'm happy to repost the issue
to [email-masked] if the chairs advise me to take it
off this list.

> try only some.  (You'll have to consult the paper to fill in the 

Thanks for the pointer.

> scale to meet demand.  In all the dancing about the issue, toes get 

In this case, the TLD server is as simple as it can be: no load
balancers or routing tricks, just a single unicast host.  And all
other servers for the TLD appeared to have been working well, at least
from the places covered by the RIPE TTM boxes according to
<http://dnsmon.ripe.net/>.  Any algorithm that results in more queries
being sent to the only unreachable server in the set seems severly
broken to me.

--
Alex

---

From: Alexander Gall <[email-masked]>
Date: Fri, 26 Jan 2007 17:23:16 +0100

Thanks for the pointer.

--
Alex

---

From: Lars-Johan Liman <[email-masked]>
Date: Fri, 26 Jan 2007 15:07:07 +0100

[email-masked]:
> implementations use?

Rickard Dahlstrand (then consultant, now at the .SE registry) did some
tests in Sweden a few years ago, which suggested that (some versions
of?)  Microsoft DNS resolver selects one of the servers in the NS RR
set, and if it responds, it locks down to that server, and pretty much
doesn't re-evaluate the NS situation. When the selected server doesn't
respond, the resolver can go nuts, and bombard the auth server with
queries.

There is a report somewhere about this test. Let me see if I can find
it.

				Cheers,
				  /Liman
#----------------------------------------------------------------------
# Lars-Johan Liman, M.Sc.	! E-mail: [email-masked]
# Senior Systems Specialist     ! HTTP  : //www.autonomica.se/
# Autonomica AB, Stockholm 	! Voice : +46 8 - 615 85 72
#----------------------------------------------------------------------

---

From: Lars-Johan Liman <[email-masked]>
Date: Fri, 26 Jan 2007 15:07:07 +0100

Cheers,
				  /Liman

---

From: "huang lican" <[email-masked]>
Date: Sun, 28 Jan 2007 15:51:59 +0800

Dear All,

I have submitted a internet draft about distributed DNS implementation in IPV6.
The following is the address:
 http://www1.ietf.org/mail-archive/web/i-d-announce/current/msg13597.html
or

http://www.ietf.org/internet-drafts/draft-licanhuang-dnsop-distributeddns-00.txt

Please give your comments.

Thanks a lot.

Lican Huang

---

From: "huang lican" <[email-masked]>
Date: Sun, 28 Jan 2007 15:51:59 +0800

Dear All,

Thanks a lot.

Lican Huang