Subject: [dnsop] Re: Root Anycast (fwd)

From: Dean Anderson <[email-masked]>
Date: Wed, 29 Sep 2004 17:41:05 -0400 (EDT)

Some time back we were talking about anycast being a bad thing on DNS Root
servers. It was suggested by that conversations typically take only one
path as a result of CEF-like caching.  I noted that providers were working
on per packet load balancing. Well, here it is, in the "major vendor":

Per-Packet Load Balancing
http://www.cisco.com/univercd/cc/td/doc/product/software/ios120/120newft/120limit/120s/120s21/pplb.htm

So, it seems that we need to review whether the use of anycast on the root
nameservers is a good idea. I suggest that we ignore for the momeent
anyway the question of whether the deployment of anycast was done with
adequate technical analysis, discussion, and approval, and just consider
whether we should continue doing it.  However, the Root server operation
and oversight issues are very important should also be discussed, too, but
probably by different forums.

		--Dean

---------- Forwarded message ----------
Date: Tue, 18 May 2004 19:03:52 -0400 (EDT)
From: Dean Anderson <[email-masked]>
To: Paul Vixie <[email-masked]>
Cc: [email-masked]
Subject: Re: Root Anycast

On 18 May 2004, Paul Vixie wrote:

> Careful design by whom?  Organic compared to what?  I assure you that f-root
> has grown by careful design.  It's only organic in that we go where we're
> invited rather than having a gigantic budget that could be used as a leash.

Do you mean "Careful Design" like the non-standard changes in Bind 9 AXFR
and IXFR?  I don't think we can take too much of that sort of thing in the
operation of the root servers before we have serious problems.

Unilateral action is not a good thing. There is no point in having an IETF
(Remember the "Internet __Engineering__ Task Force" in IETF) if you just
implement whatever you think OK at the moment (AXFR mods, IXFR mods,
Anycast, and probably more that we just don't yet know about)

		--Dean

--
Av8 Internet   Prepared to pay a premium for better service?
www.av8.net         faster, more reliable, better service
617 344 9000

.
dnsop resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/dnsop.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/dnsop/index.html

---

From: "Joe Shen" <[email-masked]>
Date: Thu, 30 Sep 2004 09:33:42 +0800

Would you please give more about  the problem of  anycast with root
server ? 

To my understanding, Per-Packet Load Balancing  works only in situation
all  DNS servers installed
behind the same router, and it  CAN NOT guarantee sequencing of TCP
packets.  

The first problem of PPLB  is ,  it could not be implemented  for a
server farm which distributes across 
internet, and  the only thing it does is  replacing traditional  load
balancer  with router. 

I don't know whether there is some research in  "how many packets does
one DNS request cost " and "how
many TCP traffic occupies in DNS traffic".  If  most of DNS request cost
more than one UDP packets, out-of-sequence 
may be a problem. Also , if TCP occupies more than 10% of traffic it
will also be a problem. 

The third,  ECMP in current  DNS server farm guarantees both per-packet
load balancing in UDP traffic and per-flow distribution
in TCP traffic.  Considering distributing DNS server across multiple AS,
I think the advange is obvious than PPLB.

Joe Shen

ps.  where can I find detailed information on Root server GTLD server
configuration  ( hardware , software, and network infrastructure)?
        I just know they use anycast but how they choose system
platform?

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of Dean Anderson
Sent: Thursday, September 30, 2004 5:41 AM
To: [email-masked]
Cc: [email-masked]
Subject: [dnsop] Re: Root Anycast (fwd)

Some time back we were talking about anycast being a bad thing on DNS
Root servers. It was suggested by that conversations typically take only
one path as a result of CEF-like caching.  I noted that providers were
working on per packet load balancing. Well, here it is, in the "major
vendor":

Per-Packet Load Balancing
http://www.cisco.com/univercd/cc/td/doc/product/software/ios120/120newft
/120limit/120s/120s21/pplb.htm

So, it seems that we need to review whether the use of anycast on the
root nameservers is a good idea. I suggest that we ignore for the
momeent anyway the question of whether the deployment of anycast was
done with adequate technical analysis, discussion, and approval, and
just consider whether we should continue doing it.  However, the Root
server operation and oversight issues are very important should also be
discussed, too, but probably by different forums.

		--Dean

---------- Forwarded message ----------
Date: Tue, 18 May 2004 19:03:52 -0400 (EDT)
From: Dean Anderson <[email-masked]>
To: Paul Vixie <[email-masked]>
Cc: [email-masked]
Subject: Re: Root Anycast

On 18 May 2004, Paul Vixie wrote:

> Careful design by whom?  Organic compared to what?  I assure you that 
> f-root has grown by careful design.  It's only organic in that we go 
> where we're invited rather than having a gigantic budget that could be

> used as a leash.

Do you mean "Careful Design" like the non-standard changes in Bind 9
AXFR and IXFR?  I don't think we can take too much of that sort of thing
in the operation of the root servers before we have serious problems.

Unilateral action is not a good thing. There is no point in having an
IETF (Remember the "Internet __Engineering__ Task Force" in IETF) if you
just implement whatever you think OK at the moment (AXFR mods, IXFR
mods, Anycast, and probably more that we just don't yet know about)

		--Dean

.

.

---

From: Paul Vixie <[email-masked]>
Date: 30 Sep 2004 06:32:47 +0000

[email-masked] ("Joe Shen") writes:

> all DNS servers installed behind the same router, and it CAN NOT
> guarantee sequencing of TCP packets.

yup.

> The first problem of PPLB is , it could not be implemented for a server
> farm which distributes across internet, and the only thing it does is
> replacing traditional load balancer with router.

yup.

> load balancing in UDP traffic and per-flow distribution in TCP traffic.
> Considering distributing DNS server across multiple AS, I think the
> advange is obvious than PPLB.

yup.

>         I just know they use anycast but how they choose system

i don't know what the gtld servers are doing.

but f-root's configuration is outlined at <http://www.isc.org/pubs/tn/>,
read ISC-TN-2003-1 and ISC-TN-2004-1.  also <http://www.isc.org/ops/f-root/>.
-- 
Paul Vixie
.
dnsop resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/dnsop.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/dnsop/index.html

---

From: Andras Salamon <[email-masked]>
Date: Thu, 30 Sep 2004 12:39:00 +0200

> configuration  ( hardware , software, and network infrastructure)?

The best info seems to be at
    http://www.root-servers.org/

-- Andras Salamon                   [email-masked]
-- http://www.dns.net/dnsrd/        DNS Resources Directory
--                                  The online DNS reference since 1994
.
dnsop resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/dnsop.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/dnsop/index.html

---

From: Dean Anderson <[email-masked]>
Date: Thu, 30 Sep 2004 15:10:19 -0400 (EDT)

> Would you please give more about  the problem of  anycast with root

I notice that Paul Vixie already replied, and as usual, oversimplified and 
gave no useful information. It unclear if he is being deliberately vague 
or if he truly doesn't understand the implications of your questions.

> guarantee sequencing of TCP packets.

That is basically correct, and is possible for a customer/end user DNS
setup with only one router. A root anycast setup is different in that root
servers are located at sites that have more than one (usually many more
than one) connection (peering or transit)  with other providers, and thus
are not 'behind the same router'.  If those providers, or their customers
or peers also have multiple paths, and if one of those customers or peers
enables PPLB, then TCP packets would go to any of the anycast servers.  

For example, the ISC-TN-2004-1 shows a diagram with 2 routers, breaking 
the "everything behind one router" requirement necessary for anycast.  
Apparently, Vixie/ISC really doesn't understand the constraints required 
for anycast operation, in particular, the "behind one router" constraint.

Also, it is not merely the sequencing of packets that causes a problem
(though this causes a minor problem), but the fact that some packets are
delivered to entirely different anycast servers and so a TCP connection
can't be established under the condition of PPLB to a group of anycast
servers behind multiple routers.

> replacing traditional load balancer with router.

PPLB is incompatible with anycast.  Any situation where the anycast server 
has multiple paths to the internet is subject to PPLB use by some other 
provider which could send packets over those paths, ultimately to 
different servers.

It is not enough for the routers at the multihomed anycast root server
farm not to do PPLB. It would be necessary to require that no one anywhere
do PPLB under the multiple router circumstance since each router will have
different cache to one of the anycast servers. And of course, the
requirement that no one anywhere can use PPLB is unreasonble.

> one DNS request cost " and "how many TCP traffic occupies in DNS
> traffic".  If most of DNS request cost more than one UDP packets,
> out-of-sequence may be a problem. Also , if TCP occupies more than 10%
> of traffic it will also be a problem.
> 

.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 30 Sep 2004 23:56:13 +0200

On 29-sep-04, at 23:41, Dean Anderson wrote:

> Root

> 120newft/120limit/120s/120s21/pplb.htm

Unless I'm very much mistaken, at least _some_ older Cisco boxes have  
been able to do this for a very long time.

Note though that it's *very* hard to create a setup where packets are  
delivered to different multicast instances, as it's hard to imagine how  
any real-world anycast setup could match the criteria in

http://www.cisco.com/en/US/tech/tk365/tk80/ 
technologies_tech_note09186a0080094431.shtml#bgpmpath

(Although a good way to guarantee this would be to use different ASes  
to source the different anycast instances.)

> root
> nameservers is a good idea.

I think the ship has sailed a long time on whether this was going to  
happen at all. However, now would be a good time to start a discussion  
on how much anycasting is enough. It would be good to keep a reasonable  
number of root servers non-anycasted.

I'll be happy to list the reasons for this (apart from general healthy  
conservatism) on some appropriate forum.

.

---

From: John Brown CT <[email-masked]>
Date: Thu, 30 Sep 2004 15:01:31 -0600

Couple of points here.

1. Typical DNS queries are via UDP, not TCP.
    Thus the noise Dean is making here about things breaking
    because of TCP issues, is well  noise.

    Keep in mind that DNS queries are UDP.  The query and the response.
    so a typical query is 2 packets, the ask and the answer.

    Having DNS be based on TCP would NOT scale very well.  Think about
    it.  Before I could even make a query I would have to deal with
    at least 3 packets for the TCP connection setup.  Then I'd send my
    query, which would also have an TCP ACK sent as well, oh then there
    is the answer to the query, with yet another TCP ACK.  So a single
    DNS query would (at a min) take 7 packets, more likely 8 to 10,
    thats 400 to 500 percent more traffic than via UDP.

    DNS uses TCP in special cases. Some of them, but not all of them
    are.  1. Packet size, 2. AXFR, 3. I think TSIG / DNS SeC stuff

    Now before Dean jumps on the See, AXFR is broke, lets understand that
    AXFR doesn't happen for anycasted root servers on their PUBLIC facing
    IP address.  AXFR is typically going to happen on a globally unique
    IP assigned to each specific Anycast'd host.  Thus TCP works just
    fine.

2. This "single router requirement" is an interesting comment.  I've not
    seen this in any RFC or BCP.  Is there one ??  I'd hope not.

    Having muliple routers in a mesh format is good.  That means if one
    router fails the other can take the traffic.

    Keep in mind that from a packet path forwarding decision process,
    these routers are speaking other protocols as well.  There is dynamic
    information being shared between these closely coupled routers that
    lets them do the right thing.

3.  With respect to Joe's question about are there numbers for DNS use
     of IP protocols (UDP vs TCP).  Yes, there are.  RIPE published or
     publishes traffic numbers.  I'm not sure if the  URL's I have are 

     "public" or not.  Ask them.   Or do a little NetFlow work of your
     own.....

I realize that many on this list don't reply to the troll food that Dean 
leaves around, but I felt it important to reply as someone thats NOT in
any shape fashion or form, ISC or its staff.   I am somone that has done 
the engineering work to make a different letter work better via Anycast.
Which letter, well that doesn't matter.

Bottom line:  Depending on your technical requirements, the protocols
in use, ANYCAST works just dandy. Its another tool in the tool bag that
in the right conditions does the job in a simple and clean manner.  Just
like a surgeon needs to know what type of tool to use, network engineers 
need to know what "tools" to use.

Would I do TCP based anycast, nope.
Would I do UDP based anycast, YES and I'm doing it for my networks

John Brown, CTO
Chagres Technologies, Inc  (Americas)
Chagres Technologies, BV   (EMEA)
Building Global IP and VOIP networks.

.

---

From: Jim Reid <[email-masked]>
Date: Thu, 30 Sep 2004 23:44:05 +0100

>>>>> "Iljitsch" == Iljitsch van Beijnum <[email-masked]> writes:

    Iljitsch> I think the ship has sailed a long time on whether this
    Iljitsch> was going to happen at all. However, now would be a good
    Iljitsch> time to start a discussion on how much anycasting is
    Iljitsch> enough. It would be good to keep a reasonable number of
    Iljitsch> root servers non-anycasted.

The usual good sense and wise judgement of the root server operators
will see that this happens anyway. I doubt if there's any group of
people who are more sensitive to avoiding single points of failure.
.
dnsop resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/dnsop.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/dnsop/index.html

---

From: Dean Anderson <[email-masked]>
Date: Thu, 30 Sep 2004 19:48:35 -0400 (EDT)

>     because of TCP issues, is well  noise.

Noise about TCP, yes.

>     Having DNS be based on TCP would NOT scale very well.  

We know. As you point out, TCP is still used.  

> Think about

We know. But people still propose things that will take big packets or 
DNSSEC, etc.

>     IP assigned to each specific Anycast'd host.  Thus TCP works just

Yes, I'll accept that roots can be updated via means other than AXFR and
updated via other than anycasted IP addresses.

>     seen this in any RFC or BCP.  Is there one ??  I'd hope not.

A BCP/RFC for what?  You mean anycast? I don't know if it is in the RFC
describing anycast.  However, that is obviously a requirement, as pointed
out previously by others.

>     router fails the other can take the traffic.

No doubt.

>     lets them do the right thing.

Really?  And what protocols are those?

		--Dean

.

---

From: John Brown CT <[email-masked]>
Date: Thu, 30 Sep 2004 18:04:01 -0600

> Really?  And what protocols are those?

Just thinking aloud here...

BGP
OSPF
HSRP
ISIS
EIGRP
IGRP

just to name some of the various protocols that routers use
to communicate with each other about the paths they know
about.
.
dnsop resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/dnsop.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/dnsop/index.html

---

From: Dean Anderson <[email-masked]>
Date: Thu, 30 Sep 2004 20:24:00 -0400 (EDT)

John Brown (a latecoming VOIP "expert" and self-described "radical") 
> Which letter, well that doesn't matter.

Perhaps this is part of the "troll food" John is talking about:  

[the issue below turned out to be genuine abuse of Dr. Bernstein in
violation of the code of conduct, and indeed in violation of common
decency, but John Brown promoted that abuse as well. The group of
Bernstein abusers then was also associated with ISC and Paul Vixie. This
rather refutes the claim made above by John that he is "NOT in any shape
fashion or form, ISC or its staff"]

---------- Forwarded message ----------
Date: Wed, 27 Nov 2002 17:47:21 -0700
From: John M. Brown <[email-masked]>
To: 'D. J. Bernstein' <[email-masked]>, [email-masked]
Cc: [email-masked], [email-masked]
Subject: RE: namedroppers mismanagement, continued

[ post by non-subscriber.  with the massive amount of spam, it is easy to
  miss and therefore delete mis-posts.  so fix subscription addresses! ]

Dan, sounds like a plan.  I say all messages from
Bernstein be handled via his option #2

Can we please BOUNCE all of Dr. Bernsteins email with
the correct procedure, in the bounce message, on how
to subscribe and be a participating member of the list
instead of bitching and wasting time.

If you have specific complaints about the List Manager
then forward them, in private, to the NomCom.

This list is NOT the place for this bitch fest.  

Now lets get back to something more important like
arguing over DNS SEC. ;)

Jeesh, what a waste...

.

---

From: Dean Anderson <[email-masked]>
Date: Thu, 30 Sep 2004 20:56:00 -0400 (EDT)

> to communicate with each other about the paths they know

Yes, but none of these prevent PPLB from sending packets across multiple 
paths. It seems you had best read some manuals or get some experience 
somewhere before you start blathering nonsense about things you apparently 
don't really understand.

		--Dean

.

---

From: Dean Anderson <[email-masked]>
Date: Thu, 30 Sep 2004 20:48:37 -0400 (EDT)

> any real-world anycast setup could match the criteria in

Its quite easy for anycast: (real names used, but not real relationships)

                Av8
              /   \
          sprint   att
              \   /
              F-root              

If Av8 turns on PPLB, traffic to F-root will go through both sprint and
att on a per-packet basis. However F-root is really anycasted, so it looks
really like this:

                              Av8

                          /           \
                  sprint                 att
                   |  |                  |  |
              ,----+--+--.            ,--+--+----.
              | router 1 |            | router 2 |  ...
              `--+-------'            `-------+--'
                 |                            |
              ,--+-------.            ,-------+--.
              | switch 1 +------------+ switch 2 |  ...
              `-+------+-'            `-+------+-'
                |      |                |      |
         ,------+-.  ,-+------.  ,------+-.  ,-+------.
         | f-root1|  | host 2 |  | f-root3|  | host 4 |  ...
         `--------'  `--------'  `--------'  `--------'
     
So, we see that some packets will go to f-root1 and some packets will go
to f-root3, on a per-packet basis. TCP will not work to an anycasted root
IF anyone on the internet uses PPLB and has the same peers/transit
providers as an anycasted root.

It might well be that packets from f-root1 and f-root3 both go through att
to Av8, assuming that f-root's operator doesn't use PPLB, but that doesn't
matter. The TCP connection can't be made or kept established.

.

---

From: Dean Anderson <[email-masked]>
Date: Thu, 30 Sep 2004 20:59:53 -0400 (EDT)

> people who are more sensitive to avoiding single points of failure.

Lack of "Good sense and wise judgement" is one of the principal problems
that got anycast deployed without adequate discussion.  Then we could talk
about incompatible and undiscussed AXFR modifications, childish personal
attacks, etc, etc, etc that also demonstrate bad judgement and bad sense.

		--Dean

.