Subject: Re: Root Anycast (fwd)

From: Dean Anderson <[email-masked]>
Date: Wed, 29 Sep 2004 17:41:05 -0400 (EDT)

Some time back we were talking about anycast being a bad thing on DNS Root
servers. It was suggested by that conversations typically take only one
path as a result of CEF-like caching.  I noted that providers were working
on per packet load balancing. Well, here it is, in the "major vendor":

Per-Packet Load Balancing
http://www.cisco.com/univercd/cc/td/doc/product/software/ios120/120newft/120limit/120s/120s21/pplb.htm

So, it seems that we need to review whether the use of anycast on the root
nameservers is a good idea. I suggest that we ignore for the momeent
anyway the question of whether the deployment of anycast was done with
adequate technical analysis, discussion, and approval, and just consider
whether we should continue doing it.  However, the Root server operation
and oversight issues are very important should also be discussed, too, but
probably by different forums.

		--Dean

---------- Forwarded message ----------
Date: Tue, 18 May 2004 19:03:52 -0400 (EDT)
From: Dean Anderson <[email-masked]>
To: Paul Vixie <[email-masked]>
Cc: [email-masked]
Subject: Re: Root Anycast

On 18 May 2004, Paul Vixie wrote:

> Careful design by whom?  Organic compared to what?  I assure you that f-root
> has grown by careful design.  It's only organic in that we go where we're
> invited rather than having a gigantic budget that could be used as a leash.

Do you mean "Careful Design" like the non-standard changes in Bind 9 AXFR
and IXFR?  I don't think we can take too much of that sort of thing in the
operation of the root servers before we have serious problems.

Unilateral action is not a good thing. There is no point in having an IETF
(Remember the "Internet __Engineering__ Task Force" in IETF) if you just
implement whatever you think OK at the moment (AXFR mods, IXFR mods,
Anycast, and probably more that we just don't yet know about)

		--Dean

--
Av8 Internet   Prepared to pay a premium for better service?
www.av8.net         faster, more reliable, better service
617 344 9000

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: Paul Vixie <[email-masked]>
Date: 30 Sep 2004 06:32:47 +0000

[email-masked] ("Joe Shen") writes:

> To my understanding, Per-Packet Load Balancing works only in situation
> all DNS servers installed behind the same router, and it CAN NOT
> guarantee sequencing of TCP packets.

yup.

> The first problem of PPLB is , it could not be implemented for a server
> farm which distributes across internet, and the only thing it does is
> replacing traditional load balancer with router.

yup.

> The third, ECMP in current DNS server farm guarantees both per-packet
> load balancing in UDP traffic and per-flow distribution in TCP traffic.
> Considering distributing DNS server across multiple AS, I think the
> advange is obvious than PPLB.

yup.

> ps.  where can I find detailed information on Root server GTLD server
> configuration  ( hardware , software, and network infrastructure)?
>         I just know they use anycast but how they choose system
> platform?

i don't know what the gtld servers are doing.

but f-root's configuration is outlined at <http://www.isc.org/pubs/tn/>,
read ISC-TN-2003-1 and ISC-TN-2004-1.  also <http://www.isc.org/ops/f-root/>.
-- 
Paul Vixie

---

From: "Joe Shen" <[email-masked]>
Date: Thu, 30 Sep 2004 09:33:42 +0800

Would you please give more about  the problem of  anycast with root
server ? 

To my understanding, Per-Packet Load Balancing  works only in situation
all  DNS servers installed
behind the same router, and it  CAN NOT guarantee sequencing of TCP
packets.  

The first problem of PPLB  is ,  it could not be implemented  for a
server farm which distributes across 
internet, and  the only thing it does is  replacing traditional  load
balancer  with router. 

I don't know whether there is some research in  "how many packets does
one DNS request cost " and "how
many TCP traffic occupies in DNS traffic".  If  most of DNS request cost
more than one UDP packets, out-of-sequence 
may be a problem. Also , if TCP occupies more than 10% of traffic it
will also be a problem. 

The third,  ECMP in current  DNS server farm guarantees both per-packet
load balancing in UDP traffic and per-flow distribution
in TCP traffic.  Considering distributing DNS server across multiple AS,
I think the advange is obvious than PPLB.

Joe Shen

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of Dean Anderson
Sent: Thursday, September 30, 2004 5:41 AM
To: [email-masked]
Cc: [email-masked]
Subject: [dnsop] Re: Root Anycast (fwd)

Some time back we were talking about anycast being a bad thing on DNS
Root servers. It was suggested by that conversations typically take only
one path as a result of CEF-like caching.  I noted that providers were
working on per packet load balancing. Well, here it is, in the "major
vendor":

Per-Packet Load Balancing
http://www.cisco.com/univercd/cc/td/doc/product/software/ios120/120newft
/120limit/120s/120s21/pplb.htm

So, it seems that we need to review whether the use of anycast on the
root nameservers is a good idea. I suggest that we ignore for the
momeent anyway the question of whether the deployment of anycast was
done with adequate technical analysis, discussion, and approval, and
just consider whether we should continue doing it.  However, the Root
server operation and oversight issues are very important should also be
discussed, too, but probably by different forums.

		--Dean

---------- Forwarded message ----------
Date: Tue, 18 May 2004 19:03:52 -0400 (EDT)
From: Dean Anderson <[email-masked]>
To: Paul Vixie <[email-masked]>
Cc: [email-masked]
Subject: Re: Root Anycast

On 18 May 2004, Paul Vixie wrote:

> Careful design by whom?  Organic compared to what?  I assure you that 
> f-root has grown by careful design.  It's only organic in that we go 
> where we're invited rather than having a gigantic budget that could be

> used as a leash.

Do you mean "Careful Design" like the non-standard changes in Bind 9
AXFR and IXFR?  I don't think we can take too much of that sort of thing
in the operation of the root servers before we have serious problems.

Unilateral action is not a good thing. There is no point in having an
IETF (Remember the "Internet __Engineering__ Task Force" in IETF) if you
just implement whatever you think OK at the moment (AXFR mods, IXFR
mods, Anycast, and probably more that we just don't yet know about)

		--Dean

.
dnsop resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/dnsop.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/dnsop/index.html

---

From: Dean Anderson <[email-masked]>
Date: Thu, 30 Sep 2004 15:10:19 -0400 (EDT)

> Would you please give more about  the problem of  anycast with root

I notice that Paul Vixie already replied, and as usual, oversimplified and 
gave no useful information. It unclear if he is being deliberately vague 
or if he truly doesn't understand the implications of your questions.

> guarantee sequencing of TCP packets.

That is basically correct, and is possible for a customer/end user DNS
setup with only one router. A root anycast setup is different in that root
servers are located at sites that have more than one (usually many more
than one) connection (peering or transit)  with other providers, and thus
are not 'behind the same router'.  If those providers, or their customers
or peers also have multiple paths, and if one of those customers or peers
enables PPLB, then TCP packets would go to any of the anycast servers.  

For example, the ISC-TN-2004-1 shows a diagram with 2 routers, breaking 
the "everything behind one router" requirement necessary for anycast.  
Apparently, Vixie/ISC really doesn't understand the constraints required 
for anycast operation, in particular, the "behind one router" constraint.

Also, it is not merely the sequencing of packets that causes a problem
(though this causes a minor problem), but the fact that some packets are
delivered to entirely different anycast servers and so a TCP connection
can't be established under the condition of PPLB to a group of anycast
servers behind multiple routers.

> replacing traditional load balancer with router.

PPLB is incompatible with anycast.  Any situation where the anycast server 
has multiple paths to the internet is subject to PPLB use by some other 
provider which could send packets over those paths, ultimately to 
different servers.

It is not enough for the routers at the multihomed anycast root server
farm not to do PPLB. It would be necessary to require that no one anywhere
do PPLB under the multiple router circumstance since each router will have
different cache to one of the anycast servers. And of course, the
requirement that no one anywhere can use PPLB is unreasonble.

> one DNS request cost " and "how many TCP traffic occupies in DNS
> traffic".  If most of DNS request cost more than one UDP packets,
> out-of-sequence may be a problem. Also , if TCP occupies more than 10%
> of traffic it will also be a problem.
>

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 30 Sep 2004 23:56:13 +0200

On 29-sep-04, at 23:41, Dean Anderson wrote:

> Root

> 120newft/120limit/120s/120s21/pplb.htm

Unless I'm very much mistaken, at least _some_ older Cisco boxes have  
been able to do this for a very long time.

Note though that it's *very* hard to create a setup where packets are  
delivered to different multicast instances, as it's hard to imagine how  
any real-world anycast setup could match the criteria in

http://www.cisco.com/en/US/tech/tk365/tk80/ 
technologies_tech_note09186a0080094431.shtml#bgpmpath

(Although a good way to guarantee this would be to use different ASes  
to source the different anycast instances.)

> root
> nameservers is a good idea.

I think the ship has sailed a long time on whether this was going to  
happen at all. However, now would be a good time to start a discussion  
on how much anycasting is enough. It would be good to keep a reasonable  
number of root servers non-anycasted.

I'll be happy to list the reasons for this (apart from general healthy  
conservatism) on some appropriate forum.

---

From: Dean Anderson <[email-masked]>
Date: Thu, 30 Sep 2004 20:48:37 -0400 (EDT)

> any real-world anycast setup could match the criteria in

Its quite easy for anycast: (real names used, but not real relationships)

                Av8
              /   \
          sprint   att
              \   /
              F-root              

If Av8 turns on PPLB, traffic to F-root will go through both sprint and
att on a per-packet basis. However F-root is really anycasted, so it looks
really like this:

                              Av8

                          /           \
                  sprint                 att
                   |  |                  |  |
              ,----+--+--.            ,--+--+----.
              | router 1 |            | router 2 |  ...
              `--+-------'            `-------+--'
                 |                            |
              ,--+-------.            ,-------+--.
              | switch 1 +------------+ switch 2 |  ...
              `-+------+-'            `-+------+-'
                |      |                |      |
         ,------+-.  ,-+------.  ,------+-.  ,-+------.
         | f-root1|  | host 2 |  | f-root3|  | host 4 |  ...
         `--------'  `--------'  `--------'  `--------'
     
So, we see that some packets will go to f-root1 and some packets will go
to f-root3, on a per-packet basis. TCP will not work to an anycasted root
IF anyone on the internet uses PPLB and has the same peers/transit
providers as an anycasted root.

It might well be that packets from f-root1 and f-root3 both go through att
to Av8, assuming that f-root's operator doesn't use PPLB, but that doesn't
matter. The TCP connection can't be made or kept established.