Subject: Re: [rrg] Anycast in the core architecture

From: [email-masked]
Date: Tue, 21 Apr 2009 18:25:39 EDT

In einer eMail vom 21.04.2009 23:54:19 Westeuropäische Normalzeit schreibt  
[email-masked]:

On Tue,  Apr 21, 2009 at 4:00 AM,  <[email-masked]> wrote:
> As  I understand Anycast  is about delivery to one out of multiple
>  destinations within a given scope.

Hi Heiner,

I would have said  "closest one from the set of valid destinations,"
but your version is good  enough.
More precisely "closest one from the set of valid destinations with respect 
 to the source,"

> Where is the center point of this scope ? Is it always  the location of 
the
> source ?

I don't understand the question.  What is a center point wrt routing?

My question is: with anycast do we always understand the above, or could it 
 also be any particular destination within the scope of an indicated remote 
 location like a different continent?
 
What do we understand by Anycast in the future ? Some one must do the  
mapping to a specific Unicast address? Is it always the ingress ? Or can't there 
 be reasonable anycast services which require to send the data to some far  
distant  anycast server where it will be mapped to some  local unicast  
addresss ?

>  What are the requirements for Anycast in a new architecture?

Well,  that's the crux of the discussion, isn't it? Anycast and unicast
are  identical in every respect in the current architecture. How do we
bound the  unforeseeable consequences if that isn't true in the new
one?

I'm  getting ready to introduce an anycast route into the table in
order to  implement a "continuing operations" system with three sites
half a world  apart. "Continuing operations" is a fancy phrase for
"disaster recovery,"  something that became really popular almost 8
years ago. Once a packet hits  any of the always-running sites, a VPN
takes it back to the site with the  servers flagged "best" for that
particular address, so holistically its  unicast but from the
perspective of the Internet core its anycast from  three distinct
locations.

Would this have any hope of working right  in an architecture that
didn't accommodate anycast? How many such uses are  out there, ready to
impede the deployment of the unwary  plan?

Sorry, I don't understand what you are hereby saying.
Heiner

Regards,
Bill Herrin

---

From: William Herrin <[email-masked]>
Date: Wed, 22 Apr 2009 13:28:58 -0400

On Wed, Apr 22, 2009 at 12:03 PM, Tony Li <[email-masked]> wrote:
>> Big important web site = at least one more route
>> in the table.
>
> Who decides what's big and important?

Hi Tony,

Historically, the people willing to pay the most money do. Like CNN.

>> I think of deterministic as "pick one direction" while
>> non-deterministic is "travel all valid directions." I'm open to a
>> better word choice.
>
> I'd favor 'unicast' for "pick one direction" and 'multicast' for "pick all
> valid directions".

Then anycast is by definition unicast since you only pick one
direction. Hence some unicast is anycast. Hence failing to support
anycast in the architecture is a failure to completely support
unicast. ;-)

>> The upper layers set requirements on the routing layer in a manner the
>> routing layer can't handle efficiently. One answer is that we redesign
>> the routing layer. Another is that we alter the requirements.
>
> Ok, but asking the routing layer to defy the laws of physics (i.e., scale
> without aggregation) is unlikely to be productive.

Then we change the requirements. Eliminate the demand that the
elements used for the next hop decision retain a constant association
with the endpoints over time. I wonder how many people still disagree
with that?

On Wed, Apr 22, 2009 at 12:03 AM, Robin Whittle <[email-masked]> wrote:
>> And as I mentioned, it's relatively trivial to get it to look
>> precisely the same in each of the two major solution strategies
>> as well.
>
> I think this is probably not true of Strategy B (core-edge
> elimination, in which new host stack and perhaps application
> functions do all the work and end-user networks never have their own
> stable address space.

Hi Robin,

What's the difference between anycast and unicast? Is it a difference
in the forwarding or data planes? Or is it a difference in how the
respondent machine(s) understand the address?

It's the latter of course. In strategy B, the host's understanding of
each packet moves out of the packet forwarding system entirely and
into the map used by the respective hosts. A strategy-B host literally
doesn't care what layer-3 addresses were attached to the received
packet. The address in the routing system only retains semantics
associated with the forwarding process itself. So, as long as the
mapping system adequately supports something that looks like anycast,
the whole strategy B system does as well.

Hi Heiner,

I don't know. Some backbones currently have a limited ability to
restrict advertisements by geography or topology using BGP communities
but I'm not sure to what degree its used.

> What do we understand by Anycast in the future ? Is it always the ingress ?

Ingress is actually two separate questions:

1. Is an anycast address only ever the respondent or does it also
initiate communications?
2. Is the anycast address ever allowed to appear in the "source"
position in the packet?

Making existing UDP and TCP work requires #2. I haven't come up with a
scenario where initiating communication from an anycast address is
desirable.

> Sorry, I don't understand what you are hereby saying.

I'll simplify the design for the sake of the discussion.

You have two sites: primary in New York (NY), disaster recovery in
Tokyo (T). Each site has one Internet link, one router and one server.

Sprint provides the Internet link to NY. Cogent provides the Internet link to T.

The NY router announces 1.2.3.0/24 via BGP at normal priority. The T
router announces 1.2.3.0/24 after prepending the AS# 3 times, causing
most of the other routers on the Internet to consider the T router
more distant than the NY router.

The two sites are normally linked together with a VPN traveling over
the Internet using 5.6.7.8 on one side and 8.7.6.5 on the other,
addresses provided by the respective ISPs. This functions as if there
was a private line directly connected between the two routers.

Ordinarily, the T router sends packets for 1.2.3.4 over the VPN to the
NY router while the NY router sends packets for 1.2.3.4 to the NY
server.

If the VPN fails, the T router will send packets for 1.2.3.4 to the T
server instead.

If the NY server fails, the NY router will send packets for 1.2.3.4
over the VPN to the T router and the T router will send packets for
1.2.3.4 to the T server.

Both servers answer to 1.2.3.4. Both return identical content.

Now, lets look at what happens in this structure. In normal operation,
1.2.3.4 is a unicast address. Requests always route to the NY server.
The T server is a hot standby that only becomes routed if the NY
server fails.

Something interesting happens during what's known as a "split brain"
scenario. Let's say Sprint gets mad at Cogent and they depeer again.
The VPN breaks and just like that, THE ADDRESS USAGE CONVERTS TO
ANYCAST. Traffic from Sprint goes to the NY server, the only one it
can see. Traffic from Cogent goes to the T server, the only one it can
see. Traffic from the rest of the Internet reaches either of the
servers depending on the length of the AS path as seen from the
requester.

So, in this disaster recovery scenario, the address usage changes back
and forth between unicast and anycast depending on the state of the
network.

Two points here:

1. Folks using a design like this will be reluctant to use a routing
system which does not support something functionally equivalent to
anycast. And they tend not to be paupers. Money speaks and their money
will impede the deployment of the new system.

2. Disaster Recovery (DR) sites are in high demand but folks find them
difficult and expensive to implement. Even after going to the expense,
DR's success rate is painfully low.

Suppose our new architecture enables the above design without
requiring support in the forwarding plane the way the above design
does? Why then it would be cheap, and you could implement DR by
renting a $50 virtual server in Tokyo instead of building a physical
site.

DR could be our killer app. Making DR cheap, accessible and easy would
be a powerful motivator for deploying our new system.

Regards,
Bill Herrin

-- 
William D. Herrin ................ [email-masked]  [email-masked]
3005 Crane Dr. ...................... Web: <http://bill.herrin.us/>
Falls Church, VA 22042-3004

---

From: Robin Whittle <[email-masked]>
Date: Thu, 23 Apr 2009 14:00:49 +1000

Hi Bill,

I will respond to what you wrote to me in the "Anycast in the core
architecture - sep. OK; elim not?" thread and what you wrote about
"Disaster Recovery" in a separate thread.

  - Robin

---

From: Tony Li <[email-masked]>
Date: Mon, 27 Apr 2009 09:55:12 -0700

Hi Bill,

Sorry for the delay.  The Real World intervened.  ;-)

> Historically, the people willing to pay the most money do. Like CNN.

Well, we've run that experiment.  We've been unable to create a market 
for routing table slots.  Handing them out any other way would seem to 
get us right back into the problem that we're trying to solve.

> anycast in the architecture is a failure to completely support

We don't support any unaggregated (or insufficiently aggregated) 
unicast.  What else is new?  ;-)

Tony