Subject: grow: Re:

From: Geoff Huston <[email-masked]>
Date: Fri, 12 Nov 2004 02:26:37 +1100

thanks for these responses Kurt

I'll only respond to the followup questions here.....

> 3.2  Goals
>>
>>
>>     4.  Triangulation of traffic sources, in the case of attack (or
>>         query) traffic which incorporates spoofed source addresses;
>>
>> Are you sure you mean 'triangulation'? I read this as being able to
>> determine precise location from three sets of relative distance
>> measurements. in a two dimensional plane
>
>I understand your point and would like to suggest a word synonym with
>what we mean, but this is not my native language :-). So, the intent is
>that you with anycast can 'home in' on real sources of spoofed traffic.
>Not sure what would be a better way to describe this with.

how about:

"Resolution of traffic source locality, to the level of granularity of the anycast distribution, in the case of attack (or query) traffic which incorporates spoofed source addresses;"

>>    5.  Improvement of query response time, by reducing the network RTT
>>         between client and server with the provision of a local Anycast
>>         Node.
>>
>> You are assuming that routing systems converge on  the lowest RTT in
>> selecting a best path. This is not universally true at either the  local or
>> the global level.
>
>Good, point and should be pointed out. Proposed new text :
>
>"             Potentially improve the query response time, by reducing
>               the network RTT between client and server with the
>               provision of a local Anycast Node. This is however
>               constrained by the attributes associated with any given
>               prefix as seen by the client, and by the best-path
>               selection algorithm in use in the routing system used at
>               the clients network and the intermediate networks. "

I agree with this proposed text in responding to the point I raisid.

>> 4.4.1  Signalling Service Availability
>>
>>     When a routing system is provided with reachability information for a
>>     Service Address from an individual node, packets addressed to that
>>     Service Address will start to arrive at the node.  Since it is
>>     desirable for the node to be ready to accept requests before they
>>     start to arrive, a coupling between the routing information and the
>>     availability of the service at a particular node is desirable.
>>
>> Some would say "essential' rather than just "desireable".
>
>Agreed.
>
>>                                                         This can be achieved using
>>     routing protocol implementations on the same servers which provide
>>     the service being distributed.
>>
>> I disagree - if the service hangs but the box remains up then this is not
>> being achieved. Perhaps you could explain this technique in further
>> detail.
>
>Ok, I guess agree that point is kind of vague. Suggested new text :
>
>"           Where a routing advertisement from a node corresponds to a
>             single Service Address, this coupling might be such that
>             availability of the service triggers the route advertisement,
>             and non-availability of the service triggers a route
>             withdrawal. This can be achieved using routing protocol
>             implementations on the same server which provide the
>             service being distributed. In addition, some form of local
>             verification of the service is needed. This verification needs
>             to ensure that the service-process is running and actually
>             answering on the intended port(s). This is in general
>             done with some form of scripting that upon detection of
>             failure signals the local routing protcol implementation
>             to do a withdrawal, or upon startup trigger an
>             announcement. "

this makes it more explict that the local 'monitor needs to check that the service process is actually functional in order to maintain the origination of the anycast route. I agree with this proposed revision.

>>      Another approach is to tunnel requests from nodes that cannot handle
>>     individual services to other nodes that can, perhaps using an IGP
>>     which extends over tunnels between nodes, in which servers
>>     participate.
>>
>> You appear to be assuming, without actually saying so, that an anycast
>> service instance has an associated unicast address and that the service
>> needs to be available over this unicast address. as well as over the
>> anycast address for this technique  to work. You should note that this may
>> be a sub-optimal solution when compared to withdrawal of the instance of
>> the anycast address from the routing system.
>
>No, that is not what we are saying. We are saying that the 'cluster' or
>node that provides the service have a unicast address, and I can't
>think of a scenario where that would not be true. In the above case you
>use the tunnels to shift traffic to another instance for packets that
>are arriving to the node while for example a BGP convergence event
>occurs in response to the local service being unavailable. This does
>not assume that the service is available at the unicast address, it
>assumes that the far end of the tunnel ends on a unicast address on a
>system that through routing local to the far tunnel end have a path to
>a working anycast service.

I expressed myself badly, and you have correctly rewritten my poor explanation
here and illustrated the assumption. While all anycast service platforms would
normally require a unicast address for a management port at the very least, it
is often the case that in attempting to secure a service delivery platform you
only respond to service traffic that is presented on the addresses  that the box
is configured to delivered traffic. If you applied this to anycast the box would only
deliver the service on the anycast prefix. For this tunnelling to work as you describe
the remote end needs to recognise the tunnel, and deliver the response. You do not
make it clear how the response is delivered. Should the response be passed back
down the tunnel to the originating anycast host ? Is this an IP level tunnel or an
application level relay. Is the IGP that extends over tunnels between nodes a
distinct routing domain?

Is this a current operating practice for anycast service overlay networks? If
so, than maybe additional text drawn from this operational experience would
help here.

>> 4.4.3  Equal-Cost Paths
>>
>>
>>     Some routing systems support equal-cost paths to the same
>>     destination.  Where multiple, equal-cost paths exist and lead to
>>     different anycast nodes, there is a risk that request packets
>>     associated with a single transaction might be delivered to more
>> than
>>     one node.
>>
>> This confuses multicast with anycast. I suspect you mean that in the
>> case
>> where a transaction requires multiple packets (or even where there is
>> fragementation of a single packet), then all the packets may not be
>> delivered to the same anycast instance. Also, strictly, its the case
>> that
>> forwarding systems support load balancing of traffic across multiple
>> paths
>> to the same destination.
>
>Correct. Reading the text above I am not really sure why you think this
>describes multicast? I actually think the text says the same as yours?

My point was that  "delivered to more than one node" can imply that the
same packet was delivered to 2 or more nodes, or that while each packet
within a packet sequence is delivered to a single node, different packets
in the sequence may be delivered to different nodes. While the latter may
be obvious to you, it is not as obvious in the text itself. I was nit picking this
source of ambiguity.

>> 4.4.4.  Route Dampening
>>
>> "especially stable Global Nodes"
>>
>> Whatever they are, it sounds like I want one. I'm not sure these  exist, in
>> that stability of a route depends of the originator, the transits and  the
>> receiver. In fact I'm really not sure what point is being made in this para.
>
>This is a forward looking statement that probably should be rewritten.
>
>Basically we are saying that a global node needs to me more stable than
>a local node, or a node that is scoped. There are several reasons for
>this, and with dampening you run the risk of a global node dampened
>effecting a lot more people than a dampened local node. This is most
>likely a more future proof design. But yes, that should be rewritten
>but I have no good proposal right now.

Neither have i at this point - maybe Joe has some ideas?

>> 4.4.5  Reverse Path Forwarding Checks
>>
>> I know what you are trying to say here, but the words are certainly
>> unclear. What you are saying is that when a router receives a response from
>> any anycast instance, it may have learned of a 'best' forwarding to another
>> instance of this anycast address that uses a different forwarding  decision.
>> Where strict RPF is in use the incoming response is appearing at the  router
>> from the 'incorrect' interface and RPF will cause the packet to be
>> discarded.
>>
>>    Care should be taken to ensure that strict-mode RPF is not enabled  in
>>     peer networks connecting to anycast nodes.
>>
>> I disagree with the equivocation here - What you are saying is that  RPF and
>> anycast are incompatible wiith asymmetrical paths, and as the Internet  is
>> today highly asymmetrical the conclusion you are driving to is that  RPF and
>> anycast are mutually incompatible. In the interests of calling a dirt
>> manipulation device a bloody spade, I would recommend you make this
>> conclusion overt (!).
>
>I think you have the first part correct, but not the second part. This
>is my text from a real world example. This occurs when you have a
>transit provider for one of the anycast nodes seen as a customer, and
>therefor preferred in one location of their network, but at the same
>time they are seeing this prefix from a direct peer in another part of
>the network. Now this particular router might be hit by a local weight
>or similar which will trigger RPF on this particular router. So it is
>not a direct effect of asymmetrical routing. While writing this, it
>strikes me that the scenario is probably close to some of the ones in
>BGP wedgies, but with other mechanisms.

we are close here - the issue of asymmetric paths is the key here in that along the
reverse path from the anycast server to the source the intermediate routers may
prefer a path to the anycast route that leads along an entirely different path that
heads towards a different instance of the anycast address, and this triggers
RPF. So its a case of asymmetry and RPF being a base issue,  and anycast has
the potential to exacerbate this.

>> 4.4.6  Propagation Scope
>>
>>    Local Nodes advertise covering routes for Service Addresses in such  a
>>     way that their propagation is restricted.  This might be done using
>>     well-known community string attributes such as NO_EXPORT [6] or
>>     NOPEER [11], or by arranging with peers to apply a conventional
>>     "peering" import policy instead of a "transit" import policy, or  some
>>     suitable combination of measures.
>>
>> If you want to talk about Bad Practices, another mechanism is to simply
>> advertise the service on a /32 and rely on other AS's using ingress  prefix
>> length filters to remove the anycast from propagating.
>
>True. But it's not very granular :-)

:-)

> 6.  Security Considerations
>>
>> I was expecting to see a discussion about the potential mitigations in  the
>> event that one instance of any anycast services is compromised without
>> touching the other instances. This form of localized attack at the  service
>> level is often undectected in an anycast instance as a single  operational
>> service monitor cannot simultaneously query all anycast instances of  the
>> service simultaneously. (you mention this in the intro under the term  of
>> monitoring the service, but there are security implications in terms of
>> detectability of security breaches of the service).
>
>Just to make sure I understand, when you say compromised you mean that
>someone manages to manipulate the data either through data insertion or
>gaining access-to the box itself? Good point.

gaining access to an anycast instance box is what was in my mind at the time.

>> Anycast also makes hijacking a service much easier, and this, too,  should
>> be noted and analysed in this section. How do I know that I am talking  to
>> an authentic instance of the anycast service? Is this a  service-specific
>> problem? In which case are some services wisely never placed on
>> anycast?
>
>I am not sure I would agree that anycast makes hijacking per se easier.
>But it does make it a lot harder to detect, and I agree that should go
>into the text.

I indeed meant to say 'harder to detect".

>> You should also talk about the cascading attack. If it is possible to
>> remove an anycast routing instance from the routing system through an
>> attack, then a sustained attack can work. By sucessively removing  anycast
>> instances the routing system automatically redirects the attack to the next
>> anycast instance without the attacker having to do anything at all.
>> This is probably not desireable.
>
>Hmm. I think this is actually a feature of anycast, so I am not sure I
>would agree that this poses a security concern, unless you are afraid
>to 'overload' the fall-back node. However, that is poor design as that
>could happen with a network failure as well.

Let me explain this another way - in the "traditional" many to one DDOS attack when many zombies that are highly distributed all attack a single unicast address, then anycast effectively makes this a many-to-many attack, and this is a feature of anycast. But when the attack concentrates the locality of the attackers such that it is a many-to-one-anycast instance, then there is the possibility of cascaded attack as described above.

>> I was looking for some comment on tradeoffs in the interdomain case  etween
>> route aggregation and the need to inject a new routing
>> entry for the anycast service - i.e.interdomain anycast is not something
>> that naturally scales as it appears to inject a routing entry per anycast
>> service, which, arguably, is at a finer granularity than per-host routes.
>> You touch upon this in 4.4.2, but it appears to be saying that in the
>> interdomain case you can't get away with a /32 advertisement, so use a  /24
>> and things will work just fine. I suspect that you need much stronger
>> caveats than that and explicitly note that an anycast service in the
>> inter-domain space burns up a routing slot for the service. and the
>> interdomain routing system does not expand infinitely to accommodate  all
>> possible services being implemented in such a manner. This implies that
>> anycast is not a general technique for service resilience and all the  other
>> Good Things you note in section 3.2, but in teh inter-domain case is
>> actually a specialized solution technique used predominately for common
>> infrastructure services, such as, surprise, surprise, instances of the
>> DNS
>> roots.
>
>I agree that we should elaborate on the aggregation risks, but I am not
>sure I think we should include references to specific services?

I was thinking of saying "anycast is applicable to some services, but probably not to any service that a reader of this document may have had in mind up until this statement." :-)

sure - omitting the ref to a specific service works for me.

  Geoff

_________________________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/grow.html
web archive:        http://darkwing.uoregon.edu/~llynch/grow/

---

From: Joe Abley <[email-masked]>
Date: Thu, 11 Nov 2004 14:42:28 -0500

Hi Geoff!

On 11 Nov 2004, at 10:26, Geoff Huston wrote:

> [tunnel stuff]

There are at least two well-known anycast deployments that use tunnels 
to route requests from anycast node A to anycast node B in the event 
that node A is unable to service the request. I don't know the details 
of how the response is routed, but I suspect it is routed from B 
directly back to the client.

My impression of how they are constructed is as follows:

1. Nodes are connected internally using tunnels. Tunnels are plumbed 
between globally-unique, unicast addresses.

2. Nodes all participate in an IGP.

3. Hosts advertise service addresses into the IGP.

Hence if a node happens to have a dead host, an IGP route will route 
the request received at that node over an external interface towards 
another node which has a host able to service the request. The response 
would be sourced from the anycast service address, and routed back 
towards the client directly from the second node.

I am trying to get some of the operators of the two anycast deployments 
that I know about to review and/or contribute to this document.

To some extent, the use of tunnels between nodes as described above 
introduces some shared fate between nodes, which could certainly be 
considered to be a Bad Thing. But on the other hand (and this is the 
reason it was done, I believe, in the two cases I keep obliquely 
mentioning) it mitigates the routing-slot-per-service problem that you 
mention elsewhere in your reply, since a single advertised anycast 
prefix can cover multiple service address.

> Neither have i at this point - maybe Joe has some ideas?

By "stable" I meant "with lots of diverse power, redundant air 
conditioning, in a building which doesn't look like it's about to fall 
over, multiply-attached to transit providers and with enough redundant 
equipment that maintenance won't cause route withdrawals".

A small local node deployed out in some remote part of the Internet 
might not enjoy all of those, but it's a good idea if the global nodes 
do.

Does the paragraph make more sense now, given that context?

> Let me explain this another way - in the "traditional" many to one 
> DDOS attack when many zombies that are highly distributed all attack a 
> single unicast address, then anycast effectively makes this a 
> many-to-many attack, and this is a feature of anycast. But when the 
> attack concentrates the locality of the attackers such that it is a 
> many-to-one-anycast instance, then there is the possibility of 
> cascaded attack as described above.

Cascading failures are worthy of mention. I had intended to talk about 
them (I talk about them in the ISC f-root anycast document) but it 
looks like I forgot.

The extent of the cascade can be limited to a subset of nodes using the 
global/local node distinction,  too, since not all nodes provide 
candidate paths to every network in that case. I will dream up some 
text for this before the next revision.

Joe