Subject: Re: grow: anycast ops draft

From: Geoff Huston <[email-masked]>
Date: Sun, 07 Nov 2004 07:32:02 +1100

[note - bounced on non-member address - delayed while list owner
 was in transit]

Hi,

some comments after reading through the doc....

2.  Terminology

    The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
    "SHOULD", "SHOULD NOT", "RECOMMENDED",  "MAY", and "OPTIONAL" in this
    document are to be interpreted as described in RFC 2119.

I see no use of these words in the document - in which case its unnecessary.

3.2  Goals

    4.  Triangulation of traffic sources, in the case of attack (or
        query) traffic which incorporates spoofed source addresses;

Are you sure you mean 'triangulation'? I read this as being able to
determine precise location from three sets of relative distance
measurements. in a two dimensional plane

   5.  Improvement of query response time, by reducing the network RTT
        between client and server with the provision of a local Anycast
        Node.

You are assuming that routing systems converge on  the lowest RTT in
selecting a best path. This is not universally true at either the local or
the global level.

4.4.1  Signalling Service Availability

    When a routing system is provided with reachability information for a
    Service Address from an individual node, packets addressed to that
    Service Address will start to arrive at the node.  Since it is
    desirable for the node to be ready to accept requests before they
    start to arrive, a coupling between the routing information and the
    availability of the service at a particular node is desirable.

Some would say "essential' rather than just "desireable".

                                                        This can be
achieved using
    routing protocol implementations on the same servers which provide
    the service being distributed.

I disagree - if the service hangs but the box remains up then this is not
being achieved. Perhaps you could explain this technique in further detail.

     Another
    approach is to tunnel requests from nodes that cannot handle
    individual services to other nodes that can, perhaps using an IGP
    which extends over tunnels between nodes, in which servers
    participate.

You appear to be assuming, without actually saying so, that an anycast
service instance has an associated unicast address and that the service
needs to be available over this unicast address. as well as over the
anycast address for this technique  to work. You should note that this may
be a sub-optimal solution when compared to withdrawal of the instance of
the anycast address from the routing system.

4.4.3  Equal-Cost Paths

    Some routing systems support equal-cost paths to the same
    destination.  Where multiple, equal-cost paths exist and lead to
    different anycast nodes, there is a risk that request packets
    associated with a single transaction might be delivered to more than
    one node.

This confuses multicast with anycast. I suspect you mean that in the case
where a transaction requires multiple packets (or even where there is
fragementation of a single packet), then all the packets may not be
delivered to the same anycast instance. Also, strictly, its the case that
forwarding systems support load balancing of traffic across multiple paths
to the same destination.

4.4.4.  Route Dampening

"especially stable Global Nodes"

Whatever they are, it sounds like I want one. I'm not sure these exist, in
that stability of a route depends of the originator, the transits and the
receiver. In fact I'm really not sure what point is being made in this para.

4.4.5  Reverse Path Forwarding Checks

I know what you are trying to say here, but the words are certainly
unclear. What you are saying is that when a router receives a response from
any anycast instance, it may have learned of a 'best' forwarding to another
instance of this anycast address that uses a different forwarding decision.
Where strict RPF is in use the incoming response is appearing at the router
from the 'incorrect' interface and RPF will cause the packet to be discarded.

   Care should be taken to ensure that strict-mode RPF is not enabled in
    peer networks connecting to anycast nodes.

I disagree with the equivocation here - What you are saying is that RPF and
anycast are incompatible wiith asymmetrical paths, and as the Internet is
today highly asymmetrical the conclusion you are driving to is that RPF and
anycast are mutually incompatible. In the interests of calling a dirt
manipulation device a bloody spade, I would recommend you make this
conclusion overt (!).

4.4.6  Propagation Scope

   Local Nodes advertise covering routes for Service Addresses in such a
    way that their propagation is restricted.  This might be done using
    well-known community string attributes such as NO_EXPORT [6] or
    NOPEER [11], or by arranging with peers to apply a conventional
    "peering" import policy instead of a "transit" import policy, or some
    suitable combination of measures.

If you want to talk about Bad Practices, another mechanism is to simply
advertise the service on a /32 and rely on other AS's using ingress prefix
length filters to remove the anycast from propagating. (whats this use of
the term "AS:es" for the plural of AS by the way?)

5.2  Self-Healing Nodes

    As is described in  having the Anycast Node avoid black-holing
    traffic in the event of a failure on the software or subsystem
    providing the service should be avoided.

As described in  --- where? (missing ref)

6.  Security Considerations

I was expecting to see a discussion about the potential mitigations in the
event that one instance of any anycast services is compromised without
touching the other instances. This form of localized attack at the service
level is often undectected in an anycast instance as a single operational
service monitor cannot simultaneously query all anycast instances of the
service simultaneously. (you mention this in the intro under the term of
monitoring the service, but there are security implications in terms of
detectability of security breaches of the service).

Anycast also makes hijacking a service much easier, and this, too, should
be noted and analysed in this section. How do I know that I am talking to
an authentic instance of the anycast service? Is this a service-specific
problem? In which case are some services wisely never placed on anycast?

You should also talk about the cascading attack. If it is possible to
remove an anycast routing instance from the routing system through an
attack, then a sustained attack can work. By sucessively removing anycast
instances the routing system automatically redirects the attack to the next
anycast instance without the attacker having to do anything at all. This is
probably not desireable.

Also:

I was looking for some comment on tradeoffs in the interdomain case between
route aggregation and the need to inject a new routing
entry for the anycast service - i.e.interdomain anycast is not  something
that naturally scales as it appears to inject a routing entry per anycast
service, which, arguably, is at a finer granularity than per-host routes.
You touch upon this in 4.4.2, but it appears to be saying that in the
interdomain case you can't get away with a /32 advertisement, so use a /24
and things will work just fine. I suspect that you need much stronger
caveats than that and explicitly note that an anycast service in the
inter-domain space burns up a routing slot for the service. and the
interdomain routing system does not expand infinitely to accommodate all
possible services being implemented in such a manner. This implies that
anycast is not a general technique for service resilience and all the other
Good Things you note in section 3.2, but in teh inter-domain case is
actually a specialized solution technique used predominately for common
infrastructure services, such as, surprise, surprise, instances of the DNS
roots.

At 07:03 AM 2/11/2004, Joe Abley wrote:
>Kurtis Lindqvist and I squeezed this draft through before the cut-off:
>
>   http://www.ietf.org/internet-drafts/draft-kurtis-anycast-bcp-00.txt
>
>>Abstract
>>
>>    As the Internet has grown, many services with high availability
>>    requirements have emerged.  The requirements of these services have
>>    increased the demands on the reliability of the infrastructure on
>>    which those services rely.
>>
>>    Many techniques have been employed to increase the availability of
>>    services deployed on the Internet.  This document presents
>>    operational experience of wide-scale service distribution using
>>    anycast, and proposes a series of recommendations for others using
>>    this approach.
>
>Anycast distribution of services (the practice of injecting reachability
>information for an address into a routing system in multiple places, from
>autonomous nodes) is not at all a new idea; however, there are an
>increasing number of people using anycast approaches to distribute
>high-profile services, and few published guidelines for doing so. The aim
>of this document is to provide some guidelines.
>
>There are many, many people with experience deploying anycast services
>whose thoughts and opinions are not in that -00 draft. Hopefully some of
>them are on this list, and they can help us make the draft better.
>
>We have a brief slot in the grow meeting next Tuesday to introduce this
>draft, and we plan to propose that it becomes a working group document. If
>people on this list have a chance to review the draft before then, that
>would be great.
>
>
>Joe
>
>_________________________________________________________________
>web user interface: http://darkwing.uoregon.edu/~llynch/grow.html
>web archive:        http://darkwing.uoregon.edu/~llynch/grow/

_________________________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/grow.html
web archive:        http://darkwing.uoregon.edu/~llynch/grow/

---

From: Kurt Erik Lindqvist <[email-masked]>
Date: Thu, 11 Nov 2004 13:39:32 +0100

Some replies from one of the authors. Joe might have more to contribute.

> I see no use of these words in the document - in which case its 

Agreed.

> measurements. in a two dimensional plane

I understand your point and would like to suggest a word synonym with 
what we mean, but this is not my native language :-). So, the intent is 
that you with anycast can 'home in' on real sources of spoofed traffic. 
Not sure what would be a better way to describe this with.

> selecting a best path. This is not universally true at either the 

Good, point and should be pointed out. Proposed new text :

"	      Potentially improve the query response time, by reducing
               the network RTT between client and server with the
               provision of a local Anycast Node. This is however
               constrained by the attributes associated with any given
               prefix as seen by the client, and by the best-path
               selection algorithm in use in the routing system used at
               the clients network and the intermediate networks. "

> Some would say "essential' rather than just "desireable".

Agreed.

> being achieved. Perhaps you could explain this technique in further 

Ok, I guess agree that point is kind of vague. Suggested new text :

"	    Where a routing advertisement from a node corresponds to a
             single Service Address, this coupling might be such that
             availability of the service triggers the route 
advertisement,
             and non-availability of the service triggers a route
             withdrawal. This can be achieved using routing protocol
             implementations on the same server which provide the
             service being distributed. In addition, some form of local
             verification of the service is needed. This verification 
needs
             to ensure that the service-process is running and actually
             answering on the intended port(s). This is in general
             done with some form of scripting that upon detection of
             failure signals the local routing protcol implementation
             to do a withdrawal, or upon startup trigger an
             announcement. "

>      Another approach is to tunnel requests from nodes that cannot 
> handle

No, that is not what we are saying. We are saying that the 'cluster' or 
node that provides the service have a unicast address, and I can't 
think of a scenario where that would not be true. In the above case you 
use the tunnels to shift traffic to another instance for packets that 
are arriving to the node while for example a BGP convergence event 
occurs in response to the local service being unavailable. This does 
not assume that the service is available at the unicast address, it 
assumes that the far end of the tunnel ends on a unicast address on a 
system that through routing local to the far tunnel end have a path to 
a working anycast service.

> forwarding systems support load balancing of traffic across multiple 

Correct. Reading the text above I am not really sure why you think this 
describes multicast? I actually think the text says the same as yours?

> receiver. In fact I'm really not sure what point is being made in this 

This is a forward looking statement that probably should be rewritten.

Basically we are saying that a global node needs to me more stable than 
a local node, or a node that is scoped. There are several reasons for 
this, and with dampening you run the risk of a global node dampened 
effecting a lot more people than a dampened local node. This is most 
likely a more future proof design. But yes, that should be rewritten 
but I have no good proposal right now.

> manipulation device a bloody spade, I would recommend you make this

I think you have the first part correct, but not the second part. This 
is my text from a real world example. This occurs when you have a 
transit provider for one of the anycast nodes seen as a customer, and 
therefor preferred in one location of their network, but at the same 
time they are seeing this prefix from a direct peer in another part of 
the network. Now this particular router might be hit by a local weight 
or similar which will trigger RPF on this particular router. So it is 
not a direct effect of asymmetrical routing. While writing this, it 
strikes me that the scenario is probably close to some of the ones in 
BGP wedgies, but with other mechanisms.

> length filters to remove the anycast from propagating.

True. But it's not very granular :-)

> (whats this use of

I think that is my bad spelling....

> As described in  --- where? (missing ref)

Thanks!

> detectability of security breaches of the service).

Just to make sure I understand, when you say compromised you mean that 
someone manages to manipulate the data either through data insertion or 
gaining access-to the box itself? Good point.

> problem? In which case are some services wisely never placed on 

I am not sure I would agree that anycast makes hijacking per se easier. 
But it does make it a lot harder to detect, and I agree that should go 
into the text.

> anycast instance without the attacker having to do anything at all. 

Hmm. I think this is actually a feature of anycast, so I am not sure I 
would agree that this poses a security concern, unless you are afraid 
to 'overload' the fall-back node. However, that is poor design as that 
could happen with a network failure as well.

> infrastructure services, such as, surprise, surprise, instances of the 

I agree that we should elaborate on the aggregation risks, but I am not 
sure I think we should include references to specific services?

- - kurtis -