Subject: Re: grow: draft-ietf-grow-anycast-03

From: Dean Anderson <[email-masked]>
Date: Fri, 4 Nov 2005 19:40:14 -0500 (EST)

I am somewhat reticent to comment, given that I have been threatened by David
Kessens with removal from IETF participation for opposing the current ISC use of
Anycast extension.  BTW, if you think my contributions have been productive and
reasonably well-documented disagreement on this subject, please let the IESG
know that you oppose the current RFC3683 action against me by Dave Crocker and
David Kessens.

There are several errors and omissions in this section. I do not think the 
document is ready for last call.

First, the TCP performance issues listed at the very end also apply to PPLB
(load splitting) over parallel links.  For some reason, you don't think this 
makes PPLB over parallel links "pathological".

Second, the TCP performance issue noted is minor, not severe. It does not cause
non-operation, or practical non-operation, nor even user-noticeable degradation.
--it only causes packets to be -unnecessarily- retransmitted 3% of the time.  
Therefore, it seems to be inappropriate to characterize this as "pathological".
Engineers and designers make performance trade-offs all the time. Its the job of
engineers and designers to make such trade-offs. Plainly, if you are designing a
network to cater to UDP, such as audio and video streaming, then TCP performance
is of little concern.  Further, completely omitted is any mention of the
benefits of diverse load-splitting, which are substantial as previously
explained.

Third, the use of Equal cost paths is not limited to BGP nor does is it
dependent on BGP exit selection algorithm. Many ISPs use "hot-potato" (closest
exit) exit routing, and so equal cost static routes and equal cost IGP routes
can lead to multiple anycast hosts.

Fourth, there seems to be an assumption that PPLB over diverse paths is not
presently used in the Internet.  This assumption is false. In fact, the Cisco
GSR Engine2 platform suffers performance degradation if PPLB is not enabled on
all interfaces (all or nothing).  GSR E2 users with parallel links must also use
diverse PPLB.  And there is increasing use of intentionally diverse
load-balancing in voice and video networks.  Av8 Internet as well as other
companies offer redundant connection services now, and have for several years.  
[I also note that when the question of PPLB and Anycast use was asked on Nanog,
that Nanog members asserted incorrectly that the GSR was incapable of PPLB]

Fifth, the references to an internal company specification, (ISC-TN-2004-1)
should be removed.  RFC2991 includes a list of methods to obtain course-grained 
load-balancing.  These methods are well understood by router vendors, and rather 
extraneous to the Anycast Extension with respect to fine-grained load-splitting.

Sixth, the notion of remote anycast "node selection" still permeates the
document, (and the section below). These references should be removed. This
notion of remote node selection is misleading and incorrect. A router or source
node cannot select the remote anycast node. Routing is "Next-Hop", and is done
hop-by-hop.{*] This terminology leads to confusion, and is not used in any other
IETF documents that I could find.

Seventh, the "PPLB" terms should generally be replaced with the term
"load-splitting", which is the term used in RFC1812. The term "PPLB" appears to
be Cisco-specific.  RFC language should be vendor-neutral.

Eighth, by describing fine-grained load splitting as "pathological", this draft
seems to update and alter RFC1812 without explicit discussion on the merits of
that alteration.

[*] Not only is there no "node selection", it is not even possible to know with
certainty what remote anycast node a packet sent by a source node will
ultimately arrive at, even without load-splitting.  This is because a BGP feed
may be filtered, leading to remote routing decisions that may not be visible to
the source node.  The source node, by examining its routing table and BGP route
table, may see only one route to a remote anycast node though another AS.
However that middle AS may have filtered out a route to a different anycast node
but still use that route in its own internal routing decisions.

         Anycast 1             Anycast 2
            |                      |
          AS3557                AS3557
           |                      |
          AS1                    AS2
           \                      |
            \                    AS3
             \                  /
              \                /
               ---- AS4 ------
                     |
                    AS5        

AS4 filters the path through AS1 from its BGP feed to AS5.  A source node in AS5 
would see only the path through AS3, and conclude that its packets would arrive 
at Anycast node 2.  This conclusion is wrong, since AS4 sees a shorter path 
through AS1, even though it does not reveal this path to AS5.

On Wed, 19 Oct 2005, Joe Abley wrote:

> Hi,
> 
> I just submitted the attached document; it should appear in the usual  
> repository in due course.
> 
> The principal changes are in section 4.4.3, the full content of which  
> can be found below; the rest of the document is substantially unchanged.
> 
> Feedback would be gratefully received.
> 
> 
> Joe
> 
> 4.4.3  Equal-Cost Paths
> 
>     Some routing systems support equal-cost paths to the same
>     destination.  Where multiple, equal-cost paths exist and lead to
>     different anycast nodes, there is a risk that different request
>     packets associated with a single transaction might be delivered to
>     more than one node.  Services provided over TCP [RFC0793]  
> necessarily
>     involve transactions with multiple request packets, due to the TCP
>     setup handshake.
> 
>     For services which are distributed across the global Internet using
>     BGP, equal-cost paths are normally not a consideration: BGP's exit
>     selection algorithm usually selects a single, consistent exit for a
>     single destination regardless of whether multiple candidate paths
>     exist.  Implementations of BGP exist that support multi-path exit
>     selection, however.
> 
>     Equal cost paths are commonly supported in IGPs.  Multi-node
>     selection for a single transaction can be avoided in most cases by
>     careful consideration of IGP link metrics, or by applying equal-cost
>     multi-path (ECMP) selection algorithms which cause a single node to
>     be selected for a single multi-packet transaction.  For an  
> example of
>     the use of hash-based ECMP selection in anycast service  
> distribution,
>     see [ISC-TN-2004-1].
> 
>     Other ECMP selection algorithms are commonly available, including
>     those in which packets from the same flow are not guaranteed to be
>     routed towards the same destination.  ECMP algorithms which select a
>     route on a per-packet basis rather than per-flow are commonly
>     referred to as performing "Per Packet Load Balancing" (PPLB).
> 
>     With respect to anycast service distribution, some uses of PPLB may
>     cause different packets from a single multi-packet transaction sent
>     by a client to be delivered to different anycast nodes, effectively
>     making the anycast service unavailable.  Whether this affects
>     specific anycast services will depend on how and where anycast nodes
>     are deployed within the routing system, and on where the PPLB is
>     being performed:
> 
>     1.  PPLB across multiple, parallel links between the same pair of
>         routers should cause no node selection problems;
> 
>     2.  PPLB across diverse paths within a single autonomous system  
> (AS),
>         where the paths converge to a single exit as they leave the AS,
>         should cause no node selection problems;
> 
>     3.  PPLB across links to different neighbour ASes where where the
>         neighbour ASes have selected different nodes for a particular
>         anycast destination will, in general, cause request packets  
> to be
>         distributed across multiple anycast nodes.  This will have the
>         effect that the anycast service is unavailable to clients
>         downstream of the router performing PPLB.
> 
>     The uses of PPLB which have the potential to interact badly with
>     anycast service distribution can also cause persistent packet
>     reordering.  A network path that persistently reorders segments will
>     degrade the performance of traffic carried by TCP [Allman2000].   
> TCP,
>     according to several documented measurements, accounts for the bulk
>     of traffic carried on the Internet ([McCreary2000], [Fomenkov2004]).
>     Consequently, in many cases it is reasonable to consider networks
>     making such use of PPLB to be pathological.
> 
> 

-- 
Av8 Internet   Prepared to pay a premium for better service?
www.av8.net         faster, more reliable, better service
617 344 9000   

_________________________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/grow.html
web archive:        http://darkwing.uoregon.edu/~llynch/grow/

---

From: Joe Abley <[email-masked]>
Date: Fri, 4 Nov 2005 21:36:15 -0500

Hi Dean,

Thanks for the technical issues you raised. Comments below.

> makes PPLB over parallel links "pathological".

I don't have a good reference to cite, but it is my experience that  
per-packet load balancing over parallel circuits leads to levels of  
packet reordering which are low enough that serious TCP performance  
degradation does not result. It may also be that the relative  
adjacency of out-of-order segments (e.g. reliably within a window)  
allows common TCP implementations to cope well, whereas topologies  
which result in greater dispersion of segments cause the pipeline to  
stall more frequently.

This would be an interesting subject for detailed examination. I  
don't think this level of detail is required in the document at hand,  
however, whose topic is differently-focussed.

> Second, the TCP performance issue noted is minor, not severe.

I have operational experience of the case in hand, and I stand by my  
description. Perhaps we differ in our interpretation of the word  
"severe".

> [...] Further, completely omitted is any mention of the

Those are out-of-scope for this document, which aims to discuss  
anycast service distribution, not per-packet load balancing.

> dependent on BGP exit selection algorithm.

Indeed, the section on PPLB specifically does not refer to BGP.

> presently used in the Internet.  This assumption is false.

There are examples of almost every kind of pathology deployed on the  
Internet. This is a sufficiently well-known truism that I don't think  
it needs to be spelt out.

> should be removed.

ISC-TN-2004-1 is a public document published by ISC. Since it is  
included as an informative (and not normative) reference, I don't  
believe this presents any problems.

> RFC2991 includes a list of methods to obtain course-grained

RFC 2991 describes equal-cost path selection algorithms which aim to  
isolate packets associated with a single flow to a single next-hop.  
These algorithms facilitate anycast distribution of TCP-based services.

I don't exactly understand what point you're making here, but I would  
not object to the idea of including a reference to RFC 2991 to  
strengthen the point that naive per-packet load balancing is poor  
practice, and can be considered pathological.

> document, (and the section below). These references should be removed.

The phrase "node selection" is intended in this document to indicate  
a selected path through one or more interconnected routing systems to  
a particular anycast node. It is certainly not a description of some  
mechanism by which a client can select a particular node for service.

I think the document is quite clear on this point, but if you have  
specific examples of ambiguous text I would be happy to review them.

> be Cisco-specific.  RFC language should be vendor-neutral.

I do not believe that "PPLB" is a cisco term. The term "load- 
splitting" is not an appropriate replacement for "PPLB" as used in  
this document, since it is too general.

> seems to update and alter RFC1812 without explicit discussion on  

The document does not describe fine-grained load splitting as  
pathological; it describes specific topologies which are likely to  
break TCP as pathological. This is an operational consideration.

> ultimately arrive at, even without load-splitting.

Correct. I believe the document describes this quite clearly, but if  
you care to illustrate specific text you find ambiguous, I would be  
glad to review it.

Joe

---

From: Kurt Erik Lindqvist <[email-masked]>
Date: Sat, 5 Nov 2005 15:26:33 +0100

Just for the record...

On 5 nov 2005, at 03.36, Joe Abley wrote:
> which result in greater dispersion of segments cause the pipeline  
> to stall more frequently.
>
> don't think this level of detail is required in the document at  
> hand, however, whose topic is differently-focussed.

Agreed.

> I have operational experience of the case in hand, and I stand by  
> my description. Perhaps we differ in our interpretation of the word  
> "severe".

Agreed.

> anycast service distribution, not per-packet load balancing

Agreed. And at least to me "load-spltting" can be done in several  
ways for various protocols and this is a protocol agnostic document  
on how to do anycast...

> Indeed, the section on PPLB specifically does not refer to BGP.

Agreed.

> There are examples of almost every kind of pathology deployed on  
> the Internet. This is a sufficiently well-known truism that I don't  
> think it needs to be spelt out.

Agreed.

> believe this presents any problems.

For Informational and non-normative documents you can AFAIK reference  
anything you want. I am pretty sure I have seen ISBN numbers, web- 
links, other SDO documents etc.

> RFC 2991 describes equal-cost path selection algorithms which aim  
> to isolate packets associated with a single flow to a single next- 
> hop. These algorithms facilitate anycast distribution of TCP-based  
> services.
>
> I don't exactly understand what point you're making here, but I  
> would not object to the idea of including a reference to RFC 2991  
> to strengthen the point that naive per-packet load balancing is  
> poor practice, and can be considered pathological.

Ok with me.

> The phrase "node selection" is intended in this document to  
> indicate a selected path through one or more interconnected routing  
> systems to a particular anycast node. It is certainly not a  
> description of some mechanism by which a client can select a  
> particular node for service.
>

Agree.

> this document, since it is too general.

I agree with Joe. Load-splitting is to me something completely  
different than PPLB.

> break TCP as pathological. This is an operational consideration.

Agreed.

- kurtis -

---

From: Dean Anderson <[email-masked]>
Date: Sun, 6 Nov 2005 21:26:58 -0500 (EST)

Hi Joe, 

BTW, forgot to say thanks for adding the part about load balancing.

inline

> which result in greater dispersion of segments cause the pipeline to  

It is not severe in either case.  Indeed, so far unicast TCP is concerned,
diverse links are also parallel.  It is only with Anycast that the parallelism
is split.  The only difference is in the number of IP-visible hops. However even
seemingly "one-hop parallel" IP links may be run over frame relay, atm, or mpls
links that are of unequal latency.  The TCP issues are the same in either case.

> description. Perhaps we differ in our interpretation of the word  

This seems vague to me.  The operational experience you cite is over parallel
links, and in that case you assert the TCP performance issue is minor, correct?

Are you asserting that diverse links suffer more severe TCP performance issues
than do parallel links?  My experience is different.  Both are minor.  Neither
case suffers non-operation, nor anything close to non-operation.  I define
"severe" as user visible delays of several seconds or more.  Load-splitting
(parallel or diverse) results in 3% unnecessary TCP retransmissions, without any
user visible delays.  

My experience is that TCP makes up about 60% of network traffic in many
networks, and this is declining. Networks that cater to VOIP or streaming audio
and video have even less TCP traffic as a percent of total. But, at 60% of
total, that means the unnecessary retransmissions amount to 2% of the total
traffic, which is well within the headroom of most networks, and so those
networks may potentially be suitable for diverse load splitting.  Plainly, the
networks that cater to VOIP and streaming media are unquestionably well-served
by benefits of diverse, fine-grained load splitting.  Many streaming protocols
can operate tolerably with a 30% or 50% packet loss, which is far preferable to
complete non-operation in the case a link fails.  Fine-grained load splitting
drastically reduces the damage of a loss of one link. Even TCP can continue to
operate with a 30 or 50% packet loss. And plainly, the greater the diversity,
the lower the packet loss during outage.

Consider the next time E911 calls fail, that you could have gotten choppy but
intelligible voice traffic through if only you had used fine-grained
load-splitting over diverse links.  Indeed, I suspect that there is a good
chance that such architectural features may one day be required by law or
regulation.

> anycast service distribution, not per-packet load balancing.

If you decide in the document to dismiss fine grained load splitting as
"pathological", the consquences of this decision should be discussed.

> Indeed, the section on PPLB specifically does not refer to BGP.

Perhaps I misread something. I read the PPLB discussion in section 
4.4.3 "Equal Cost Paths", discussing specifically BGP equal cost paths. Its not 
terribly clear when you start discsussing PPLB that the problem is more general 
than one of BGP exit policy.

> it needs to be spelt out.

There is no reason to think that PPLB is a pathology.  My criticism in this area
is meant to spur a justification of this view, not mere restatement of the same
canard.

> > Fifth, the references to an internal company specification, (ISC- TN-2004-1)
> > should be removed.
> 

OK, I just don't see many references to cisco manuals and such in RFCs. I
thought there might be some policy against this. But I'm certainly not the RFC
editor.

> > RFC2991 includes a list of methods to obtain course-grained load-balancing.  
> > These methods are well understood by router vendors, and rather extraneous
> > to the Anycast Extension with respect to fine-grained load-splitting.
> 

RFC2991 offers 3 methods of flow-based load balancing. Its other assertions
about the harms of per-packet load balancing have been shown to be wrong. They
listed TCP performance and Path MTU discovery is primary reasons for the harm.
Their views on both subjects has been discredited.  They miscalculated the TCP 
performance issue, and the PMTUD issue was also a red-herring. PMTU discovery 
works correctly with PPLB.

> specific examples of ambiguous text I would be happy to review them.

I gave an example. Your above description is vague and incorrect, but you
plainly do not see it. Paths are not selected. Paths are never selected.  Only
next hops are selected.  Even diagnotics which show paths are not reliable due 
to filtering, and probably other reasons.

> this document, since it is too general.

This I don't understand. "load-splitting" and "PPLB" seem to be synonymous, and 
"load splitting" is the term used by RFC1812.

> break TCP as pathological. This is an operational consideration.

Well, PPLB doesn't break TCP. So it must not be pathological.  This is the crux
of my objection to this language.  I suggest that you describe diverse load
splitting as being "incompatible" with the Anycast Extension, rather than
using the term "pathological".  Plainly, you can't use both at the same time.

> > [*] Not only is there no "node selection", it is not even possible to know
> > with certainty what remote anycast node a packet sent by a source node will

The term "node selection" is itself vague and incorrect with respect to packet
routing.  So, everywhere you use the term, it is vague and incorrect.  This term
is not found in any other IETF document that I could find involving routing. The
term "node selection" is used in XML and XSL, and does not apply cleanly to IP
packet routing.  I don't think the document makes the distinction anywhere
between the intuitive meaning of "node selection" as used by XML.  But rather
than redefining this term with a new meaning, it would be clearer if you stuck
with the accepted terms previously used in other IETF routing documents, and
couched your explanation in the previously accepted terms that are familiar from
other IP routing standards.

---

From: Dean Anderson <[email-masked]>
Date: Sun, 6 Nov 2005 21:49:05 -0500 (EST)

> I agree with Joe. Load-splitting is to me something completely  

Are you sure?  Just to make sure we are on the same page, PPLB means "Per Packet
Load Balancing"

RFC1812 describes "load splitting". in section 5.2.4.5 Load Splitting.

This cisco explanation of PPLB
http://www.cisco.com/univercd/cc/td/doc/product/software/ios120/120newft/120limit/120s/120s21/pplb.htm 
seems to match right up with RFC1812 load splitting.

I think they are synonymous.  However, if you think they are different,
especially if you think they are _completely_ different, then some explanation
of the meaning of these terms and their differences should also be in the draft.
And doubly so if you really think the differences are important.

Do you assert that load-splitting is compatible with the anycast extension in
some way that PPLB is not? 

If they are completely different, perhaps there should be a section on the 
compability of load-splitting as well.

But I'm rather doubtful they are different.

		--Dean

---

From: Joe Abley <[email-masked]>
Date: Mon, 7 Nov 2005 09:51:47 -0500

On 6-Nov-2005, at 21:49, Dean Anderson wrote:

> http://www.cisco.com/univercd/cc/td/doc/product/software/ 
> ios120/120newft/120limit/120s/120s21/pplb.htm
> I think they are synonymous.

No; per-packet load balancing is one load-splitting mechanism; there  
are others.

Mechanisms which group packets associated with a single flow such  
that a consistent next-hop is chosen for each of them, for example,  
avoid the issue that you are concerned about (that of different  
packets in a single client request being delivered to different  
anycast nodes).

Joe

---

From: Joe Abley <[email-masked]>
Date: Mon, 7 Nov 2005 10:21:04 -0500

> BTW, forgot to say thanks for adding the part about load balancing.

Thanks for bringing it up.

> links that are of unequal latency.  The TCP issues are the same in  

Let's follow this diversion in a little more detail. Precise  
modelling of TCP over a complete, general set of network conditions  
is a horrendous problem (at least for me) and I don't pretend to be  
an experienced researcher in this area. However, I think there are  
some coarse, high-level observations we can probably agree on which  
will help here.

In the case of multiple, similarly-loaded links between a pair of  
routers, the paths between source and unicast destination will be  
markedly similar in terms of the metrics which have an impact on TCP  
performance (e.g. propagation latency, jitter, segment re-ordering).

In the case of links which connect to different routers which are not  
topologically close the paths between source and unicast destination  
is much more likely to be different over each path.

TCP functions most effectively when the variation in network  
conditions between source and destination is of relatively low  
frequency; this allows TCP to adapt its sending rate such that the  
pipeline remains reasonably full.

As network conditions change (e.g. a congestion point appears, and  
packets become delayed or dropped) TCP adapts its sending behaviour  
to maximise throughput. This change cannot happen instantaneously,  
since all the signalling and inference happens in-band. Effective  
transmission for TCP relies on the fact that network conditions tend  
not to change abruptly, and hence incremental modifications in  
sending behaviour can keep a TCP session tuned without the window  
draining and everything restarting from scratch.

A topology which involves successive segments following paths which  
are markedly different represents the pathological case of abrupt  
network change; not only is the change abrupt, but it happens with  
near peak frequency.

> links, and in that case you assert the TCP performance issue is  

I think there is widespread support for this (I think I see you  
agreeing with it, in fact :-)

> retransmissions, without any

It certainly seems possible to do PPLB across paths from a single  
router which land in different upstream ISPs, and see minimal harm as  
you describe above. I am not accusing you of making this stuff up.  
However, it is my experience (and that of others here) that this is  
the exception rather than the rule.

> 4.4.3 "Equal Cost Paths", discussing specifically BGP equal cost  
> paths.

It mentions BGP with the specific intention of mentioning that equal- 
cost, multi-path topologies are possible with BGP. It then goes on to  
mention other routing protocols in which opportunities to acquire  
ECMP routes are far more common (in every BGP implementation I've  
seen, there is no multi-path capability by default: you have to  
explicitly turn it on).

> Its not

It seems clear to me. However, please feel free to suggest text to  
make that more clear.

> plainly do not see it.

I think you are being over-literal, and judging from the response (or  
non-response) of others I think your objections to this turn of  
phrase are not widely shared.

Note that the process we are participating in is one of rough  
consensus and not universal agreement. This is not the ITU :-)

Joe

---

From: Joe Abley <[email-masked]>
Date: Mon, 7 Nov 2005 09:51:47 -0500

Joe

---

From: Joe Abley <[email-masked]>
Date: Mon, 7 Nov 2005 10:21:04 -0500

Joe