Subject: RE: on documenting ECMP (was on the mpls oam framework)

From: "Busschbach, Peter B (Peter)" <[email-masked]>
Date: Tue, 18 Nov 2003 14:35:09 -0500

Curtis,

Something must have been lost in the translation, because you seem to interpret most of my statements opposite to what I intended. I like ECMP. My email was intended as a request to accept ECMP as a given and accept that it leads to non-deterministic behavior in the network.

For fear of further slipping away from the compromise that you referred to, I will leave most of your misinterpretations unanswered, but I do have a question about one of your statements. Please see in-line.

Peter

> -----Original Message-----
> From: Curtis Villamizar [mailto:[email-masked]]
> Sent: Tuesday, November 18, 2003 2:03 PM
> To: Busschbach, Peter B (Peter)
> Cc: 'David Allan'; [email-masked]'; [email-masked]';
> [email-masked]
> Subject: Re: on documenting ECMP (was on the mpls oam framework) 
> 
> 
> 
> In message 
> <[email-masked]
> om>, "Busschbach, Peter B (Peter)" writes:
> > Dave, et.al.
> > 
> > In one of the side-branches of this discussion, the name 
> Dijkstra came up a c
> > ouple of times. Apart from inventing the famous algorithm, 
> Dijkstra did impor
> > tant work in the area of correctness proofs and concurrent 
> processes. One of 
> > his insights was that non-determinism is a powerful concept 
> and that a progra
> > m should be written in such a way that its correctness can 
> be proven, even in
> >  the face of non-determinism.
> > 
> > We have to be careful translating this to OAM (after all, 
> you don't want your
> >  service provider to blame non-determinism when your 
> connections fail), but l
> > et's give it a try.
> 
> Nice thought followed by a cheap shot, but fine.  I understand that
> you don't like ECMP and no one is asking you to like it, simply to
> accept that it is part of the architecture and it will be covered by
> whatever OAM emerges from the IETF.
> 
> > I think that ECMP is an issue only for a subset of traffic. 
> ECMP is not an is
> > sue for PW-traffic, since VCCV provides a solution. ECMP is 
> not an issue for 
> > services with QoS guarantees, where RSVP-TE is used to set 
> up the connection,
> >  because an explicit path is nailed down. 
> 
> QoS and explicit path are orthogonal.  Some provider may feel that
> they must combine the two in their service but it is not generally the
> case.  QoS and ECMP are also orthogonal.  Lets not try to sneak in
> assertions that may be true in your case but are not generally true.

I don't understand why RSVP-TE and ECMP are orthogonal. If one uses RSVP-TE to reserve bandwidth between two points, doesn't that nail down an exact end-to-end path? 

> 
> > The question in my mind is: for the traffic that is subject 
> to ECMP, is it re
> > ally crucial to know exactly how traffic is routed? 
> Shouldn't we just embrace
> >  non-determinism for the advantages that it provides and 
> find ways to work ar
> > ound it?
> 
> We did that in MPLS Ping.  The ingress can determine from the midpoint
> (essentially by the midpoint telling it) where ECMP branches exists
> and how to exercise each branch.  This method is deterministic and
> will exercise all paths.  The alternate is to randomly spray across
> the 127/8 space, which is non-deterministic but arguably will also
> exercise all paths.  The latter method can detect loss but not measure
> the loss experienced on any given path (loss may be concentrated in a
> subset of the traffic).
> 
> If other techniques want to do something similar to determine how to
> exercise ECMP paths, then they become applicable where ECMP is used.
> If they don't they are simply not applicable when ECMP is used.  One
> provider may choose to disable ECMP and use a technique that works
> only with ECMP disabled.  Another provider may choose to keep ECMP
> enabled and use a different OAM technique that works with ECMP.
> 
> > For example, when I surf the web, I don't need an ISP to be 
> able to send OAM 
> > packets along the exact same path that I used five minutes 
> ago. For such appl
> > ications, end-to-end OAM would not be relevant, but local 
> loopback tests migh
> > t be much more useful.
> 
> I'm quite confident that the PC in my home has not sent any OAM
> packets to the IETF mail servers to verify the path to the mail
> server.  But quite honestly this has nothing to do with the discussion
> of the OAM framework.  If I want to test to see if my ISP is live or
> the path to the IETF mail server is OK, I just use IP ping and
> traceroute.  Loopback tests are accomplished for your example.
> 
> > Peter 
> 
> We had some compromise and I fear that we are now slipping back to
> arguing about whether ECMP is part of the architecture.  I hope that
> is not the case.
> 
> Curtis
>

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 18 Nov 2003 15:31:37 -0500

In message <[email-masked]
om>, "Busschbach, Peter B (Peter)" writes:
> Curtis,
> 
> Something must have been lost in the translation, because you seem to interpr
> et most of my statements opposite to what I intended. I like ECMP. My email w
> as intended as a request to accept ECMP as a given and accept that it leads t
> o non-deterministic behavior in the network.
> 
> For fear of further slipping away from the compromise that you referred to, I
>  will leave most of your misinterpretations unanswered, but I do have a quest
> ion about one of your statements. Please see in-line.
> 
> Peter

Sorry.  I did very much misread your response.  Hopefully I'll do
better with the clarification below.

> I don't understand why RSVP-TE and ECMP are orthogonal. If one uses RSVP-TE t
> o reserve bandwidth between two points, doesn't that nail down an exact end-t
> o-end path? 

First of all:  RSVP/TE != explicit-path

They are related but not equal.  RSVP/TE paths can be determined by
the ingress based on the CSPF computation.  Explicit-path means the
path is predetermined and fixed.  RSVP/TE supports explicit-path as an
option.  When determined by the CSPF (explicit-path is not being
used), the path is in a sense nailed down, at least until the
adaptivity timer fires and a later CSPF decides there is a better path
now.

Also reserving bandwidth != QoS.  In most applications this is an
accounting action only - no queueing is allocated according to the
amount of reservation unless explicitly supported and configured to do
so and most routers don't even offer the ability to set quueuing
according to the bandwidth reservation as an option.  While I
personally think that setting WFQ weights according to the bandwidth
reservation is a good idea (and it appears you do too), ISPs don't
seem very interested in doing this.

There is also no reason why QoS can't exist with LDP and in fact it
does.  The simplest QoS is based on the EXP bits only.  Queueing is
set to favor some EXP values over others.  The amount of traffic with
the preferred EXP values must be limited.  In most cases it is limited
by market acceptance (very few willing to pay extra for higher QoS).

Another issue is whether a MPLS path is a single path.  In LDP,
clearly it need not be - if an ECMP situation exists anywhere along
the path traffic will split if ECMP is enabled.  With RSVP/TE if one
of the hops is a hierarchical hop, then although it is single logical
hop, there may be two or more LSPs configured between the pair of
nodes that are used to split the traffic.  Even if it were an
explicit-path (configured by operator or NMS) if a hop was a
hierarchical hop and ECMP was configured, the traffic would split (if
traffic is such that microflows could be identified).

In some routers a very large logical link between a pair of nodes may
be a concatonated set of lower bandwidth links.  For example, eight
OC48c can form a 20 GB logical link or eight OC192c can form a 80 GB
link (this is not a hypothetical btw).  Since this is a logical link,
it can either be assumed that the pair of nodes can handle monitoring
the link (which has proven safe, but..) or this can be treated as a
form of hierarchy and as an ECMP case to explicitly test all insegment
mappings from a tool such as MPLS Ping.

Without hierarchical LSPs, RSVP/TE does not have multipath unless 1)
more than one LSP is configured to the same destination with equal
cost and MP is enabled, or 2) more than one LSP is on a path to
different egress and the total cost is the same, and this form of MP
is both supported and enabled.  This form of MP is easy to deal with
from an OAM standpoint because the only branch is at ingress.

QoS based on EXP can be enabled and ECMP can also be enabled.  If each
branch of the path honors the EXP bits, QoS still works and exists in
the presense of ECMP, over LDP or RSVP/TE.  There is very clear
existance proof of this.

Curtis

---

From: David Charlap <[email-masked]>
Date: Tue, 18 Nov 2003 16:35:49 -0500

> adaptivity timer fires and a later CSPF decides there is a better path

RSVP-TE's definition of "explicit" can be rather fluid, depending on the 
nature of the ERO object and how it is generated.

- An LSP may be signaled without an ERO object.  In which case, it 
follows the IGP's route to the egress address.  The route table may be 
populated by the result of CSPF, ECMP, or other mechanisms.  When the 
route table changes, the LSP reroutes to follow it.  This is obviously 
not in any way explicit.

- The path described by an ERO may terminate before the egress node is 
reached, in which case the remaining hops are determined using the route 
table and the egress address.  This is only explicit up to the end of 
the ERO.

- An ERO object may contain loose hops.  The next-hop chosen in response 
to a loose hop is a function of the routing table.  This is explicit in 
the sense that the LSP will pass through certain nodes, but the route 
taken between those nodes is not explicit.

- Since RSVP ERO hop addresses identify nodes, not links (except when 
the GMPLS IF_ID/unnumbered HOP extensions are used), the local route 
table is used to pick a link when multiple links exist between a pair of 
nodes along the path.

- Finally, the ERO used by the ingress node may be manually generated by 
an operator, or it may be computed by software external to RSVP.  If it 
is computed, then the ERO-computation software may choose to generate a 
new ERO and reroute the LSP as the network topology changes.  This is 
explicit from RSVP's perspective, but not necessarily from the 
operator's perspective.

RSVP's path is only fixed in the intuitive sense if an ERO consisting 
entirely of strict subobjects is used, and these subobjects lead all the 
way to the egress node, and the entity that computes the ERO (human or 
software) chooses not to reroute the LSP as the network changes.

-- David

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 18 Nov 2003 21:01:12 -0500

> software) chooses not to reroute the LSP as the network changes.

I was using the RFC2702 definition of explicit-path.

  5.6.1 Administratively Specified Explicit Paths

   An administratively specified explicit path for a traffic trunk is
   one which is configured through operator action. An administratively
   specified path can be completely specified or partially specified. A
   path is completely specified if all of the required hops between the
   endpoints are indicated. A path is partially specified if only a
   subset of intermediate hops are indicated. In this case, the
   underlying protocols are required to complete the path.  [... etc]

Note that the term explicit path is used *only* for this case at least
in RFC2702 and AFAIK is not redefined differently in any other RFC.

My statement still stands: RSVP/TE != explicit-path.  And the rest of
it remains true.  I neglected to mention that an explicit-path may
contain loose hops.

Your statement is also entirely correct.  I think we are going off
into detail on a tangent.  Just wanted to clarify where the definition
comes from and that it really does mean "configured" (which can
include SNMP - to fend off another possible tangient).

Curtis

btw - in case anyone thinks ECMP is some new requirement (also from
RFC2702, dated Sep 1999):

  5.6.5 Load Distribution Across Parallel Traffic Trunks

   Load distribution across multiple parallel traffic trunks between two
   nodes is an important consideration.  In many practical contexts, the
   aggregate traffic between two nodes may be such that no single link
   (hence no single path) can carry the load. However, the aggregate
   flow might be less than the maximum permissible flow across a "min-
   cut" that partitions the two nodes. In this case, the only feasible
   solution is to appropriately divide the aggregate traffic into sub-
   streams and route the sub-streams through multiple paths between the
   two nodes.

---

From: David Charlap <[email-masked]>
Date: Wed, 19 Nov 2003 11:04:02 -0500

Curtis Villamizar wrote:
> David Charlap writes:
>> Curtis Villamizar wrote:
>>> 
>>> First of all:  RSVP/TE != explicit-path ...
>> 
>> RSVP-TE's definition of "explicit" can be rather fluid, depending
>> on the nature of the ERO object and how it is generated. ...
> 
> I was using the RFC2702 definition of explicit-path. ...
> 

I think you misunderstood my post.  I was posting more detailed
information about RSVP in support of your statement, not as a
counterexample.

RSVP-TE can only be considered equivalent with explicit-path for a small 
subset of the various mechanisms that RSVP implementations may generate 
and use the EXPLICIT_ROUTE object - for other usages, they are not even 
close to equivalent.

> into detail on a tangent.  Just wanted to clarify where the
> definition comes from and that it really does mean "configured"
> (which can include SNMP - to fend off another possible tangient).

-- David