Subject: RE: on documenting ECMP (was on the mpls oam framework)

From: "Busschbach, Peter B (Peter)" <[email-masked]>
Date: Tue, 18 Nov 2003 17:31:53 -0500

> -----Original Message-----
> From: Naidu, Venkata [mailto:[email-masked]]
> Sent: Tuesday, November 18, 2003 4:14 PM
> To: [email-masked]'; Busschbach, Peter B (Peter)
> Cc: 'David Allan'; [email-masked]'; [email-masked]
> Subject: RE: on documenting ECMP (was on the mpls oam framework) 
> 
> 
> Curtis,
> 
> -> Without hierarchical LSPs, RSVP/TE does not have multipath 
> unless 1)
> -> more than one LSP is configured to the same destination with equal
> -> cost and MP is enabled, or 2) more than one LSP is on a path to
> -> different egress and the total cost is the same, and this 
> form of MP
> -> is both supported and enabled.  This form of MP is easy to 
> deal with
> -> from an OAM standpoint because the only branch is at ingress.
> 
>   Very delightful to read such a good technical explanation. But...
>   what ever you said above that current MPLS signaling can't support
>   ECMP at non-ingress branch point is a limitation. From MPLS-TE
>   point of view, all the necessary information to split the traffic
>   at non-ingress can be calculated easily. Unfortunately, signaling
>   doesn't support that.
> 
>   If given a chance, such an ECMP split can be computed by ingress
>   CSPF by finding all augmented "equal min-cost max-flow" paths to 
>   the destination. Chosing any of least/farthest such common ancestor 
>   split can be made as part of singaling decision.
> 
> -> QoS based on EXP can be enabled and ECMP can also be 
> -> enabled.  If each
> -> branch of the path honors the EXP bits, QoS still works 
> and exists in
> -> the presense of ECMP, over LDP or RSVP/TE.  There is very clear
> -> existance proof of this.
> 
>   I view ECMP as a TE mechanism than a QoS workhorse. TE and QoS are
>   infact orthogonal.

String-theorist must be right: we live in a 10-dimensional world, because  everything seems to be orthogonal to everything else :-)

I would agree that TE is neither necessary nor sufficient for QoS, but to call them orthogonal is a little extreme.

It may be helpful if I reword my original point, which was:

1) ECMP leads to non-deterministic behavior. We should develop OAM mechanisms that accept that as a given

2) Nevertheless, for certain types of traffic it might be possible to use tools from the connection-oriented world. E.g. if a Service Provider uses RSVP-TE to reserve bandwidth between two points, it will result in a path without intermediate splits. 

That last statement was my assumption. Curtis argued that there are exceptions, such as the case of hierarchical hops where a logical link consists of multiple physical links. You argue that path calculations can theoretically deal with ECMP splits. 

I stand corrected. I do wonder how routers will distribute traffic over the multiple paths with bandwidth guarantees. As far as I know, current hashing algorithms leave the packet sequence of micro flows intact, but there is nothing that prevents them from sending 90% of the traffic over one path and 10% over another. Or is there?

Peter

> Venkata.

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 18 Nov 2003 20:30:08 -0500

In message <[email-masked]
om>, "Busschbach, Peter B (Peter)" writes:
[snip]
>  that accept that as a given

Agreed.

> 2) Nevertheless, for certain types of traffic it might be possible to use too
> ls from the connection-oriented world. E.g. if a Service Provider uses RSVP-T
> E to reserve bandwidth between two points, it will result in a path without i
> ntermediate splits. 
> 
> That last statement was my assumption. Curtis argued that there are exception
> s, such as the case of hierarchical hops where a logical link consists of mul
> tiple physical links. You argue that path calculations can theoretically deal
>  with ECMP splits. 

But the ISP will know that these exist unless the links are going
across another providers infrastructure in which case the provider
should not see any adverse affects (but could infer by reordering of
packet from different microflows that ECMP was used).

> I stand corrected. I do wonder how routers will distribute traffic over the m
> ultiple paths with bandwidth guarantees. As far as I know, current hashing al
> gorithms leave the packet sequence of micro flows intact, but there is nothin
> g that prevents them from sending 90% of the traffic over one path and 10% ov
> er another. Or is there?
> 
> Peter

There is no guarentee but even when the Internet was made up of T1
lines there was enough diversity in the flows to keep the balance
reasonably good.  Back then and in the T3-NSFNET days the NSF funded
supercomputer centers could open a TCP connection that would seriously
bias the split to one branch.  Single wide area host to host TCP flows
much over OC12c have not even been demonstrated.  The fastest wide
area host flows remain well under 100 mb/s even though 1 Gb/s on the
LAN is not so hard to do.  At 100 Mb/s over 10 msec RTT (5 msec each
way) you need a 125 MB TCP send and receive buffer and machines
configured to allow that much buffering per TCP flow are quite rare
(again, most likely "big science" still at it).  Any packet loss
whatsoever would greatly reduce the throughput.  Typical wide area
flows would be well 1 mb/s.  Lots of these on a 10 Gb/s link tend to
spread out quite evenly.  They even spread out nicely over OC3c.  When
you get down below DS3, load split can be quite uneven.

Back in the late 1990s there was speculation that single host to host
microflows might reach OC12c or even OC48c speeds due to traffic
between IPsec gateways that appeared as a single flow.  That hasn't
happenned at all.

So the answer is - in theory it could be a problem.  In practice, for
IP it never is.  For PW that is not expected to be the case so very
large PW LSPs may have to be handled by ISPs with some caution.

If QoS is EXP based, and preferred traffic remains very small, whether
you have 90% of a very small number or 50% of a very small number on
one side doesn't matter.  What does matter is how much of the total
traffic ends up on either leg and there it is more likely that the
split will be even, and the consequence of it not being even is
limited.  If the non-QoS traffic is all IP then in practice split will
be very even.

btw- probability dictates that the gases in a room will be more or
less evenly distributed, but it not entirely deterministic so the
distribution is never perfect, and there is no guarentee that all of
the oxygen won't migrate to one side of the room such that people on
the other side will suffocate.  I think you get my point.  :-)

Curtis