Subject: RE: on documenting ECMP (was on the mpls oam framework)

From: "Busschbach, Peter B (Peter)" <[email-masked]>
Date: Tue, 18 Nov 2003 21:24:42 -0500

Thanks, Curtis. You make a convincing argument.

One last question:

What happens with RSVP-TE messages at an ECMP split? Suppose a source node sends a Path message requesting reservation of 10 Mb/s of bandwidth. When it reaches the ECMP split, will that router generate two Path messages, each requesting 5 Mb/s, and forward them along the two equal-cost paths to the destination?

Peter

> -----Original Message-----
> From: Curtis Villamizar [mailto:[email-masked]]
> Sent: Tuesday, November 18, 2003 8:30 PM
> To: Busschbach, Peter B (Peter)
> Cc: 'Naidu, Venkata'; [email-masked]'; 'David Allan';
> [email-masked]'; [email-masked]
> Subject: Re: on documenting ECMP (was on the mpls oam framework) 
> 
> 
> 
> In message 
> <[email-masked]
> om>, "Busschbach, Peter B (Peter)" writes:
> > 
> [snip]
> > 
> > I would agree that TE is neither necessary nor sufficient 
> for QoS, but to cal
> > l them orthogonal is a little extreme.
> > 
> > It may be helpful if I reword my original point, which was:
> > 
> > 1) ECMP leads to non-deterministic behavior. We should 
> develop OAM mechanisms
> >  that accept that as a given
> 
> Agreed.
> 
> > 2) Nevertheless, for certain types of traffic it might be 
> possible to use too
> > ls from the connection-oriented world. E.g. if a Service 
> Provider uses RSVP-T
> > E to reserve bandwidth between two points, it will result 
> in a path without i
> > ntermediate splits. 
> > 
> > That last statement was my assumption. Curtis argued that 
> there are exception
> > s, such as the case of hierarchical hops where a logical 
> link consists of mul
> > tiple physical links. You argue that path calculations can 
> theoretically deal
> >  with ECMP splits. 
> 
> But the ISP will know that these exist unless the links are going
> across another providers infrastructure in which case the provider
> should not see any adverse affects (but could infer by reordering of
> packet from different microflows that ECMP was used).
> 
> > I stand corrected. I do wonder how routers will distribute 
> traffic over the m
> > ultiple paths with bandwidth guarantees. As far as I know, 
> current hashing al
> > gorithms leave the packet sequence of micro flows intact, 
> but there is nothin
> > g that prevents them from sending 90% of the traffic over 
> one path and 10% ov
> > er another. Or is there?
> > 
> > Peter
> 
> There is no guarentee but even when the Internet was made up of T1
> lines there was enough diversity in the flows to keep the balance
> reasonably good.  Back then and in the T3-NSFNET days the NSF funded
> supercomputer centers could open a TCP connection that would seriously
> bias the split to one branch.  Single wide area host to host TCP flows
> much over OC12c have not even been demonstrated.  The fastest wide
> area host flows remain well under 100 mb/s even though 1 Gb/s on the
> LAN is not so hard to do.  At 100 Mb/s over 10 msec RTT (5 msec each
> way) you need a 125 MB TCP send and receive buffer and machines
> configured to allow that much buffering per TCP flow are quite rare
> (again, most likely "big science" still at it).  Any packet loss
> whatsoever would greatly reduce the throughput.  Typical wide area
> flows would be well 1 mb/s.  Lots of these on a 10 Gb/s link tend to
> spread out quite evenly.  They even spread out nicely over OC3c.  When
> you get down below DS3, load split can be quite uneven.
> 
> Back in the late 1990s there was speculation that single host to host
> microflows might reach OC12c or even OC48c speeds due to traffic
> between IPsec gateways that appeared as a single flow.  That hasn't
> happenned at all.
> 
> So the answer is - in theory it could be a problem.  In practice, for
> IP it never is.  For PW that is not expected to be the case so very
> large PW LSPs may have to be handled by ISPs with some caution.
> 
> If QoS is EXP based, and preferred traffic remains very small, whether
> you have 90% of a very small number or 50% of a very small number on
> one side doesn't matter.  What does matter is how much of the total
> traffic ends up on either leg and there it is more likely that the
> split will be even, and the consequence of it not being even is
> limited.  If the non-QoS traffic is all IP then in practice split will
> be very even.
> 
> btw- probability dictates that the gases in a room will be more or
> less evenly distributed, but it not entirely deterministic so the
> distribution is never perfect, and there is no guarentee that all of
> the oxygen won't migrate to one side of the room such that people on
> the other side will suffocate.  I think you get my point.  :-)
> 
> Curtis
>

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 18 Nov 2003 22:42:48 -0500

In message <[email-masked]
om>, "Busschbach, Peter B (Peter)" writes:
> What happens with RSVP-TE messages at an ECMP split? Suppose a source node se
> nds a Path message requesting reservation of 10 Mb/s of bandwidth. When it re
> aches the ECMP split, will that router generate two Path messages, each reque
> sting 5 Mb/s, and forward them along the two equal-cost paths to the destinat
> ion?
> 
> Peter

Peter,

This email thread has a life of its own.  Not a comment to you but
maybe I should be appologizing for the email bandwidht I've used up.

I did try to answer that in another response.  Briefly, at a
hierarchical link you have a logical link so if you ask for 10 mb/s it
has to find 10 mb/s.  If its a bundle it finds 10 mb/s on a single
component.  If it is load spliting, it may split two ways in effect
taking 5 mb/s from each side.  Obviously it has to have 5 mb/s
available on each component LSP of the hierarchical LSP to do so.

Keep in mind that how a LSR decides how much bandwidth to allocate to
a hierarchical LSP and when to change this allocation is not defined.
[Lots of room for innovation.  The ITU guys are gonna hate it.  And
now that its been brought up we may end up spending a lot of time
explaining why it will interoperate. :) ]

Curtis