Subject: Load balancing draft

From: Shahram Davari <[email-masked]>
Date: Tue, 19 Nov 2002 07:36:35 -0800

Hi,

One of the criticism for this draft was that it is designed to overcome
Y.1711 limitation, in which a reserved label is used. As I mentioned
in the meeting the MPLS-Ping also requires usage of an extra reserved
label for non-IP carrying LSPs:

  "To test an LSP that carries non-IP traffic, before injecting ICMP and
   MPLS ping messages into the LSP, the IPv4 Explicit NULL label should
   be prepended to such messages."

So this draft is equally applicable to MPLS-ping. 

Yours,
Shahram

---

From: Shahram Davari <[email-masked]>
Date: Tue, 19 Nov 2002 08:12:10 -0800

Eric,

Understood.

-Shahram

> -----Original Message-----
> From: Gray, Eric [mailto:[email-masked]]
> Sent: Tuesday, November 19, 2002 11:09 AM
> To: Shahram Davari; [email-masked]'
> Subject: RE: Load balancing draft
> 
> 
> Shahram,
> 
> 	I don't think the issue is whether or not people feel
> that this work is applicable.  I believe it might be better
> to characterize the issue as whether or not it is practical
> (or even feasible) to do this given stated constraints.
> 
> 	As with QoS and other types of value add functionality,
> the 9 letter acronym TANSTAAFL applies.  There ain't no such
> thing as a free lunch - in this case or in any other.  Given
> the current situation with respect to existing and deployed
> hardware, about the only short term approach to ensure that
> probe packets follow the data path is likely to be disabling 
> load sharing.
> 
> 	In terms of specifying another approach, at least for
> the alternatives discussed, these are potentially areas of
> product differentiation and/or long term "fixes".  Hence it
> is probably not something we necessarily want to undertake
> at this point.
> 
> Eric W. Gray
> Systems Architect
> Celox Networks, Inc.
> [email-masked]
> 508 305 7214
> 
> 
> > -----Original Message-----
> > From: Shahram Davari [mailto:[email-masked]]
> > Sent: Tuesday, November 19, 2002 10:37 AM
> > To: [email-masked]'
> > Subject: Load balancing draft
> > 
> > Hi,
> >

---

From: "Gray, Eric" <[email-masked]>
Date: Tue, 19 Nov 2002 11:09:22 -0500

Shahram,

Eric W. Gray
Systems Architect
Celox Networks, Inc.
[email-masked]
508 305 7214

---

From: "David Allan" <[email-masked]>
Date: Tue, 19 Nov 2002 16:29:16 -0500

Tom:

> >The concept of avoiding the reserved labels in an MPLS hash 
> as a "best 
> >way" makes some sense,
> 
> 
>          Does this mean that we then have to mandate how IP 
> (L2TPV3) does
> traffic distribution too?  What is next? 
   
L2TPv3 from what I can tell does not have a mechanism by which I could
distinguish any type of PE-PE probe of a PW so the example is moot. I can
only use payload probes (I.610, ICMP, whatever).

I think that this 
> solution looks
> good in theory, but in practice there are huge problems with it.

Only problem I can think of is that one cannot simply hash an arbitrary
length of the MPLS frame. The implementation is a bit more complex, that's
the price of testabilty.

> 
> >though it would probably only be gradually available as 
> hardware changes.
> 
>          Kireeti or you made a good point that
> this solution is not evolutionary; it is revolutionary as it 
> will require
> hardware changes.  

Many things eventually migrate to hardware. Some formerly hardware functions
migrate to microcode. This years core LSR is next years edge box. Are we
saying that MPLS is frozen in time based upon last year or the year before's
implementation? If so, lets skip draft and simply promote everything to STD
and go home.

On the surface it might sound reasonable to
> say that well, hardware will evolve and as it does it must support
> this sort of load-balancing. However, the fact of the matter is that
> one size does not fit all in this case, and thus this will be a burden
> on both existing and future implementations.

At the moment you're stipulating that one size fits all. That's what the
draft was trying to avoid.

rgds
Dave
<snipped to the end>

---

From: [email-masked]
Date: Tue, 19 Nov 2002 23:08:04 -0000

David Allan wrote 19 November 2002 21:29
> only use payload probes (I.610, ICMP, whatever).
NH=> Agreed Dave but this misses a key point....IP does not have the same
potential failure modes as LDP mp2p.  IP is architecturally a cnls network
so it *multiplexes* pkts, it does not *merge* pkts......indeed, merging in a
co pkt-sw network forwarding paradigm is a very weird thing to do and it
demands some level of 'demerging' above it.  This is a fundamentally
important observation.
> length of the MPLS frame. The implementation is a bit more 
NH=> It's one of the hidden charges in the search for a free-lunch.
> implementation? If so, lets skip draft and simply promote 
NH=> Totally agreed.
> draft was trying to avoid.
NH=> Agreed....speaking as an operator and looking ahead, we want choice
here.  Bit spooky but we were discussing these very issues today in a CTO
mtg.

---

From: "David Allan" <[email-masked]>
Date: Wed, 20 Nov 2002 01:23:00 -0500

Hi Tom:

<snipped>
> 
>          Not necessarily. I think you can test/trace the 
> paths using LSP ping,
> but of course, there is a price to pay for that too; there is 
> no free lunch.
> The advantage of the latter is that it works with all of the hardware
> deployed today.

Agreed on the free lunch piece, in this case it is that LSP PING needs to
run cycling the destination address until a message is missed. I'd assume
this is data plane return so I am loading both ends of the LSP . Mind you
with the one way option I guess availability can be measured at the far end
(and skip the "is the return path broken?" step) although unless I am
sending multiple copies of each address tuple, I still have simple
congestive loss to consider, or port rate limiting at the egress. In other
words although the ingress is cycling the address conbinations, I cannot
really infer anything from missing one message at the egress, and if I have
to send each address tuple multiple times as matter of course, testing all
useful combinations is going to be very slow. 

So lets call one way mode a non-starter in this case and stick with real
ping.

So I remember the address tuple used on the suspect non-return case, then I
try it a few more times to ensure it is not congestive loss or port rate
limiting, then I try it with various return options to make sure it is not a
break in the return path, (which loads the control plane of all intervening
LSRs) at which point I then believe I have a real problem and I try to
traceroute it. Of course I need to run much of this at a slower rate than
the network convergence time as otherwise simple non-converged network state
will give me spurious results.

I understand your argument w.r.t. a solution retrofittable to existing gear,
bit I find the sheer number of messages required to a) identify a potential
problem and then b) determine it is real followed by c) isolating it, a
rather depressing prospect. Esp. with a protocol that will not lend itself
to easy hardware assist. I'm echoing control plane information in TLVs.....
this is the essence of my concern with bounded detection times and I would
observe that we are painted into a bit of a corner.

If we can work towards fate sharing between the probes and the path or at
least provide that option for folks who care, then we obviate the need for
address cycling or multiple resends, can move to egress detection and I
elimintate 90% of the maintenance overhead required to detect a problem.

rgds
Dave

>          No, I am saying that there are always going to be 
> low-end platforms
> that have hardware limitations and thus might not be able to 
> use the load
> share algorithm you specify in this document.  What do we do 
> for those?
> I think we are just back in the situation that we are in 
> today: proprietary
> load balancing.  It works for IP and other technologies, why 
> do we need
> to mandate an algorithm here?
> 
>          No, I am saying that what we have today is many sizes
> fit all (in the case of load balancing) because that is a reality
> of routers/switches.
> 
>          --Tom
> 
> 
> >rgds
> >Dave
> ><snipped to the end>
> 
> Success is relative; the more success, the more relatives. -Anonymous
> 
> 
>

---

From: Shahram Davari <[email-masked]>
Date: Wed, 20 Nov 2002 06:44:14 -0800

This equally applies to MPLS-ping for non-IP carrying LSPs. Because 
for non-IP carrying LSPs there is no IP address to be cycled through (as is the case for IP-carrying LSPs) to exercise all the ICMP paths. And the
inclusion of Explicit Null label in the hashing could cause MPLS-ping to take a different path from data.

May be this could be captured in an Information draft.

-Shahram

> From: Vach Kompella [mailto:[email-masked]]
> Sent: Wednesday, November 20, 2002 9:33 AM
> To: Thomas D. Nadeau; Alia Atlas
> Cc: David Allan; [email-masked]'
> Yes, the "tunnel" labels don't change.  However, if an implementation
> "indiscriminately" uses all labels to hash on, then it would 
> use the VC label
> for data packets, and VC + OAM label for OAM packets, 
> yielding a not so nice
> result.
> 
> In that sense, it "would be nice" to suggest that a BCP 
> should be to ignore
> reserved labels.  While various implementations may not be 
> able to follow this
> guideline, it is still a reasonable suggestion, somewhat 
> along the lines of
> saying don't use certain fields in the IP header for hashing.
> 
> -Vach
> 
> > From: [email-masked] [mailto:[email-masked]]On Behalf 
> Of Thomas D.
> > Nadeau
> > Sent: Wednesday, November 20, 2002 6:15 AM
> > To: Alia Atlas
> > Cc: David Allan; [email-masked]'
> > At 10:40 PM 11/19/2002 -0500, Alia Atlas wrote:
> > >At 05:45 PM 11/19/2002 -0500, Thomas D. Nadeau wrote:
> paths using LSP
> > >> ping,
> > >>but of course, there is a price to pay for that too; 
> there is no free lunch.
> > >How does it work today with the pseudo-wire case?  From 
> the draft, it
> > >seems that the same PW label is used, but it is not the 
> bottom of stack,
> > >b/c there is an explicit NULL underneath.
> > >
> > >So, suddenly one must forward based not only on the MPLS 
> label but also
> > >the  bottom-of-stack bit...
> > >
> > >Tell me I'm missing something here, but if the above is 
> really what is
> > >intended in the draft, I don't understand how you can 
> claim that it is
> > >compatible with current MPLS hardware.
> >
> >          It is my understanding that tracing the PW 
> "tunnel" LSP is the
> > same as any
> > other MPLS LSP.
> >
> >          --Tom
> >
> >
> >
>

---

From: Shahram Davari <[email-masked]>
Date: Wed, 20 Nov 2002 08:44:31 -0800

>But most of the time you can
> be pretty sure with much lighter overhead.  

How?

-Shahram

---

From: "David Allan" <[email-masked]>
Date: Thu, 21 Nov 2002 13:20:59 -0500

I don't follow why specific knowledge of boxes is required. The general
effect is to englarge the scope of fate sharing between the probe stream and
associated traffic. This would be independent of specific knowledge. If box
'A' randomizes stuff between B, C and D, and the probe pathologically
follows B then C and D are not tested. If B randomizes stuff to boxes E F
and G, and probes sticks with the associated traffic then E F and G are
tested. 

The only truly pathological case I can think of is when all load sharing
points implement exactly the same algorithm and have exactly the same number
of candidate NHLFEs and are only probed from a single point, otherwise
incremental deployment of probe fate sharing pays dividends.

rgds
Dave

> From: Thomas D. Nadeau [mailto:[email-masked]]
> Sent: Thursday, November 21, 2002 11:49 AM
> To: Allan, David [CAR:NS00:EXCH]
> Cc: [email-masked]'
> >Not entirely true, any point where there is useful 
> predictability reduces 
> >the number of
> >possible faults that may only be caught by random process. 
> IMHO that is 
> >useful.
> 
>          In some cases it may help, but in others it does not which
> IMHO is the point.  It is only useful if you know which
> boxes support which algorithms, which gets us back to
> where we are today.
> 
>          --tom
> 
> 
> 
> 
> >cheers
> >Dave
> >
> > > From: Thomas D. Nadeau 
> > [<mailto:[email-masked]>mailto:[email-masked]]
> > > Sent: Thursday, November 21, 2002 10:59 AM
> > > To: Fedyk, Don [BL60:1A00:EXCH]
> > > Cc: George Swallow; Fedyk, Don [BL60:1A00:EXCH]; Allan, David
> > > [CAR:NS00:EXCH]; Alia Atlas; Shahram Davari; [email-masked]'
> > > Subject: Re: Load balancing draft
> > >
> > >
> > >
> > > >George
> > > >
> > > >Let me restate. You Said "All load balancing done in
> > > prescribed manner."
> > > >Which is certainly restrictive. The Draft for load balancing
> > > says  not to
> > > >use certain fields. That is not the same. So I added different
> > > >implementations for load balancing following the draft still
> > > have freedoms
> > > >and different vendors can choose different algorithms.
> > >
> > >          Not really. If you want things to work the way the
> > > draft aspires
> > > them to, then everyone in the MPLS domain must use the same,
> > > predictable load balancing algorithm.  Going halfway doesn't work,
> > > and is arguably no better than what we have today.
> > >
> > >          --Tom
> > >
> > >
> > > >Hope that makes it clear,
> > > >Don .
> > > >George Swallow wrote:
> > > >
> > > >>Don -
> > > >>
> > > >>This seems to a non-sequitor.  I don't see how this comment
> > > relates to
> > > >>what I wrote below.  Were you trying to answer some 
> other message?
> > > >>
> > > >>...George
> > > >>
> > > >>
> > > >>
> > > >>>George to be fair I heard Dave say  that the load
> > > balancing should not
> > > >>>use certain fields which are part of the label stack. This
> > > does not
> > > >>>require that balancing be done in a prescribed manner,
> > > freedom is still
> > > >>>there. Instead load balancing should be correct with
> > > respect to all of
> > > >>>MPLS formats  including reserved labels.
> > > >>>
> > > >>>Don
> > > >>>
> > > >>>George Swallow wrote:
> > > >>>
> > > >>>
> > > >>>
> > > >>>>>If you look at my post from rather late last night, I'd
> > > question the
> > > >>>>>lighter
> > > >>>>>overhead assertion. This discussion is really orthogonal
> > > to the probe
> > > >>>>>frequency and is more concerned with the probe
> > > quantity/quality to
> > > >>>>>determine
> > > >>>>>there is a problem. Goodness and scalability (and again
> > > my favorite
> > > >>>>>"bounded
> > > >>>>>detection time") can be expressed as how authoritative a
> > > test result is in
> > > >>>>>proportion to the messages expended. Also the fewer
> > > heuristics required to
> > > >>>>>interpret the test results, the more reliable the
> > > overall system is....
> > > >>>>>
> > > >>>>>
> > > >>>>My only point is that if your measurement is based on the
> > > assumption
> > > >>>>that all load-balancing is being done in a prescribed
> > > manner, then the
> > > >>>>quality of that measurement is very much dependent on the
> > > certainty
> > > >>>>that all of the nodes in the path are actually behaving
> > > according to
> > > >>>>that assumption.  On node behaving differently could
> > > reroute a large
> > > >>>>portion (or all) of the probes.  Thus the measurement 
> could be far
> > > >>>>
> > > >>>>
> > > >>>>from the actual behavior.
> > > >>>
> > > >>
> > > 
> >>==================================================================
> > > >>George Swallow       Cisco Systems                  
> (978) 497-8143
> > > >>                     250 Apollo Drive
> > > >>                     Chelmsford, Ma 01824
> > > >>
> > > >>
> > > >
> > >

---

From: "Thomas D. Nadeau" <[email-masked]>
Date: Thu, 21 Nov 2002 11:49:03 -0500

>Not entirely true, any point where there is useful predictability reduces 
>the number of
>possible faults that may only be caught by random process. IMHO that is 
>useful.

         --tom

> > >>what I wrote below.  Were you trying to answer some other message?
> > >>
> > >>...George
> > >>
> > >>
> > >>
> > >>>>portion (or all) of the probes.  Thus the measurement could be far
> > >>>>
> > >>>>
> > >>George Swallow       Cisco Systems                  (978) 497-8143
> > >>                     250 Apollo Drive
> > >>                     Chelmsford, Ma 01824
> > >>
> > >>
> > >
> >