Subject: [mpls] MPLS-RT review of draft-villamizar-mpls-multipath-use

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Sun, 20 Jan 2013 21:08:24 +0000

Hello,

I have been selected as an MPLS Review team reviewer for draft-villamizar-mpls-multipath-use-00.

I read this document, and I believe it is ready to be considered for WG adoption and it is a really good start -- although I also believe that more work and discussion needs to happen in Section 3 (where solutions and requirements are presented).

I also believe this is a very well written document. It describes a real problem space adequately providing citations with context and background, and explains cases that are useful in real operational networks.

Additionally, I have some (minor and editorial) comments and questions for consideration.

Minor:

1. I was surprised that Section 3.1.1 of RFC 5960 was not cited, as that defines the MPLS-TP requirements for ECMP load balancing.

2. In the requirement MP#1, what does "identify" mean? Identify in the edges where nodes have visibility to both layers, tag, identify in mid-point nodes?

3. I think the concepts of utilizing {ELI; EL} need further discussion in the document. Similarly, so do the implications on the topic of the differences of LAG vs. ECMP (both defined explicitly).

4. The concepts of unequal load split or otherwise load balancing proportionally to some ratio in different links/paths are briefly mentioned but could be more explicitly covered. The ECMP definition mentions this (proportionally to capacity), but in that case it would not be "Equal". Similarly, concepts like "fairly evenly distributed" are used in definitions but not explained -- might not be needed perhaps, but what is fair and what is even in different scenarios can be understood differently.

5. In requirement MP#2, is the requirement to "completely exclude from the hash", or to "always coming up with the same output, either by excluding or including but always resulting in the same"? (i.e., is excluding a way of getting to the result, or is it the result?)

Nits:

1. Some times, I think that in many instances, "MPLS LSP" and "MPLS-TP LSP" should be plural (s/LSP/LSPs/). Consequently, not always clear when is talking about splitting a single LSP as opposed to splitting multiple LSPs.

2. Some acronyms are not expanded on first use (or not defined), like CSPF, PSC, ILM, others.

3. s/overa/over a/

I hope these are clear and useful.

Thank you,

Carlos Pignataro.

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 22 Jan 2013 22:40:18 -0500

In message <[email-masked]>
"Carlos Pignataro (cpignata)" writes:
> I read this document, and I believe it is ready to be considered for
> WG adoption and it is a really good start -- although I also believe
> that more work and discussion needs to happen in Section 3 (where
> solutions and requirements are presented).

Looks like you and Dave are in agreement on that.

I look forward to specific suggestions, but I will reread the section
and try to make it more clear and more explicit.

> I also believe this is a very well written document. It describes a
> real problem space adequately providing citations with context and
> background, and explains cases that are useful in real operational
> networks.
>  
> 1. I was surprised that Section 3.1.1 of RFC 5960 was not cited, as
>    that defines the MPLS-TP requirements for ECMP load balancing.

Thank you for pointing that out.  The requirement to not load split
MPLS-TP is mentioned but the citation is not made.  In fact I will
quote the following paragraph verbatim.

   Equal-Cost Multi-Path (ECMP) load-balancing MUST NOT be performed
   on an MPLS-TP LSP.  MPLS-TP LSPs as defined in this document MAY
   operate over a server layer that supports load-balancing, but this
   load-balancing MUST operate in such a manner that it is transparent
   to MPLS-TP.  This does not preclude the future definition of new
   MPLS-TP LSP types that have different requirements regarding the
   use of ECMP in the server layer.

Use of ECMP is allowed in the server layer, as long as it is
transparent to the MPLS-TP client layer.

> 2. In the requirement MP#1, what does "identify" mean? Identify in the
>    edges where nodes have visibility to both layers, tag, identify in
>    mid-point nodes?

A MPLS-TP LSP may be aggregated by a midpoint LSR into a very large
MPLS LSP, as would be the case in a core node to core node MPLS LSP
between major cities.  In this case the ingress of the MPLS LSP cannot
through any existing signaling mechanism identify those client LSP
contained with it as MPLS-TP or not MPLS-TP.  For those client LSP
that are MPLS-TP LSP, a single EL value must be chosen.  For those
client LSP that are MPLS LSP, per packet entropy below the top label
must be (for practical reasons "must" not "should") used to determine
the entropy label value.

Also, a MPLS-TP ingress may not know that an LSP it is setting up is
going over a RFC 6790 capable LAG.  If the LSP is not identified as
being MPLS-TP, then load splitting would occur.

OTOH - If the MPLS-TP ingress picked a random number and used that in
an EL, even if it had no multipath interface the problem is solved,
but that is already pointed out in the document.

Do I need to change the text to clarify what is meant by "identify" in
this context?

Perhaps the requirement should be:

   MP#1  It MUST be possible to identify MPLS-TP LSP or otherwise
         accommodate the MPLS-TP LSP requirement to avoid reordering
         of packets.

The latter phrase acknowledges that adding an ELI and EL is fine.

> 3. I think the concepts of utilizing {ELI; EL} need further discussion
>    in the document. Similarly, so do the implications on the topic of
>    the differences of LAG vs. ECMP (both defined explicitly).

Perhaps the use of ELI and EL needs to be more explicit.  Dave seems
to share your opinion that more detail is needed.

> 4. The concepts of unequal load split or otherwise load balancing
>    proportionally to some ratio in different links/paths are briefly
>    mentioned but could be more explicitly covered. The ECMP definition
>    mentions this (proportionally to capacity), but in that case it
>    would not be "Equal". Similarly, concepts like "fairly evenly
>    distributed" are used in definitions but not explained -- might not
>    be needed perhaps, but what is fair and what is even in different
>    scenarios can be understood differently.

Familiarity with prior work in multipath in core routers would help
the reader.  Unfortunately not much has been published.

Some useful information can be found in draft-ietf-rtgwg-cl-use-cases
in Appendix B.  There is also useful descriptions of expectations
regarding limiting the frequency of load balancing and the inevitable
imperfect balancing in that document and the CL documents in RTGWG:
draft-ietf-rtgwg-cl-{requirements,framework}

> 5. In requirement MP#2, is the requirement to "completely exclude from
>    the hash", or to "always coming up with the same output, either by
>    excluding or including but always resulting in the same"? (i.e., is
>    excluding a way of getting to the result, or is it the result?)

In this context "completely excluding the MPLS-TP LSP ..." is the
functional equivalent to MPLS Link Bundling with pinning.

There is very little distinction between MP#2 and MP#3, except that
MP#3 allows that the LSP could be moved if requirements could no
longer be met (component link down, LSP preempted from the component
link, etc).

> Nits:
>  
> 1. Some times, I think that in many instances, "MPLS LSP" and "MPLS-TP
>    LSP" should be plural (s/LSP/LSPs/). Consequently, not always clear
>    when is talking about splitting a single LSP as opposed to
>    splitting multiple LSPs.

OK.  I'll look for instances of LSP that should be LSPs.

> 2. Some acronyms are not expanded on first use (or not defined), like

ILM is expanded in RFC 3031 (MPLS Architecture) in the table of
contents.  :-)

PSC is expanded in RFC 3471 (GMPLS Signaling Functional Description).

CSPF is expanded in RFC 3945 (GMPLS Architecture).

These are all base MPLS documents, but I agree that expanding even
well known acronyms makes the document more readable.

Thanks for pointing this out.

> 3. s/overa/over a/

Got it.

> I hope these are clear and useful.

They are both clear and useful.  I will try to reword the sections
where clarity may be lacking or where I need to be more explicit.
Thank you for the review.

> Carlos Pignataro.

Regards,

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 24 Jan 2013 17:41:40 -0500

Carlos,

Looking over prior email, a minor clarification came to mind.

In message <[email-masked]>
> "Carlos Pignataro (cpignata)" writes:

[ ... big snip ... ]

> draft-ietf-rtgwg-cl-{requirements,framework}

The "equal" in "ECMP" applies to equal cost.  So if a 10GbE server
layer LSP and 100GbE server layer LSP (ie: Packet Switch Capable LSP
or PSC LSP) have equal administrative costs, then it still makes sense
to split the load unequally.  The same applies to ECMP when all of the
links have the same next-hop, though I'm not sure who if anyone still
supports that.

In any case the term "multipath" is used in this document for a number
or reasons.  One reason is to avoid any confusion about limitations of
some ECMP implementations or preconceived notions of ECMP limitations,
such as any notion that the load must be split evenly.  Another reason
is to include Ethernet Link Aggregation, Link Bundling, as well as
ECMP, but exclude inverse-mux, hence the definitions (in Section 2):

   Multipath
       The term multipath includes all techniques in which

       1.  Traffic can take more than one path from one node to a
           destination.

       2.  Individual packets take one path only.  Packets are not
           subdivided and reassembled at the receiving end.

       3.  Packets are not resequenced at the receiving end.

       4.  The paths may be:

           a.  parallel links between two nodes, or

           b.  may be specific paths across a network to a destination
               node, or

           c.  may be links or paths to an intermediate node used to
               reach a common destination.

   [...]

   Composite Link
       The term Composite Link had been a registered trademark of
       Avici Systems, but was abandoned in 2007.  The term composite
       link is now defined by the ITU in [ITU-T.G.800].  The ITU
       definition includes multipath as defined here, plus inverse
       multiplexing which is explicitly excluded from the definition
       of multipath.

Regards,

Curtis

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Thu, 24 Jan 2013 23:27:26 +0000

Curtis,

> ECMP, but exclude inverse-mux, hence the definitions (in Section 2):

I should have said this in my review: I like the approach of using a new term, "Multipath", for this, and abstract from the implications of "ECMP" and group various techniques. I think it is really good. Normalizing definitions with draft-ietf-rtgwg-cl-* would be ideal as well.

Thanks,

-- Carlos.

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Thu, 24 Jan 2013 23:27:18 +0000

Hi, Curtis,

> The latter phrase acknowledges that adding an ELI and EL is fine.

Yes, I agree this is good refinement of the requirement.

> to share your opinion that more detail is needed.

I would think so.

> the reader.  Unfortunately not much has been published.

Do you think that an Informative pointer would help? These rtgwg documents do apply to IP and MPLS.

> longer be met (component link down, LSP preempted from the component

I see. Thanks.

> Thank you for the review.

Anytime! Thanks for considering the comments.

-- Carlos.

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 24 Jan 2013 20:43:45 -0500

In message <[email-masked]>

[ ... big snip ... ]

> > draft-ietf-rtgwg-cl-{requirements,framework}

The documents draft-ietf-rtgwg-cl-{requirements,use-cases,framework}
are expected to advance together in RTGWG.  This means that
cl-requirements whish was done and last called a long time ago and
cl-use-cases which is not likely to change and probably could be last
called, will both be held up by cl-framework.

The cl-framework is in turn held up by documents it references that
are in very early stages, or expired, or are needed but don't exist
yet.  Cleaning up the cl-framework is somewhere on my to-do-list.

Given the long time non-advancement of the CL work in RTGWG, I'd
rather not reference that work.

Rather than reference one of these, I'd rather break out the part of
draft-ietf-rtgwg-cl-use-cases that describes existing MPLS multipath
usage as a separate document and advance that separately (perhaps in
MPLS WG) and reference that since it would describe only the current
practices and would be more likely to advance in finite time.

Things like unequal load split and dynamic multipath are not critical
to this document, so my preference is to either not mention either
technique, or very briefly mention it with enough wording in place to
convey what the technique is.  I think what we have is enough, but I
could be convinced otherwise.

Curtis

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Fri, 25 Jan 2013 01:58:44 +0000

> could be convinced otherwise.

That approach seems OK. Frankly my only concern is to end up with multiple definitions of the same terms and concepts in different documents, that invariably end up desynchronized. 

Thanks,

-- Carlos.

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 24 Jan 2013 21:08:40 -0500

In message <[email-masked]>
"Carlos Pignataro (cpignata)" writes:
 
> I should have said this in my review: I like the approach of using a
> new term, "Multipath", for this, and abstract from the implications of
> "ECMP" and group various techniques. I think it is really
> good. Normalizing definitions with draft-ietf-rtgwg-cl-* would be
> ideal as well.
>  
> Thanks,
>  
> -- Carlos.

AFAIK the definitions in the RTGWG CL work and the definition here are
not identical but are consistent.

In cl-requirements "composite link" is redefined slightly from the
ITU-T definition to exclude inverse-multiplexing, but then a whole lot
of requrements are piled on that have nothing to do with ITU-T CL
minus inverse-mux.

Existing practice is defined in cl-use-cases as "classic multipath".
The distinction in the CL work between RTGWG CL (something new) and
"classic multipath".  The latter (classic multipath) lacks support for
new capabilities - for example: explicit support for heterogeneous
component links - groups of component links having different metrics
for things such as delay and jitter.

So "multipath" here is the same as "classic multipath" in the CL
documents.  The definition here is more precise though and it wouldn't
hurt to move that to a common document.

This document does not refer to RTGWG definition of "composite link",
only the ITU-T definition.  I personally don't think the RTGWG
documents should have used the term "composite link", given that the
ITU-T already had a definition for CL but instead used a different
term and then piled the new requirements on that new term.

If anything, the definitions in the RTGWG CL work need to be improved,
but that is a topic for the rtgwg mailing list.

Curtis