Subject: Re: [mpls] New Version Notification for draft-akiya-mpls-lsp-ping-lag-multipath-00.txt

From: "Nagendra Kumar Nainar (naikumar)" <[email-masked]>
Date: Mon, 7 Jul 2014 15:23:49 +0000

Hi Nobo,

Apologies for the delay. Please see below:

>Hi Nagendra,
>
>> -----Original Message-----
>> From: Nagendra Kumar Nainar (naikumar)
>> Sent: Saturday, June 14, 2014 1:54 PM
>> To: Nobo Akiya (nobo)
>> Subject: Re: New Version Notification for draft-akiya-mpls-lsp-ping-lag-
>> multipath-00.txt
>> 
>> Hi Nobo,
>> 
>> As usual, interesting draft. Please see below for initial comments,
>
>Thank you for comments, and thank you for agreeing to allow me to include
>the MPLS WG in this response.
>
>I'm glad you found this document to be interesting. Although it became a
>bit more complicated that I was originally hoping. However, this
>extension add a great value to a tool which can be part of the something
>larger to "fight" the traffic black hole happening on MPLS networks :)
> 
>> 
>> From section 3, It appears that responder MUST set the LAG Info TLV. Is
>>it the
>> case even if the downstream is not LAG?. Or did you meant that if the
>> responder have LAG based donwstream interface, then it MUST include the
>> TLV?.
>
>First bullet in the last portion of the Section 3 was exactly to explain
>why this is a MUST, but clearly additional text will be helpful. How
>about we update the bullet:
>
>[old]
>   o  Identify whether responder LSR understands this mechanism.
>
>[new]
>   o  Mandating the responder to always add the LAG Interface Info TLV
>      in the MPLS echo reply allows the initiating LSR to identify
>      whether responder LSR understands this mechanism.

<Nagendra> I am trying to understand the behavior when a responder wants
back to echo received with LAG Interface Info TLV. In figure 1, assume
there is one more node connected to E (say F) and A wants to trace the
path from A to F (via A-B-LAG-C-E-F). When C receives thhe echo with LAG
interface Info  TLV, since it doenst have any upstream/downstream LAG
interface, will it still include the TLV in reply?. If so, is it like it
unset both U/D bit?.

>
>> 
>> 
>> In Section, I am trying to understand what does the below means,
>> 
>> "All fields and Sub-TLVs, except for Multipath Data Sub-TLV and
>>          Interface Index Sub-TLV, are set/added to DDMAP to describe
>>          this LAG interface, as per [RFC6424].
>> "
>> 
>> The above statement followed by statement to add "Interface Index Sub-
>> TLV"
>> appears to be confusing.
>
>I see. Perhaps re-wording it can help clarify things. How about something
>like this?
>
>[old]
>      *  All fields and Sub-TLVs, except for Multipath Data Sub-TLV and
>         Interface Index Sub-TLV, are set/added to DDMAP to describe
>         this LAG interface, as per [RFC6424].
>
>[new]
>      *  In the DDMAP, Multipath Data Sub-TLV and Interface Index Sub-
>         TLV are to describe each LAG member link.  All other fields of
>         the DDMAP are to describe the LAG interface.

<Nagendra> Sounds good.
>
>> 
>> 
>> In section 3, the below 2 statements appears to be redundant. Do we
>>really
>> need both?.
>> 
>> "*  For each LAG member link of this LAG interface:
>> 
>>          +  MUST add Interface Index Sub-TLV (described in Section 7)
>>             with LAG Member Link Indicator flag set in Interface Index
>>             Flags, describing this LAG member link.
>> 
>>          +  MUST add Multipath Data Sub-TLV for this LAG member link, if
>>             received DDMAP requested multipath information.
>> 
>>    Each LAG member link is described with Interface Index Sub-TLV and
>>    conditionally with Multipath Data Sub-TLV (if multipath information
>>    is requested). "
>
>True. Let me re-word this one too.
>
>[old]
>   Each LAG member link is described with Interface Index Sub-TLV and
>   conditionally with Multipath Data Sub-TLV (if multipath information
>   is requested).  If both Sub-TLVs are placed in the DDMAP to describe
>   a LAG member link, Interface Index Sub-TLV MUST be added first with
>   Multipath Data Sub-TLV immediately following.
>
>[new]
>   When both the Interface Index Sub-TLV and the Multipath Data Sub-TLV
>   is placed in the DDMAP to describe a LAG member link, Interface Index
>   Sub-TLV MUST be added first with Multipath Data Sub-TLV immediately
>   following.

<Nagendra> Sounds good.

>
>> 
>> 
>> Section 4 says the inclusion of Detailed interface TLV is a MUST while
>> section 8 sys it is a MAY statement as below,
>> 
>> section 4:
>> "
>> Responder LSR MUST:
>> 
>>    o  Add LAG Interface Info TLV in the MPLS echo reply to provide
>>       acknowledgement back to the initiator.  Upstream LAG Info
>>       Accommodation flag MUST be set in LAG Interface Info Flags.
>> 
>>    o  Add the Detailed Interface and Label Stack TLV (described in
>>       Section 8) in the MPLS echo reply.
>> "
>> 
>> section 8:
>> "The Detailed Interface and Label Stack object is a TLV that MAY be
>>    included in a MPLS echo reply message to report the interface on
>>    which the MPLS echo request message was received and the label stack
>>    that was on the packet when it was received.  A responder LSR MUST
>>    NOT insert more than one instance of this TLV. "
>> 
>
>Right, perhaps above snippet being preceded by the initiating LSR sending
>the MPLS echo request with DS Flags I bit set is not sufficient to elude
>that responder is functioning with an assumption that it needs to include
>interface/label-stack TLV. Let me change some texts:
>
>[old]
>   o  Add the Detailed Interface and Label Stack TLV (described in
>      Section 8) in the MPLS echo reply.
>
>   o  Add the Incoming Interface Index Sub-TLV (described in
>      Section 8.1.2) for LAG interfaces.  The LAG Member Link Indicator
>      flag MUST be set in Interface Index Flags, and the incoming
>      Interface Index set to LAG member link which received the MPLS
>      echo request.
>
>[new]
>   o  When the received DDMAP had DS Flags I set, add the Detailed
>      Interface and Label Stack TLV (described in Section 8) in the MPLS
>      echo reply.
>
>   o  When the received DDMAP had DS Flags I set, add the Incoming
>      Interface Index Sub-TLV (described in Section 8.1.2) for LAG
>      interfaces.  The LAG Member Link Indicator flag MUST be set in
>      Interface Index Flags, and the incoming Interface Index set to LAG
>      member link which received the MPLS echo request.

<Nagendra> Ok.

>
>> 
>> 
>> Section 7 specifies as below,
>> "The Interface Index Sub-TLV describes the index assigned by the
>>    upstream LSR to the interface."
>> 
>> Can you also point some reference to what type of index is it?. Is it
>> something assigned and advertised by LAG protocol like LACP or PAGP?.
>
>It's a locally allocated persistent index for local interfaces. It may or
>may not be advertised, and that's not really important for this context.
>We are just interested in getting a unique value (local to that node)
>that identifies a specific interface. I could add more texts, but the
>description on this sub-TLV was mostly ported from RFC4379. I think it's
>ok as is.

<Nagendra> Ok. So the responder will include the locally assigned index
value (of the egress interface) to this Sub-TLV and reply back. So should
we modify the below statement?,

"The Interface Index Sub-TLV describes the index assigned by the
   upstream LSR to the interface.²

I think it is not the upstream LSR to the interface right?.

>
>> 
>> In association to above, Section 4 states the below,
>> 
>> "Note that defined procedures will provide a deterministic result for
>>    LAG interfaces that are back-to-back connected between routers (i.e.
>>    no L2 switch in between).  If there is a L2 switch between LSR at
>>    TTL=n and LSR at TTL=n+1, there is no guarantee that traversal of
>>    every LAG member link at TTL=n will result in reaching different
>>    interface index at TTL=n+1."
>> 
>> When there is L2 switch(es) in between, the LAG termination/negotiation
>> wil be between the LSR and L2 switch and so we may need some way to
>> exchange the Interface Index for each LAG member via L2 switch to be
>> exchanged between the LSRs. I am not aware of such exchange. If I am
>> missing something, I think it is worth referring the same for clarity :)
>
>That is exactly why L2-in-between is outside the scope of this document
>:) Appendix A describes various flavors and why it is complicated to
>extend support for those flavors.

<Nagendra> Ok.
>
>> 
>> Does the below sounds like a (direct) SHOULD statement?
>> 
>> "If the responder LSR is able to accommodate this request,
>>    then the LAG Interface Info object MUST be included in the MPLS echo
>>    reply message."
>> 
>> In the below topology,
>> 
>>         +-------+
>>              |       |
>>      A-------B=======C-------E
>>              |       |       |
>>              +-------D-------G
>> 
>> Assume A is validating all ECMP paths from A to G. Assume the metric
>>from
>> B to D is 2 and rest all link are 1. So it will be
>> 
>> A-B-LAG-C-E-G
>> A-B-LAG-C-D-G
>> A-B-D-G
>> 
>> Assume the list of multipath data is (1-10)
>> 
>> Now B will reply as below,
>> 
>> (1,2,3) - {Link1, Downstream:C}
>> (4,5,6) - {Link2 Downstream:C}
>> (7,8,9,10) - {LinkBC, Downstream:D}
>> 
>> A will then send 2 request as below:
>> 
>> Echo1 (to C) - (multipath:1,2,3,4,5,6),
>> Echo2 (to D) - (multipath:7,8,9,10)
>> 
>> C will reply back with
>> 
>> (1,2,3) - {LinkCE, Downstream E}
>> (4,5,6) - {LinkCD, Downstream D}
>> 
>> So we dont have anything common between Link2 of LAG and downtream E.
>> Similarly we dont have anyhting common between Link1 of LAG and
>> downstream D (from C). I think we need the downstream node of LAG to
>> reply back with multipath info for each downstream from each LAG range.
>> Any thoughts?.
>
>The main problem with above example is that the range of multipath info
>is too small, thus you ran out of intersect in the downstream.
>Additionally, if I was to implement this, then I would not carry over
>treating each LAG member link separately in the LSP Tree Trace
>operations, once TTL=n+1 is reached. Once LAG merges to a same downstream
>node, then I would aggregate the result and move forward in the
>downstream (i.e. treat the LAG as one logical interface).

<Nagendra> Ok. Let me check this.

Thanks,
Nagendra

>
>Thanks!
>
>-Nobo
>
>> 
>> Thanks,
>> Nagendra
>> 
>> 
>> 
>> 
>> On 6/13/14, 5:33 PM, "Nobo Akiya (nobo)" <[email-masked]> wrote:
>> 
>> >FYI, new LSP Ping extension draft.
>> >
>> >> -----Original Message-----
>> >> From: mpls [mailto:[email-masked]] On Behalf Of Nobo Akiya
>> >> (nobo)
>> >> Sent: Friday, June 13, 2014 5:15 PM
>> >> To: [email-masked]
>> >> Subject: [mpls] FW: New Version Notification for
>> >>draft-akiya-mpls-lsp-ping-
>> >> lag-multipath-00.txt
>> >>
>> >> Dear MPLS WG,
>> >>
>> >> On MPLS data plane, one aspect which OAM is blind to traffic black
>> >>holing is  when LSPs are traversing over LAG interfaces. We would like
>> >>to propose an  extension to RFC4379 to provide the tool an ability to
>> >>discover and traverse  LAG members.
>> >>
>> >> It will be very helpful if you could review this document, provide
>> >>comments  and help us standardize this extension.
>> >>
>> >> Thanks!
>> >>
>> >> -Nobo, on behalf of authors
>> >>
>> >> > -----Original Message-----
>> >> > From: [email-masked] [mailto:[email-masked]]
>> >> > Sent: Friday, June 13, 2014 5:04 PM
>> >> > To: John E. Drake; Nobo Akiya (nobo); George Swallow (swallow);
>> >> > George Swallow (swallow); Bruno Decraene; Stephane Litkowski; Bruno
>> >> > Decraene; Nobo Akiya (nobo); John Drake; Stephane Litkowski
>> >> > Subject: New Version Notification for
>> >> > draft-akiya-mpls-lsp-ping-lag- multipath-00.txt
>> >> >
>> >> >
>> >> > A new version of I-D,
>> >> > draft-akiya-mpls-lsp-ping-lag-multipath-00.txt
>> >> > has been successfully submitted by Nobo Akiya and posted to the
>> >> > IETF repository.
>> >> >
>> >> > Name:		draft-akiya-mpls-lsp-ping-lag-multipath
>> >> > Revision:	00
>> >> > Title:		Label Switched Path (LSP) Ping/Trace Multipath
>> Support for
>> >> > Link Aggregation Group (LAG) Interfaces
>> >> > Document date:	2014-06-13
>> >> > Group:		Individual Submission
>> >> > Pages:		18
>> >> > URL:
>> >>http://www.ietf.org/internet-drafts/draft-akiya-mpls-lsp-ping-
>> >> > lag-multipath-00.txt
>> >> > Status:
>> >>https://datatracker.ietf.org/doc/draft-akiya-mpls-lsp-ping-lag-
>> >> > multipath/
>> >> > Htmlized:
>> >>http://tools.ietf.org/html/draft-akiya-mpls-lsp-ping-lag-
>> >> > multipath-00
>> >> >
>> >> >
>> >> > Abstract:
>> >> >    This document defines an extension to the Multiprotocol Label
>> >> >    Switching (MPLS) Label Switched Path (LSP) Ping and Traceroute
>>to
>> >> >    describe Multipath Information for Link Aggregation (LAG) member
>> >> >    links separately, thus allowing MPLS LSP Ping and Traceroute to
>> >> >    discover and exercise specific paths of layer 2 Equal-Cost
>> >>Multipath
>> >> >    (ECMP) over LAG interfaces.
>> >> >
>> >> >    This document updates RFC4379 and RFC6424.
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > Please note that it may take a couple of minutes from the time of
>> >> > submission until the htmlized version and diff are available at
>> >> tools.ietf.org.
>> >> >
>> >> > The IETF Secretariat
>> >>
>> >> _______________________________________________
>> >> mpls mailing list
>> >> [email-masked]
>> >> https://www.ietf.org/mailman/listinfo/mpls
>

---

From: "Nobo Akiya (nobo)" <[email-masked]>
Date: Mon, 7 Jul 2014 23:52:51 +0000

Hi Nagendra,

Thanks for further comments!
Please see in-line.

> >> From section 3, It appears that responder MUST set the LAG Info TLV.
> >>Is it the  case even if the downstream is not LAG?. Or did you meant
> >>that if the  responder have LAG based donwstream interface, then it
> >>MUST include the  TLV?.
> >
> >First bullet in the last portion of the Section 3 was exactly to
> >explain why this is a MUST, but clearly additional text will be
> >helpful. How about we update the bullet:
> >
> >[old]
> back to echo received with LAG Interface Info TLV. In figure 1, assume there
> is one more node connected to E (say F) and A wants to trace the path from
> A to F (via A-B-LAG-C-E-F). When C receives thhe echo with LAG interface
> Info  TLV, since it doenst have any upstream/downstream LAG interface, will
> it still include the TLV in reply?. If so, is it like it unset both U/D bit?.

Good question. I think you meant E and not C, since C has LAG as upstream?

Assuming you were asking what will E send, then:

Even if E does not have any upstream/downstream LAG, E will still include LAG Intf Info TLV in the MPLS echo reply, if it understands this TLV. This is how A learns whether or not E is capable of following the procedures described in this document. Otherwise lack of LAG upstream/downstream information becomes ambiguous to A: there is no upstream/downstream LAG on E vs. E didn't understand the LAG procedures (i.e. ignored the option TLV and proceeded) but may have upstream/downstream LAG.

So, LAG Intf Info TLV in MPLS echo reply is to let the initiator know:
- Responder understands this mechanism.
- U bit to indicate that responder is capable of providing Detailed Interface and Label Stack TLV.
- D bit to indicate that responder is capable of providing DDMAP with LAG info.

I wanted to break up the (U)pstream and (D)ownstream capabilities, since on some products, doing one could be more difficult than the other one.

> >It's a locally allocated persistent index for local interfaces. It may
> >or may not be advertised, and that's not really important for this context.
> >description on this sub-TLV was mostly ported from RFC4379. I think
> >it's ok as is.
> 
> value (of the egress interface) to this Sub-TLV and reply back. So should we
> modify the below statement?,
> 

Hmm... yes, that text was copied from RFC4379, but it's a bit confusing. Let me change that:

[old]
   The Interface Index Sub-TLV describes the index assigned by the
   upstream LSR to the interface.

[new]
   The Interface Index Sub-TLV describes the index assigned by this
   LSR to the interface.

Would that work?

Thanks!

-Nobo

---

From: Sam Aldrin <[email-masked]>
Date: Mon, 7 Jul 2014 18:06:37 -0700

Hi Nobo and authors,

Slow in catching up on this ID.

I have some basic questions about the draft. Want to get them clarified before doing a detailed review.

1. I thought the ECMP is about the LSP's and not about L3 interfaces or L2 interfaces. i.e. at MPLS LSP level where LSP's are established irrespective of L3 or L2 interfaces. But section 1.2. talks about L3 and L2 and implies that RFC4379 does only L3 interfaces ECMP, could you kindly clarify? Atleast as I read RFC4379 section 4.1 doesn't distinguish as defined in this draft.

2. Could you clarify the following? In the case of LAG interface, for a given FEC, will there be different label stack for each sub-interface type? If not, i.e. same label stack but differ sub i.f, which multipath type one could use to determine hash computation?

3. If in Q #2 Label stack there exists different label stack for each lag member, is there a downside to using separate DDMAP for each LAG member? i.e. w/o any enhancement?

4. Also, how is DDMAP interface info validated? I believe the LAG member link # is local. What all fields being validated for LAG interface?

Responses will clarify for me to review better. :D

-sam

On Jun 13, 2014, at 2:14 PM, Nobo Akiya (nobo) <[email-masked]> wrote:

> Dear MPLS WG,
> 
> On MPLS data plane, one aspect which OAM is blind to traffic black holing is when LSPs are traversing over LAG interfaces. We would like to propose an extension to RFC4379 to provide the tool an ability to discover and traverse LAG members.
> 
> It will be very helpful if you could review this document, provide comments and help us standardize this extension.
> 
> Thanks!
> 
>> To: John E. Drake; Nobo Akiya (nobo); George Swallow (swallow); George
>> Swallow (swallow); Bruno Decraene; Stephane Litkowski; Bruno Decraene;
>> Nobo Akiya (nobo); John Drake; Stephane Litkowski
>> Subject: New Version Notification for draft-akiya-mpls-lsp-ping-lag-
>> multipath-00.txt
>> 
>> 
>> A new version of I-D, draft-akiya-mpls-lsp-ping-lag-multipath-00.txt
>> has been successfully submitted by Nobo Akiya and posted to the IETF
>> repository.
>> 
>> Title:		Label Switched Path (LSP) Ping/Trace Multipath Support for
>> URL:            http://www.ietf.org/internet-drafts/draft-akiya-mpls-lsp-ping-
>> lag-multipath-00.txt
>> Status:         https://datatracker.ietf.org/doc/draft-akiya-mpls-lsp-ping-lag-
>> multipath/
>> Htmlized:       http://tools.ietf.org/html/draft-akiya-mpls-lsp-ping-lag-
>> multipath-00
>> 
>> 
>> Abstract:
>>   Switching (MPLS) Label Switched Path (LSP) Ping and Traceroute to
>>   discover and exercise specific paths of layer 2 Equal-Cost Multipath
>> submission until the htmlized version and diff are available at tools.ietf.org.
>> 
>> The IETF Secretariat
>

---

From: "Nobo Akiya (nobo)" <[email-masked]>
Date: Tue, 8 Jul 2014 01:41:49 +0000

Hi Sam,

Always appreciate your comments/questions :)

Please see in-line.

> From: Sam Aldrin [mailto:[email-masked]]
> Sent: Monday, July 07, 2014 9:07 PM
> To: Nobo Akiya (nobo)
> Cc: [email-masked]
> Subject: Re: [mpls] New Version Notification for draft-akiya-mpls-lsp-ping-
> lag-multipath-00.txt
> 
> Hi Nobo and authors,
> 
> 1. I thought the ECMP is about the LSP's and not about L3 interfaces or L2
> interfaces. i.e. at MPLS LSP level where LSP's are established irrespective of
> L3 or L2 interfaces. But section 1.2. talks about L3 and L2 and implies that
> RFC4379 does only L3 interfaces ECMP, could you kindly clarify? Atleast as I
> read RFC4379 section 4.1 doesn't distinguish as defined in this draft.

RFC4379/RFC6424 do not explicitly spell this out, but it's the DDMAP (as well as DSMP) fields that imply L3.

      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |               MTU             | Address Type  |    DS Flags   |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |             Downstream IP Address (4 or 16 octets)            |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |         Downstream Interface Address (4 or 16 octets)         |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

And the Address Type can be:

         Type #        Address Type           K Octets
         ------        ------------           --------
              1        IPv4 Numbered                16
              2        IPv4 Unnumbered              16
              3        IPv6 Numbered                40
              4        IPv6 Unnumbered              28

Type 5 (Non IP) was also added by RFC6426 for TP.

If an LSR was to describe each LAG member in a DDMAP using procedures described in RFC4379/RFC6424, then all fields expect for Multipath Info will be exactly the same. This is sort of "weird" in the first place, but what is worse is that this leads to the difficulty of the second portion of the draft-akiya-mpls-lsp-ping-lag-multipath, which is to validate the different traversal for each LAG member.

> 
> 2. Could you clarify the following? In the case of LAG interface, for a given
> FEC, will there be different label stack for each sub-interface type? If not,
> i.e. same label stack but differ sub i.f, which multipath type one could use
> to determine hash computation?

I think I understand your question ... but if my answers are not what you were asking, do elaborate. For a given FEC, the incoming & outgoing labels will be the same for all LAG members when traversing over a LAG to reach the downstream. Of course SR could assign an Adj-SID for a LAG member, and explicitly steer the packets over the specific LAG member, but it would no longer be considered to travel over the LAG at that point. Similar if RSVP was to pin-down a specific LAG member. So the LAG is treated as one logical interface by the label binding procedures, thus same labels for every LAG member.

As far as how an incoming packet is load balanced to each LAG member, that could be, in the context of LSP trace, either destination IP addresses (multipath type {2, 4, 8}) or Entropy Labels (multipath type {9}) or both (multipath type {10}).

> 
> 3. If in Q #2 Label stack there exists different label stack for each lag
> member, is there a downside to using separate DDMAP for each LAG
> member? i.e. w/o any enhancement?

Same label, see above.

> 
> 4. Also, how is DDMAP interface info validated? I believe the LAG member
> link # is local. What all fields being validated for LAG interface?

When MPLS echo request traverses over all LAG members (i.e. different interface index) to the nexthop, then the packets should have terminated on different incoming LAG members at the downstream (different interface index).

For supported/unsupported scenarios, please view: http://tools.ietf.org/html/draft-akiya-mpls-lsp-ping-lag-multipath-00#appendix-A

> Responses will clarify for me to review better. :D

Looking forward to seeing more comments from you, hopefully not as harsh as the Reply Mode extension draft :)

Thanks!

-Nobo

> > On MPLS data plane, one aspect which OAM is blind to traffic black holing
> is when LSPs are traversing over LAG interfaces. We would like to propose
> an extension to RFC4379 to provide the tool an ability to discover and
> traverse LAG members.
> >

---

From: "Nagendra Kumar Nainar (naikumar)" <[email-masked]>
Date: Tue, 8 Jul 2014 02:25:58 +0000

Hi Nobo,

Even if E does not have any upstream/downstream LAG, E will still include
LAG Intf Info TLV in the MPLS echo reply, if it understands this TLV. This
is how A learns whether or not E is capable of following the procedures
described in this document. Otherwise lack of LAG upstream/downstream
information becomes ambiguous to A: there is no upstream/downstream LAG on
E vs. E didn't understand the LAG procedures (i.e. ignored the option TLV
and proceeded) but may have upstream/downstream LAG.

So, LAG Intf Info TLV in MPLS echo reply is to let the initiator know:
- Responder understands this mechanism.
- U bit to indicate that responder is capable of providing Detailed
Interface and Label Stack TLV.
- D bit to indicate that responder is capable of providing DDMAP with LAG
info.

I wanted to break up the (U)pstream and (D)ownstream capabilities, since
on some products, doing one could be more difficult than the other one.

<Nagendra> Yes, it is E :). That makes sense. So I think E will set U and
D to zero to indicate there is no LAG details but I understand this
feature :). May be this can be clarified in the draft?.

Hmm... yes, that text was copied from RFC4379, but it's a bit confusing.
Let me change that:

[old]

[new]

Would that work?

<Nagendra> Or “The Interface Index Sub-TLV describes the index assigned by
local LSR to the egress interface”?.

Thanks,
Nagendra

On 7/7/14, 7:52 PM, "Nobo Akiya (nobo)" <[email-masked]> wrote:

>>there
>Even if E does not have any upstream/downstream LAG, E will still include
>LAG Intf Info TLV in the MPLS echo reply, if it understands this TLV.
>This is how A learns whether or not E is capable of following the
>procedures described in this document. Otherwise lack of LAG
>upstream/downstream information becomes ambiguous to A: there is no
>upstream/downstream LAG on E vs. E didn't understand the LAG procedures
>(i.e. ignored the option TLV and proceeded) but may have
>upstream/downstream LAG.
>
>I wanted to break up the (U)pstream and (D)ownstream capabilities, since
>on some products, doing one could be more difficult than the other one.
>

---

From: Sam Aldrin <[email-masked]>
Date: Mon, 7 Jul 2014 19:32:02 -0700

Thanks, Nobo, for quick response.

Inline for my comments.

> RFC4379/RFC6424 do not explicitly spell this out, but it's the DDMAP (as well as DSMP) fields that imply L3.
May be the wording in your draft should change, because neither RFC explicitly say that. It could be just that LAG interface type was not explicitly identified. 
> If an LSR was to describe each LAG member in a DDMAP using procedures described in RFC4379/RFC6424, then all fields expect for Multipath Info will be exactly the same. This is sort of "weird" in th…
ok
> I think I understand your question ... but if my answers are not what you were asking, do elaborate. For a given FEC, the incoming & outgoing labels will be the same for all LAG members when traver…
Few questions,

Because you mentioned it is L2 interfaces only, I was wondering whether L3 header is taken into consideration or not! 
Secondly, assuming if member is not pinned to a specific LSP, is the validation for FEC? If so, how is that being done? when do you consider LSP ping/trace failure? As it is sub member failure, the interface is still running and packets are still fwd'd. 

> As far as how an incoming packet is load balanced to each LAG member, that could be, in the context of LSP trace, either destination IP addresses (multipath type {2, 4, 8}) or Entropy Labels (multi…
You tripped me with L2 only interfaces/sub if. Hence the question, whether L3 header is taken into consideration.
> For supported/unsupported scenarios, please view: http://tools.ietf.org/html/draft-akiya-mpls-lsp-ping-lag-multipath-00#appendix-A
I think appendix sections are very imp to consider and should be in main sections.

From high level, this addition will help discover various LAG members associated with the data path of an LSP. But I think it is difficult to validate against a given FEC. Because the DDMAP info of LAG member sent cannot be validated. If it is indeed could be validated, should be documented in the ID. 

> Looking forward to seeing more comments from you, hopefully not as harsh as the Reply Mode extension draft :)
It is just a beginning and hope you won't conclude with reply mode 1. :D

-sam

---

From: "Nobo Akiya (nobo)" <[email-masked]>
Date: Wed, 9 Jul 2014 23:41:15 +0000

Hi Nagendra,

> Sent: Monday, July 07, 2014 10:26 PM
> To: Nobo Akiya (nobo)
> Cc: [email-masked]
> LAG Intf Info TLV in the MPLS echo reply, if it understands this TLV. This is
> how A learns whether or not E is capable of following the procedures
> information becomes ambiguous to A: there is no upstream/downstream
> LAG on E vs. E didn't understand the LAG procedures (i.e. ignored the option
> TLV and proceeded) but may have upstream/downstream LAG.
> 
> <Nagendra> Yes, it is E :). That makes sense. So I think E will set U and D to
> zero to indicate there is no LAG details but I understand this feature :). May
> be this can be clarified in the draft?.

Sure, how about changes in Section 3.

[old]
   These procedures allow initiating LSR to:

   o  Identify whether each DDMAP describes a LAG interface or a non-LAG
      interface.

[new]
   These procedures allow initiating LSR to:

   o  Mandating the responder LSR to always add the LAG Interface Info TLV
      in the MPLS echo reply allows the initiating LSR to identify whether
      responder LSR understands this mechanism

   o  The value of the LAG Description Indicator flag in DS Flags will allow
      the initiating LSR to identify whether each DDMAP describes a LAG
      interface or a non-LAG interface

Will that work?

> <Nagendra> Or "The Interface Index Sub-TLV describes the index assigned
> by local LSR to the egress interface"?.

Accepted.

Thank!

-Nobo

---

From: "Nobo Akiya (nobo)" <[email-masked]>
Date: Thu, 10 Jul 2014 00:34:34 +0000

Hi Sam,

> Thanks, Nobo, for quick response.

Sorry for the delay in my second response :D

> >> 2. Could you clarify the following? In the case of LAG interface, for
> >> a given FEC, will there be different label stack for each
> >> sub-interface type? If not, i.e. same label stack but differ sub i.f,
> >> which multipath type one could use to determine hash computation?
> >
> > I think I understand your question ... but if my answers are not what you
> were asking, do elaborate. For a given FEC, the incoming & outgoing labels
> will be the same for all LAG members when traversing over a LAG to reach
> the downstream. Of course SR could assign an Adj-SID for a LAG member,
> and explicitly steer the packets over the specific LAG member, but it would
> no longer be considered to travel over the LAG at that point. Similar if RSVP
> was to pin-down a specific LAG member. So the LAG is treated as one logical
> interface by the label binding procedures, thus same labels for every LAG
> member.
> Few questions,
> 
> Because you mentioned it is L2 interfaces only, I was wondering whether L3
> header is taken into consideration or not!

I see, actual data carried on the LSP may be "whatever", but for the purpose of LSP Traceroute multipath discovery, we can assume that multipath is going to be performed with destination IP or Flow/Entropy Label.

> Secondly, assuming if member is not pinned to a specific LSP, is the
> validation for FEC? If so, how is that being done? when do you consider LSP
> ping/trace failure? As it is sub member failure, the interface is still running
> and packets are still fwd'd.

If a LAG member is pinned to a specific LSP, then this document does not impose any changes. Thanks for pointing this out though, I think it's worth spelling that out in the document. What do you think about following text at the end of Section 2 (Overview).

   Note that the mechanism described in this document does not impose
   any changes to scenarios where an LSP is pinned down to a particular
   LAG member (i.e. the LAG is not treated as one logical interface by the
   LSP).

> >> 4. Also, how is DDMAP interface info validated? I believe the LAG
> >> member link # is local. What all fields being validated for LAG interface?
> >
> > When MPLS echo request traverses over all LAG members (i.e. different
> interface index) to the nexthop, then the packets should have terminated
> on different incoming LAG members at the downstream (different interface
> index).
> >
> > For supported/unsupported scenarios, please view:
> > http://tools.ietf.org/html/draft-akiya-mpls-lsp-ping-lag-multipath-00#
> > appendix-A

I agree that Appendix is useful. At the same time, I'd like the main to focus on the problem and the solution (otherwise the document could become even more difficult to read). How about I add explicit xref to Appendix from the Overview.

...
   o  In Section 7, the Interface Index Sub-TLV;

   o  In Section 8, the Detailed Interface and Label Stack TLV;

   o  In Appendix A, issues with LAG having L2 Switch. <---- this entry!

> From high level, this addition will help discover various LAG members
> associated with the data path of an LSP. But I think it is difficult to validate
> against a given FEC. Because the DDMAP info of LAG member sent cannot be
> validated. If it is indeed could be validated, should be documented in the ID.

Good point. I think Section 4 (Mechanism to Validate L2 ECMP Traversal) can use more texts to describe the what can be validated. How about we make some changes in the Section 4 (Mechanism to Validate L2 ECMP Traversal).

[old]
   Described procedures allow initiating LSR to know:

   o  The expected load balance information of every LAG member link, at
      LSR with TTL=n.

   o  The actual incoming interface at LSR with TTL=n+1, including the
      interface index of LAG member link if incoming interface is a LAG
      interface.

[new]
   Described procedures allow initiating LSR to know:

   o  The expected load balance information of every LAG member link, at
      LSR with TTL=n.

   o  With specific entropy, the expected interface index of the
      outgoing LAG member link at TTL=n.

   o  With specific entropy, the interface index of the incoming LAG
      member link at TTL=n+1.

   Expectation is that there's a relationship between the interface
   index of the outgoing LAG member link at TTL=n and the interface
   index of the incoming LAG member link at TTL=n+1 for all discovered
   entropies.  In other words, set of entropies that load balances to
   outgoing LAG member link X at TTL=n should all reach the nexthop on
   same incoming LAG member link Y at TTL=n+1.

Well, you are already providing good set of comments, much appreciate it!

Thanks!

-Nobo

---

From: Sam Aldrin <[email-masked]>
Date: Wed, 9 Jul 2014 22:47:09 -0700

Hi Nobo,

Thanks for the responses.
Snipped out things so as not to bloat the discussion. 
I could review draft when you add those changes.
For now, have some points which I feel are more imp :D Hope you don’t mind.
>   same incoming LAG member link Y at TTL=n+1.
I think this is very important, for the reasons below.
1. How does one know same LAG bundle originates at TTL=n terminates at TTL=n+1. I am talking from discovery/treetrace/someother side? Both TTL=n and TTL=n+1 could tell it is LAG but doesn’t mean that is 1:1 correspondence per link/link member.
IOW, how does sec #3 be performed for the scenario described in last paragraph in sec #4? Better yet, how does one know not to perform if any?
2. Let us take this scenario,  I receive DDMAP at TTL=n with various LAG interface+members and their multipath info. I send TTL=n+1 with a chosen LAG interface and multi path info. At TTL=n+1, what am I going to validate?  For the FEC validation and interface validation, I need to check for specific information, in this case the LAG member. My understanding is, unlike load balanced interfaces, if LAG member is not pinned to specific LSP, the even if one member link fails, it load balances over other member link, because MPLS will only see it as single interface from LIB point of view, no?
3. If the LAG member info be only used to indicate that there is LAG but NOT for validation, then text should be added to describe that. Especially on how FEC interface and label validation is done. 
4. As you said, if a LAG member is pinned to a specific LSP, then we will be using a separate DSMAP/DDMAP anyway. If so, could explain the benefit of LAG with multi path info really used for? In other words, how much beneficial compared to BFD over LAG?
5. When do you consider a LAG member has failed and how do you indicate or inform the requestor?

Will save more questions for later :D

-sam

---

From: "Nobo Akiya (nobo)" <[email-masked]>
Date: Thu, 10 Jul 2014 21:39:44 +0000

Hi Sam,

> For now, have some points which I feel are more imp :D Hope you don't
> mind.

No problem, thanks for continued discussions!

> >> associated with the data path of an LSP. But I think it is difficult
> >> to validate against a given FEC. Because the DDMAP info of LAG member
> >> sent cannot be validated. If it is indeed could be validated, should be
> documented in the ID.
> >
> > Good point. I think Section 4 (Mechanism to Validate L2 ECMP Traversal)
> can use more texts to describe the what can be validated. How about we
> make some changes in the Section 4 (Mechanism to Validate L2 ECMP
> Traversal).
> >
> > [old]
> 1. How does one know same LAG bundle originates at TTL=n terminates at
> TTL=n+1. I am talking from discovery/treetrace/someother side? Both TTL=n
> and TTL=n+1 could tell it is LAG but doesn't mean that is 1:1 correspondence
> per link/link member.

If the LAG is back to back connected, then the LAG should have LAG member links that are 1:1. What we are providing here is a way for the initiator to traverse each LAG member and to allow the initiator perform some sanity check on this 1:1.

> IOW, how does sec #3 be performed for the scenario described in last
> paragraph in sec #4? Better yet, how does one know not to perform if any?

This document is providing a way for the initiator to obtain more information from downstream LSRs having LAG as outgoing/incoming interfaces. What the initiator can do with collected information is a bit of an implementation matter, which is why above suggested text only described that at high level.

> 2. Let us take this scenario,  I receive DDMAP at TTL=n with various LAG
> interface+members and their multipath info. I send TTL=n+1 with a chosen
> LAG interface and multi path info. At TTL=n+1, what am I going to validate?

DDMAP validation procedures are not updated by this document. However, this document should have described how the initiator updates the DDMAP received back from TTL=n before sending the DDMAP to TTL=n+1. If it's indeed this missing information that's generating some questions, I do apologize. Let me plan to add some texts on this.

> For the FEC validation and interface validation, I need to check for specific
> information, in this case the LAG member. My understanding is, unlike load
> balanced interfaces, if LAG member is not pinned to specific LSP, the even if
> one member link fails, it load balances over other member link, because
> MPLS will only see it as single interface from LIB point of view, no?

Again DDMAP validation procedures are not updated by this document, let me be clear on this point in the next revision as well.

> 3. If the LAG member info be only used to indicate that there is LAG but NOT
> for validation, then text should be added to describe that. Especially on how
> FEC interface and label validation is done.

True, I will add that LAG intf info TLV is not used for any FEC validation purpose.

> 4. As you said, if a LAG member is pinned to a specific LSP, then we will be
> using a separate DSMAP/DDMAP anyway. If so, could explain the benefit of
> LAG with multi path info really used for? In other words, how much
> beneficial compared to BFD over LAG?

There are things like LACP and BFD on LAG which is intended to verify the L1/L2/L3 functionality of each LAG member. However, the missing (and important) piece is that when there are in/out label corruption (i.e. inconsistencies between control plane and data plane) that only affects subset (one or more) of the LAG member links, then there is nothing that can deterministically test this at the MPLS layer.

> 5. When do you consider a LAG member has failed and how do you indicate

No DDMAP validation changes, it's up to the initiator to gather and compare the information.

Thanks!

-Nobo

---

From: Sam Aldrin <[email-masked]>
Date: Thu, 10 Jul 2014 14:55:32 -0700

Hi Nobo,

Thanks for the reply. Provides better clarity now.
Inline for few of my comments.

On Thu, Jul 10, 2014 at 2:39 PM, Nobo Akiya (nobo) <[email-masked]> wrote:

> If the LAG is back to back connected, then the LAG should have LAG member
> links that are 1:1. What we are providing here is a way for the initiator
> to traverse each LAG member and to allow the initiator perform some sanity
> check on this 1:1.
>
%sam - Idea is noble. The only problem is, when you check at TTL=n+1, you
do not know whether to expect from LAG member =1 or 2 or something. Hence,
the question.

> This document is providing a way for the initiator to obtain more
> information from downstream LSRs having LAG as outgoing/incoming
> interfaces. What the initiator can do with collected information is a bit
> of an implementation matter, which is why above suggested text only
> described that at high level.
>
%sam - If it is purely for the purpose of knowing a LAG interface exisits
and to indicate packet ingress/egress is that interface, I have no problem.
But that multipath info for each member is the itch. ;-)

> DDMAP validation procedures are not updated by this document. However,
> this document should have described how the initiator updates the DDMAP
> received back from TTL=n before sending the DDMAP to TTL=n+1. If it's
> indeed this missing information that's generating some questions, I do
> apologize. Let me plan to add some texts on this.
>
%sam - NP. Will wait for next version.

> Again DDMAP validation procedures are not updated by this document, let me
> be clear on this point in the next revision as well.
>
> There are things like LACP and BFD on LAG which is intended to verify the
> L1/L2/L3 functionality of each LAG member. However, the missing (and
> important) piece is that when there are in/out label corruption (i.e.
> inconsistencies between control plane and data plane) that only affects
> subset (one or more) of the LAG member links, then there is nothing that
> can deterministically test this at the MPLS layer.
>
%sam - So, does that mean the status reported is per bundled interface and
not per Link member?

-sam

---

From: "Nobo Akiya (nobo)" <[email-masked]>
Date: Thu, 10 Jul 2014 23:44:10 +0000

Hi Sam,

> Sent: Thursday, July 10, 2014 5:56 PM
> To: Nobo Akiya (nobo)
> Cc: [email-masked]

I'm very glad to hear that, and thanks again for many comments!

Also, I'm glad you didn't ask all of these questions at the mic :)

> links that are 1:1. What we are providing here is a way for the initiator to
> traverse each LAG member and to allow the initiator perform some sanity
> check on this 1:1.
> do not know whether to expect from LAG member =1 or 2 or something.
> Hence, the question.

Yes these are good questions. What we do know is that if:
- traversing LAG member=1 at TTL=n will result in LAG member=1' as the incoming interface at TTL=n+1.
- traversing LAG member=2 at TTL=n will result in LAG member=2' as the incoming interface at TTL=n+1.

The error is detected when:
- traversing LAG member=1 at TTL=n will result in LAG member=1' as the incoming interface at TTL=n+1.
- traversing LAG member=2 at TTL=n will result in LAG member=1' as the incoming interface at TTL=n+1.

So it's all up to the initiating LSR to add logic to verify this.

> interfaces. What the initiator can do with collected information is a bit of an
> implementation matter, which is why above suggested text only described
> that at high level.
> %sam - If it is purely for the purpose of knowing a LAG interface exisits and
> to indicate packet ingress/egress is that interface, I have no problem. But
> that multipath info for each member is the itch. ;-)

Yes it's itchy but necessary, as we need to deterministically traverse each LAG member in the MPLS layer, hence the need for multipath.

The primary goal here is to ensure we have a necessary OAM tool to mitigate traffic black hole in MPLS networks, and I'm afraid this is a necessary (or very very beneficial) change.

> %sam - NP. Will wait for next version.

Thanks to you and Nagendra, numbers of beneficial changes were suggested. I should have a revision ready in couple of days, but you'll need to wait until the gate is open again for its publication. If you want to take an early look, drop me a line.

> Again DDMAP validation procedures are not updated by this document, let
> me be clear on this point in the next revision as well.
> 
> > 5. When do you consider a LAG member has failed and how do you
> indicate
> > or inform the requestor?

DDMAP is still per LAG, but some Sub-TLVs (multipath info and interface index) are per LAG member link.

Thanks!

-Nobo

---

From: Sam Aldrin <[email-masked]>
Date: Fri, 11 Jul 2014 10:28:52 -0700

Hi Nobo,

Few comments inline.

> So it's all up to the initiating LSR to add logic to verify this. 
So, the validation is pretty much offline and not built into the OAM packets and initiator has to know topology etc via offline means and specifically whether the links traverse switches or not.
> Yes it's itchy but necessary, as we need to deterministically traverse each LAG member in the MPLS layer, hence the need for multipath.
Itchiness is not because of the info, rather, this is applicable only for a specific topology, which cannot be discovered for validation, via inline means. Ofcourse that doesn't diminish the value.
> The primary goal here is to ensure we have a necessary OAM tool to mitigate traffic black hole in MPLS networks, and I'm afraid this is a necessary (or very very beneficial) change.
OAM is meant for that :D
> Thanks to you and Nagendra, numbers of beneficial changes were suggested. I should have a revision ready in couple of days, but you'll need to wait until the gate is open again for its publication.…
No time to review before the mtg. Sorry.
> DDMAP is still per LAG, but some Sub-TLVs (multipath info and interface index) are per LAG member link.
nod.

-sam