Subject: Re: [mpls] working group last call on draft-ietf-mpls-spring-entropy-label

From: Stewart Bryant <[email-masked]>
Date: Sun, 1 May 2016 09:58:57 +0100

On 30/04/2016 21:05, Carlos Pignataro (cpignata) wrote:
> The RLD relevant to the calculation seems to be a characteristic of 
> the path on a segment, spanning multiple nodes. 

At first sight it would seem to be min(RLD) of all of the possible paths 
that might be taken between the segment end-points. This is quite a 
large calculation, since it requires a per egress node, all paths 
calculation to any possible next segment node calculation.

However is an IPFRR repair is configured then the packet may travel on a 
path that would not normally be considered in a formal shortest path 
determination.

In a network that might configure IPFRR, I think that RLD approximates 
to min(RLD) of the network, and will of course change as the network 
evolves in both the short term (failure, re-instatement, management 
change) and the long term (reconfiguration, new links, new equipment).

The per network approximation of course massively simplifies things.

There was some work in RTGWG that Mike, Alia and I did (sorry if I 
forgot anyone else) on how to advertise such domain wide semi-constants 
since it was needed for convergence time.

- Stewart

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Sun, 1 May 2016 12:05:31 +0000

> On May 1, 2016, at 04:59, Stewart Bryant <[email-masked]> wrote:
> 

If this is used, there's also non insignificant implications. You either:
1. Need the whole network to be upgraded in order to use this feature, or
2. Need to make network-wide assumptions about node's capabilities. 

Thanks,

Carlos.

---

From: Stewart Bryant <[email-masked]>
Date: Sun, 1 May 2016 18:54:39 +0100

> On 1 May 2016, at 13:05, Carlos Pignataro (cpignata) <[email-masked]> wrote:
> 
> 

In an SDN network it's pretty simple of course, since the controller should know all the equipment types and hence parameters.

With a distributed routing system, the change is limited to the control plane - one new TLV - which is not great but reasonably tractable.

- Stewary

---

From: <[email-masked]>
Date: Mon, 2 May 2016 12:59:14 +0000

Hi authors,

Please find below some comments on the document (§4).

- I would expect that network operator may need some flexibility to influence the placement of the EL(s) based on the set of constraints. 
- The need for load-balancing seems significantly dependent on the 1) relative size of the flow (which is typically egress dependent) and 2) the presence of ECMP paths (IOW, I may not care if node N does not see the EL, if node N has a single link/path toward the egress). This may lead to additional requirements (e.g. advertising L2 ECMP which may not be currently visible to the L3 ingress. For spring, eventually draft-ietf-isis-l2bundles may already do the job)

Both points are not touched in the draft, and I think these additions would improve the benefit of the draft.

Thanks,
Regards,
-- Bruno

> Working Group,
> 
> This is to initiate a two week working group last call on draft-ietf-mpls-spring-
> entropy-label.
> 
> Please send your comments to the mpls wg mailing list ([email-masked]).
> 
> There are no IPR disclosures against this document.
> 
> All the authors and contributors (with one exception) have stated on the
> working group mailing list that they are not aware of any other IPRs that
> relates to this draft.
> 
> This working group last call ends May 12, 2016.
> 
> 
> /Loa
> for the MPLS wg chairs
> --
> 
> 
> Loa Andersson                        email: [email-masked]
> Senior MPLS Expert                          [email-masked]
> Huawei Technologies (consultant)     phone: +46 739 81 21 64
> 
> _______________________________________________
> mpls mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/mpls

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: Rob Shakir <[email-masked]>
Date: Tue, 3 May 2016 17:11:30 -0700

[readding mpls@]

> On May 3, 2016, at 14:22, Carlos Pignataro (cpignata) <[email-masked]> wrote:
> 
> There’s another unanswered piece for me, which is: All those routers are advertising ELC and RLDs. As long as R5 and R10 are ELC, then you can add EL/ELIs there.
> BUT, what is the RLD of the *loose* path R1..R5 and of the *loose* path R6..R10? 

RLD is unfortunately something that we have to deal with. There is nothing really in the MPLS specs that tells us that an assumption that one can optimise hardware to have a shallow RLD is OK AIUI, but still we have many systems that have a depth to which they can read.

To this end, the way that we have implemented things is to assume that RLD = min(network). This actually has implications for service implementations, since your customer L2VPN MTU will be impacted by the number of <ELI, EL> pairs you need to allow room for within the core. Since a customer service must work over all paths that reroute might happen over; most transport LSPs will have at least a loose backup path option; and the fact that old hardware is long-lived - then min(network) seems to be the manageable and most robust option to take. 

So, your question, to me - please correct me if you don't feel it is accurate - comes down to whether the entity processing some label can find some entropy within its readable depth. At the point that we have considered min(network) for the RLD then it does not matter whether the next label is a adjacency or node segment we know that within our RLD then there is a <ELI, EL> pair. 

Note that the problem case I see here occurs when the RLD for the network becomes sufficiently low. If it is set to 1 then we need to be able to push more labels at the head end. In the worst case, the iLER can push fewer labels than the least capable LSR can read. At this point, we are stuck - there is nothing that can be done to satisfy all requirements. The approach that the coauthors of this draft discussed tries to do the best it can to make entropy available to all systems,  since we know that not having entropy really hurts for load-sharing - but it also does not mandate that a system does not send a packet that one system in the network cannot hash properly. 

This comes back to the problem you presented around whether all systems must be upgraded or not, the answer is no, rather like the existing EL behaviours. We essentially need to make an assumption about RLDs of the nodes that do not advertise RLD. The first option is assume that RLD = 1, however, this means potentially over-inserting ELs (the worst case would be that we do not know whether they, as midpoints, even know about the EL - and insert them even though they cannot hash on them). The alternative is to assume they know how to hash themselves (or simply are expected not to hash) when EL is not present, since this has been true for systems historically. This results in some cases where we do not have entropy but would desire it. Operators need to make choices themselves as to what to do here - which may include limiting the label stack depth imposed by their policies on a head end, or doing LSP stitching or segment-expansion somewhere else in the network. 

There are many procedures here which come down to combining the various standardised and non-standardised procedures that may be available to an operator to balance their requirements for entropy against their requirements for path strictness, and the number of approaches deployed in their network. 

Please let me know if there are things here that not clear, I am happy to expand on them. 

r.

---

From: Gmail <[email-masked]>
Date: Wed, 4 May 2016 07:48:33 +0100

When you get to an RLD of one, you ought to look at whether the (sub)stack is smaller if you use adjacency labels. 

Stewart

Sent from my iPad

> On 4 May 2016, at 01:11, Rob Shakir <[email-masked]> wrote:
> 
> [readding mpls@]
>

---

From: <[email-masked]>
Date: Wed, 4 May 2016 12:45:01 +0000

Hi authors,

As an additional comment, I'd like a see a more normative definition of "RLD" as this is an important point for the interop.
In particular, the grey zone is whether RLD is before the ELI, or include the EL, or include both the EL and ELI. Also, is RLD only applicable when EL is used or is it also applicable when load-balancing is performed based on the hashing of a stack of labels (a stack of RLD labels)?

>From the description of the text, I think I read that both EL and ELI needs to be within the RLD. But on the other hand, there has been a discussion on the mailing list about RLD=1, which would not even allow reading the ELI label.

Thanks,
Regards,
Bruno

> From: DECRAENE Bruno IMT/OLN > Sent: Monday, May 02, 2016 2:59 PM
> 
> Hi authors,
> 
> - I would expect that network operator may need some flexibility to
> influence the placement of the EL(s) based on the set of constraints.
> - The need for load-balancing seems significantly dependent on the 1)
> relative size of the flow (which is typically egress dependent) and 2) the
> presence of ECMP paths (IOW, I may not care if node N does not see the EL,
> if node N has a single link/path toward the egress). This may lead to
> additional requirements (e.g. advertising L2 ECMP which may not be
> currently visible to the L3 ingress. For spring, eventually draft-ietf-isis-
> l2bundles may already do the job)
>

---

From: <[email-masked]>
Date: Wed, 4 May 2016 13:19:31 +0000

Hi,

I think RLD is clearly expressed in :

"An LSR may have a limitation in its ability to read and process the

   label stack in order to do multipath load balancing.  This limitation

   expressed in terms of the number of label stack entries that the LSR

   can read is henceforth referred to as the Readable Label Depth (RLD)

   capability of that LSR.  If an EL does not occur within the RLD of an

   LSR in the label stack of the MPLS packet that it receives, then it

   would lead to poor load balancing at that LSR."

It refers to the number of label that the LSR can read. So as a consequence, if you want to be able to read the EL, ELI+EL must be within the RLD. Otherwise the also described behavior will apply (poor loadbalancing).

Is it fine ?

Thanks

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
Sent: Wednesday, May 04, 2016 14:45
To: [email-masked]; [email-masked]
Cc: [email-masked]; Loa Andersson
Subject: RE: [mpls] working group last call on draft-ietf-mpls-spring-entropy-label

Hi authors,

Thanks,

Regards,

Bruno

> relative size of the flow (which is typically egress dependent) and 2)

> the presence of ECMP paths (IOW, I may not care if node N does not see

> the EL, if node N has a single link/path toward the egress). This may

> lead to additional requirements (e.g. advertising L2 ECMP which may

> not be currently visible to the L3 ingress. For spring, eventually

> draft-ietf-isis- l2bundles may already do the job)

> Both points are not touched in the draft, and I think these additions

> would improve the benefit of the draft.

> > This is to initiate a two week working group last call on

> > draft-ietf-mpls-

> spring-

> > Please send your comments to the mpls wg mailing list ([email-masked]<mailto:[email-masked]>).

> > All the authors and contributors (with one exception) have stated on

> > the working group mailing list that they are not aware of any other

> > IPRs that relates to this draft.

> > Loa Andersson                        email: [email-masked]<mailto:[email-masked]>

> > Senior MPLS Expert                          [email-masked]<mailto:[email-masked]>

> > [email-masked]<mailto:[email-masked]>

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.

Thank you.

---

From: <[email-masked]>
Date: Wed, 4 May 2016 13:45:09 +0000

Hi Stéphane,

Thanks for the quote. I had read that sentence, and I think it could be made more precise.
e.g.

OLD: This limitation expressed in terms of the number of label stack entries that the LSR  can read is henceforth referred to as the Readable Label Depth (RLD)  capability of that LSR.

NEW: This limitation expressed in terms of the number of label stack entries that the LSR  can read. This document defines the Readable Label Depth (RLD) as the number of labels that a transit LSR can read for load-balancing purpose.  When EL is used, both ELI and EL MUST be within the RLD, in order for the EL to be used during load-balancing.

Additionally, you did not answer my second related question:

> is RLD only applicable when EL is used or is it also applicable when load-balancing is performed based on the hashing of a stack of labels (a stack of RLD labels)?

A related point is whether a transit LSR not supporting EL, MAY/SHOULD/MUST NOT advertise the RLD.  (because transit LSR don't need to support EL to benefit from the improved load-balancing  provided by EL)

The question could further be refined when MPLS carry an IP packet and the LSR perform load-balancing based on some hashing of this inner IP packet.

Thanks,

--Bruno

From: LITKOWSKI Stephane OBS/OINIS
Sent: Wednesday, May 04, 2016 3:20 PM
To: DECRAENE Bruno IMT/OLN; [email-masked]; [email-masked]
Cc: [email-masked]; Loa Andersson
Subject: RE: [mpls] working group last call on draft-ietf-mpls-spring-entropy-label

Hi,

Is it fine ?

Thanks

-----Original Message-----
From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]
Sent: Wednesday, May 04, 2016 14:45
To: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>; Loa Andersson
Subject: RE: [mpls] working group last call on draft-ietf-mpls-spring-entropy-label

Hi authors,

Thanks,

Regards,

Bruno

Thank you.

---

From: Sri <[email-masked]>
Date: Wed, 4 May 2016 06:55:54 -0700

Hi Bruno,

On Wed, May 4, 2016 at 6:45 AM, <[email-masked]> wrote:

> OLD: This limitation expressed in terms of the number of label stack
> entries that the LSR  can read is henceforth referred to as the Readable
> Label Depth (RLD)  capability of that LSR.
>
> NEW: This limitation expressed in terms of the number of label stack
> entries that the LSR  can read. This document defines the Readable Label
> Depth (RLD) as the number of labels that a transit LSR can read for
> load-balancing purpose.  When EL is used, both ELI and EL MUST be within
> the RLD, in order for the EL to be used during load-balancing.
>

The definition of RLD must not specify conditions on when an EL must be
used. RLD is a per LSR characteristic. Any recommendations on placing the
EL must be specified outside of the definition.

Sri

---

From: <[email-masked]>
Date: Wed, 4 May 2016 14:09:10 +0000

Hi Sri,

> The definition of RLD must not specify conditions on when an EL must be used. RLD is a per LSR characteristic. Any recommendations on placing the EL must be specified outside of the definition.

IMO, the goal of RLD is specifically to say that the EL needs to be within the RLD in order for the EL to be used for load-balancing. Otherwise, can you clarify the goal of signaling this RLD?

(That’s a different point compared to specifying the position of the EL in the stack, which is a freedom of the ingress. (although a secondary goal of this goal seems to be to reduce this freedom))

-- Bruno

From: Sri [mailto:[email-masked]]
Sent: Wednesday, May 04, 2016 3:56 PM
To: DECRAENE Bruno IMT/OLN
Cc: LITKOWSKI Stephane OBS/OINIS; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [mpls] working group last call on draft-ietf-mpls-spring-entropy-label

Hi Bruno,

On Wed, May 4, 2016 at 6:45 AM, <[email-masked]<mailto:[email-masked]>> wrote:
Hi Stéphane,

The definition of RLD must not specify conditions on when an EL must be used. RLD is a per LSR characteristic. Any recommendations on placing the EL must be specified outside of the definition.

Sri

---

From: Sri <[email-masked]>
Date: Wed, 4 May 2016 10:07:43 -0700

Hi Bruno,

Thanks for the clarification. On my first read of your last sentence under
NEW it had seemed to me that it is placing a restriction on the ingress. On
reading your clarification I understand that it is not your intent. So I
accept the comment for the NEW definition but suggest that that we refer to
RFC6790 rather than re-stating that EL is used for load-balancing. So how
about this for sec 4 para2 (changes highlighted) -

   An LSR may have a limitation on the depth of the label stack that
it can read and process
    in order to do multipath load balancing as described in [RFC6790].
This limitation
   expressed in terms of the number of label stack entries that the LSR
   can read is defined as the Readable Label Depth (RLD)
   capability of that LSR.  If an EL does not occur within the RLD of an
   LSR in the label stack of the MPLS packet that it receives, then it
   would lead to poor load balancing at that LSR.  The RLD of an LSR is
   a characteristic of the forwarding plane of that LSR's implementation
   and determining it is outside the scope of this document.

I have not used your suggested sentence "...both the ELI and EL MUST be.."
since the ELI is placed above the EL and if the LSR can read the EL, it can
obviously have read the ELI.

Thanks
Sri

On Wed, May 4, 2016 at 7:09 AM, <[email-masked]> wrote:

> IMO, the goal of RLD is specifically to say that the EL needs to be within
> the RLD in order for the EL to be used for load-balancing. Otherwise, can
> you clarify the goal of signaling this RLD?
>
>
>
> (That’s a different point compared to specifying the position of the EL in
> the stack, which is a freedom of the ingress. (although a secondary goal of
> this goal seems to be to reduce this freedom))
>
>
>
> -- Bruno
>
>
>
> *From:* Sri [mailto:[email-masked]]
> *Sent:* Wednesday, May 04, 2016 3:56 PM
> *To:* DECRAENE Bruno IMT/OLN
> *Cc:* LITKOWSKI Stephane OBS/OINIS;
> [email-masked]; [email-masked];
> [email-masked]
> *Subject:* Re: [mpls] working group last call on
> draft-ietf-mpls-spring-entropy-label
>
>
>
> Hi Bruno,
>
>
>

---

From: <[email-masked]>
Date: Fri, 6 May 2016 09:34:41 +0000

Hi Sri,

Thanks for the follow up.
As previously asked, is RLD specific to the use of the entropy label or not? Let’s remember that multipath load balancing existed before EL, and that transit LSR not compliant with EL can still benefit from the presence of the EL label.
If RLD is not specific to EL, I’m not certain that adding the reference to RFC6790 is the best way to clarify.
In all cases, this point should be stated in the document defining RLD. (which may be this one, of maybe the IGP extensions if draft-ietf-mpls-spring-entropy-label gets informational).

> I have not used your suggested sentence "...both the ELI and EL MUST be.." since the ELI is placed above the EL and if the LSR can read the EL, it can obviously have read the ELI.
Agreed. Yet mentioning both labels helps the reader to remember that the limit for the number of “useful” labels is RLD-2. (I mean useful for steering the packet, I’m not implying that EL is not useful). This may be obvious for people coming from the EL space, but less for people coming from the spring space, especially since the text often refers to a/the (i.e. one) EL label. But this points could indeed be stated elsewhere.

Thanks,
-- Bruno

Sent: Wednesday, May 04, 2016 7:08 PM

Hi Bruno,

Thanks for the clarification. On my first read of your last sentence under NEW it had seemed to me that it is placing a restriction on the ingress. On reading your clarification I understand that it is not your intent. So I accept the comment for the NEW definition but suggest that that we refer to RFC6790 rather than re-stating that EL is used for load-balancing. So how about this for sec 4 para2 (changes highlighted) -

   An LSR may have a limitation on the depth of the label stack that it can read and process

    in order to do multipath load balancing as described in [RFC6790].  This limitation

I have not used your suggested sentence "...both the ELI and EL MUST be.." since the ELI is placed above the EL and if the LSR can read the EL, it can obviously have read the ELI.

Thanks
Sri

On Wed, May 4, 2016 at 7:09 AM, <[email-masked]<mailto:[email-masked]>> wrote:
Hi Sri,

-- Bruno

From: Sri [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Wednesday, May 04, 2016 3:56 PM
To: DECRAENE Bruno IMT/OLN
Cc: LITKOWSKI Stephane OBS/OINIS; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [mpls] working group last call on draft-ietf-mpls-spring-entropy-label

Hi Bruno,

Sri

Thank you.

---

From: Sri <[email-masked]>
Date: Fri, 6 May 2016 07:21:49 -0700

Hi Bruno,

See inline.

On Fri, May 6, 2016 at 2:34 AM, <[email-masked]> wrote:

> As previously asked, is RLD specific to the use of the entropy label or
> not? Let’s remember that multipath load balancing existed before EL, and
> that transit LSR not compliant with EL can still benefit from the presence
> of the EL label.
>

Sri> RLD is not specific to EL. RFC6790 Section 1 gives a very good
explanation of how load-balancing is done (even before it gets to EL) and
such text is not present in other RFCs. But I can understand how this
reference can be interpreted as EL specific. So I am fine with not using
that reference as I suggested.

> In all cases, this point should be stated in the document defining RLD.
> (which may be this one, of maybe the IGP extensions if draft-ietf-mpls-spring-entropy-label
> gets informational).
>
>
>
> > I have not used your suggested sentence "...both the ELI and EL MUST
> be.." since the ELI is placed above the EL and if the LSR can read the EL,
> it can obviously have read the ELI.
>
> Agreed. Yet mentioning both labels helps the reader to remember that the
> limit for the number of “useful” labels is RLD-2. (I mean useful for
> steering the packet, I’m not implying that EL is not useful). This may be
> obvious for people coming from the EL space, but less for people coming
> from the spring space, especially since the text often refers to a/the
> (i.e. one) EL label. But this points could indeed be stated elsewhere.
>
>
Sri> I see your point. I will include that.

> *Sent:* Wednesday, May 04, 2016 7:08 PM
>

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Sat, 7 May 2016 16:58:10 +0000

Hi Stephane,

I had seen this definition, and had this follow-on thought:

The RLD is defined as a node attribute (per-LSR), as the min-LSRs of all line-cards. So for a network, you have RLDi (with i a node index). However, draft-ietf-mpls-spring-entropy-label is treating RLD as a single value. That seems like something that should be deeply considered. Which RLDi is used in the pseudocode?

If there’s no RLD advertised for a node, what’s assumed? Three, zero, other?

Are there FRR considerations for the calculation, since a node repair can add labels, etc?

Also, as Bruno also pointed out, the RLD is a generic attribute beyond EL/ELI, correct? In that case, how can a node advertise that it can read deep labels to RLD, but it does not support ELC? Take a FAT-PW case for example.

Thanks!

— Carlos.

> On May 4, 2016, at 9:19 AM, [email-masked] wrote:
> 
> Hi,
> 
> From: [email-masked] <mailto:[email-masked]> [mailto:[email-masked] <mailto:[email-masked]>]
> To: [email-masked] <mailto:[email-masked]>; [email-masked] <mailto:[email-masked]>
> Cc: [email-masked] <mailto:[email-masked]>; Loa Andersson
> > > Please send your comments to the mpls wg mailing list ([email-masked] <mailto:[email-masked]>).
> > >
> > > Loa Andersson                        email: [email-masked] <mailto:[email-masked]>
> > > Senior MPLS Expert                          [email-masked] <mailto:[email-masked]>
> > > [email-masked] <mailto:[email-masked]>
> > > https://www.ietf.org/mailman/listinfo/mpls <https://www.ietf.org/mailman/listinfo/mpls>
> 
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/mpls <https://www.ietf.org/mailman/listinfo/mpls>

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Sat, 7 May 2016 17:20:17 +0000

Hi, Rob,

> On May 3, 2016, at 8:11 PM, Rob Shakir <[email-masked]> wrote:
> 
> [readding mpls@]
> 
> To this end, the way that we have implemented things is to assume that RLD = min(network).

I agree with the realities of hardware.

However, that’s a big assumption and design principle that I could not find mentioned in draft-ietf-mpls-spring-entropy-label. Did I miss it? (in fact, I could not find “min”).

Additionally, if RLD = min (RLDi for i=..network), what happens if there’s a single node that does not support RLD?

> This actually has implications for service implementations, since your customer L2VPN MTU will be impacted by the number of <ELI, EL> pairs you need to allow room for within the core. Since a customer service must work over all paths that reroute might happen over; most transport LSPs will have at least a loose backup path option; and the fact that old hardware is long-lived - then min(network) seems to be the manageable and most robust option to take.
> 

I do not disagree with that pragmatic approach. But there are implications which ought to be spelled out.

> So, your question, to me - please correct me if you don't feel it is accurate - comes down to whether the entity processing some label can find some entropy within its readable depth. At the point…

The paraphrasing is accurate, but I feel it does matter. Say you have a series of Adj-labels. For example (N=node, A=Adj): N1-A1-A2-A3-N2-Payload. N1 is top, N2 is bottom of stack. There’s an RLD of 4. Placing a EL/ELI after A2 or A3 is effectively of null value practically. (E.g., A2 and A3 are specific links, but N1 and N2 are long loose paths). The places where LB could happen while the packet is switched with N1 and N2 will not have an EL/ELI readable, and A2 A3 won’t use EL/ELI.

> 
> Note that the problem case I see here occurs when the RLD for the network becomes sufficiently low. If it is set to 1 then we need to be able to push more labels at the head end. In the worst case, the iLER can push fewer labels than the least capable LSR can read. At this point, we are stuck

I think RLD lower than 3 will get us in the same place.

> - there is nothing that can be done to satisfy all requirements.

I agree, so the function is one to maximize effect, not satisfy all requirements.

> The approach that the coauthors of this draft discussed tries to do the best it can to make entropy available to all systems,  since we know that not having entropy really hurts for load-sharing - but it also does not mandate that a system does not send a packet that one system in the network cannot hash properly.

Right — hard requirement is that an EL/ELI gets placed in a way where the receiving node will understand it. This is important or the packet gets dropped.
The rest is the best that can be done, given RLDi of nodes in the (potential) path, and given how many nodes are traveled using a single label stack entry. I think the latter is not considered or mentioned, and the former is implicitly oversimplified.

> 
> This comes back to the problem you presented around whether all systems must be upgraded or not, the answer is no, rather like the existing EL behaviours. We essentially need to make an assumption about RLDs of the nodes that do not advertise RLD.

Absolutely agree. My point is only that those assumption should be listed as they carry operational considerations.

> The first option is assume that RLD = 1,

Sorry, here I got lost: RLD of 1 means there’s no point in inserting any EL/ELI pairs. An RLD (min RLD of the whole path or network) of 3 means insert in-between every label. Or did I misunderstood?

> however, this means potentially over-inserting ELs (the worst case would be that we do not know whether they, as midpoints, even know about the EL - and insert them even though they cannot hash on them). The alternative is to assume they know how to hash themselves (or simply are expected not to hash) when EL is not present, since this has been true for systems historically.

To me, and again I may be misunderstanding, RLD does not mean that a node understands Entropy Labels. A node with an RLD of 7 can still not use EL/ELI and only use the deepest readable label, conceptually.

This, I know, compounds more the complexity of the problem this draft is aiming at solving.

> This results in some cases where we do not have entropy but would desire it. Operators need to make choices themselves as to what to do here - which may include limiting the label stack depth imposed by their policies on a head end, or doing LSP stitching or segment-expansion somewhere else in the network.
> 

As you can see, I still have either questions or misunderstandings, or I am pointing out things that might be broken.

Please do let me know if anything I commented about is not clear.

Thanks!

— Carlos.

---

From: Stewart Bryant <[email-masked]>
Date: Sat, 7 May 2016 21:47:50 +0100

> Are there FRR considerations for the calculation, since a node repair 

Both a link repair and a node repair can add labels.

I would have thought that the ingress node would assume no repair (the 
normal case) and the repairing node would have worried about RLD - 
assuming it worried about ECMPing the repair at all.

- Stewart

---

From: Stewart Bryant <[email-masked]>
Date: Sat, 7 May 2016 21:52:46 +0100

> The paraphrasing is accurate, but I feel it does matter. Say you have a series of Adj-labels. For example (N=node, A=Adj): N1-A1-A2-A3-N2-Payload. N1 is top, N2 is bottom of stack. There’s an RLD o…
Why would you put an ELI in the middle of a set of adj labels? Surely 
you want the packet to unconditionally go through those labels?

- Stewart

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Sun, 8 May 2016 01:19:22 +0000

Stewart, see inline. 

> On May 7, 2016, at 16:52, Stewart Bryant <[email-masked]> wrote:
> 
> 
> 
> Why would you put an ELI in the middle of a set of adj labels? Surely you want the packet to unconditionally go through those labels?
> 

Well, that was exactly my point...

What I tried to exemplify is that the algorithm does not differentiate between Node labels and Adj labels, and I am asking if it should...

Thumb typed by Carlos Pignataro.
Excuze typofraphicak errows

---

From: Gmail <[email-masked]>
Date: Sun, 8 May 2016 07:39:25 +0100

Sent from my iPad

> On 8 May 2016, at 02:19, Carlos Pignataro (cpignata) <[email-masked]> wrote:
> 
> Stewart, see inline. 
> 

Carlos,

Sorry I misunderstood your point.

A point that I made much earlier in the thread was that it was a local matter and that although the properties of various algorithms might be discussed, no algorithm should be recommended.

Stewart

---

From: <[email-masked]>
Date: Mon, 9 May 2016 15:29:03 +0000

Hi Carlos,

In your example (N1-A1-A2-A3-N2-Payload), if you consider RLD=4 network wide, with a bottom_to_top EL insertion you may have (considering no limit in push) : N1-ELI-EL-A1-A2-ELI-EL-A3-N2-ELI - EL - Payload. 
With a top_to_bottom insertion, you may have : N1-N2-ELI-EL-A1-A2-ELI-EL-N2-ELI-EL-Payload

Keeping EL for Adj-SID is still good as Adj-SID can represent a bundle of links (L2 or L3). In theory, there is a bigger advantage for Node-SID, but only if we are sure that all nodes along the path can process the ELI/EL. Now from a practical point of view, we have no insurance.

Best Regards,

-----Original Message-----
From: mpls [mailto:[email-masked]] On Behalf Of Carlos Pignataro (cpignata)
Sent: Sunday, May 08, 2016 03:19
To: Stewart Bryant
Cc: [email-masked]
Subject: Re: [mpls] working group last call on draft-ietf-mpls-spring-entropy-label

Stewart, see inline. 

>> The paraphrasing is accurate, but I feel it does matter. Say you have a series of Adj-labels. For example (N=node, A=Adj): N1-A1-A2-A3-N2-Payload. N1 is top, N2 is bottom of stack. There's an RLD of 4. Placing a EL/ELI after A2 or A3 is effectively of null value practically. (E.g., A2 and A3 are specific links, but N1 and N2 are long loose paths). The places where LB could happen while the packet is switched with N1 and N2 will not have an EL/ELI readable, and A2 A3 won't use EL/ELI.

---

From: <[email-masked]>
Date: Thu, 19 May 2016 07:54:36 +0000

Hi Carlos,

Sorry I missed to answer this comment …

Please find some inline comments.

From: Carlos Pignataro (cpignata) [mailto:[email-masked]]
Sent: Saturday, May 07, 2016 18:58
To: LITKOWSKI Stephane OBS/OINIS
Cc: DECRAENE Bruno IMT/OLN; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [mpls] working group last call on draft-ietf-mpls-spring-entropy-label

Hi Stephane,

The RLD is defined as a node attribute (per-LSR), as the min-LSRs of all line-cards. So for a network, you have RLDi (with i a node index). However, draft-ietf-mpls-spring-entropy-label is treating RLD as a single value. That seems like something that should be deeply considered. Which RLDi is used in the pseudocode?
[SLI] Agree, we already discussed this offline and pseudocode should be enhanced to be more clear on that point.

Are there FRR considerations for the calculation, since a node repair can add labels, etc?
[SLI] FRR is not taken into account. As FRR is usually a new push (except for TE DETOUR afaik), an implementation may decide to push additional EL for FRR independently. But IMO, as we can see with implementations, ECMP is not well addressed for FRR yet. Only usage of LAGs will use loadbalancing during FRR.

Also, as Bruno also pointed out, the RLD is a generic attribute beyond EL/ELI, correct? In that case, how can a node advertise that it can read deep labels to RLD, but it does not support ELC? Take a FAT-PW case for example.
[SLI] For ISIS , you have two different subTLVs. You should be able to advertise one and not the other. I agree that it’s not clearly expressed in the isis draft.

Thanks!

— Carlos.

On May 4, 2016, at 9:19 AM, [email-masked]<mailto:[email-masked]> wrote:

Hi,

Is it fine ?

Thanks

-----Original Message-----
From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]
Sent: Wednesday, May 04, 2016 14:45
To: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>; Loa Andersson
Subject: RE: [mpls] working group last call on draft-ietf-mpls-spring-entropy-label

Hi authors,

Thanks,
Regards,
Bruno

Thank you.

---

From: Loa Andersson <[email-masked]>
Date: Tue, 24 May 2016 15:40:08 +0800

Working Group, Authors,

This wglc is closed.

I let it stay open for a while since I thought the discussion on RLD
had not really converge. It now seem to have stopped.

There are there groups of comments:

- comments on the treatment of the RLD and the protocol to
   negotiate it
- comments that relates to whether a correct treatment of RLDs
   have an impact on whether the draft can become Informational or not
- other comments

Can the authors please address the comments, and work with the reviewers
(on the mailing list) to converge on a joint text.

/Loa

mpls wg co-chair

On 2016-04-23 15:53, Loa Andersson wrote:
> draft-ietf-mpls-spring-entropy-label.
>

---

From: Sri <[email-masked]>
Date: Tue, 24 May 2016 06:28:50 -0700

Hi Loa,

We are already in the process of updating the draft and working with the
reviewers. We will post a new version in a few weeks.

Thanks
Sri