Subject: [trill] Suggestion on RFC 6325 section 4.5.2, single parent ECMP link selection.

From: "Ramkumar Parameswaran (ramkumar)" <[email-masked]>
Date: Wed, 22 Aug 2012 15:34:47 +0000

Hi,

With regard to the Parallel Link check in section 4.5.2 of RFC 6325, item 3 talks about how to select links in multi-destination trees when a node in a multi-destination tree has a single parent connected to it by multiple links. The section in the RFC identifies how to select a link in the bundle,over which all multi-destination traffic may traverse, when the links are of type P2P or LAN links with pseudonode suppressed. Specifically, a single link is selected and traffic for all multi-destination trees is carried over the selected link.

When a node is connected to a single parent across several LAN links which advertise a pseudonode (when pseudonode is not suppressed), RFC 6325 section 4.5.1 already specifies a way to load balance multi-destination traffic across these links by pulling a unique link into each multi-destination tree.

This is illustrated in the following example

         |   Node B      | -> upstream, closer to tree root.

         -----------------

             /  |   | \

            |   |   |  |

            |01 02  |03|04

            |   |   |  |

             \  |   |  /

         -----------------

         |   Node A      |

Consider Node A and node B being connected to each other on 4 independent LAN links. Each of the four links above is a LAN link with a DRB elected and pseudo-node advertised. The number to the right of the link identifies the pseudo-node id operational on the link. In this situation, per 4.5.1, since seven byte system id is considered in load-balancing links across trees, other factors holding, link 01 will be assigned to Tree 1, link 02

will be assigned to tree 2 and so on.

However, if the links were P2P links with 01, 02, 03, and 04 being the extended circuit id of the node with higher system id (B, say), then per 4.5.2, link 04 would be pulled into all multi-destination trees, with no load-balancing on any of the other links.

For P2P links and pseudonode suppressed LAN links, we were wondering whether the following variant can be accommodated, so that multi-destination traffic can be  spread over more links, with better link utilization:

Modified Approach: Once a dominant link-type is identified using the tie-break rule specified in item 3 (P2P or LAN link with suppressed pseudo node), if there is more than one link of the dominant type, then such links are arranged in ascending order of parent's Extended Circuit ID or Pseudonode ID,  given serial numbers starting with 0, and assigned to trees with a logic similar to the multi-parent ECMP case:

Link on Tree T = (T-1) mod N where N is the number of parallel links

The parallel link check in section 4.5.2 must be modified to allow only the

adjacency that is selected for the tree that the packet arrived on.

No change in metric advertisement is needed with regard to the rest of the network, and in line with what is mentioned in 6325, the notion of mapping continues to remain local to the two adjacent nodes RB1-RB2.

Please let us know if the approach suggested above is feasible or if we may have missed something.

Thanks,

Ramkumar

---

From: Donald Eastlake <[email-masked]>
Date: Wed, 22 Aug 2012 14:12:26 -0400

Hi Ramkumar,

On Wed, Aug 22, 2012 at 11:34 AM, Ramkumar Parameswaran (ramkumar)
<[email-masked]> wrote:

> Hi,
>
> With regard to the Parallel Link check in section 4.5.2 of RFC 6325,
> item 3 talks about how to select links in multi-destination trees
> when a node in a multi-destination tree has a single parent
> connected to it by multiple links. The section in the RFC identifies
> how to select a link in the bundle, over which all multi-destination
> traffic may traverse, when the links are of type P2P or LAN links
> with pseudonode suppressed. Specifically, a single link is selected
> and traffic for all multi-destination trees is carried over the
> selected link.

Generally speaking, if the links are equal speed P2P Ethernet links, I
would recommend just aggregating them with 802.1AX, which resolves
these issues.

> When a node is connected to a single parent across several LAN links
> which advertise a pseudonode (when pseudonode is not suppressed),
> RFC 6325 section 4.5.1 already specifies a way to load balance
> multi-destination traffic across these links by pulling a unique
> link into each multi-destination tree.
>
>           |   Node B      | -> upstream, closer to tree root
>           -----------------
>              /  |   |  \
>             |   |   |   |
>             |01 02  |03 |04
>             |   |   |   |
>              \  |   |  /
>          -----------------
>          |   Node A      |
>

> Consider Node A and node B being connected to each other on 4
> independent LAN links. Each of the four links above is a LAN link
> with a DRB elected and pseudo-node advertised. The number to the
> right of the link identifies the pseudo-node id operational on the
> link. In this situation, per 4.5.1, since seven byte system id is
> considered in load-balancing links across trees, other factors
> holding, link 01 will be assigned to Tree 1, link 02 will be
> assigned to tree 2 and so on.
>
> However, if the links were P2P links with 01, 02, 03, and 04 being
> the extended circuit id of the node with higher system id (B, say),
> then per 4.5.2, link 04 would be pulled into all multi-destination
> trees, with no load-balancing on any of the other links.

This provision in RFC 6325 didn't come out of nowhere. It was
requested due, if I recall correctly, to limitation of some hardware
in implementing the Reverse Path Forwarding Check for
multi-destinaiton frames.

> For P2P links and pseudonode suppressed LAN links, we were wondering
> whether the following variant can be accommodated, so that
> multi-destination traffic can be spread over more links, with better
> link utilization:

When there are multiple parallel unlabeled links visible to TRILL
between two RBridges, this is reported to the campus as a single
adjacency. If the two RBridges can come to agreement, it is really a
private matter between them how those RBridges handle it. The
provisions in RFC 6325 are just a minimum to assure that the default
behavior will interoperate, taking into account the hardware
limitations that, at the time, indicated only one of such links could
be used for multi-destinaiton frames.

There are lots of possible optimizations that are left out so that
implementations have room to distinguish themselves. For example, the
sending RBridge gets to chose which of the unlabeled parallel links to
send unicast frames on. In doing so, it might want to avoid the link
chosen for multi-destination.

> Modified Approach: Once a dominant link-type is identified using the
> tie-break rule specified in item 3 (P2P or LAN link with suppressed
> pseudo node), if there is more than one link of the dominant type,
> then such links are arranged in ascending order of parent's Extended
> Circuit ID or Pseudonode ID, given serial numbers starting with 0,
> and assigned to trees with a logic similar to the multi-parent ECMP
> case:
>
> The parallel link check in section 4.5.2 must be modified to allow
> only the adjacency that is selected for the tree that the packet
> arrived on.

I don't see any particular problem with such a change if we could get
assurance that the previous reported hardware limitation was no longer
a factor. However, it is pretty hard to prove a negative. And if you
could, so different multi-destination frames could be sent over
different links of such a bundle of unlabeled links, then why have any
limiations at all? Just permit the sending RBridge to send over
whichever link its private hash algorithm and weighting indicate...

In fact, the capability of receiving multi-destination frames over any
link in such a unlabeled bundle could be advertised by a capability
bit, which would make such a relaxation of the Parallel Links Check
backwards compatible. If the capability was not signalled, you would
still have to do what is in RFC 6325. This seems like a reasonable way
to go to me.

Thanks,
Donald
=============================
 Donald E. Eastlake 3rd   +1-508-333-2270 (cell)
 155 Beaver Street, Milford, MA 01757 USA
 [email-masked]

> No change in metric advertisement is needed with regard to the rest of the
> network, and in line with what is mentioned in 6325, the notion of mapping
> continues to remain local to the two adjacent nodes RB1-RB2.
>

---

From: Anoop Ghanwani <[email-masked]>
Date: Wed, 22 Aug 2012 11:34:21 -0700

> multi-destinaiton frames.

I do remember discussions around this issue, but can't remember
exactly what it was...I think it might have had to do with trying
to use these links for ECMP within the same tree.

Anoop

---

From: "Ramkumar Parameswaran (ramkumar)" <[email-masked]>
Date: Fri, 24 Aug 2012 02:43:06 +0000

Hi Donald,

Thanks for the consideration. One comment inline, tagged [**]:
________________________________________
From: [email-masked] [[email-masked]] on behalf of Donald Eastlake [[email-masked]]
Sent: Wednesday, August 22, 2012 11:12 AM
To: Ramkumar Parameswaran (ramkumar)
Cc: [email-masked]
Subject: Re: [trill] Suggestion on RFC 6325 section 4.5.2, single parent ECMP link selection.

Hi Ramkumar,

[**]: Agree in general, but wanted to point out the following - 
the unrestricted approach of sending on any link in the bundle (on a tree) may have issues with RPF  (Parallel link) check.  
Waiving the RPF check may not be a good option, for whatever reasons that RPF check exists in networks today. If the RPF check is 
retained, the hardware may have to maintain a list of interfaces to cross check against and this would be expensive - Table space for a single interface is probably the norm. 
It may also dilute the check. Or, the load-distribution hash may need to be applied on the receive side before the RPF check as well, again this may be expensive in 
hardware. 

When it was discussed in the past, were you considering multiple links in the same tree, or one link per tree? 
The approach we are suggesting needs only one RPF interface (per tree), and makes the P2P and pseudo-node suppressed LAN cases 
symmetric with the treatment of regular LAN.  Treating it as optional tied to a capability bit is fine. 

thanks,

ramkumar 

Thanks,
Donald

> Please let us know if the approach suggested above is feasible or if we may
_______________________________________________
trill mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/trill

---

From: Donald Eastlake <[email-masked]>
Date: Fri, 24 Aug 2012 15:34:35 -0400

Hi Ramkumar,

On Thu, Aug 23, 2012 at 10:43 PM, Ramkumar Parameswaran (ramkumar)
<[email-masked]> wrote:
>>> ...

> the unrestricted approach of sending on any link in the bundle (on a
> tree) may have issues with RPF (Parallel link) check.  Waiving the
> RPF check may not be a good option, for whatever reasons that RPF
> check exists in networks today. If the RPF check is retained, the
> hardware may have to maintain a list of interfaces to cross check
> against and this would be expensive - Table space for a single
> interface is probably the norm.  It may also dilute the check. Or,
> the load-distribution hash may need to be applied on the receive
> side before the RPF check as well, again this may be expensive in
> hardware.

I think you have re-discovered the reason for the current provisions
in the specification. Of course, if the RPF check used a pseudo port
number which was the same for all ports in such a TRILL port group,
then it would not take any more RPF table space... So this sort of
thing is critically dependent on just how the fast path hardware is
implemented.

> When it was discussed in the past, were you considering multiple
> links in the same tree, or one link per tree?

I don't remember. You can go back and look in the mail archives if you
want.

>  The approach we are suggesting needs only one RPF interface (per
> tree), and makes the P2P and pseudo-node suppressed LAN cases
> symmetric with the treatment of regular LAN.  Treating it as
> optional tied to a capability bit is fine.

I think in practice this situation would usually be solved by either
using link aggregation or, for LAN links, by enabling pseudo-nodes on
the parallel links. (With pseudo-nodes, you would typically get ECMP
between the fastest links for unicast and different trees on different
ones of the fastest links for multi-destination, with slower links
ignored.) Furthermore, if both RBridges want to cooperate (like they
are from the same manufacturer or something), they can do whatever
fancy thing they want.

As I said, the current provisions are to get something that will
operate and interoperate correctly if such a configuration actually
occurs. Is it worth the complexity to have additional logic, a
configuration bit, etc., to make it somewhat better?

In fact, you would need additional suggestions or provisions to really
improve things for a wide variety of conditions. For example, what
should you do if there are four unlabeled links between RB1 and RB2
that are 10Meg, 100Meg, 1Gig, and 10Gig bits per second repsectively?
Auto-configuration can only get you so far before it becomes
unreasonably complex and, to get the best performance, you need
configuration. For example, one factor in such configuration might be
what percentage of traffic overall is multi-destiation, something it
would be a pain for TRILL to figure out.

Thanks,
Donald