Subject: [rbridge] Can someone explain Multi-Topology...its purpose, use, etc?

From: Radia Perlman <[email-masked]>
Date: Thu, 1 Mar 2012 20:05:03 -0800

I understand the concept of adding per-topology costs to IS-IS, and
having different forwarding tables computed for each topology.
However, especially in order to imagine how it might (as claimed)
solve various problems, I'd like to have people explain the following,
or any other questions that people might like to toss into the
discussion.  If it will help, I'll try to make a stab at answering my
own questions.

1) What problem was multi-topology intended to solve?

There are two things I could imagine, or that people have mentioned to
me.  One is having different paths for IPv4 and IPv6, presumably
because some of the routers only support one or the other, and
wouldn't be able to know how to forward the other type.  (if all
routers support both, I can't quite imagine why you'd want different
paths for IPv4 and IPv6.  Can anyone explain that?).  The other is to
solve some of the problems addressed by MPLS traffic engineering; to
have certain paths reserved for certain types of traffic (e.g., voice
vs data, or reserved paths for some customers who have negotiated an
SLA).

I could imagine different paths being useful in TRILL for
jumbo-frames, but the path could be computed based on the "link MTU
size" rather than "topology", I'd think, though one could certainly
create a "topology" that is for jumbo-frames, and report infinite cost
for links that don't support whatever size "jumbo" is, in that
topology.

2) How do switches/routers know which topology to use when forwarding a packet?

I'd think the safest thing would be to have a topology name in the
LSPs (say "topology 2"), and then have a field in the data packet that
specifies "topology 2".  If there's an indirect way, as in, IPv4 goes
on topology 1, and IPv6 goes on topology 2, that is individually
configered at each router, then there is the danger that some router,
presumably one that supports both IPv4 and IPv6, forwarding it
according to a different topology than other routers.

If it's indirect (because there's no room for a "topology" field in a
data packet, then it would be good to at least have some router inform
the others about how to map a data packet onto a topology. (based on
protocol type?  TCP port? packet size?...whatever)

For TRILL, there was one proposal (that I understood), which was to
allocate some of the nickname bits to indicate "topology".  Although I
understand this one, it makes me nervous because various people have
been complaining that they see 16 bits for nickname as not enough...so
if we steal 2 or 3 bits for "topology", that leaves even fewer bits
for nickname.

3) Specifically for TRILL, why would multi-topology be useful?

Thanks,

Radia
_______________________________________________
rbridge mailing list
[email-masked]
http://mailman.postel.org/mailman/listinfo/rbridge

---

From: Thomas Narten <[email-masked]>
Date: Fri, 02 Mar 2012 08:25:59 -0500

> 3) Specifically for TRILL, why would multi-topology be useful?

Disjoint multipathing for FCoE. Today, many FC deployments use
physically replicated SANs. Twice the cabling/cost/etc.

But, there is real value in terms of fault isolation in having
completely disjoint paths.

TRILL, today, does not really do this.

Thomas

---

From: <[email-masked]>
Date: Fri, 2 Mar 2012 13:21:32 -0500

Note that using FCoE over TRILL essentially requires that TRILL support DCB Ethernet.

Thanks,
--David

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Narten
> Sent: Friday, March 02, 2012 8:26 AM
> To: Radia Perlman
> Cc: [email-masked]
> Subject: Re: [rbridge] Can someone explain Multi-Topology...its purpose, use, etc?
>

---

From: Donald Eastlake <[email-masked]>
Date: Fri, 2 Mar 2012 14:39:24 -0500

> Note that using FCoE over TRILL essentially requires that TRILL support DCB Ethernet.

See https://datatracker.ietf.org/doc/draft-eastlake-trill-rbridge-dcb/

Thanks,
Donald
=============================
 Donald E. Eastlake 3rd   +1-508-333-2270 (cell)
 155 Beaver Street, Milford, MA 01757 USA
 [email-masked]

---

From: <[email-masked]>
Date: Fri, 2 Mar 2012 14:48:25 -0500

That was a timely submission.

My understanding is that while congestion notification was originally part of DCB, that's
no longer considered to be the case.  Deployments of DCB Ethernet for FCoE typically do
not include congestion notification.

Thanks,
--David

> From: Donald Eastlake [mailto:[email-masked]]
> Sent: Friday, March 02, 2012 2:39 PM
> To: Black, David
> Cc: [email-masked]; [email-masked]

---

From: Donald Eastlake <[email-masked]>
Date: Fri, 2 Mar 2012 15:23:29 -0500

Some people think congestion notification (CN) is important and some
think it isn't.

If you are just talking about Priority Based Flow control (PFC, aka
"per priority pause") or just PFC and ETS, they are down in the
queueing control at ports and orthogonal to what sort of switch
protocol you are running above the ports. Could be a bridge, a layer 3
router, an RBridge, or whatever. PFC and ETS are pretty much the same.

Thanks,
Donald

---

From: Vishwas Manral <[email-masked]>
Date: Fri, 2 Mar 2012 13:49:23 -0800

Agree with you totally here Donald.

-Vishwas

---

From: Jon Hudson <[email-masked]>
Date: Fri, 2 Mar 2012 15:30:59 -0800

A couple things.

1.) For SANs it's just about logical separation. It's about physical
separation. No electrons bumping into each other between path A and path B.

I have customers today that do this today with FCoE. This issue becomes
that you now have two interfaces on a host from two different networks.
FCoE doesn't care as the multipathing driver takes care of it, and it's all
layer 2. However many applications are not designed to listen on two
different networks. There has been talk of a vRouter at a hypervisor level
to solve this for virtual hosts, but it's early.

2.) While I agree with Donald that TRILL should play well with DCB. The
fact is that both Cisco and Brocade offer FCoE over their "TRILL-ish"
products. And they both only support FCoE with DCB. So you can use PFC &
ETS without interacting with TRILL at all. Not saying that it's optimal,
but I can personally say it works.

3.) If you are from the camp that believes in CN then you really also want
some interaction between TRILL and CN. There is a lot of value in that.

However since there are no working implementations of CN today, it's not
yet a problem.

While there is not a lot of FCoE deployed today, there is some. My guess is
less than 100 production sites, but that is a guess.

-J

-- 
"Do not lie. And do not do what you hate."
"Don't worry, be happy"
_______________________________________________
rbridge mailing list
[email-masked]
http://mailman.postel.org/mailman/listinfo/rbridge

---

From: Sam Aldrin <[email-masked]>
Date: Fri, 2 Mar 2012 16:35:18 -0800

Nicely put, Jon.

Understand that there are no existing deployments with CN for TRILL today.
Forward looking, it would be very helpful to understand the need and it would be good to hear how much of usefulness and real case scenarios exist for TRILL to work with CN. As this question keeps popping up every now and then, would like to hear more details on this.

Thanks
Sam

Sent from my iPad

On Mar 2, 2012, at 3:30 PM, Jon Hudson <[email-masked]> wrote:

> A couple things. 
> 
> 1.) For SANs it's just about logical separation. It's about physical separation. No electrons bumping into each other between path A and path B. 
> 
> I have customers today that do this today with FCoE. This issue becomes that you now have two interfaces on a host from two different networks. FCoE doesn't care as the multipathing driver takes care of it, and it's all layer 2. However many applications are not designed to listen on two different networks. There has been talk of a vRouter at a hypervisor level to solve this for virtual hosts, but it's early. 
> 
> 2.) While I agree with Donald that TRILL should play well with DCB. The fact is that both Cisco and Brocade offer FCoE over their "TRILL-ish" products. And they both only support FCoE with DCB. So you can use PFC & ETS without interacting with TRILL at all. Not saying that it's optimal, but I can personally say it works. 
> 
> 3.) If you are from the camp that believes in CN then you really also want some interaction between TRILL and CN. There is a lot of value in that. 
> 
> However since there are no working implementations of CN today, it's not yet a problem. 
> 
> While there is not a lot of FCoE deployed today, there is some. My guess is less than 100 production sites, but that is a guess. 
> 
> -J
> 
> On Fri, Mar 2, 2012 at 1:49 PM, Vishwas Manral <[email-masked]> wrote:

---

From: Mingui Zhang <[email-masked]>
Date: Mon, 05 Mar 2012 02:05:50 +0000

Multi-Topology (MT) provides physical separation of TRILL traffic. Similar as the MT in layer3, it can be used to achieve a more load-balanced TRILL campus than ECMP; If MTs are set up intentionally (e.g., disjoint MTs), it can be used to realize fast rerouting under failures. But there are some differences between the layer2 MT and layer3 MT. For example, the use of layer2 MT further confines the broadcast domain besides VLAN; The MAC sharing between different topologies is a new topic. The MAC sharing among different topologies future pertain to the asymmetric use of MTs.

One usage of MT I want to add into the discussion is to use MT to distinguish the traffic flow ingressed/egressed by multi-homing members. In this way, we can get rid of the RPFC issue caused by active-active multi-homing. For more information, one can read the draft, http://tools.ietf.org/html/draft-zhang-trill-multi-topo-rpfc-00.

Thanks,
Mingui

   > From: [email-masked] [mailto:[email-masked]] On
   > Behalf Of Radia Perlman
   > Sent: Friday, March 02, 2012 12:05 PM
   > To: [email-masked]
   > Subject: [rbridge] Can someone explain Multi-Topology...its purpose, use,
   > etc?
   >

---

From: Vishwas Manral <[email-masked]>
Date: Mon, 5 Mar 2012 12:52:26 -0800

Hi Jon,

> to solve this for virtual hosts, but it's early.
I know SAN vendors would want to have separate interfaces altogether on
servers. However using a CNA and a single Ethernet interface on the host,
the nirvana we were supposed to achieve with FCoE.

I agree in a lot of cases even if we use FCoE, we would want separate
networks (after first hop) for different kinds of traffic. In which case we
would have some parts of the network, that could  be common and some that
would not be. Am I missing the point you are making?

> I guess what you are talking about is not end-to-end Ethernet forwarding
with FCoE, but a hop by hop, where the forwarding is done by FCF (and not
Ethernet switching).

We are talking about a case where we have Ethernet forwarding as a lot of
the Enterprises may like. We do not need TRILL to be DCB aware except for
the case of end-to-end messaging like PFC.

> Correct.

> However since there are no working implementations of CN today, it's not
Ok.

>
> While there is not a lot of FCoE deployed today, there is some. My guess
> is less than 100 production sites, but that is a guess.
>
> Do you mean end-to-end FCoE here or do you mean single hop?

> -J

Thanks,
Vishwas

>> On Fri, Mar 2, 2012 at 12:23 PM, Donald Eastlake <[email-masked]>wrote:
>>
>>> >> See
>>> https://datatracker.ietf.org/doc/draft-eastlake-trill-rbridge-dcb/
>>> >>
>>> >> Thanks,
>>> >> Donald
>>> >> >> From: [email-masked] [mailto:
>>> [email-masked]] On Behalf Of Thomas Narten

---

From: Vishwas Manral <[email-masked]>
Date: Mon, 5 Mar 2012 12:54:40 -0800

Hi Mingui,

I agree the case you mention could be a use case for MT too.

In my view MT provides tools we can use for multiple purposes - including
gaining scalability as Tissa from Cisco pointed out. :)

Thanks,
Vishwas

On Sun, Mar 4, 2012 at 6:05 PM, Mingui Zhang <[email-masked]> wrote:

> Multi-Topology (MT) provides physical separation of TRILL traffic. Similar
> as the MT in layer3, it can be used to achieve a more load-balanced TRILL
> campus than ECMP; If MTs are set up intentionally (e.g., disjoint MTs), it
> can be used to realize fast rerouting under failures. But there are some
> differences between the layer2 MT and layer3 MT. For example, the use of
> layer2 MT further confines the broadcast domain besides VLAN; The MAC
> sharing between different topologies is a new topic. The MAC sharing among
> different topologies future pertain to the asymmetric use of MTs.
>
> One usage of MT I want to add into the discussion is to use MT to
> distinguish the traffic flow ingressed/egressed by multi-homing members. In
> this way, we can get rid of the RPFC issue caused by active-active
> multi-homing. For more information, one can read the draft,
> http://tools.ietf.org/html/draft-zhang-trill-multi-topo-rpfc-00.
>
> Thanks,
> Mingui
>

---

From: Vishwas Manral <[email-masked]>
Date: Mon, 5 Mar 2012 16:02:12 -0800

Hi Jon,

We seem to have moved to more vendor specific pitches. :-)

We can continue the discussion offline from here.

Thanks,
Vishwas

On Mon, Mar 5, 2012 at 3:50 PM, Jon Hudson <[email-masked]> wrote:

> inline below
> The One Cable idea is a myth. It's nice Marketing. Not even one card with
> dual ports will do. Two cards, one cable each is the minimum for most
> customers I work with. This is about failure domains, statistics & MTBFs.
> This has nothing to do with what SAN vendors want. This has to do with IT
> folks who will lose their job if they lose data. If a SAN vendor sells a
> dual switch setup connected or air separated, they still just sell 2
> switches. This comes from the paranoia and experience of running
> datacenters. The last time I had filesystem damage due to single path
> failure I was restoring data for 26hrs. It's not worth it.
>
> If you are going to be accessing a remote filesystem most customers have
> learned that two physically separate paths, are the only way to provide
> permanent availability.
>
> but keep in mind, I work with some very large customers, your local flower
> shop might feel very differently.
>
>
>>
>>>
> I am talking about both "sparse" and "dense" style FCoE. TRILL can provide
> a multi-hop multi-path layer for which DCB runs on top. PFC can send it's
> pause frames and cause upper level protocols to do what they need to and
> TRILL just keeps on moving.
>
> It's like freeways today. Do freeways need to be aware of traffic flow and
> congestion to work? No. Would they work much better with that knowledge.
> Yes.
>
>
>>  3.) If you are from the camp that believes in CN then you really also
>>> want some interaction between TRILL and CN. There is a lot of value in
>>> that.
>>>
>>> Correct.
>>
>>
> I mean both TOR->FC single hop and TOR->Aggregation/Core-> FC dual hop.
>
> This does not include all the blades servers out there using FCoE
> internally.
>
> The number could be higher, but the truth is no one running FCoE wants
> anyone to know they are running FCoE so getting data you can talk about is
> very difficult =)
> On Mon, Mar 5, 2012 at 12:52 PM, Vishwas Manral <[email-masked]>wrote:
>
>>
>>
>
>
> --

---

From: Jon Hudson <[email-masked]>
Date: Mon, 5 Mar 2012 15:50:42 -0800

inline below

---

From: David Klebanov <[email-masked]>
Date: Wed, 07 Mar 2012 14:06:58 -0800

Hi Donald,

I wanted to comment on your reply here. In the context of TRiLL or any other
Ethernet encapsulation technology, how do you deal with intermediate nodes
that do not support DCB, but yet run TRiLL. Will you basically "loose" DCB
function on that link inside the TRiLL environment?

Thanks,
David

On 3/2/12 12:23 PM, "Donald Eastlake" <[email-masked]> wrote:

>>>>> Behalf Of Thomas Narten

---

From: Donald Eastlake <[email-masked]>
Date: Wed, 7 Mar 2012 18:46:26 -0500

Hi David,

> function on that link inside the TRiLL environment?

Short answer: Yes -- but this has nothing to do with TRILL and
everything to do with having nodes that do not support DCB.

Long answer:

The goal of the DCB facilities we are talking about is to eliminate
loss due to queue overflow. To do that over a path for some particular
type of traffic (usually classified by priority) every queue along
that path (usually queues just exist at output ports) must support the
facility. So any device with more than two ports where you could have
multiple streams converging on one output port or even a device with
only two ports if they could be different speed, must support the
facility or you have lost the guarantees. And you also have to meet
various engineering criterion for bandwidth-delay-buffering.

I don't see that it has anything particular to do with "TRILL" or
"encapsulation". If any device along the path has a queue that could
overflow and it doesn't implement the flow control facilities you are
counting on, then you can't depend on not losing data in that queue.

If a path is appropriately engineered and the devices along it are
some mix of TRILL switches and 802.1 bridges and they all implement,
for example, PFC and ETS and have those enabled for priority-x frames,
you should never lose a priority-x frame along that path due to queue
overflow.

Thanks,
Donald

---

From: Radia Perlman <[email-masked]>
Date: Wed, 7 Mar 2012 16:22:11 -0800

That's what I'd assume....that you'd lose DCB functionality on links
in which the switches don't do DCB.  So, if a link doesn't do
pause/resume (priority based flow control), then packets might
experience congestion loss on that link.   And the other thing in
DCB...congestion notification.... the switch that doesn't support it
wouldn't inform the source.

I like that feature of DCB...that it can be phased in...not every
switch needs to support it.

Radia