Subject: WG review for draft-lapukhov-bgp-routing-large-dc

From: Petr Lapukhov <[email-masked]>
Date: Sun, 27 Jul 2014 17:30:58 +0000

Hi,

Authors of draft-lapukhov-bgp-routing-large-dc would like to submit it for WG review and possible adoption as a workgroup document. In this draft we summarize our experience of using BGP as the only protocol for routing in large-scale data-center networks (100K+ bare metal servers, multiple thousands of routers). 

Document location: http://tools.ietf.org/html/draft-lapukhov-bgp-routing-large-dc/

Thank you!

Petr

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Mon, 28 Jul 2014 13:48:28 +0000

Support

Jim Uttaro

-----Original Message-----
From: rtgwg [mailto:[email-masked]] On Behalf Of Petr Lapukhov
Sent: Sunday, July 27, 2014 1:31 PM
To: [email-masked]
Subject: WG review for draft-lapukhov-bgp-routing-large-dc

Hi,

Thank you!

Petr
_______________________________________________
rtgwg mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/rtgwg

---

From: <[email-masked]>
Date: Mon, 28 Jul 2014 14:39:01 +0000

Support.
Wouldn't GROW be a better home than RTGWG?

> Sent: Sunday, July 27, 2014 7:31 PM
> To: [email-masked]
> Authors of draft-lapukhov-bgp-routing-large-dc would like to submit it for
> WG review and possible adoption as a workgroup document. In this draft we
> summarize our experience of using BGP as the only protocol for routing in
> large-scale data-center networks (100K+ bare metal servers, multiple
> thousands of routers).
> 

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: Hannes Gredler <[email-masked]>
Date: Mon, 28 Jul 2014 18:06:39 +0200

support;

On Sun, Jul 27, 2014 at 05:30:58PM +0000, Petr Lapukhov wrote:
| Hi,
| 
| Authors of draft-lapukhov-bgp-routing-large-dc would like to submit it for WG review and possible adoption as a workgroup document. In this draft we summarize our experience of using BGP as the only protocol for routing in large-scale data-center networks (100K+ bare metal servers, multiple thousands of routers). 
| 
| Document location: http://tools.ietf.org/html/draft-lapukhov-bgp-routing-large-dc/
| 
| Thank you!
| 
| Petr
| _______________________________________________
| rtgwg mailing list
| [email-masked]
| https://www.ietf.org/mailman/listinfo/rtgwg

---

From: Antoni Przygienda <[email-masked]>
Date: Mon, 28 Jul 2014 16:26:48 +0000

Personally, would tend to agree that GROW is a better place than RTGWG looking @ charters or possibly even NVO3 (albeit this work does not seem to address multi TS or mobility @ first glance or maybe the intention is for this to be an 'underlay' for lack of better terms) ? 

Work is surely interesting and would be nice to see a presentation and get a floor discussion going

--- tony

> From: rtgwg [mailto:[email-masked]] On Behalf Of
> [email-masked]
> Sent: Monday, July 28, 2014 7:39 AM
> To: Petr Lapukhov; [email-masked]
> Cc: [email-masked]
> Subject: RE: WG review for draft-lapukhov-bgp-routing-large-dc
> 
> Support.

---

From: Alia Atlas <[email-masked]>
Date: Mon, 28 Jul 2014 12:35:10 -0400

GROW & IDR both agreed that it wasn't in their charter.  This falls under
rtgwg having individual drafts without a good WG.  I have agreed to
AD-sponsor it, but would prefer that it get the focus and review and
consensus from being in a WG.

There are 2 drafts (this & abno) that we are trying this with.

Alia
On Jul 28, 2014 10:39 AM, <[email-masked]> wrote:

> > Document location:
> http://tools.ietf.org/html/draft-lapukhov-bgp-routing-
> > large-dc/
> >
> > Thank you!
> >
> > Petr
> Ce message et ses pieces jointes peuvent contenir des informations
> confidentielles ou privilegiees et ne doivent donc
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez
> recu ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
> electroniques etant susceptibles d'alteration,
> This message and its attachments may contain confidential or privileged
> information that may be protected by law;

---

From: Jeff Tantsura <[email-masked]>
Date: Mon, 28 Jul 2014 17:50:02 +0000

Hi,

Both, this document and abno are architectural frameworks and to my thinking rtgwg as a good home for those. I'm glad to see support from the AD's here!
To second Alia - please do express your support/ lack of such, we would like the decision to be driven by consensus (so far only "support" mails have been received)

In the future we are going to bring more of routing architecture relevant discussions, WAN, as well as DC.

Thanks!

Regards,
Jeff

On Jul 28, 2014, at 9:35 AM, "Alia Atlas" <[email-masked]<mailto:[email-masked]>> wrote:

GROW & IDR both agreed that it wasn't in their charter.  This falls under rtgwg having individual drafts without a good WG.  I have agreed to AD-sponsor it, but would prefer that it get the focus and review and consensus from being in a WG.

Alia

On Jul 28, 2014 10:39 AM, <[email-masked]<mailto:[email-masked]>> wrote:
Support.
Wouldn't GROW be a better home than RTGWG?

> From: rtgwg [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Petr Lapukhov
> To: [email-masked]<mailto:[email-masked]>
> [email-masked]<mailto:[email-masked]>

_______________________________________________
rtgwg mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/rtgwg
_______________________________________________
rtgwg mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/rtgwg

---

From: Jeffrey Haas <[email-masked]>
Date: Mon, 28 Jul 2014 14:20:13 -0400

> To second Alia - please do express your support/ lack of such, we would like the decision to be driven by consensus (so far only "support" mails have been received)

I think rtgwg is a good place for such things.

For this document in particular, it did seem a better fit for GROW than
rtgwg but as long as it's getting adopted somewhere.

I had read a prior version of this document and support its adoption.  I
should read the latest rev at some point.

-- Jeff

---

From: Ebben Aries <[email-masked]>
Date: Mon, 28 Jul 2014 14:34:52 -0400

support

On 07/27/2014 01:30 PM, Petr Lapukhov wrote:
> Document location: https://urldefense.proofpoint.com/v1/url?u=http://tools.ietf.org/html/draft-lapukhov-bgp-routing-large-dc/&k=ZVNjlDMF0FElm4dQtryO4A%3D%3D%0A&r=GJQFPrZyyh453ywaGV%2FvoQ%3D%3D%0A&m=NH0rpk5PHrLAtv7k7%2B2fqWVgPDTxdgavnWs1mHERgKQ%3D%0A&s=eb9cad1de263bd8b7ff2d6e57b8dfd062bfe375db7117dd7c137d26771d0d908
>
> Thank you!
>
> Petr
> https://urldefense.proofpoint.com/v1/url?u=https://www.ietf.org/mailman/listinfo/rtgwg&k=ZVNjlDMF0FElm4dQtryO4A%3D%3D%0A&r=GJQFPrZyyh453ywaGV%2FvoQ%3D%3D%0A&m=NH0rpk5PHrLAtv7k7%2B2fqWVgPDTxdgavnWs1mHERgKQ%3D%0A&s=0dd339199588341ad4a2f1fed78bc8ff3f22d8a58d8836d46a94a810473bb68a
>

---

From: Mohan Nanduri <[email-masked]>
Date: Mon, 28 Jul 2014 18:45:27 -0400

Support.

Cheers,
-Mohan

On Mon, Jul 28, 2014 at 2:34 PM, Ebben Aries <[email-masked]> wrote:

>> Authors of draft-lapukhov-bgp-routing-large-dc would like to submit it
>> for WG review and possible adoption as a workgroup document. In this draft
>> we summarize our experience of using BGP as the only protocol for routing
>> in large-scale data-center networks (100K+ bare metal servers, multiple
>> thousands of routers).
>>
>> Document location: https://urldefense.proofpoint.
>> com/v1/url?u=http://tools.ietf.org/html/draft-lapukhov-
>> bgp-routing-large-dc/&k=ZVNjlDMF0FElm4dQtryO4A%3D%3D%
>> 0A&r=GJQFPrZyyh453ywaGV%2FvoQ%3D%3D%0A&m=NH0rpk5PHrLAtv7k7%
>> 2B2fqWVgPDTxdgavnWs1mHERgKQ%3D%0A&s=eb9cad1de263bd8b7ff2d6e57b8dfd
>> 062bfe375db7117dd7c137d26771d0d908
>>
>>
>> Thank you!
>>
>> Petr
>> https://urldefense.proofpoint.com/v1/url?u=https://www.ietf.
>> org/mailman/listinfo/rtgwg&k=ZVNjlDMF0FElm4dQtryO4A%3D%3D%
>> 0A&r=GJQFPrZyyh453ywaGV%2FvoQ%3D%3D%0A&m=NH0rpk5PHrLAtv7k7%
>> 2B2fqWVgPDTxdgavnWs1mHERgKQ%3D%0A&s=0dd339199588341ad4a2f1fed78bc8
>> ff3f22d8a58d8836d46a94a810473bb68a
>>
>>

---

From: Hannes Gredler <[email-masked]>
Date: Tue, 29 Jul 2014 11:42:34 +0200

hi antoni,

i am all for accepting it as a WG item - 
IMO its an excellent proofpoint that really
large datacenters can be run based
on exisiting protocols and even getting to
10000s of routing nodes is not the end of the world.

tx,

/hannes

On Mon, Jul 28, 2014 at 04:26:48PM +0000, Antoni Przygienda wrote:
| Personally, would tend to agree that GROW is a better place than RTGWG looking @ charters or possibly even NVO3 (albeit this work does not seem to address multi TS or mobility @ first glance or maybe the intention is for this to be an 'underlay' for lack of better terms) ? 
| 
| Work is surely interesting and would be nice to see a presentation and get a floor discussion going
| 
| --- tony
| 
| > -----Original Message-----
| > From: rtgwg [mailto:[email-masked]] On Behalf Of
| > [email-masked]
| > Sent: Monday, July 28, 2014 7:39 AM
| > To: Petr Lapukhov; [email-masked]
| > Cc: [email-masked]
| > Subject: RE: WG review for draft-lapukhov-bgp-routing-large-dc
| > 
| > Support.
| > Wouldn't GROW be a better home than RTGWG?
| > 
| 
| _______________________________________________
| rtgwg mailing list
| [email-masked]
| https://www.ietf.org/mailman/listinfo/rtgwg

---

From: Antoni Przygienda <[email-masked]>
Date: Tue, 29 Jul 2014 14:45:33 +0000

> hi antoni,
> 
> i am all for accepting it as a WG item - IMO its an excellent proofpoint that
> really large datacenters can be run based on exisiting protocols and 

[Tony said]  Hannes, yepp, first, work's interesting & fact that it works gives it its own merit, should be possibly taken up by someone IMO. My points were though:

	1. I didn't see 'running large datacenters' as something on RTGWG charter and it's also typically something that is first driven by larger set of requirements rather than a single datapoint. 
	2. A blueprint of a particular solution is exactly that.  It is not a generic protocol specification or guideline that will fit e'one. If you have multi-TS which bring their own existing addresses or need MAC mobility or have to run L2 applications or don't have BGP implementation with necessary twists or other tid-nits which tons of DC happen to carry about then the shoe may not fit. 

> even getting to 10000s of routing nodes is not the end of the world.
> 
[Tony said]  We know that from a running thing called the 'Internet'  ;-P  I know I took it out the context but I couldn't resist the tongue-in-cheek pun possible ;-)

Again, looking fwd' to presentation and discussion on the floor. 

--- tony

---

From: Alia Atlas <[email-masked]>
Date: Tue, 29 Jul 2014 11:23:03 -0400

Hi Tony,

On Tue, Jul 29, 2014 at 10:45 AM, Antoni Przygienda <
[email-masked]> wrote:

> [Tony said]  Hannes, yepp, first, work's interesting & fact that it works
> gives it its own merit, should be possibly taken up by someone IMO. My
> points were though:
>
>         1. I didn't see 'running large datacenters' as something on RTGWG
> charter and it's also typically something that is first driven by larger
> set of requirements rather than a single datapoint.
>

As I said earlier, this falls into the other work and handling individual
drafts without a home.  It's a way of getting the review and consensus for
drafts that would otherwise be AD-sponsored.  I agree that this is a change
from how rtgwg has been used in the past.

>         2. A blueprint of a particular solution is exactly that.  It is
> not a generic protocol specification or guideline that will fi t e'one. If
> you have multi-TS which bring their own existing addresses or need MAC
> mobility or have to run L2 applications or don't have BGP implementation
> with necessary twists or other tid-nits which tons of DC happen to carry
> about then the shoe may not fit.

Absolutely - this is a starting point that gives one idea.

> [Tony said]  We know that from a running thing called the 'Internet'  ;-P
>  I know I took it out the context but I couldn't resist the tongue-in-cheek
> pun possible ;-)
>

Please - start the discussion here and now!  I was delighted to see how
many people were clearly familiar with the work and thought it was a good
idea to discuss.  Let's get some good reviews and suggestions so the draft
can be much better before the next IETF.

Alia

---

From: Antoni Przygienda <[email-masked]>
Date: Tue, 29 Jul 2014 15:35:23 +0000

Alia, ack 100%. I’m just hanging in the bleachers with my helmet strapped on and waiting for IDR & L2VPN wizards to open the fray but so far stuff’s been astonishingly quiet ;-)   tony

From: Alia Atlas [mailto:[email-masked]]
Sent: Tuesday, July 29, 2014 8:23 AM
To: Antoni Przygienda
Cc: Hannes Gredler; [email-masked]; [email-masked]
Subject: Re: WG review for draft-lapukhov-bgp-routing-large-dc

Hi Tony,

As I said earlier, this falls into the other work and handling individual drafts without a home.  It's a way of getting the review and consensus for drafts that would otherwise be AD-sponsored.  I agree that this is a change from how rtgwg has been used in the past.

        2. A blueprint of a particular solution is exactly that.  It is not a generic protocol specification or guideline that will fi t e'one. If you have multi-TS which bring their own existing addresses or need MAC mobility or have to run L2 applications or don't have BGP implementation with necessary twists or other tid-nits which tons of DC happen to carry about then the shoe may not fit.

Please - start the discussion here and now!  I was delighted to see how many people were clearly familiar with the work and thought it was a good idea to discuss.  Let's get some good reviews and suggestions so the draft can be much better before the next IETF.

Alia

--- tony

---

From: Petr Lapukhov <[email-masked]>
Date: Tue, 29 Jul 2014 15:43:54 +0000

Hi Tony,

I do agree that the document is rather operational/blueprint in its nature. However, I think it's beneficial to have a practical example and start discussion around how large-scale routing in "dense" networks could be accomplished using existing routing protocols. This could be used as a factual ground in evaluation various new routing protocol designs for similar environments. As we progress, I hope to add more practical data (e.g. convergence times). There are some interesting theoretical aspects (e.g. using route summarization with simple virtual aggregation) that haven't been explored yet and could be an interesting discussion topic as well.

Thank you!

Petr

________________________________
From: rtgwg [[email-masked]] on behalf of Alia Atlas [[email-masked]]
Sent: Tuesday, July 29, 2014 8:23 AM
To: Antoni Przygienda
Cc: [email-masked]; [email-masked]
Subject: Re: WG review for draft-lapukhov-bgp-routing-large-dc

Hi Tony,

Alia

--- tony

_______________________________________________
rtgwg mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/rtgwg<https://urldefense.proofpoint.com/v1/url?u=https://www.ietf.org/mailman/listinfo/rtgwg&k=ZVNjlDMF0FElm4dQtryO4A%3D%3D%0A&r=4Cgrr4FbaWmFJmFxSc8F6Q%3D%3D%0A&m=p6%2BjkvZVeCkwqto%2FcW%2F0sFjBDtWiked%2BB7ujDChwB7s%3D%0A&s=459951d713904ffe2c9fba132e7c24489d25231fc72c0738fb0700c648cc1053>

---

From: Jeff Tantsura <[email-masked]>
Date: Tue, 29 Jul 2014 15:55:47 +0000

Hi Petr,

While creating the blueprint you must have considered different options, eBGP vs iBGP, mixed, summarization (different places), MPLS over foo, etc
Would be great if you could elaborate why the design looks the way it is.

Thanks!

Cheers,
Jeff

From: Petr Lapukhov <[email-masked]<mailto:[email-masked]>>
Date: Tuesday, July 29, 2014 at 8:43 AM
To: Alia Atlas <[email-masked]<mailto:[email-masked]>>, Antoni Przygienda <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: RE: WG review for draft-lapukhov-bgp-routing-large-dc

Hi Tony,

Thank you!

Petr

________________________________
From: rtgwg [[email-masked]<mailto:[email-masked]>] on behalf of Alia Atlas [[email-masked]<mailto:[email-masked]>]
Sent: Tuesday, July 29, 2014 8:23 AM
To: Antoni Przygienda
Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: WG review for draft-lapukhov-bgp-routing-large-dc

Hi Tony,

Alia

--- tony

---

From: Hannes Gredler <[email-masked]>
Date: Tue, 29 Jul 2014 21:41:05 +0200

On Tue, Jul 29, 2014 at 02:45:33PM +0000, Antoni Przygienda wrote:
| 	1. I didn't see 'running large datacenters' as something on RTGWG charter and it's also typically something that is first driven by larger set of requirements rather than a single datapoint. 

current thinking is that RTGWG is going to be a place where "new"
routing approaches not strictly fitting into exisiting WGs can
be discussed ... but i hear you - its yet to be seen how much abuse
is going to happen on anyone's given definition of "new".

| 	2. A blueprint of a particular solution is exactly that.  It is not a generic protocol specification or guideline that will fit e'one. If you have multi-TS which bring their own existing addresses or need MAC mobility or have to run L2 applications or don't have BGP implementation with necessary twists or other tid-nits which tons of DC happen to carry about then the shoe may not fit. 
| 
| > even getting to 10000s of routing nodes is not the end of the world.
| > 
| [Tony said]  We know that from a running thing called the 'Internet'  ;-P  I know I took it out the context but I couldn't resist the tongue-in-cheek pun possible ;-)
| 
| Again, looking fwd' to presentation and discussion on the floor. 

i found the nanog preso on this matter pretty interesting ...
https://www.nanog.org/meetings/nanog55/presentations/Monday/Lapukhov.pdf

---

From: Antoni Przygienda <[email-masked]>
Date: Tue, 29 Jul 2014 20:05:52 +0000

Yepp, I'm quite familiar with the work since a good while. M$ evolved their DC since 2008 or so in very interesting way around the ecosystem they bred around Azure (always a _big_ advantage if you can strong-arm your kernel to do things that makes your DC better or even understand what traffic characteristics you run & how you can mash them to shape ;-). Earlier papers (very insightful) are centered around VL2 & Monsoon DC architectures.  Newest output I saw from there points however to yet a different data point (I think it's hinted @ in this draft) where the computation is being actually largely moved out to something akin to a PCS and very impressive DC sizes.  Peter may leak more once he warms up showing the work ;-)    

-- tony

---

From: Petr Lapukhov <[email-masked]>
Date: Wed, 30 Jul 2014 05:10:40 +0000

Jeff,

Sure thing, some of the reasoning is provided in the document, let me elaborate specifically on the questions you have asked.

On the eBGP vs iBGP topic. There have been some implementation constraints we bumped into when we considered iBGP. For example, if route-reflection is employed, then using ECMP for anycast prefixes (typically announced by load-balancer) could get you in trouble, since all implementations we had on hands did not consider cluster list length when picking up ECMP-equivalent paths. This could be worked around by applying a policy to enforce tie breaking (e.g. prefer downstream routes over upstream), but that means a bit more complexity in the configuration. There have been a few other issues (e.g. iBGP might try to use default route to establish peering even if the directly connected subnet is not reachable). This being said, I'm aware of iBGP based deployments and there has been work to document those. For the most part, routing mechanics and convergence is the same in both designs.

It is possible to use 'hybrid' approach, e.g. use iBGP between Tier-2 and Tier-3 switches, with each 'cluster' wrapped in its own ASN. There are deployments that use this approach as well, and it works just fine - same routing properties. With the design described in the document, we eventually turned the hybrid option down as it looked less uniform in terms of BGP features used. Potential benefit if eBGP *only* design is the ability to come up with more light-weight BGP implementation that only has the concept of eBGP sessions (less code). It could be a matter of operational preference, but having eBGP everywhere also allows to easily trace a particular prefix to its originating Tier-3 switch (though one can also use communities for that).

Route summarization is probably the most interesting aspect. We actively employ route summarization at FB, and document has some discussion around this topic. Put it short, with route summarization you need either of the following:

1) Creating a bypass path between Tier-2 switches (or any tier doing aggregation) - e.g. a ring interconnecting them together. This adds some concerns with managing the ring capacity and wasting ports on Tier-2 switches. You can see a sample topology that uses this design here (older FB design): http://nathanfarrington.com/papers/facebook-oic13.pdf
2) Use simple virtual aggregation when doing summarization and leaking, but not using, specifics - i.e. only installing specifics when some of the paths fail. This is mentioned in the document, but to my knowledge there is currently no widespread implementation for DC-class network devices.
3) Use intelligent dis-aggregation on Tier-3 switches - once a switch detects that one of its uplinks is dead, it starts announcing its prefixes with a BGP community that excludes it from summarization on upstream devices. Technically it is similar to (2), but relies purely on the control plane.
4) As opposed to using the bypass ring between Tier-2 devices, use paths via Tier-3 switches as alternates: this means allowing some of Tier-3 switches to re-adverte routes back to Tier-2. This has a problem with significantly increased routing and policy complexity.

Route summarization adds the benefit of reduced fault domain scopes (which could be very handy in preventing certain positive feedback loops) and reduced FIB footprint. The latter is especially important with IPv6 -  FB is dual-stack everywhere, and majority of internal traffic is IPv6, see http://www.swissipv6council.ch/sites/default/files/docs/day_2_7_paul_saab.pdf. 

MPLS is an interesting topic, mostly because it allows for simple and uniform data-plane and permits seamless handover b/w DC and backbone. There are also possible use-cases with virtual-aggregation. However complexity tradeoffs need better understanding (e.g. if going with encapsulation on the server). We currently have no experience of using it inside data-center, but that's an actively discussed topic.

I believe the major contribution of the document is debunking some fears associated with BGP (path hunting, MRAI etc) and demonstrating its feasibility as a routing protocol for hierarchical, dense, up-down routed network. The proposed design avoids the issues with path hunting and demonstrates convergence times well under one second (for single failure, full table loads make take longer with FIB update times) - mostly bounded by event propagation time + FIB update time. In case of ECMP, the failover is done in hardware, and with local bypass available (e.g. ring interconnecting Tier-2 switches) the repair can happen locally both for upstream and downstream traffic. 

Best regards,

Petr

____
From: Jeff Tantsura [[email-masked]]
Sent: Tuesday, July 29, 2014 8:55 AM
To: Petr Lapukhov; Alia Atlas; Antoni Przygienda
Cc: [email-masked]; [email-masked]
Subject: Re: WG review for draft-lapukhov-bgp-routing-large-dc

Hi Petr,

Thanks!

Cheers,
Jeff

Hi Tony,

Thank you!

Petr

Hi Tony,

Alia

--- tony

---

From: Vitkovský Adam <[email-masked]>
Date: Thu, 31 Jul 2014 09:48:19 +0000

Hello Peter,

I'd like to ask couple of questions regarding the design and confirm my understanding please. 

What is the recommended fan out ratio for Tier3 to Tier2 and Tier2 to Tier1 please? 
Tier3 to Tier2 would be 1/4 (so in case one Tier2 device fails the remaining BW still available for a cluster is 75%) (Tier3 device has 4 ECMP paths). 
Tier2 to Tier1 would be 1/2 (so in case one Tier1 device fails the remaining BW still available for cluster is 87.5%) (Tier2 device has 2 ECMP paths). 

Or is it more like 1/32 for the Tier3 to Tier2 (Tier3 device has 32 ECMP paths) please?  

8.2.1. Collapsing Tier-1 Devices Layer. 
 - I think that as a result of collapsing the number of Tier1 devices to a 1/2 the impact of the failure of a single Tier1 device will increase in 50%. 
 - Thus wouldn't it be more desirable to leave the same number of Tier1 devices and only add links from a particular tier2 device to another/neighboring pair of Tier1 devices please? 
 - The reduction in port capacity would remain the same. 
 - However the impact of a failure of a single link or single Tier1 device is unchanged. 

8.2. Route Summarization within Clos Topology. 
 - Since you have mentioned that all the devices are preferably of the same type to accommodate REQ2. 
 - I'm thinking they would probably have the same FIB capacity right please? 
 - So if Tier1 device can hold all the DC routes than Tier2 and Tier3 devices can as well right? 
 - If the FIB size differs between the devices used in various Tiers than summarization is beneficial indeed. 

If FIB size is a cause for concern would it be possible to utilize a scheme where servers are grouped into server groups, then to define which server groups need to communicate with other set of server groups or everybody or the internet please? 
This way prefixes could be marked and filters on Tier3 and Tier 2 devices set accordingly -to only allow the necessary prefixes to be accepted form a Peer or inserted from BGP into RIB/FIB. 
Drawback is of course the increased operational complexity maintaining filters as well as troubleshooting. 
Though with clear server groups to Tier3 devices (or clusters) mapping scheme the filters would be set once than maintained only occasionally. 
Also with a clear communities scheme troubleshooting would be straight forward I believe. 

I'm thinking like in MPLS VRFs if a particular PE (Tier3 device) is serving only a subset of VRFs(server groups) it doesn't really need to hold all the DC routes. 

8.3. ICMP Unreachable Message Masquerading. 
 Another option is to make the network device perform IP address
 masquerading, 
 - Does that mean the network device will respond with RID/Loopback IP during traceroute please? 
 - If so it would be than impossible to pinpoint the link used to forward traffic to the next hop so if there are two IP paths between directly connected devices we wouldn't be able to distinguish the failed one. 
 - But I guess this kind of setup is not going to be used. 

And just some nit picking. 
7.1. Fault Detection Timing. 
This feature is sometimes called as "fast fallover" 
 - Do you mean the "fast external failover" as it only applies to eBGP sessions? 
 - Or you mean the "fast peering session deactivation" functionality that brings the same functionality to iBGP sessions please? 

Thank you very much

adam