Subject: /32s in DC and BGP

From: Robert Raszuk <[email-masked]>
Date: Tue, 11 Feb 2014 09:22:41 +0100

<changing subject to reflect more broader l3vpn related topic>

Hi,

Could those who claim that that sending /32 or /64 or /128 in BGP mainly
within contained DC zone environment will not scale be a bit more precise
and kindly indicate what the real problem is ?

* Which control or data plane element will not scale ?

or

* Which part of BGP state machine will not scale ?

Just curious ....

Cheers,
R.

On Tue, Feb 11, 2014 at 3:03 AM, Thomas Nadeau

<[email-masked]> wrote:

>
>         Thomas,
>
>         I too object to it's adoption based on the /32 point Giles made.
> 
>

On Feb 10, 2014:2:29 PM, at 2:29 PM, Giles Heron <[email-masked]>
> wrote:
>
> > I don't support adoption of this draft as a WG item (speaking as a
> non-author but name-checked commenter).
> >
> > The draft has a major limitation (no support for interconnecting
> routers, but only for interconnecting hosts), and I'm unconvinced that
> passing /32 host routes around in BGP will scale.
>

---

From: Thomas Morin <[email-masked]>
Date: Tue, 11 Feb 2014 10:56:08 +0100

Hi Giles, Thomas,

I would echo Robert's questions below, and ask an additional question: 
advertising non aggregated host routes (v4 /32 routes or v6 /128 routes) 
in BGP VPNv4 routes is not fundamentally different from a scaling 
standpoint than advertising MAC addresses in E-VPN BGP routes.
What would be the reason to believe it would be an issue in one case but 
not in the other  ?

-Thomas

2014-02-11, Robert Raszuk:
> Could those who claim that that sending /32 or /64 or /128 in BGP 
> mainly within contained DC zone environment will not scale be a bit 
> more precise and kindly indicate what the real problem is ?
>
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>
>             Thomas,
>
>     On Feb 10, 2014:2:29 PM, at 2:29 PM, Giles Heron
>     <[email-masked] <mailto:[email-masked]>> wrote:
>
>     > I don't support adoption of this draft as a WG item (speaking as
>     a non-author but name-checked commenter).
>     >
>     routers, but only for interconnecting hosts), and I'm unconvinced
>     that passing /32 host routes around in BGP will scale.
>
>
>

---

From: Giles Heron <[email-masked]>
Date: Tue, 11 Feb 2014 11:36:14 +0000

Hi Thomas,

There are two key differences between E-VPN and virtual-subnet wrt scaling:

1) E-VPN supports interconnection of routers.  virtual-subnet doesn't.  One of the key use-cases for a multi-point L2 service is to interconnect routers.
2) As well as E-VPN we are standardising PBB E-VPN in L2VPN.  PBB E-VPN only advertises provider MAC addresses in BGP.  Client MACs are learned/cached in the data plane.

Giles

On 11 Feb 2014, at 09:56, Thomas Morin <[email-masked]> wrote:

> Hi Giles, Thomas,
> 
> I would echo Robert's questions below, and ask an additional question: advertising non aggregated host routes (v4 /32 routes or v6 /128 routes) in BGP VPNv4 routes is not fundamentally different from a scaling standpoint than advertising MAC addresses in E-VPN BGP routes.  
> What would be the reason to believe it would be an issue in one case but not in the other  ?
> 
> -Thomas
> 
>> Could those who claim that that sending /32 or /64 or /128 in BGP mainly within contained DC zone environment will not scale be a bit more precise and kindly indicate what the real problem is ? 
>> 
>> On Feb 10, 2014:2:29 PM, at 2:29 PM, Giles Heron <[email-masked]> wrote:
>> 
>> > I don't support adoption of this draft as a WG item (speaking as a non-author but name-checked commenter).
>> >
>> > The draft has a major limitation (no support for interconnecting routers, but only for interconnecting hosts), and I'm unconvinced that passing /32 host routes around in BGP will scale.
>> 
>> 
>

---

From: Yakov Rekhter <[email-masked]>
Date: Tue, 11 Feb 2014 06:21:36 -0800

Giles,

> 1) E-VPN supports interconnection of routers.  virtual-subnet
> doesn't.  One of the key use-cases for a multi-point L2 service
> is to interconnect routers.

>From draft-ietf-l2vpn-evpn-04:
                                     An EVPN instance comprises
   CEs that are connected to PEs that form the edge of the MPLS
   infrastructure. A CE may be a host, a router or a switch. The PEs
   provide virtual Layer 2 bridged connectivity between the CEs.

>From the above quote it should be clear that while it is true that
"E-VPN supports interconnection of routers", it is also true that
E-VPN supports interconnection of hosts and/or switches.

Yakov.

> 
> 2) As well as E-VPN we are standardising PBB E-VPN in L2VPN.  PBB
> E-VPN only advertises provider MAC addresses in BGP.  Client MACs
> are learned/cached in th e data plane.
> 
> Giles
> 
> > I would echo Robert's questions below, and ask an additional question: adve
rtising non aggregated host routes (v4 /32 routes or v6 /128 routes) in BGP VPN
v4 routes is not fundamentally different from a scaling standpoint than adverti
sing MAC addresses in E-VPN BGP routes.  
> >> Could those who claim that that sending /32 or /64 or /128 in BGP mainly w
ithin contained DC zone environment will not scale be a bit more precise and ki
ndly indicate what the real problem is ? 
> >> > The draft has a major limitation (no support for interconnecting routers
, but only for interconnecting hosts), and I'm unconvinced that passing /32 hos
t routes around in BGP will scale.

---

From: Thomas Nadeau <[email-masked]>
Date: Tue, 11 Feb 2014 09:27:55 -0500

> What would be the reason to believe it would be an issue in one case but not in the other  ?

	I think you are missing the point that was raised: its not that BGP will not scale well; as a protocol its clearly shown that it can ship around millions of routes and lots of other stuff as it has become the "dump truck" of the protocol world. The question is more of the approach - do we want to be shuffling around /32s (regardless of protocol) given that there are other solutions that work well without doing so? I would also question the use of BGP in a Data Center given past operational feedback in NVO3 for example, but that is orthogonal to the question of the mechanics of this solution.

	--Tom

---

From: Thomas Morin <[email-masked]>
Date: Tue, 11 Feb 2014 16:04:39 +0100

Hi Thomas,

2014-02-11, Thomas Nadeau:
>> I would echo Robert's questions below, and ask an additional 
>> question: advertising non aggregated host routes (v4 /32 routes or v6 
>> /128 routes) in BGP VPNv4 routes is not fundamentally different from 
>> a scaling standpoint than advertising MAC addresses in E-VPN BGP routes.
> I think you are missing the point that was raised: its not that BGP 
> will not scale well;

(Giles initial point, that you supported, was a that he was unconvinced 
it would scale.)

> as a protocol its clearly shown that it can ship around millions of 
> routes and lots of other stuff as it has become the "dump truck" of 
> the protocol world. The question is more of the approach - do we want 
> to be shuffling around /32s (regardless of protocol) given that there 
> are other solutions that work well without doing so? I would also 
> question the use of BGP in a Data Center given past operational 
> feedback in NVO3 for example, but that is orthogonal to the question 
> of the mechanics of this solution.

This discussion about putting /32s in BGP is absolutely not specific to 
draft-..-virtual-subnet (see E-VPN, end-system l3vpn).

-Thomas

2014-02-11, Robert Raszuk:
>>>     > I don't support adoption of this draft as a WG item (speaking
>>>     as a non-author but name-checked commenter).
>>>     >
>>>     > The draft has a major limitation (no support for
>>>     interconnecting routers, but only for interconnecting hosts),
>>>     and I'm unconvinced that passing /32 host routes around in BGP
>>>     will scale.
>>>
>>>
>>>
>>
>
>

---

From: Giles Heron <[email-masked]>
Date: Tue, 11 Feb 2014 15:16:43 +0000

Hi Yakov,

> E-VPN supports interconnection of hosts and/or switches.

Indeed.  And I wasn't claiming that it didn't.

What I was trying to make clear was that E-VPN provides two ways to scale that are missing in virtual-subnet:
1) you can use it to interconnect routers.  So you get one FIB entry per router.
2) you can use it to interconnect PBB switches.  So you get one FIB entry per switch.

AFAIK virtual-subnet only provides the model where you interconnect hosts and get one FIB entry per host.

Giles

---

From: Thomas Nadeau <[email-masked]>
Date: Tue, 11 Feb 2014 10:19:10 -0500

On Feb 11, 2014:10:04 AM, at 10:04 AM, Thomas Morin <[email-masked]> wrote:

>>  I think you are missing the point that was raised: its not that BGP will not scale well;
> 
> (Giles initial point, that you supported, was a that he was unconvinced it would scale.)

	Anyways, MY point then is not around BGP per se; its the approach. *)

>> as a protocol its clearly shown that it can ship around millions of routes and lots of other stuff as it has become the "dump truck" of the protocol world. The question is more of the approach - do we want to be shuffling around /32s (regardless of protocol) given that there are other solutions that work well without doing so? I would also question the use of BGP in a Data Center given past operational feedback in NVO3 for example, but that is orthogonal to the question of the mechanics of this solution.
> 
> This discussion about putting /32s in BGP is absolutely not specific to draft-..-virtual-subnet (see E-VPN, end-system l3vpn).

	Cool.

	--Tom

---

From: "Susan Hares" <[email-masked]>
Date: Tue, 11 Feb 2014 11:00:18 -0500

Tom: 

The argument on /32 for IPv4 in BGP is an old one that was resolved with
"what makes business sense" (aka Shakespere in BGP)  instead the "purist"
mandate on what BGP can do. 

As to Data Center's use of the BGP, I suggest you review the draft:

http://datatracker.ietf.org/doc/draft-lapukhov-bgp-routing-large-dc/

Some data centers pass host routes for a variety of purposes.  Yakov
indicates host routes are being ship in EVPN.  IMHO documenting on these
deployed use cases aids researchers and implementers.   

The IDR WG has moved on from the "purist" view to allowing DCs private AS
space and allowing link-state information to be carried in BGP.    These
usage are not Internet-wide usage, but adapt BGP to a portion of the
Internet.  If you feel host routes should not be carried, please comment to
[email-masked]  on draft-ietf-idr-ls-distribution solution which passes LSP
Info in BGP.  It is being suggest for early allocation of code points - so
it vital to make your concerns known. 

Sue Hares

From: Thomas Nadeau [mailto:[email-masked]] 
Sent: Tuesday, February 11, 2014 9:28 AM
To: Thomas Morin
Cc: Giles Heron; Robert Raszuk;
[email-masked]; L3VPN
Subject: Re: /32s in DC and BGP

Hi Giles, Thomas,

I would echo Robert's questions below, and ask an additional question:
advertising non aggregated host routes (v4 /32 routes or v6 /128 routes) in
BGP VPNv4 routes is not fundamentally different from a scaling standpoint
than advertising MAC addresses in E-VPN BGP routes.  
What would be the reason to believe it would be an issue in one case but not
in the other  ?

            I think you are missing the point that was raised: its not that
BGP will not scale well; as a protocol its clearly shown that it can ship
around millions of routes and lots of other stuff as it has become the "dump
truck" of the protocol world. The question is more of the approach - do we
want to be shuffling around /32s (regardless of protocol) given that there
are other solutions that work well without doing so? I would also question
the use of BGP in a Data Center given past operational feedback in NVO3 for
example, but that is orthogonal to the question of the mechanics of this
solution.

            --Tom

-Thomas

Hi,

or 

Just curious ....

Cheers,
R.

        Thomas,

 

>
> The draft has a major limitation (no support for interconnecting routers,
but only for interconnecting hosts), and I'm unconvinced that passing /32
host routes around in BGP will scale.

---

From: Thomas Nadeau <[email-masked]>
Date: Tue, 11 Feb 2014 11:57:41 -0500

I am still not convinced that shipping around what effectively is the entire FIB of a DC, is a good idea. 
There just seem to be better alternatives that have been pointed out that scale well. I also do not think it is 
a purist idea to question routing /32s in a DC. Look at what big DCs are doing today for routes, and its not
/32s.  I will also point out that one of the advantages to routing protocols is their aggregation/summarization 
abilities; in this case we are completely defeating that.

	--Tom

On Feb 11, 2014:11:00 AM, at 11:00 AM, Susan Hares <[email-masked]> wrote:

> Tom:
>  
> The argument on /32 for IPv4 in BGP is an old one that was resolved with “what makes business sense” (aka Shakespere in BGP)  instead the “purist” mandate on what BGP can do.
>  
> As to Data Center’s use of the BGP, I suggest you review the draft:
> Some data centers pass host routes for a variety of purposes.  Yakov indicates host routes are being ship in EVPN.  IMHO documenting on these deployed use cases aids researchers and implementers.   
>  
> The IDR WG has moved on from the “purist” view to allowing DCs private AS space and allowing link-state information to be carried in BGP.    These usage are not Internet-wide usage, but adapt BGP to a portion of the Internet.  If you feel host routes should not be carried, please comment to [email-masked]  on draft-ietf-idr-ls-distribution solution which passes LSP  Info in BGP.  It is being suggest for early allocation of code points – so it vital to make your concerns known.
>  
> Sue Hares
>  
>  
> Cc: Giles Heron; Robert Raszuk; [email-masked]; L3VPN

---

From: "Susan Hares" <[email-masked]>
Date: Tue, 11 Feb 2014 12:05:58 -0500

Tom:

My point was - don't legislate the BGP usage - let the market decide.
Document what is as informational.  Not going to debate your IHMO or my
IHMO.  

Sue 

From: Thomas Nadeau [mailto:[email-masked]] 
Sent: Tuesday, February 11, 2014 11:58 AM
To: Susan Hares
Cc: Thomas Morin; Giles Heron; Robert Raszuk;
[email-masked]; L3VPN
Subject: Re: /32s in DC and BGP

            I am still not convinced that shipping around what effectively
is the entire FIB of a DC, is a good idea. 

There just seem to be better alternatives that have been pointed out that
scale well. I also do not think it is 

a purist idea to question routing /32s in a DC. Look at what big DCs are
doing today for routes, and its not

/32s.  I will also point out that one of the advantages to routing protocols
is their aggregation/summarization 

            --Tom

Tom:

 <http://datatracker.ietf.org/doc/draft-lapukhov-bgp-routing-large-dc/>
http://datatracker.ietf.org/doc/draft-lapukhov-bgp-routing-large-dc/

The IDR WG has moved on from the "purist" view to allowing DCs private AS
space and allowing link-state information to be carried in BGP.    These
usage are not Internet-wide usage, but adapt BGP to a portion of the
Internet.  If you feel host routes should not be carried, please comment to
<mailto:[email-masked]> [email-masked]  on draft-ietf-idr-ls-distribution
solution which passes LSP  Info in BGP.  It is being suggest for early
allocation of code points - so it vital to make your concerns known.

Sue Hares

From: Thomas Nadeau [ <mailto:[email-masked]>
mailto:[email-masked]] 
Sent: Tuesday, February 11, 2014 9:28 AM
To: Thomas Morin
Cc: Giles Heron; Robert Raszuk;
<mailto:[email-masked]>
[email-masked]; L3VPN
Subject: Re: /32s in DC and BGP

Hi Giles, Thomas,

            --Tom

-Thomas

Hi,

or 

Just curious ....

Cheers,
R.

< <mailto:[email-masked]> [email-masked]> wrote:

        Thomas,

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 11 Feb 2014 18:08:14 +0100

Tom,

It is common to have orchestration layer spreading given tenant VMs or
appliances across different hosts.

To reach those you need to have a route to it. And that is an overlay route
not a transport one. That's already an important distinction.

I am not sure what is your definition of "shipping around" as it is pretty
easy to divide your tenants into separate and disjoined control plane
blocks. Then you do not need to carry it all in one basket.

I think you are here not really arguing against virtual-subnet draft as
this proposes just an enhancement. You are fighting against
l3vpn-for-end-systems proposal which is based on carrying all /32s of all
VMs within a DC zone.

Please observe that this has nothing to do with FIBs .. each data plane
will still use only what it needs to serve locally attached tenant
resources.

If you aggregate at the network in the PE sure you may summarize a bit, but
then you will have all those folks asking for seamless and optimal VM
mobility across such PEs which does require precise reachability.

rgs,
R.

On Tue, Feb 11, 2014 at 5:57 PM, Thomas Nadeau <[email-masked]>wrote:

> /32s.  I will also point out that one of the advantages to routing
> protocols is their aggregation/summarization
>  [email-masked]  on draft-ietf-idr-ls-distribution solution which passes
> LSP  Info in BGP.  It is being suggest for early allocation of code points
> - so it vital to make your concerns known.
>
> Sue Hares
>
>
> *From:* Thomas Nadeau [mailto:[email-masked]<[email-masked]>
> ]
> *Sent:* Tuesday, February 11, 2014 9:28 AM
> *To:* Thomas Morin
> *Cc:* Giles Heron; Robert Raszuk;
> *Subject:* Re: /32s in DC and BGP
>
>
>
>
> Hi Giles, Thomas,
>
>             I think you are missing the point that was raised: its not
> that BGP will not scale well; as a protocol its clearly shown that it can
> ship around millions of routes and lots of other stuff as it has become the
> "dump truck" of the protocol world. The question is more of the approach -
> do we want to be shuffling around /32s (regardless of protocol) given that
> there are other solutions that work well without doing so? I would also
> question the use of BGP in a Data Center given past operational feedback in
> NVO3 for example, but that is orthogonal to the question of the mechanics
> of this solution.
>
>             --Tom
>
>
>
>
> -Thomas
>

---

From: "Susan Hares" <[email-masked]>
Date: Tue, 11 Feb 2014 12:31:41 -0500

Reposting to list . 

Sue 

From: Susan Hares [mailto:[email-masked]] 
Sent: Tuesday, February 11, 2014 11:00 AM
To: 'Thomas Nadeau'; 'Thomas Morin'
Cc: 'Giles Heron'; 'Robert Raszuk';
[email-masked]; 'L3VPN'
Subject: RE: /32s in DC and BGP

Tom: 

Sue Hares

 

Hi Giles, Thomas,

            --Tom

-Thomas

Hi,

or 

Just curious ....

Cheers,
R.

        Thomas,

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 12 Feb 2014 01:56:32 +0000

发件人: L3VPN [mailto:[email-masked]] 代表 Thomas Nadeau
发送时间: 2014年2月12日 0:58
收件人: Susan Hares
抄送: Thomas Morin; [email-masked]; L3VPN; Robert Raszuk
主题: Re: /32s in DC and BGP

Hi Tom,

Would you please share us your knowledge about what big DC are doing today for routes and how the route aggregation/summarization is performed? Thanks in advance.

Best regards,
Xiaohu

     --Tom

On Feb 11, 2014:11:00 AM, at 11:00 AM, Susan Hares <[email-masked]<mailto:[email-masked]>> wrote:

Tom:

The IDR WG has moved on from the “purist” view to allowing DCs private AS space and allowing link-state information to be carried in BGP.    These usage are not Internet-wide usage, but adapt BGP to a portion of the Internet.  If you feel host routes should not be carried, please comment to [email-masked]<mailto:[email-masked]>  on draft-ietf-idr-ls-distribution solution which passes LSP  Info in BGP.  It is being suggest for early allocation of code points – so it vital to make your concerns known.

Sue Hares

From: Thomas Nadeau [mailto:[email-masked]]
Sent: Tuesday, February 11, 2014 9:28 AM
To: Thomas Morin
Cc: Giles Heron; Robert Raszuk; [email-masked]<mailto:[email-masked]>; L3VPN
Subject: Re: /32s in DC and BGP

Hi Giles, Thomas,

            --Tom

-Thomas

Hi,

or

Just curious ....

Cheers,
R.

<[email-masked]<mailto:[email-masked]>> wrote:

        Thomas,

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 12 Feb 2014 03:19:35 +0000

Cool.

Tom,

Please don’t scrimp your word at the critical moment.  Since we really want to identify what your real concern is, say on please.

Xiaohu

     --Tom

---

From: Eric Rosen <[email-masked]>
Date: Wed, 12 Feb 2014 11:43:53 -0500

Robert> Could those who claim that that sending /32 or /64 or /128 in BGP
Robert> mainly within contained DC zone environment will not scale be a bit
Robert> more precise and kindly indicate what the real problem is ?

I think the issue is really straightforward, and has nothing specifically to
do with BGP.  My interpretation of the draft is that it is trying to meet
the following requirements:

- Allow the number of VMs to increase without bound.

- Give each VM a host address that is location-independent (and hence not
  summarizable within routing).

- Provide optimal routing from any client (presumably anywhere on the
  Internet) to any VM.

I think it is difficult to meet these requirements jointly without running
into some problems of scale.

Perhaps I'm just misunderstanding the requirements that the draft is trying
to meet.  Your mention of "contained DC zone environment" certainly suggests
that I'm overlooking some context.  Similarly, your remark in another
thread:

Robert> Inter-DC or DC to user rather always depends on careful choice and
Robert> when possible aggregation at the gateway.

suggests that summarization at some level is still going to be important for
good scaling.  However, the draft does not talk about that at all, nor does
it have normative references to other drafts that set the context.

I suppose it's possible, as Pedro seems to suggest, that my thinking is
hopelessly trapped in the last century ;-) Perhaps new developments in
hardware and/or virtualization mean that address summarizability is no
longer an important factor in scaling the routing system.  If that's the
assumption, it would be good to state it.

Note that the draft itself calls attention to these scalability issues by
explicitly positioning itself as being more scalable than an L2 solution.
While L3 is certainly more scalable than L2, much of that increased
scalability comes from L3's ability to summarize addresses on a topological
basis.

The second scaling issue has to do with rate of change.  Pedro has presented
some facts indicating that the time to complete a VM move is quite long
compared with routing convergence time.  I don't question that.  However,
that by itself does not imply anything about the rate of VM movement.  It's
not clear to me whether the rate of VM movement is supposed to be able to
increase without bound, or whether realistically this rate is expected to
always remain small compared to what routing can handle.  If the latter,
that would certainly be worth mentioning.

The third scaling issue is specific to the "ARP snooping" scheme, and to the
way that enduser activity (the generation of ARP responses) can lead to the
auto-origination of BGP routes.  Perhaps MVPN has inured us to this sort of
phenomenon.  But RFC6514 does talk of rate limiting the generation of MVPN
BGP routes, while this draft does not mention rate limiting at all.  A
related area of concern is the feedback loop where ARP responses cause BGP
activity, and BGP activity causes ARP responses. (This feedback loop doesn't
exist if the PEs get their information from the orchestration system, rather
than from snooping ARPs.)

In a previous message you said:

Robert> Just presence of learned entry in the ARP table should not trigger
Robert> the host route auto-generation.

So, where are the detailed rules relating ARP snooping to host route
auto-generation?

I do like Xiaohu's suggestion to de-emphasize the ARP snooping procedure and
to better document its applicability restrictions.  

I didn't mean to start a food fight, or to start yet another religious
discussion about the use of BGP.  I just think that the draft makes claims
of scalability that depend upon certain assumptions, and the assumptions are
not made explicit.  It doesn't bother me if the assumptions are
controversial, but I would like to know what they are.

---

From: Pedro Roque Marques <[email-masked]>
Date: Wed, 12 Feb 2014 08:55:26 -0800

> abilities; in this case we are completely defeating that.

I'll present my data.

There are two types of scaling tests that the team I work with has performed:

1. Simulation of BGP control plane scale.
2. Scale tests with functioning nested OpenStack clusters: the first layer cluster provides virtual machines that simulate hypervisors which then run nested virtual machines.

In simulation we have set a target of 4M routes and a demux factor of 2k compute-nodes per control node. 4M routes is 2 to 3 orders of magnitude higher than the maximum scale that is possible with existing commercial orchestration system enterprise offerings. There is 1 (maybe 2) organizations that have proprietary systems that can reach that scale but these are not generally available.

The simulation tests the worse case scenario of all compute-nodes rebooting simultaneously (which is impossible from a power management perspective). Expected reboot rates are about 1 per day. Proprietary orchestration systems will be able to spawn tens of VMs per minute at peek activity. Commercial offerings to not have such dynamic workloads so from a perspective of updates compute-node reboot is the stress factor.

Control plane:
- Hardware: 24 virtual cores, 128GB RAM [ Intel Xeon E5-2640 ]
- Software: opencontrail control-node

Load generator: https://github.com/Juniper/contrail-controller/blob/master/src/bgp/test/bgp_stress_test.cc
The load generator allows one to vary the number of simulated compute-nodes per process as well as number of VRFs and routes per VRF.

Some of the results:

Given that the software is freely available i would invite others to try to replicate the results.

We found the tests to be very useful in identifying concurrency issues with the software. But 60k updates per second is 3 orders of magnitude higher than the expected max requirement for a solution.

Solutions based on BGP can scale horizontally. The results above are for a pair of Route Servers. By deploying additional pairs of route servers the total numbers of compute-nodes served can be increase; one can also decrease the number of updates seen by each Route Server.

The teams conclusion is that the system is over-designed beyond what is actually a reasonable safety margin.

Becca Nitzan and Ananth Suryanarayana worked on generating the results above. They did a fantastic job over multiple months to create as much stress is possible in simulation.

With full cluster level deployment (openstack on openstack) the largest setup we where able to simulate was 800 compute-nodes running 4k virtual-machine instances. This is using 80 physical servers.
While it is a challenge to bring such as system up, the scaling of the control plane or the number of routes has never entered the picture. Please note that the simulation above is targeting a scenario where there are 1000 times more virtual-machines present.

The system above provides a full distributed routing implementation which reduces latency from inter-VM traffic. Traffic is not required to traverse the data-center fabric multiple times. It scales orders of magnitude above the requirements.

I hope that WG looks at the questions being raised from the perspective of analyzing concrete data.

   Pedro.

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 12 Feb 2014 18:39:29 +0100

Hi Eric,

My interpretation of the draft is that it is trying to meet
> into some problems of scale.

I think your interpretation is almost correct. Almost since no one is
claiming to use this "anywhere on the Internet".

Access to data centers or WAN VPNs from internet is happening via secure
gateways and this is natural place where summarization can and will happen.

Likewise if you need to join same-subnet between DC and WAN L3VPN (I
personally would recommend to avoid it) you can leak some host routes, but
this is still done within VPN scope.

> it have normative references to other drafts that set the context.

I look at this as an optional enhancement working together with L3VPN. So
L3VPN will be summarized at the zone/DC gateway when extending it to join
WAN L3VPN instances.

In the same time you can inject some host routes when needed. But this
would not mean in any deployment I would use same-subnet for everything
within DC and WAN. No need.

> assumption, it would be good to state it.
I think it all depends what the routing system you are really talking
about. It also depends if this is an overlay or not. Last and not least it
really matters if your routing system is running on new x86 CPUs or being
squeezed to run on some underpowered RP/RE blades of traditional last
century routers ;-). Should we design all protocol extensions to run on
those black sheeps ?

> scalability comes from L3's ability to summarize addresses on a topological

I guess no one will argue that. But does this means that we can not have
solutions which do not use summarization in some of their parts ?

> that would certainly be worth mentioning.

Eric .. let's keep in mind that you inject /32 when the VM get's created
.. when it's ready to take over you may prefer it over former /32 if it was
the same address.

So one can argue that starting VM will always be slower then BGP Update or
BGP Withdraw messages. Wouldn't you agree ?

>
> The third
> s
> caling issue is specific to the "ARP snooping" scheme, and to the

I would perhaps go as far as GARP trigger. But honestly this solution
works well in concert with orchestration system. 

> to better document its applicability restrictions.
+1

> controversial, but I would like to know what they are.

I think your points make perfect sense now. In fact the entire idea to
make this a WG document is not to have it blessed by IETF, but get more
feedback within the working group where people can contribute to make the
document and its applicability clearly articulated.

Best,
R.

---

From: Saku Ytti <[email-masked]>
Date: Sat, 22 Feb 2014 21:48:50 +0200

> 	I think you are missing the point that was raised: its not that BGP will not scale well; as a protocol its clearly shown that it can ship around millions of routes and lots of other stuff as it ha…

One example why I think people might want L3 (i.e. shuffling around  /32s)

Imagine you have in your own network DC in San Francisco and New York. Subnet
is advertised from San Fran.
Client in New York complains high latency, so you use XLAN or what not
to migrate their server to New York, to be closer to customer.

Before:
client -> ny -> sf ->  ny -> client
After:
client -> ny -> sf -> ny -> sf -> ny -> client

You moved server closer to customer and you doubled your latencies and reduced
MTBF.

It seems quite simple, vswitch runs BGP to advertise /32 hosts, uses proxy-arp
to connect disconnected hosts in subnet. During convergence GRE tunnel to
bounce wrongly routed packets to correct vswitch, hitlesss L3 migration with
optimal routing.

Why would you not want L3?

-- 
  ++ytti