Subject: [nvo3] To B(GP) or not to B(GP)

From: Kireeti Kompella <[email-masked]>
Date: Mon, 2 Apr 2012 18:06:58 -0700

Hi,

It would be helpful for carrier and non-carrier cloud providers to chime in
and say whether a BGP-based solution to the massive data center scaling
problem is viable.  If yes, are there caveats?  If no, would some sort of
simplification help?

Several opinions have been expressed from a wide variety of IETF
participants -- providers, vendors, hangers-on, kibitzers, etc.; the call
here is specifically for (potential) cloud providers.

If you'd rather not reply to the list, perhaps you could reply to the BoF
chairs, who can then anonymize and summarize the result?

-- 
Kireeti

PS: Benson, Matthew: hope you don't mind!

---

From: David Freedman <[email-masked]>
Date: Tue, 3 Apr 2012 01:20:57 +0000

Cloud provider wearing an SP hat,  I like the idea, I'm thinking about convergence(*), error handling (**) and path selection (***)
, but nothing that can't be solved with a number of drafts which are currently sitting inside IDR now.

I think it would be useful to more clearly realise the architecture being proposed here (for BGP) , in order to understand the
applicability of any WIP enhancements (and how they could be prioritised if need be in order to deliver something of sufficient robustness…)

Dave.

Example problems:

* mainly recovery related to failure detection
** malformed / bad updates killing large reachability bearing sessions
*** related to convergence/fast recovery and / or load balancing

From: Kireeti Kompella <[email-masked]<mailto:[email-masked]>>
Date: Mon, 2 Apr 2012 18:06:58 -0700
To: <[email-masked]<mailto:[email-masked]>>
Subject: [nvo3] To B(GP) or not to B(GP)

Hi,

It would be helpful for carrier and non-carrier cloud providers to chime in and say whether a BGP-based solution to the massive data center scaling problem is viable.  If yes, are there caveats?  If no, would some sort of simplification help?

Several opinions have been expressed from a wide variety of IETF participants -- providers, vendors, hangers-on, kibitzers, etc.; the call here is specifically for (potential) cloud providers.

If you'd rather not reply to the list, perhaps you could reply to the BoF chairs, who can then anonymize and summarize the result?

--
Kireeti

_______________________________________________ nvo3 mailing list [email-masked]<mailto:[email-masked]> https://www.ietf.org/mailman/listinfo/nvo3

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 03 Apr 2012 07:32:12 +0200

Hi Kireeti,

 > call here is specifically for (potential) cloud providers.

Since this is exactly the project of my employer's new "Cloud Vision" 
let me provide my comments.

I don't know why you keep insisting on taking the problem completely 
backwords. Should we use BGP or not really depends for what (ie to 
define session end points, NLRI format, dynamics of information, p2p vs 
p2mp nature of information distribution, requirement for day one RTC etc 
..... essentially to define service primitives I asked for in the other 
email)

If one requires that the encap/decap is done at each hypervisor please 
explain what robust and production quality BGP implementation is part of 
standard linux distribution today ?

Next is assuming I have 10K-100K hypervisors/hosts *and there are bigger 
data centers too* please show me which BGP RR can scale to such number 
of BGP sessions today ? And let's not assume folks around would be using 
hierarchy of route reflectors day one.

I hope this WG will not lock on one protocol then try to limit the 
solution scope to match such protocols capabilities. That would be 
completely not helpful to the progress or to the industry acceptance of 
this WG specifications.

Best regards,
R.

> Hi,
>
> It would be helpful for carrier and non-carrier cloud providers to chime
> in and say whether a BGP-based solution to the massive data center
> scaling problem is viable.  If yes, are there caveats?  If no, would
> some sort of simplification help?
>
> participants -- providers, vendors, hangers-on, kibitzers, etc.; the
> call here is specifically for (potential) cloud providers.
>
> If you'd rather not reply to the list, perhaps you could reply to the
> BoF chairs, who can then anonymize and summarize the result?
>
> --
> Kireeti
>
> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3

---

From: <[email-masked]>
Date: Thu, 5 Apr 2012 11:51:20 +0200

Hi Robert,

In NVO3, we are looking both at the dataplane/encap issue, and on the 
control plane issue.

On the control plane issue, there seems to be a large agreement that the 
pseudo-control plane based on dataplane-learning, is not good enough.

To me,  considering BGP as an option for the control plane, does *not* 
sound as "taking the problem backwards".
Indeed, as far as I know, there aren't many options on the table for 
this control plane.  MP-BGP E-VPN would be one, 
draft-marques-l3vpn-end-system is another (although not focusing on what 
is proposed as the primary goal of NVO3, i.e. Ethernet connectivity), 
but where are the alternatives described, including how they would map 
to an NVO3 use-case ?

So, my take is that for this discussing to progress, we need to 
understand the delta between what already existing solutions can achieve 
and the requirements.  We might very well conclude we need something 
new, but we aren't there yet.  For instance, we can hear different 
things about why BGP wouldn't be good enough, here and there. But the 
discussions on these aspects has for now been, at best, very diffuse : 
would not scale to 10k/100k hosts ? need for transactional strong 
consistency... ?   If we end up exploring this, we'll need to see what 
the atlernatives would be and why they would achieve the goals where 
current solutions would presumably fail.

Robert:
> completely not helpful to the progress or to the industry acceptance 
> of this WG specifications. 

Agreed.
But locking one option out, a priori, would not help either.  The reason 
is that "the industry" is multiplayer here, this is especially true when 
you consider that solutions developed by the hypervisor/server industry 
need to bridge with the network device industry -- which we'll need to 
do to get NVO3 traffic out of NVO3 islands.

-Thomas

Robert Raszuk :
> define session end points, NLRI format, dynamics of information, p2p 
> vs p2mp nature of information distribution, requirement for day one 
> RTC etc ..... essentially to define service primitives I asked for in 
> the other email)
>
> explain what robust and production quality BGP implementation is part 
> of standard linux distribution today ?
>
> Next is assuming I have 10K-100K hypervisors/hosts *and there are 
> bigger data centers too* please show me which BGP RR can scale to such 
> number of BGP sessions today ? And let's not assume folks around would 
> be using hierarchy of route reflectors day one.
>
>
>
> Best regards,
> R.
>
>> Hi,
>>

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 05 Apr 2012 12:20:10 +0200

Hi Thomas,

Just few points ..

 > sound as "taking the problem backwards".

I think there is fundamental misunderstanding what one understands by 
"control plane" to start with.

All who push BGP assume NVO3 is a plumber/sealer to interconnect pipes 
between VMs. I don't see it that way at all.

For me NVO3 solution should be complete. VMs do not appear out of the 
starts in DCs. They are instantiated by orchestration systems and I 
would highly recommend we start from there to see what is missing to 
interconnect such VMs in topology of choice of the users/customers.

As this is over L3 we can assume we have any to any underlining 
transport connectivity. So by use of proper encapsulation one could 
point out that said orchestration system is completely capable to 
deliver tenant's VM virtualization within and across data centers.

Network itself or for that matter any network protocol does not need to 
participate in the game at all. What is missing is common language 
between DC orchestration systems. That is IMHO the most important 
specification NVO3 could produce.

Unfortunately most people who push BGP think that DC virtualization is 
just yet another L2/L3VPN and of course they see opportunity to make it 
network based provided by service provider as opposed to think of it as 
host/hypervisor based constructed by the layer above.

I think what would be really helpful at this time is for the charter to 
explicitly state that any solution this group would come up with has to 
include hosts/hypervisors as the virtualization mux/demux points.

If someone who is in the process of building data center visualization 
would like to exclude multi-tenancy virtualization to start at that 
level it would be very constructive to provide reasoning behind it.

Of course such demarcation point could, but does not need to be 
exclusive. However let's notice that if someone wants to put PEs at TOR 
or DC aggregation points and just pipe automagically instantiated VMs to 
such device I guess there is nothing that prevents him from doing it 
already today so I am not sure how NVO3 work and outcome would be helpful.

Regards,
R.

---

From: <[email-masked]>
Date: Thu, 5 Apr 2012 13:42:38 +0200

Hi Robert,

> include hosts/hypervisors as the virtualization mux/demux points.

I agree that including hosts/hypervisors as the virtualization mux/demux 
points has to be one option. But I don't see why we should take it for 
granted that the use of BGP implies that the mux/demux point has to be a 
network device such as the ToR.

We'll have to bridge the cultural gap between "server folks" and 
"network devices" folks at some point. For this to be possible, you 
can't take it for granted that an approach that for now has been mostly 
implemented only on one side, cannot be implemented on the other. If we 
do, we won't solve the cases where both need to interconnect.

> already today so I am not sure how NVO3 work and outcome would be helpful.

I don't agree here. There is a missing piece in the "just pipe 
automagically instantiated VMs to such device".

-Thomas

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 05 Apr 2012 14:02:40 +0200

Hi Thomas,

> do, we won't solve the cases where both need to interconnect.

Take example of IPSec VPNs, take example of skype, take example of http.

Where do they "interconnect" with network devices ? Last time I checked 
they are happily use network as transport as any of us use airlines to 
move from A to B without talking much with pilots nor ATC leave alone 
sitting in the cockpits.

And Shane's "interconnect" point is very valid and needed however it is 
not what you describe as bridging cultural gap. IMO the optimal place to 
accommodate his point for application optimization is at the VM itself 
level via DNS, ALTO, NPS etc.. In the same time I would trust network 
level without interacting with it that BGP and IGP will not send packets 
round the world to reach B.

Best,
R.

---

From: <[email-masked]>
Date: Thu, 5 Apr 2012 17:03:05 +0200

Hi Robert,

> Take example of IPSec VPNs, take example of skype, take example of http.

I'm not sure analogy-based reasoning always helps...

> move from A to B without talking much with pilots nor ATC leave alone

Beyond IP connectivity, IPSec does not need cooperation from the network 
carrying IPsec packets, but IPsec does, in many cases, need to terminate 
on network devices. So, it has to "interconnect" in some kind of way. 
But, again, I'm not sure what analogy-based reasoning brings here...

( Skype and HTTP are not packet forwarding, in no place do they need to 
terminate on a router - I don't get where the analogy can help)

Let me clarify what I meant when saying "interconnect": the need to have 
a router gateways NVO3 packets to the outside world, outside world being 
possibly the Internet, or for some use cases, IP VPNs or Ethernet VPNs. 
For all these cases, if the cultural gap is not bridged in some kind of 
way, the interconnect will likely have to rely on a lowest common 
denominator, such as mapping overlays to VLANs on one side, and VLANs to 
the GRT or a VRF/VSI on the other side. This is a kind of poor man's 
interconnect which, based on experience, reveals as being very inefficient.

-Thomas

---

From: Paul Unbehagen <[email-masked]>
Date: Thu, 5 Apr 2012 09:12:32 -0600

I tend to agree with Robert and Thomas on this topic as in many data centers, especially enterprise, BGP will be overkill.  Between data enters though, BGP could be very useful especially when considered with the presentation that David Black gave on domain of ownership, but still an end to end service.

BGP is a very useful protocol, but not the answer to every situation or network type.  More important to think about is Shane's comment on routing metrics.  We need to look at how routing metrics underneath these overlays can affect forwarding or risk VM moves taking more circuitous paths than a famous San Francisco road.

--
Paul Unbehagen

Sent from my iPad