Subject: [nvo3] Second Draft Charter Update for Discussion

From: Benson Schliesser <[email-masked]>
Date: Sat, 23 Aug 2014 12:49:21 -0400

Dear NVO3 Contributors -

Over the past couple weeks, as you’ve reviewed the proposed charter update, Matthew and I have received a great deal of feedback both on- and off-list. Based on all of this input we have developed a revision that we hope is more clear and accurate. Please review the text below and provide any feedback that you may have within the next week.

Thanks,
-Benson & Matthew

http://svn.tools.ietf.org/svn/wg/nvo3/charter-ietf-nvo3-01-rev-20140823.txt

An NVO3 solution is a set of protocols and/or protocol extensions that enable
network virtualization within a data center (DC) environment using an IP-based
overlay approach. It provides layer 2 and layer 3 services for virtual networks
enabling multi-tenancy, workload mobility, optimization, management, and
security, addressing the issues described in the problem statement and
consistent with the framework previously produced by the NVO3 WG.

The NVO3 WG will develop solutions for network virtualization based on the
following architectural tenets:
 - Support for an IP-based underlay data plane
 - A logically centralized control plane for network virtualization
Network virtualization approaches that do not adhere to these tenets are
explicitly outside of the scope of the NVO3 WG.

In pursuit of the solutions described above, the NVO3 WG will document an
architecture for network virtualization within a data center environment.

The NVO3 WG may produce requirements for a network virtualization control
plane, and will select, extend, and/or develop one or more control plane
protocols to support the architecture. Such protocols are expected to fulfill
the communication requirements between a Tenant System (TS) and Network
Virtualization Edge (NVE), and between an NVE and the Network Virtualization
Authority (NVA). The internal mechanisms and protocols of a logically
centralized NVA are explicitly out of scope of the NVO3 WG.  Architectural
issues raised by coexistence of multiple logically centralized control planes
in the same data center may be considered by the WG.  Inter-DC mechanisms are
not in scope of the NVO3 WG at this time.

The NVO3 WG may produce requirements for network virtualization data planes
based on encapsulation of virtual network traffic over an IP-based underlay
data plane. Such requirements should consider OAM and security. Based on these
requirements the WG will select, extend, and/or develop one or more data plane
encapsulation format(s).

Additionally, the WG may document common use-cases for NVO3 solutions.

The working group may choose to adopt a protocol or data encapsulation that was
previously worked on outside the IETF as the basis for the WG's work.  If the
NVO3 WG anticipates the adoption of the technologies of another SDO as part of
the selected protocols or data encapsulation, the NVO3 WG will first liaise
with that SDO.

BGP-based solutions to network virtualization within a data center environment
will be developed in the BGP-Enabled Services (BESS) WG.

MILESTONES

Done - Problem Statement submitted for IESG review
Done - Framework document submitted for IESG review
TBD - Architecture submitted for IESG review
TBD - TS to NVE Control Plane Protocol Adopted by WG
TBD - NVE to NVA Control Plane Protocol Adopted by WG
TBD - NVE Data Plane Protocol Adopted by WG
TBD - TS to NVE Control Plane Protocol Submitted for IESG review
TBD - NVA to NVA Control Plane Protocol Submitted for IESG review
TBD - NVE Data Plane Protocol Submitted for IESG review
TBD - Recharter or close WG

OPTIONAL DOCUMENTS
(we intend to produce these if they're helpful to the WG, but are not obliged
to do so)

Control Plane Requirements
Data Plane Requirements
Use-cases

---

From: Tina TSOU <[email-masked]>
Date: Sun, 24 Aug 2014 23:00:46 +0000

Dear Benson,

How are we going to deal with draft like http://datatracker.ietf.org/doc/draft-ashwood-nvo3-oam-requirements/
?

Thank you,
Tina

On Aug 24, 2014, at 12:49 AM, "Benson Schliesser" <[email-masked]<mailto:[email-masked]>> wrote:

Dear NVO3 Contributors -

Thanks,
-Benson & Matthew

MILESTONES

OPTIONAL DOCUMENTS

_______________________________________________
nvo3 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nvo3

---

From: Osama Zia <[email-masked]>
Date: Mon, 25 Aug 2014 21:30:45 +0000

Hi Benson,

This is indeed clearer than before. Thanks for putting in the effort.

I have some comments.

1. I am getting the impression that NVO3 WG will only look at logically
centralized control plane. All decentralized solutions such as BGP based
should be looked separately. Is this correct?
2. The statement says that "The NVO3 WG will develop solutions for network
virtualization based on the following architectural tenets". However, the
control plane requirements and data plane requirements are optional.
Shouldn't architecture be based on certain requirements?

Regards,
Osama

-----Original Message-----
From: nvo3 [mailto:[email-masked]] On Behalf Of Benson Schliesser
Sent: Saturday, August 23, 2014 9:49 AM
To: [email-masked]
Subject: [nvo3] Second Draft Charter Update for Discussion

Dear NVO3 Contributors -

Over the past couple weeks, as you've reviewed the proposed charter update,
Matthew and I have received a great deal of feedback both on- and off-list.
Based on all of this input we have developed a revision that we hope is more
clear and accurate. Please review the text below and provide any feedback
that you may have within the next week.

Thanks,
-Benson & Matthew

An NVO3 solution is a set of protocols and/or protocol extensions that
enable network virtualization within a data center (DC) environment using an
IP-based overlay approach. It provides layer 2 and layer 3 services for
virtual networks enabling multi-tenancy, workload mobility, optimization,
management, and security, addressing the issues described in the problem
statement and consistent with the framework previously produced by the NVO3
WG.

The NVO3 WG will develop solutions for network virtualization based on the
following architectural tenets:
 - Support for an IP-based underlay data plane
 - A logically centralized control plane for network virtualization Network
virtualization approaches that do not adhere to these tenets are explicitly
outside of the scope of the NVO3 WG.

The NVO3 WG may produce requirements for a network virtualization control
plane, and will select, extend, and/or develop one or more control plane
protocols to support the architecture. Such protocols are expected to
fulfill the communication requirements between a Tenant System (TS) and
Network Virtualization Edge (NVE), and between an NVE and the Network
Virtualization Authority (NVA). The internal mechanisms and protocols of a
logically centralized NVA are explicitly out of scope of the NVO3 WG.
Architectural issues raised by coexistence of multiple logically centralized
control planes in the same data center may be considered by the WG.
Inter-DC mechanisms are not in scope of the NVO3 WG at this time.

The NVO3 WG may produce requirements for network virtualization data planes
based on encapsulation of virtual network traffic over an IP-based underlay
data plane. Such requirements should consider OAM and security. Based on
these requirements the WG will select, extend, and/or develop one or more
data plane encapsulation format(s).

The working group may choose to adopt a protocol or data encapsulation that
was previously worked on outside the IETF as the basis for the WG's work.
If the
NVO3 WG anticipates the adoption of the technologies of another SDO as part
of the selected protocols or data encapsulation, the NVO3 WG will first
liaise with that SDO.

BGP-based solutions to network virtualization within a data center
environment will be developed in the BGP-Enabled Services (BESS) WG.

MILESTONES

Done - Problem Statement submitted for IESG review Done - Framework document
submitted for IESG review TBD - Architecture submitted for IESG review TBD -
TS to NVE Control Plane Protocol Adopted by WG TBD - NVE to NVA Control
Plane Protocol Adopted by WG TBD - NVE Data Plane Protocol Adopted by WG TBD
- TS to NVE Control Plane Protocol Submitted for IESG review TBD - NVA to
NVA Control Plane Protocol Submitted for IESG review TBD - NVE Data Plane
Protocol Submitted for IESG review TBD - Recharter or close WG

OPTIONAL DOCUMENTS
(we intend to produce these if they're helpful to the WG, but are not
obliged to do so)

[email-masked]

---

From: Benson Schliesser <[email-masked]>
Date: Mon, 25 Aug 2014 18:16:17 -0500

Hi, Tina.

Speaking broadly about requirements drafts: The proposed charter allows the
WG to publish requirements. They are described as items the WG "may
produce" because they're not blocking further work. (I.e. requirements are
non-gating, and the work on solutions is not serialized.) The purpose of
working on requirements is to assist the WG in developing solutions.

The current OAM requirements draft is (by its own description) high level
and broad. The WG could choose to publish it in this form, to simply set
high level expectations as we continue our work on solutions. And/or the WG
could choose to integrate it into control plane and/or data plane
requirements drafts, elaborating on the general text in ways that are more
specific to each draft's topic. And/or the WG could choose to abandon it.
But we would need to show WG consensus for doing any of these things. And I
don't think that consensus is yet clear in this case.

In my personal opinion, I'd like to see the text evolve to be more specific
as a stand-alone document *and* have the OAM requirement ideas integrated
more tightly into the various data plane and control plane drafts. I have
not heard (nor imagined) any compelling reasons to abandon it. But I also
do not think it's quite "done" at this point.

Cheers,
-Benson

On Sun, Aug 24, 2014 at 6:00 PM, Tina TSOU <[email-masked]>
wrote:

>  Dear Benson,
>
>  How are we going to deal with draft like
> http://datatracker.ietf.org/doc/draft-ashwood-nvo3-oam-requirements/
> ?
>
>
>  Thank you,
> Tina
>
> On Aug 24, 2014, at 12:49 AM, "Benson Schliesser" <[email-masked]>
> wrote:
>
>  Dear NVO3 Contributors -
>
> Over the past couple weeks, as you’ve reviewed the proposed charter
> update, Matthew and I have received a great deal of feedback both on- and
> off-list. Based on all of this input we have developed a revision that we
> hope is more clear and accurate. Please review the text below and provide
> any feedback that you may have within the next week.
>
> Thanks,
> -Benson & Matthew
>
> enable
> fulfill
> these
> environment
> obliged

---

From: Tina TSOU <[email-masked]>
Date: Mon, 25 Aug 2014 23:47:40 +0000

Dear Benson,

Fair enough, thank you for your prompt reply.

We will work on the revision to meet this expectation.

Thank you,
Tina

On Aug 26, 2014, at 7:16 AM, "Benson Schliesser" <[email-masked]<mailto:[email-masked]>> wrote:

Hi, Tina.

Speaking broadly about requirements drafts: The proposed charter allows the WG to publish requirements. They are described as items the WG "may produce" because they're not blocking further work. (I.e. requirements are non-gating, and the work on solutions is not serialized.) The purpose of working on requirements is to assist the WG in developing solutions.

The current OAM requirements draft is (by its own description) high level and broad. The WG could choose to publish it in this form, to simply set high level expectations as we continue our work on solutions. And/or the WG could choose to integrate it into control plane and/or data plane requirements drafts, elaborating on the general text in ways that are more specific to each draft's topic. And/or the WG could choose to abandon it. But we would need to show WG consensus for doing any of these things. And I don't think that consensus is yet clear in this case.

In my personal opinion, I'd like to see the text evolve to be more specific as a stand-alone document *and* have the OAM requirement ideas integrated more tightly into the various data plane and control plane drafts. I have not heard (nor imagined) any compelling reasons to abandon it. But I also do not think it's quite "done" at this point.

Cheers,
-Benson

On Sun, Aug 24, 2014 at 6:00 PM, Tina TSOU <[email-masked]<mailto:[email-masked]>> wrote:
Dear Benson,

Thank you,
Tina

Dear NVO3 Contributors -

Thanks,
-Benson & Matthew

MILESTONES

OPTIONAL DOCUMENTS

---

From: Benson Schliesser <[email-masked]>
Date: Mon, 25 Aug 2014 18:48:54 -0500

Hi, Osama.

> should be looked separately. Is this correct?

Yes. To elaborate:

Existing and new solutions that are based on BGP to provide network
virtualization would be developed in the (to-be-formed) BESS working group.
This would be the case for all BGP-based solutions, regardless of whether
they have a control plane architecture that is distributed or centralized.

It is possible that NVO3 solutions could leverage BGP and/or that BGP-based
solutions could leverage NVO3 work. In these cases we would work cross-WG
to make sure that the solution is documented completely and accurately,
with work happening in the most appropriate place(s). For example one could
imagine combining a NVO3-developed data plane encap, a NVO3-developed
TS-NVE control plane, and a BESS-developed NVA control plane. (There are
many other possible combinations here; I'm not trying to be exhaustive,
just illustrate the idea.)

Similar to what I've described above, NVO3 solutions might leverage
distributed protocols in various ways, resulting in cross-WG work with
other WGs. For example, it is possible that the centralized NVE-NVA control
plane protocol is responsible for certain mapping functions whilst other
functions are provided by distributed protocols, such as liveness detection
via IGP-provided link state etc. (Again, not trying to be exhaustive, just
illustrating the idea.)

> Shouldn't architecture be based on certain requirements?

Requirements are certainly helpful, and as an NVO3 chair I do intend to
encourage their development. But one of the goals of this recharter is to
begin working on solutions in parallel, rather than in serial, with
requirements. Indeed, the purpose of working on requirements is to assist
the WG in developing solutions (requirements on their own have little
value). And so with this new charter text we have given ourselves the
flexibility to manage the production of requirements in whatever way is
best for the WG.

Cheers,
-Benson

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Tue, 26 Aug 2014 01:22:02 +0000

Hi Benson,

Overall, I think this is much better.  I have a few nits inline below.

Thanks, Larry

On 8/23/14 9:49 AM, "Benson Schliesser" <[email-masked]> wrote:

>Dear NVO3 Contributors -
>
>Over the past couple weeks, as you¹ve reviewed the proposed charter
>overlay approach. It provides layer 2 and layer 3 services

LK> I would say "layer 2 and/or layer 3", saying only and implies both
services are always present in an NVO3 solution.

>for virtual networks
>Virtualization Edge (NVE),

LK> I think when you say TS above, you really mean the Hypervisor or an
End Device such as a Network Service Appliance where the NVE is split with
the encap/decap happening in another device, such as a TOR switch.  We
have been steering clear of having the untrusted Tenant Systems signaling
what networks they want to connect to.

>and between an NVE and the Network Virtualization

---

From: "Fedyk, Don" <[email-masked]>
Date: Tue, 26 Aug 2014 14:32:00 +0000

Hi Benson

> will be developed in the BGP-Enabled Services (BESS) WG.

It almost sounds as if BGP solutions are out of scope for NVO3. 

What I think is important is:
If an NVO3 solution specifies an existing protocol developed in another IETF WG, any specific extensions to that protocol should be coordinated with and may be developed within the respective WG. For example any BGP-based extensions for NVO3 BGP-based solutions will be developed in the BGP-Enabled Services (BESS) WG.

I see BGP solutions as in scope for NVO3 but the BESS WG has the oversight on the extensions. 

Cheers,
Don

---

From: Osama Zia <[email-masked]>
Date: Tue, 26 Aug 2014 16:34:58 +0000

Hi Benson,

Sounds good to me. Thanks for the clarification.

Regards,

Osama

From: Benson Schliesser [mailto:[email-masked]]
Sent: Monday, August 25, 2014 4:49 PM
To: Osama Zia
Cc: [email-masked]
Subject: Re: [nvo3] Second Draft Charter Update for Discussion

Hi, Osama.

On Mon, Aug 25, 2014 at 4:30 PM, Osama Zia <[email-masked] 
<mailto:[email-masked]> > wrote:

Yes. To elaborate:

Existing and new solutions that are based on BGP to provide network 
virtualization would be developed in the (to-be-formed) BESS working group. 
This would be the case for all BGP-based solutions, regardless of whether they 
have a control plane architecture that is distributed or centralized.

It is possible that NVO3 solutions could leverage BGP and/or that BGP-based 
solutions could leverage NVO3 work. In these cases we would work cross-WG to 
make sure that the solution is documented completely and accurately, with work 
happening in the most appropriate place(s). For example one could imagine 
combining a NVO3-developed data plane encap, a NVO3-developed TS-NVE control 
plane, and a BESS-developed NVA control plane. (There are many other possible 
combinations here; I'm not trying to be exhaustive, just illustrate the idea.)

Similar to what I've described above, NVO3 solutions might leverage 
distributed protocols in various ways, resulting in cross-WG work with other 
WGs. For example, it is possible that the centralized NVE-NVA control plane 
protocol is responsible for certain mapping functions whilst other functions 
are provided by distributed protocols, such as liveness detection via 
IGP-provided link state etc. (Again, not trying to be exhaustive, just 
illustrating the idea.)

Requirements are certainly helpful, and as an NVO3 chair I do intend to 
encourage their development. But one of the goals of this recharter is to 
begin working on solutions in parallel, rather than in serial, with 
requirements. Indeed, the purpose of working on requirements is to assist the 
WG in developing solutions (requirements on their own have little value). And 
so with this new charter text we have given ourselves the flexibility to 
manage the production of requirements in whatever way is best for the WG.

Cheers,

-Benson

---

From: Benson Schliesser <[email-masked]>
Date: Tue, 26 Aug 2014 21:46:42 -0400

Hi, Larry.

> Overall, I think this is much better.  I have a few nits inline below.

Thanks for the constructive feedback. Comments inline.

> services are always present in an NVO3 solution.

Agreed.

> what networks they want to connect to.

Your interpretation is correct, and I can see that my text is not right. If I change TS to "End Device” does that make it correct? Or do you have a different suggestion for the text?

Thanks,
-Benson

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Wed, 27 Aug 2014 03:47:12 +0000

Hi Benson,

Yes, the term "End Device" is consistent with
draft-ietf-nvo3-hpvr2nve-cp-req-00
<https://datatracker.ietf.org/doc/draft-ietf-nvo3-hpvr2nve-cp-req/>, and
the NVO3 Framework document.

Thanks, Larry

On 8/26/14 6:46 PM, "Benson Schliesser" <[email-masked]> wrote:

>Your interpretation is correct, and I can see that my text is not right.
>If I change TS to "End Device² does that make it correct? Or do you have
>a different suggestion for the text?
>
>Thanks,
>-Benson
>
>
>

---

From: Pat Thaler <[email-masked]>
Date: Thu, 28 Aug 2014 00:23:28 +0000

With those changes, the draft charter update looks good to me. Pat

-----Original Message-----
From: nvo3 [mailto:[email-masked]] On Behalf Of Larry Kreeger (kreeger)
Sent: Tuesday, August 26, 2014 8:47 PM
To: Benson Schliesser
Cc: [email-masked]
Subject: Re: [nvo3] Second Draft Charter Update for Discussion

Hi Benson,

Thanks, Larry

---

From: Benson Schliesser <[email-masked]>
Date: Wed, 27 Aug 2014 20:54:09 -0400

Hi, Don.

Thanks for your comment. It is actually a very good question and a difficult one to answer, which is why I took and extra day to respond...

> It almost sounds as if BGP solutions are out of scope for NVO3. 

Yes, that is the idea. BGP solutions are out of scope. But it is probably worth expanding on what that word “scope” means in this case, because it’s slightly ambiguous. See my next comment.

> I see BGP solutions as in scope for NVO3 but the BESS WG has the oversight on the extensions. 

The difficult question is how to apply the idea of scope to “solutions”. It’s somewhat easier to decide where specific protocol work should happen, based on the text of the proposed charter. So let’s assume that as a starting point - we know where protocols will be developed. (And if we don’t, then we’ll figure it out when examples arise.)

But if a solution is a combination of multiple protocols (e.g. data and control plane), and some of those protocols were developed by different WG(s), then it might be less clear where the solution belongs. 

Before we can figure out the right answer, I suppose we ought to define what a “solution” actually is in this context: I posit that it is a document describing how to use one or more protocols to solve the problem. In other words, it is an Applicability Statement (AS) document.

If an AS describes how to solve for the issues outlined in the NVO3 Problem Statement in a manner consistent with the Architecture, then arguably that AS document should be developed in the NVO3 WG. If the AS solves for a different problem and/or is inconsistent with our Architecture, then it probably belongs to a different WG. Alternatively (and perhaps very likely), if the AS crosses WG boundaries and isn’t clear, then it’s up to the WG management (chairs and ADs) to make sure it gets cross-WG review and ultimately decide which WG should develop it.

In the case of BGP-based solutions, we are agreeing up-front that AS documents with BGP-based control planes will be developed in the BESS WG. This is being called out explicitly because it is possible to imagine BGP used in a centralized fashion (such as a route-reflector NVA model), and we intend for that work to happen in BESS even though it resembles the NVO3 architecture.

All that being said, if we develop a solution that leverages BGP protocol extensions but also has a significant non-BGP NVA control plane, then we will evaluate where it belongs - and it may very well belong in NVO3. However it isn’t a good idea to pre-judge where such work would belong until we have a real example… I’m just explaining my thought process, in case it helps you (and others) understand how it might evolve.

Of course, I’d welcome any additional feedback on this. But at this time I’m going to leave the referenced charter text as-is.

Cheers,
-Benson

---

From: "Manish Kumar (manishkr)" <[email-masked]>
Date: Thu, 28 Aug 2014 16:21:13 +0000

Hi Benson,

The same text ("A logically centralized control plane for network virtualization")  was bothering me for days as well! However, having been out of touch with developments in the working group for the past few months, I thought I better do the homework to check if this has already been covered in some discussions earlier. I also see the intent of keeping out BGP and or MPLS (L2VPN/L3VPN) based solutions because work for the same is going on (or would be done) in other working groups.

The final solution, as you say, could very well involve centralised and/or distributed components. Even the updated framework document talks about a balance needed - "Domain and/or deployment specific constraints define the balance between centralized and distributed approaches". Given these, as well as based on other limitations of a centralised model, I still don't understand the rationale behind the text - "A logically centralized control plane for network virtualization" in the charter update. Excluding BGP and/or MPLS based solutions for the reason above makes sense but not distributed models summarily. Am I missing something here ?

Thanks,
Manish

From: Benson Schliesser <[email-masked]<mailto:[email-masked]>>
Date: Tuesday, 26 August 2014 5:18 AM
To: Osama Zia <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [nvo3] Second Draft Charter Update for Discussion

Hi, Osama.

On Mon, Aug 25, 2014 at 4:30 PM, Osama Zia <[email-masked]<mailto:[email-masked]>> wrote:
1. I am getting the impression that NVO3 WG will only look at logically
centralized control plane. All decentralized solutions such as BGP based
should be looked separately. Is this correct?

Yes. To elaborate:

Existing and new solutions that are based on BGP to provide network virtualization would be developed in the (to-be-formed) BESS working group. This would be the case for all BGP-based solutions, regardless of whether they have a control plane architecture that is distributed or centralized.

It is possible that NVO3 solutions could leverage BGP and/or that BGP-based solutions could leverage NVO3 work. In these cases we would work cross-WG to make sure that the solution is documented completely and accurately, with work happening in the most appropriate place(s). For example one could imagine combining a NVO3-developed data plane encap, a NVO3-developed TS-NVE control plane, and a BESS-developed NVA control plane. (There are many other possible combinations here; I'm not trying to be exhaustive, just illustrate the idea.)

Similar to what I've described above, NVO3 solutions might leverage distributed protocols in various ways, resulting in cross-WG work with other WGs. For example, it is possible that the centralized NVE-NVA control plane protocol is responsible for certain mapping functions whilst other functions are provided by distributed protocols, such as liveness detection via IGP-provided link state etc. (Again, not trying to be exhaustive, just illustrating the idea.)

Requirements are certainly helpful, and as an NVO3 chair I do intend to encourage their development. But one of the goals of this recharter is to begin working on solutions in parallel, rather than in serial, with requirements. Indeed, the purpose of working on requirements is to assist the WG in developing solutions (requirements on their own have little value). And so with this new charter text we have given ourselves the flexibility to manage the production of requirements in whatever way is best for the WG.

Cheers,
-Benson

---

From: "Thomas D. Nadeau" <[email-masked]>
Date: Thu, 28 Aug 2014 12:25:53 -0400

> The final solution, as you say, could very well involve centralised and/or distributed components. Even the updated framework document talks about a balance needed - "Domain and/or deployment speci…

	I think you are. Logically centralized covers physically centralized and only allows you to have more resiliency/scale/performance than a single instance.  I would also think that no one in a production environment would install a single centralized instance anyways, so we need logically centralized to cover real-world scenarios.

	--Tom

> 
> Thanks,
> Manish
> 
> From: Benson Schliesser <[email-masked]>
> To: Osama Zia <[email-masked]>
> Cc: "[email-masked]" <[email-masked]>

---

From: "Manish Kumar (manishkr)" <[email-masked]>
Date: Thu, 28 Aug 2014 16:37:52 +0000

Tom,

My question/concern was quite the opposite. Why logically centralised; why not even logically distributed ? I would rather think leaving it open to accommodate both (logically centralised or distributed) depending on deployment scenario, etc would be more sensible. A physically centralised one, obviously, as you said, would hardly be practically unusable.

- Manish

From: "Thomas D. Nadeau" <[email-masked]<mailto:[email-masked]>>
Date: Thursday, 28 August 2014 9:55 PM
To: Manish Kumar <[email-masked]<mailto:[email-masked]>>
Cc: Benson Schliesser <[email-masked]<mailto:[email-masked]>>, Osama Zia <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [nvo3] Second Draft Charter Update for Discussion

On Aug 28, 2014:12:21 PM, at 12:21 PM, Manish Kumar (manishkr) <[email-masked]<mailto:[email-masked]>> wrote:

Hi Benson,

--Tom

Thanks,
Manish

Hi, Osama.

Yes. To elaborate:

Cheers,
-Benson

---

From: Sharon Barkai <[email-masked]>
Date: Thu, 28 Aug 2014 18:20:21 +0000

Must say I also struggled with the term logically centralized because for all practical purposes people will distribute the NVA, but then realized the point was - we could distribute the NVA based on the fact that there is fully functional IP underlay network that can be used for that. It is there by definition. And so this workgroup should not specify how to implement distributed software on an existing IP network, it should specify what this software does using which interfaces. If then people insist on using BGP, PNNI, IN to distribute states it's their call, other people may use simpler and more functional means.

--szb

On Aug 28, 2014, at 9:26, "Thomas D. Nadeau" <[email-masked]<mailto:[email-masked]>> wrote:

Hi Benson,

--Tom

Thanks,
Manish

Hi, Osama.

Yes. To elaborate:

Cheers,
-Benson

---

From: Benson Schliesser <[email-masked]>
Date: Fri, 29 Aug 2014 00:09:21 -0400

Hi, Manish. Your question about centralized vs distributed control planes is actually very important, and something that I’ve discussed offline quite a bit. So let me see if I can answer it here, for your benefit as well as the rest of the WG.

I think Sharon described this accurately. To elaborate: In a logically centralized NVA architecture, we could envision multiple interfaces including NVE-NVA, intra-NVA (clustering), and inter-NVA (federation). In a fully distributed control plane, however, we might imagine having only a single NVE-to-NVE control interface.

In the proposed charter text, we have said that a fully distributed control plane is not in NVO3 WG’s scope. This is not making a statement about whether that approach is valid, invalid, good, bad, etc, but rather is intended to focus our current work on the centralized architecture.

Likewise, we have said that the intra-NVA (clustering) interface is not in scope. Again, this is not making a statement about whether it is necessary, good, bad, etc - because it obviously is necessary - but rather is intended to focus our work. Further, it seems likely that such an interface will not benefit from open specification and/or standardization in the near future, as implementors may use various mechanisms of their own choosing within their implementation.

Of course, work on things like clustering, fully distributed control planes, etc, might be done in other WGs now. And/or the NVO3 WG might recharter to address them in the future. But our proposed charter text doesn’t take these into account at this time, just for the sake of focus.

I hope this helps answer your question.

Cheers,
-Benson

On Aug 28, 2014, at 2:20 PM, Sharon Barkai <[email-masked]> wrote:

> On Aug 28, 2014, at 9:26, "Thomas D. Nadeau" <[email-masked]> wrote:
> 
>>

---

From: Anoop Ghanwani <[email-masked]>
Date: Thu, 28 Aug 2014 22:08:18 -0700

To elaborate: In a logically centralized NVA architecture, we could
envision multiple interfaces including NVE-NVA, intra-NVA (clustering), and
inter-NVA (federation). In a fully distributed control plane, however, we
might imagine having only a single NVE-to-NVE control interface.

I think it would be worthwhile to add these sentences (or some version of
them) to the charter to clarify what is meant by a logically centralized
(vs distributed) NVA architecture.  I had expressed a similar concern
earlier and it's only now become clear to me what a distributed control
plane architecture is.  I don't think we can have an NVE-to-NVE control
plane...what you probably mean by that is that the NVA function is
co-resident with every NVE.  I am not aware of any implementation that does
that, so I can see why it makes sense for the WG to exclude that from the
charter.

Anoop

On Thu, Aug 28, 2014 at 9:09 PM, Benson Schliesser <[email-masked]>
wrote:

> Hi, Manish. Your question about centralized vs distributed control planes
> is actually very important, and something that I've discussed offline quite
> a bit. So let me see if I can answer it here, for your benefit as well as
> the rest of the WG.
>
> I think Sharon described this accurately. To elaborate: In a logically
> centralized NVA architecture, we could envision multiple interfaces
> including NVE-NVA, intra-NVA (clustering), and inter-NVA (federation). In a
> fully distributed control plane, however, we might imagine having only a
> single NVE-to-NVE control interface.
>
> In the proposed charter text, we have said that a fully distributed
> control plane is not in NVO3 WG's scope. This is not making a statement
> about whether that approach is valid, invalid, good, bad, etc, but rather
> is intended to focus our current work on the centralized architecture.
>
> Likewise, we have said that the intra-NVA (clustering) interface is not in
> scope. Again, this is not making a statement about whether it is necessary,
> good, bad, etc - because it obviously is necessary - but rather is intended
> to focus our work. Further, it seems likely that such an interface will not
> benefit from open specification and/or standardization in the near future,
> as implementors may use various mechanisms of their own choosing within
> their implementation.
>
> Of course, work on things like clustering, fully distributed control
> planes, etc, might be done in other WGs now. And/or the NVO3 WG might
> recharter to address them in the future. But our proposed charter text
> doesn't take these into account at this time, just for the sake of focus.
>
>  Must say I also struggled with the term logically centralized because
> for all practical purposes people will distribute the NVA, but then
> realized the point was - we could distribute the NVA based on the fact that
> there is fully functional IP underlay network that can be used for that. It
> is there by definition. And so this workgroup should not specify how to
> implement distributed software on an existing IP network, it should specify
> what this software does using which interfaces. If then people insist on
> using BGP, PNNI, IN to distribute states it's their call, other people may
> use simpler and more functional means.
>
> --szb
>
>  The same text ("A logically centralized control plane for network
> virtualization")  was bothering me for days as well! However, having been
> out of touch with developments in the working group for the past few
> months, I thought I better do the homework to check if this has already
> been covered in some discussions earlier. I also see the intent of keeping
> out BGP and or MPLS (L2VPN/L3VPN) based solutions because work for the same
> is going on (or would be done) in other working groups.
>
>  The final solution, as you say, could very well involve centralised
> and/or distributed components. Even the updated framework document talks
> about a balance needed - "Domain and/or deployment specific constraints
> define the balance between centralized and distributed approaches". Given
> these, as well as based on other limitations of a centralised model, I
> still don't understand the rationale behind the text - "A logically
> centralized control plane for network virtualization" in the charter
> update. Excluding BGP and/or MPLS based solutions for the reason above
> makes sense but not distributed models summarily. Am I missing something
> here ?
>
>
>  I think you are. Logically centralized covers physically centralized and
> only allows you to have more resiliency/scale/performance than a single
> instance.  I would also think that no one in a production environment would
> install a single centralized instance anyways, so we need logically
> centralized to cover real-world scenarios.
>
>  --Tom
>
>
>
>  Thanks,
> Manish
>
>  It is possible that NVO3 solutions could leverage BGP and/or that
> BGP-based solutions could leverage NVO3 work. In these cases we would work
> cross-WG to make sure that the solution is documented completely and
> accurately, with work happening in the most appropriate place(s). For
> example one could imagine combining a NVO3-developed data plane encap, a
> NVO3-developed TS-NVE control plane, and a BESS-developed NVA control
> plane. (There are many other possible combinations here; I'm not trying to
> be exhaustive, just illustrate the idea.)
>

---

From: "Manish Kumar (manishkr)" <[email-masked]>
Date: Fri, 29 Aug 2014 08:01:04 +0000

Hi Sharon, Benson,

Thanks for quick clarifications! It helps although I don't think I'm on the same page yet :). Let me attempt getting there!

Im totally with Sharon's point about the WG not entering into specifying how the network state is distributed (for a distributed model; or for that matter how it is aggregated for a centralised model) but rather focus on the behaviour of interfaces and the functional expectation from the various pieces. Using the existing IP underlay is just one way to distribute that information though and we perhaps may not choose to even go into those semantics though. However, it shouldn't imply that the NVA architecture be logically centralized – it could be centralized, distributed or a mix of both (even within the same network) using whichever model suits better.

I feel 'a fully distributed control plane is not in NVO3 WG’s scope' could be still better:) as it at least leaves the scope for a mixed model open between the two extremes of a logically centralised and fully distributed control planes (more so the NVA part of the function). Nevertheless, forcing a 'logically centralised control plane' (specifically the NVA) seems limiting at the start and also increases chances for fragmented and/or overlapping solutions. I thought of this as a good place to integrate these various models into a common architecture based on interfaces and behavioural expectations irrespective of whether network state is either centralized or distributed and how it's aggregated or distributed. Each model could obviously leverage works done in other WGs. Also, just to highlight, the management plane could still be centralised (which is where this need is felt IMO). My two cents!

Cheers,
Manish

From: Benson Schliesser <[email-masked]<mailto:[email-masked]>>
Date: Friday, 29 August 2014 9:39 AM
To: Manish Kumar <[email-masked]<mailto:[email-masked]>>
Cc: Sharon Barkai <[email-masked]<mailto:[email-masked]>>, Thomas Nadeau <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, Osama Zia <[email-masked]<mailto:[email-masked]>>
Subject: Re: [nvo3] Second Draft Charter Update for Discussion

Cheers,
-Benson

On Aug 28, 2014, at 2:20 PM, Sharon Barkai <[email-masked]<mailto:[email-masked]>> wrote:

--szb

Hi Benson,

--Tom

Thanks,
Manish

Hi, Osama.

Yes. To elaborate:

Cheers,
-Benson

---

From: "Manish Kumar (manishkr)" <[email-masked]>
Date: Fri, 29 Aug 2014 08:24:26 +0000

Hi Anoop,

I agree a clarification could definitely help! As far as the implementations go, I don't know whether it would be appropriate to name them but I can vouch for at least one (and multiple others claim interoperability with that) that has been around for a decade, if not more, and does have millions of network nodes deployed (including instances of 25k+ in single networks) although primarily in L3 space so far. The mechanism (NVA co-resident with every NVE) is an integral part of the same although it also uses the other mechanism (NVA-NVE not co-resident) and allows for a centralised model as well. The last one obviously doesn't scale as well though unless you use a brute force means of having to keep increasing the capacity of the central authority/NVA.

Cheers,
Manish

From: Anoop Ghanwani <[email-masked]<mailto:[email-masked]>>
Date: Friday, 29 August 2014 10:38 AM
To: Benson Schliesser <[email-masked]<mailto:[email-masked]>>
Cc: Manish Kumar <[email-masked]<mailto:[email-masked]>>, Osama Zia <[email-masked]<mailto:[email-masked]>>, Thomas Nadeau <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, Sharon Barkai <[email-masked]<mailto:[email-masked]>>
Subject: Re: [nvo3] Second Draft Charter Update for Discussion

To elaborate: In a logically centralized NVA architecture, we could envision multiple interfaces including NVE-NVA, intra-NVA (clustering), and inter-NVA (federation). In a fully distributed control plane, however, we might imagine having only a single NVE-to-NVE control interface.

I think it would be worthwhile to add these sentences (or some version of them) to the charter to clarify what is meant by a logically centralized (vs distributed) NVA architecture.  I had expressed a similar concern earlier and it's only now become clear to me what a distributed control plane architecture is.  I don't think we can have an NVE-to-NVE control plane...what you probably mean by that is that the NVA function is co-resident with every NVE.  I am not aware of any implementation that does that, so I can see why it makes sense for the WG to exclude that from the charter.

Anoop

On Thu, Aug 28, 2014 at 9:09 PM, Benson Schliesser <[email-masked]<mailto:[email-masked]>> wrote:
Hi, Manish. Your question about centralized vs distributed control planes is actually very important, and something that I’ve discussed offline quite a bit. So let me see if I can answer it here, for your benefit as well as the rest of the WG.

Cheers,
-Benson

--szb

Hi Benson,

--Tom

Thanks,
Manish

Hi, Osama.

Yes. To elaborate:

Cheers,
-Benson

---

From: Benson Schliesser <[email-masked]>
Date: Fri, 29 Aug 2014 21:54:21 -0400

Hi, Manish and other contributors -

(Changing the message Subject to better describe the topic.)

Thanks, I think I understand the question you’re asking now. And it’s possible that we need to revise the proposed charter text to be more clear. But first, let’s explore the topic of NVA centralization, what it means and doesn’t mean, etc.

In my previous note I described three NVA interfaces: NVE-NVA, Intra-NVA (Clustering), and Inter-NVA (Federation). I think that we have understanding of these interfaces and rough consensus about what’s in scope here. However, there is a broader question about what the NVA does, and whether a centralized NVA obviates the need for all control plane functionality at the NVE-NVE interface. Matthew and I have discussed this and we agree that the NVO3 scope needs to be somewhat flexible in this regard.

For example, I can imagine a solution that uses a NVE-NVA control plane for overlay-underlay mapping, combined with a NVE-NVE control plane for liveness detection. (This is just a simple example, and is not meant to be prescriptive nor exhaustive of the various ways that functionality might be split up between these interfaces.) As a counter-example, we do not want to work on solutions where effectively all of the control plane is implemented as a NVE-NVE protocol, such as a traditional routing protocol. (As I said previously, this isn’t a criticism of such approaches, but rather a choice on how to focus the NVO3 work.)

Having said that, it isn’t perfectly clear to me how to improve the proposed charter text. It already says that solutions must have a logically centralized NVA. And it isn’t prescriptive about what functionality a NVA provides. So, is that adequate to allow the flexibility the WG needs? Or is there some concise way to characterize the balance between centralized-distributed control plane functionality? I’m hesitant to add too much detail, too much text in the charter - I’d really like to keep it concise. But I also understand the point of confusion here, and I’m open to suggestions on how to fix it.

Feedback from the WG would be appreciated.

Cheers,
-Benson

On Aug 29, 2014, at 4:01 AM, Manish Kumar (manishkr) <[email-masked]> wrote:

> To: Manish Kumar <[email-masked]>
> Cc: Sharon Barkai <[email-masked]>, Thomas Nadeau <[email-masked]>, "[email-masked]" <[email-masked]>, Osama Zia <[email-masked]>

---

From: Sharon <[email-masked]>
Date: Sat, 30 Aug 2014 08:56:07 -0700

Looks like very good phrasing.

Eg in environments where there is no consistent topo-hierarchical addressing like overlays, where any EID can show up anywhere you either -

1) make the NVE-NVA indirection explicit as suggested and decouple the location of information from it's producers and consumers 

2) do the indirection implicitly or hidden with "reflectors" and such, make hard to evolve and extend because you are presumably specifying a peer to peer protocol

3) have no reason to expect to be more scalable then transparent bridging

As was stated, once you make the NVE-NVA indirection and interface explicit then there is no problem implementing the intra-NVA clustering or inter-NVA federation peer-to-peer over the underlay as most no-sqls do.

To illustrate, a non decoupled topo-data exchange of just 500 blade servers (assume one large non fragmented cluster), that would have a minimum of 1000 NVEs aggregating 1000 VMs or 1000 contained OS images each, would need a B state updates just to get going. Where as a decoupled solution will need a maximum of this many updates over a lengthy periods of flat VM to VM communcation.

--szb

> On Aug 29, 2014, at 6:54 PM, Benson Schliesser <[email-masked]> wrote:
>

---

From: Linda Dunbar <[email-masked]>
Date: Sun, 31 Aug 2014 13:21:44 +0000

It is really less about “Centralized NVA” because NVA can be distributed (that is why you need NVA-NVA protocols), but more about an entity with  “Authoritative Information coming from   databases or configurations” instead from network protocol self discovery.

This entity (NVA) can be centralized or distributed.

Linda

From: nvo3 [mailto:[email-masked]] On Behalf Of Benson Schliesser
Sent: Friday, August 29, 2014 8:54 PM
To: [email-masked]
Subject: [nvo3] Understanding "Centralized NVA" in the Charter

Cheers,
-Benson

On Aug 29, 2014, at 4:01 AM, Manish Kumar (manishkr) <[email-masked]<mailto:[email-masked]>> wrote:

Hi Sharon, Benson,

Cheers,
Manish

Cheers,
-Benson

---

From: Haoweiguo <[email-masked]>
Date: Mon, 1 Sep 2014 02:27:00 +0000

Hi Benson,

I have two comments.

The first one:

NVA can use intra-NVA(clustering) and inter-NVA(federation) archetecture. Many people are confused about the scenarios of these two archetecture.
It will be better to provide usecases for these two archetecture. For example:
Intra-NVA(clustering) is applied to a single data center and inter-NVA(federation)is applied to DCI.
Intra-NVA(clustering) is applied to homogenious envionment and inter-NVA(federation) is applied to heterogenius envionment.
I also agree we should not emphasize on one single logical NVA. Sinlge logical NVA means only intra-NVA(clustering)archetecture is used. If we use inter-NVA(federation) archetecture, logically there are multiple NVAs.

Another comment:

I agree we should focus on definition of the interaction interface behavior and functionality devision among different components(NVA and NVE),rather than definition of specific protocol.

My two cents.

weiguo

________________________________
发件人: Benson Schliesser [[email-masked]]
发送时间: 2014年8月30日 9:54
收件人: [email-masked]
主题: [nvo3] Understanding "Centralized NVA" in the Charter

Cheers,
-Benson

Hi Sharon, Benson,

Cheers,
Manish

Cheers,
-Benson