Subject: [nvo3] NVO3 charter 1530UK 12April

From: Stewart Bryant <[email-masked]>
Date: Thu, 12 Apr 2012 15:35:21 +0100

In this revised version of the charter I have attempted
to incorporate all of the charter discussion on the list.

In particular I have used the term Data Center Virtual Private
Network (DCVPN) in place of Overlay. This has lead to
a slight change in WG name, but one that is consistent
with the L3 goals.

I have tried to take the solutions references out.

I have added in the scaling requirements which are
a key  feature of this work.

- Stewart

NVO3: Network Virtualization Over Layer 3

Chairs - TBD
Area - Routing
Area Director - Stewart Bryant
OPS Area Adviser - TBD

Support for  multi-tenancy has become a core requirement of data centers
(DCs), especially in the context of data centers supporting virtualized
hosts known as virtual machines (VMs).  A key multi-tenancy requirement
is traffic isolation, so that a tenant's traffic (and internal address
usage) is not visible to any other tenant and does not collide with
addresses used within the data center itself. Another key requirement is
to support the placement and migration of VMs anywhere within the data
center, without being limited by DC network constraints such as the IP
subnet boundaries of the underlying DC network.

An NVO3 solution (known here as a Data Center Virtual Private
Network (DCVPN)) is a VPN that is viable across a scaling range of
a few thousand VMs to over two million VMs running on greater
than 100K physical servers. It thus has good scaling properties
from relatively small networks to networks with hundreds of
thousands of DCVPN endpoints and several million DCVPNs within
a single administrative domain.

Note that although this charter uses the term VM throughout, NVO3 must
also support connectivity to traditional hosts e.g. hosts that do not
have hypervisors.

NVO3 will develop an approach to multi-tenancy that uses an
encapsulation header that resides above IP rather than relying on
traditional L2 isolation mechanisms (e.g., VLANs) to support
multi-tenancy. The approach will provide an emulated ethernet
service capable of satisfying typical data center deployments.

NVO3 will document the problem statement, the applicability, and an
architectural framework for DCVPNs within a data center
environment. Within this framework, functional blocks will be defined to
allow the dynamic attachment / detachment of VMs to their DCVPN,
and the interconnection of elements of the DCVPNs over
the underlying physical network. This will support delivery of packets
to the destination VM, and provide the network functions required for
the migration of VMs within the network in a sub-second timeframe.

Based on this framework, the WG will develop requirements for both
control plane protocol(s) and data plane encapsulation format(s), and
perform a gap analysis of existing candidate mechanisms. In addition to
functional and architectural requirements, NVO3 will develop operational,
maintenance, troubleshooting, and security requirements.

The WG will investigate the interconnection of the DCVPNs
and their tenants with non-NVO3 IP network(s) to determine if
any specific work is needed.

The NVO3 will write the following informational RFCs, which
must be substantially complete before rechartering can be
considered:
     Problem Statement
     Framework document
     Control plane requirements document
     Data plane requirements document
     Operational Requirements
     Gap Analysis

Driven by the requirements and consistent with the gap analysis,
it is anticipated that the WG will request being rechartered to
document solutions consisting of one or more data plane
encapsulations and control plane protocols as applicable.
Any documented solutions will use existing mechanisms
if suitable, or will develop new mechanisms if necessary.

If the WG anticipates the adoption  of the technologies of
another SDO, such as the IEEE, as part of the solution, it
will liaise with that SDO to ensure the compatibility of
the approach.

Milestones:

Dec 2012 Problem Statement submitted for IESG review
Dec 2012 Framework document submitted for IESG review
Dec 2012 Control plane requirements submitted for IESG review
Dec 2012 Data plane requirements submitted for IESG review
Dec 2012 Gap Analysis submitted for IESG review
Dec 2012 Recharter or close Working Group

---

From: John E Drake <[email-masked]>
Date: Thu, 12 Apr 2012 07:50:05 -0700

Stewart,

I would strike the following, which is inconsistent with the rest of the charter:

"NVO3 will develop an approach to multi-tenancy that uses an encapsulation header that resides above IP rather than relying on
traditional L2 isolation mechanisms (e.g., VLANs) to support multi-tenancy. The approach will provide an emulated Ethernet service capable of satisfying typical data center deployments."

Thanks,

John

Sent from my iPhone

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Stewart Bryant
> Sent: Thursday, April 12, 2012 7:35 AM
> To: [email-masked]
> Subject: [nvo3] NVO3 charter 1530UK 12April
> 
> 
> In this revised version of the charter I have attempted to incorporate
> all of the charter discussion on the list.
> 
> In particular I have used the term Data Center Virtual Private Network
> (DCVPN) in place of Overlay. This has lead to a slight change in WG
> name, but one that is consistent with the L3 goals.
> 
> I have added in the scaling requirements which are a key  feature of
> this work.
> 
> - Stewart
> 
> 
> Support for  multi-tenancy has become a core requirement of data
> centers (DCs), especially in the context of data centers supporting
> virtualized hosts known as virtual machines (VMs).  A key multi-tenancy
> requirement is traffic isolation, so that a tenant's traffic (and
> internal address
> addresses used within the data center itself. Another key requirement
> is to support the placement and migration of VMs anywhere within the
> data center, without being limited by DC network constraints such as
> the IP subnet boundaries of the underlying DC network.
> 
> An NVO3 solution (known here as a Data Center Virtual Private Network
> (DCVPN)) is a VPN that is viable across a scaling range of a few
> thousand VMs to over two million VMs running on greater than 100K
> physical servers. It thus has good scaling properties from relatively
> small networks to networks with hundreds of thousands of DCVPN
> endpoints and several million DCVPNs within a single administrative
> domain.
> 
> traditional L2 isolation mechanisms (e.g., VLANs) to support multi-
> tenancy. The approach will provide an emulated ethernet service capable
> of satisfying typical data center deployments.
> 
> architectural framework for DCVPNs within a data center environment.
> Within this framework, functional blocks will be defined to allow the
> dynamic attachment / detachment of VMs to their DCVPN, and the
> interconnection of elements of the DCVPNs over the underlying physical
> network. This will support delivery of packets to the destination VM,
> and provide the network functions required for the migration of VMs
> within the network in a sub-second timeframe.
> 
> functional and architectural requirements, NVO3 will develop
> operational, maintenance, troubleshooting, and security requirements.
> 
> The WG will investigate the interconnection of the DCVPNs and their
> tenants with non-NVO3 IP network(s) to determine if any specific work
> is needed.
> 
> The NVO3 will write the following informational RFCs, which must be
> substantially complete before rechartering can be
> considered:
>      Problem Statement
>      Framework document
> Driven by the requirements and consistent with the gap analysis, it is
> anticipated that the WG will request being rechartered to document
> solutions consisting of one or more data plane encapsulations and
> control plane protocols as applicable.
> Any documented solutions will use existing mechanisms if suitable, or
> will develop new mechanisms if necessary.
> 
> If the WG anticipates the adoption  of the technologies of another SDO,
> such as the IEEE, as part of the solution, it will liaise with that SDO
> to ensure the compatibility of the approach.
> 
> 
> Milestones:
> 
> Dec 2012 Problem Statement submitted for IESG review Dec 2012 Framework
> document submitted for IESG review Dec 2012 Control plane requirements
> submitted for IESG review Dec 2012 Data plane requirements submitted
> for IESG review Dec 2012 Gap Analysis submitted for IESG review Dec
> 2012 Recharter or close Working Group
> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3

---

From: Stewart Bryant <[email-masked]>
Date: Thu, 12 Apr 2012 16:11:33 +0100

John

I think that there is some useful material in those sentences.

"rather than relying on
traditional L2 isolation mechanisms (e.g., VLANs) to support multi-tenancy."

Set out how it is mostly done today

"NVO3 will develop an approach to multi-tenancy that uses an encapsulation header that resides above IP"

Says we will do it over IP. I suppose your point is that excludes MPLS, so we could say

"NVO3 will develop an approach to multi-tenancy that uses an encapsulation header that resides above Layer 3"
since that is what the WG name says.

"The approach will provide an emulated Ethernet service capable of 
satisfying typical data center deployments"

Seems to be the DC expectation WRT handoff

So how about we s/ip/layer3/ ?

Stewart

-- 
For corporate legal information go to:

http://www.cisco.com/web/about/doing_business/legal/cri/index.html

---

From: John E Drake <[email-masked]>
Date: Thu, 12 Apr 2012 09:01:14 -0700

Stewart,

BTW, I forgot to complement you on your revisions to the charter;  I think it looks generally very nice.

However, I do still have several issues with the paragraph I suggested that you strike.  Generally, MPLS is considered layer 2.5, so 'layer 3' may not capture its potential inclusiveness.

Also, the phrase 'an emulated Ethernet service' has some problems.  It sounds an awful lot like what the L2VPN group is supposed to be doing and it may be restrictive in that I thought we wanted to send traffic between nodes in different subnets, i.e., routing.

Thanks,

John 

Sent from my iPhone

> From: Stewart Bryant [mailto:[email-masked]]
> Sent: Thursday, April 12, 2012 8:12 AM
> To: John E Drake
> Cc: [email-masked]; [email-masked]
> Subject: Re: [nvo3] NVO3 charter 1530UK 12April
> 
> > "NVO3 will develop an approach to multi-tenancy that uses an
> tenancy. The approach will provide an emulated Ethernet service capable
> of satisfying typical data center deployments."
> >
> > Thanks,
> >
> > John
> >
> > Sent from my iPhone
> >
> John
> 
> tenancy."
> "NVO3 will develop an approach to multi-tenancy that uses an
> encapsulation header that resides above IP"
> 
> "NVO3 will develop an approach to multi-tenancy that uses an
> encapsulation header that resides above Layer 3"
> >> From: [email-masked] [mailto:[email-masked]] On Behalf
> >> Of Stewart Bryant
> >> In this revised version of the charter I have attempted to
> >> incorporate all of the charter discussion on the list.
> >>
> >> Network
> >> virtualized hosts known as virtual machines (VMs).  A key
> >> multi-tenancy requirement is traffic isolation, so that a tenant's
> >> traffic (and internal address
> Network
> >> Note that although this charter uses the term VM throughout, NVO3
> >> must also support connectivity to traditional hosts e.g. hosts that
> >> do not have hypervisors.
> >>
> >> tenancy. The approach will provide an emulated ethernet service
> >> capable of satisfying typical data center deployments.
> >>
> >> interconnection of elements of the DCVPNs over the underlying
> >> physical network. This will support delivery of packets to the
> >> destination VM, and provide the network functions required for the
> >> migration of VMs within the network in a sub-second timeframe.
> >>
> >> perform a gap analysis of existing candidate mechanisms. In addition
> >> to functional and architectural requirements, NVO3 will develop
> >> Driven by the requirements and consistent with the gap analysis, it
> >> is anticipated that the WG will request being rechartered to
> document
> >> If the WG anticipates the adoption  of the technologies of another
> >> SDO, such as the IEEE, as part of the solution, it will liaise with
> >> that SDO to ensure the compatibility of the approach.
> >>
> >>
> >> Milestones:
> >>
> >> Dec 2012 Problem Statement submitted for IESG review Dec 2012
> >> Framework document submitted for IESG review Dec 2012 Control plane
> >> requirements submitted for IESG review Dec 2012 Data plane
> >> requirements submitted for IESG review Dec 2012 Gap Analysis
> >> submitted for IESG review Dec

---

From: Stewart Bryant <[email-masked]>
Date: Thu, 12 Apr 2012 17:08:18 +0100

Thank you John

Whilst I am sympathetic to what you say, I think we need to find
better words, that than completely remove the material.

The key question, is whether we are close enough that we can
thrash out these details in IETF Last Call?

Stewart

---

From: David Meyer <[email-masked]>
Date: Thu, 12 Apr 2012 09:36:35 -0700

Hey Stuart,

For the most part this is very nice. Search for dmm> for a few comments in-line.

Thanks,

Dave

On Thu, Apr 12, 2012 at 7:35 AM, Stewart Bryant <[email-masked]> wrote:
> addresses used within the data center itself.

dmm> this sentence is a bit awkward. Perhaps the following:
dmm>
dmm> "Two key requirements needed to support multi-tenancy are traffic
dmm> isolation, so that a tenant's traffic is not visible to any other tenant,
dmm> and address independence, so that one tenant's addressing does
dmm> not collide with other tenants addressing schemes or with addresses
dmm> used within the data center itself."
dmm>
dmm> or something like that...
dmm>
dmm> BTW, the term "visible" might be to loose; For example, "visible" might
dmm> mean traffic isolation (as you might find via a VLAN) or
something stronger
dmm> (like encryption)? Are both (and/or others) envisioned here?

> Another key requirement is

> encapsulation header that resides above IP rather than relying on

dmm> the term "above" might be ambiguous here. Would it be useful
dmm> to draw an abstract picture to describe exactly what is intended?
dmm> In addition, where does MPLS fit here (being neither L2 or L3)?

> environment. Within this framework, functional blocks will be defined to

dmm> might be useful to define what a "functional block" is and what
dmm> kinds of properties it needs to have in this context.

---

From: John E Drake <[email-masked]>
Date: Thu, 12 Apr 2012 09:43:16 -0700

That's fine.

Sent from my iPhone

> Sent: Thursday, April 12, 2012 9:08 AM
> To: John E Drake
> Whilst I am sympathetic to what you say, I think we need to find better
> words, that than completely remove the material.
> 
> The key question, is whether we are close enough that we can thrash out
> these details in IETF Last Call?
> 
> Stewart
> 
> 
> 
> > However, I do still have several issues with the paragraph I
> suggested that you strike.  Generally, MPLS is considered layer 2.5, so
> 'layer 3' may not capture its potential inclusiveness.
> >
> > Also, the phrase 'an emulated Ethernet service' has some problems.
> It sounds an awful lot like what the L2VPN group is supposed to be
> doing and it may be restrictive in that I thought we wanted to send
> traffic between nodes in different subnets, i.e., routing.
> >
> > Thanks,
> >
> > John
> >
> > Sent from my iPhone
> >
> >
> >>> I would strike the following, which is inconsistent with the rest
> of
> >> the charter:
> >> tenancy. The approach will provide an emulated Ethernet service
> >> capable of satisfying typical data center deployments."
> >>> Thanks,
> >>>
> >>> John
> >>>
> >>> Sent from my iPhone
> >>>
> >> John
> >>
> >>>> From: [email-masked] [mailto:[email-masked]] On
> >>>> Behalf Of Stewart Bryant
> >>>> I have added in the scaling requirements which are a key  feature
> >>>> of this work.
> >>>>
> >>>> - Stewart
> >>>>
> >>>>
> >>>> is to support the placement and migration of VMs anywhere within
> >>>> the data center, without being limited by DC network constraints
> >>>> such as the IP subnet boundaries of the underlying DC network.
> >>>>
> multi-
> environment.
> >>>> perform a gap analysis of existing candidate mechanisms. In
> >>>> addition to functional and architectural requirements, NVO3 will
> >>>> develop operational, maintenance, troubleshooting, and security
> >> requirements.
> it

---

From: Kireeti Kompella <[email-masked]>
Date: Thu, 12 Apr 2012 11:24:01 -0700

Hi Stewart,

> a slight change in WG name, but one that is consistent

Thanks, that's a big improvement!

> a key  feature of this work.

Thanks again -- although see inline.

In general, this update reads much better.

> a few thousand VMs to over two million VMs running on greater

Why "two"?  Perhaps "several" ...

> a single administrative domain.

Perhaps the last two should be swapped:
"to networks with of several million DCVPN endpoints and hundreds of
thousands DCVPNs within a single administrative domain."

> service capable of satisfying typical data center deployments.

I'll go with your suggestion to change "above IP" to "at Layer 3", with the
proviso that MPLS is Layer 3-ish.

Thanks again -- good changes!
Kireeti.

> ______________________________**_________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/**listinfo/nvo3<https://www.ietf.org/mailman/listinfo/nvo3>
>

-- 
Kireeti

---

From: "Puneet Agarwal" <[email-masked]>
Date: Thu, 12 Apr 2012 18:37:06 +0000

Hi Stewart,

1) This is a minor quibble but we never really defined what a data center is - it would be good to have a generic definition. In the case of DCVPN, would it be fair to say it is limited by the underlying IGP domain?
2) Agree with Kireeti on rewording "two million VMs" to "several million VMs".
3) It is unclear whether the charter scope allows for a DCVPN to span multiple DCs or not (see #1). It would be good to state explicitly if that is in/out of scope. 

Thanks.
 
-Puneet

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Stewart Bryant
Sent: Thursday, April 12, 2012 7:35 AM
To: [email-masked]
Subject: [nvo3] NVO3 charter 1530UK 12April

- Stewart

Chairs - TBD
Area - Routing

Milestones:

---

From: Stewart Bryant <[email-masked]>
Date: Thu, 12 Apr 2012 20:18:21 +0100

> I'll go with your suggestion to change "above IP" to "at Layer 3", 
> with the proviso that MPLS is Layer 3-ish.
>
>
I rolled in your other suggestions.

I could make that say "network layer", let's discuss in LC.

- Stewart

---

From: Stewart Bryant <[email-masked]>
Date: Thu, 12 Apr 2012 20:25:25 +0100

Puneet

> 1) This is a minor quibble but we never really defined what a data center is - it would be good to have a generic definition. In the case of DCVPN, would it be fair to say it is limited by the unde…
Please can you float a one sentence definition during LC and we will see 
it it gets traction.

> 2) Agree with Kireeti on rewording "two million VMs" to "several million VMs".
Done
> 3) It is unclear whether the charter scope allows for a DCVPN to span multiple DCs or not (see #1). It would be good to state explicitly if that is in/out of scope.

I have had a few people say span and a few people say not.

We need to be able to interconnect DCs, but it's not clear that
we need the same fast switching dynamics. Can we look
at this during the requirement's phase and see whether
interconnect is easy/hard and as a problem integral/separable?

Stewart

---

From: Stewart Bryant <[email-masked]>
Date: Thu, 12 Apr 2012 20:44:05 +0100

On 12/04/2012 17:36, David Meyer wrote:
>
>   A key multi-tenancy requirement
Sentence changed
> dmm>  to draw an abstract picture to describe exactly what is intended?
We can't have pictures in a charter :(
> dmm>  In addition, where does MPLS fit here (being neither L2 or L3)?
That has come up before. Let's discuss in LC. Would network layer
work better.

> dmm>  kinds of properties it needs to have in this context.
I don't think there is space in the charter for that, unless you can propose
a sentence.

Stewart

---

From: "Puneet Agarwal" <[email-masked]>
Date: Thu, 12 Apr 2012 21:46:16 +0000

Hi Stewart,

1) from http://searchdatacenter.techtarget.com/definition/data-center", (though I am not too keen on it myself - Wikipedia was even more vague).
A data center  is a centralized repository, either physical or virtual, for the storage, management, and dissemination of data and information organized around a particular body of knowledge or pertaining to a particular business. 

Or from dictionary .com - a facility equipped with or connected to one or more computers, used for processing or transmitting data.

3) I am fine with looking at it in detail during the requirements phase. Then let's just say in the charter  that "DCVPN can span multiple data centers. The WG will look at requirements for spanning multiple data centers - however this  is a lower priority than single data center". 

Thanks.
 
-Puneet

-----Original Message-----
From: Stewart Bryant [mailto:[email-masked]] 
Sent: Thursday, April 12, 2012 12:25 PM
To: Puneet Agarwal
Cc: [email-masked]
Subject: Re: [nvo3] NVO3 charter 1530UK 12April

Puneet

Done

Stewart

---

From: Yakov Rekhter <[email-masked]>
Date: Thu, 12 Apr 2012 15:31:44 -0700

Kireeti,

> proviso that MPLS is Layer 3-ish.

Rather than arguing about layers, I think it would be much better
to just explicitly spell out that "NVO3 will develop an approach...
that uses encapsulation over IP or MPLS enabled IP packet switched
networks (PSN)."

Yakov.

P.S. The last part I took almost verbatim from the pwe3 WG charter.

---

From: Larry Kreeger <[email-masked]>
Date: Thu, 12 Apr 2012 17:17:28 -0700

On 4/12/12 11:37 AM, "Puneet Agarwal" <[email-masked]> wrote:

> Hi Stewart,
> 
> 1) This is a minor quibble but we never really defined what a data center is -
> it would be good to have a generic definition. In the case of DCVPN, would it
> be fair to say it is limited by the underlying IGP domain?

In draft-kreeger-nvo3-overlay-cp-00 we had the following as the definition
of a data center: 

"A physical complex housing physical servers, network switches and routers,
Network Service Appliances and networked storage.  The purpose of a Data
Center is to provide application and/or compute and/or storage services."

> 3) It is unclear whether the charter scope allows for a DCVPN to span multiple
> DCs or not (see #1). It would be good to state explicitly if that is in/out of
> scope. 
> 
> Thanks.
>  
> -Puneet
>

---

From: Xuxiaohu <[email-masked]>
Date: Fri, 13 Apr 2012 02:11:08 +0000

> that uses encapsulation over IP or MPLS enabled IP packet switched

It looks more concrete and more clear. I like it. By the way, how about "IP or MPLS enabled packet switched network (PSN)"

Best regards.
Xiaohu

---

From: Xuxiaohu <[email-masked]>
Date: Fri, 13 Apr 2012 02:28:17 +0000

[Xiaohu]  I think the "endpoints" here mean DCVPN PE nodes, rather than VMs. If so, the number of endpoints should be no more than that of physical servers as mentioned above (i.e., greater than 100K physical servers) . Of course, I agree the granularity of the number of DCVPN instances (i.e., hundreds of thousands) is reasonable.

Xiaohu

---

From: David Allan I <[email-masked]>
Date: Thu, 12 Apr 2012 22:51:35 -0400

>	a few thousand VMs to over two million VMs ....

Are we discussing a subnet with a /11 or shorter prefix?

We may want to make it clear that we are not discussing a single tenant instance....

Cheers
D

---

From: Kireeti Kompella <[email-masked]>
Date: Thu, 12 Apr 2012 20:06:59 -0700

Hi Yakov,

> Kireeti,

<snipped>

> that uses encapsulation over IP or MPLS enabled IP packet switched

That's even better ... and since it was the approach taken by another
"overlay" technology, namely PWE3, it seems fitting.

> P.S. The last part I took almost verbatim from the pwe3 WG charter.

Yes, it sounded familiar -- should be even more familiar to Stewart :-)

-- 
Kireeti

---

From: Kireeti Kompella <[email-masked]>
Date: Thu, 12 Apr 2012 20:07:22 -0700

Hi Xiaohu,

On Thu, Apr 12, 2012 at 7:11 PM, Xuxiaohu <[email-masked]> wrote:

> It looks more concrete and more clear. I like it. By the way, how about
> "IP or MPLS enabled packet switched network (PSN)"
>

Agreed.

-- 
Kireeti

---

From: Xuxiaohu <[email-masked]>
Date: Fri, 13 Apr 2012 03:14:03 +0000

> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代表 David
> Allan I
> 发送时间: 2012年4月13日 10:52
> 收件人: Xuxiaohu; Kireeti Kompella; [email-masked]
> 抄送: [email-masked]
> 主题: Re: [nvo3] NVO3 charter 1530UK 12April
> 
> 
> 

Your observation is correct. Maybe it should be said as "...is a VPN technology/solution that is viable..."

Best regards,
Xiaohu

---

From: Kireeti Kompella <[email-masked]>
Date: Thu, 12 Apr 2012 20:17:25 -0700

Hi Dave,

> Are we discussing a subnet with a /11 or shorter prefix?

The range of sizes of a single tenant will vary widely -- which is not to
say it will be of the order of millions.

However, let's not push the numbers too far -- I'm happy that some sense of
the scale has been captured, and I'd leave it at that.

> We may want to make it clear that we are not discussing a single tenant

The number of tenants, actually, DCVPNs, is of the order of 10s to 100s of
thousands, so I think we're covered.

> D

Regards,
Kireeti

---

From: Stewart Bryant <[email-masked]>
Date: Fri, 13 Apr 2012 08:59:21 +0100

On 13/04/2012 01:17, Larry Kreeger wrote:
> On 4/12/12 11:37 AM, "Puneet Agarwal"<[email-masked]>  wrote:
>
>> Hi Stewart,
>>

That is a great definition for the req/frmwrk but does it really belong 
in a charter?

The charter talks about scale, isolation  and mobility which are the
key differentiators. As far as the charter is concerned, a DC is an abstract
notation for a place where you find that set of new requirements.

Stewart

---

From: James Kempf <[email-masked]>
Date: Fri, 13 Apr 2012 12:20:43 -0400

Hi,

I'd suggest modifying:

"and provide the network functions required for the migration of VMs within the network in a sub-second timeframe."

to:
"and provide the network functions required to ensure that the networking component of VM migration stays within the sub-second timeframe."

because the time required for VM migration varies widely and will be a function of the size of the VM image. A large VM image might require many seconds to migrate, with pages dirtied during the migration having to be recopied. I think the important point for this working group is that the networking component not require complex network functions (re-arping, updating care-of addresses, etc.) so that it doesn't become the bottleneck.

		jak

> Sent: Thursday, April 12, 2012 12:44 PM
> To: David Meyer
> Cc: [email-masked]
> > dmm>  "Two key requirements needed to support multi-tenancy are 
> > dmm> traffic  isolation, so that a tenant's traffic is not 
> visible to 
> > dmm> any other tenant,  and address independence, so that 
> one tenant's 
> > dmm> addressing does  not collide with other tenants addressing 
> > dmm> schemes or with addresses  used within the data center itself."
> > dmm>
> > dmm>  BTW, the term "visible" might be to loose; For example, 
> > dmm> "visible" might  mean traffic isolation (as you might 
> find via a 
> > dmm> VLAN) or
> > something stronger
> >> to support the placement and migration of VMs anywhere within the 
> > dmm>  the term "above" might be ambiguous here. Would it be 
> useful  to 
> > dmm> draw an abstract picture to describe exactly what is intended?
> That has come up before. Let's discuss in LC. Would network 
> layer work better.
> 
> >> multi-tenancy. The approach will provide an emulated 
> ethernet service 
> >> Within this framework, functional blocks will be defined to
> I don't think there is space in the charter for that, unless 
> you can propose a sentence.
> 
> Stewart

---

From: Kireeti Kompella <[email-masked]>
Date: Fri, 13 Apr 2012 09:43:43 -0700

Hi James,

> "and provide the network functions required to ensure that the networking component of VM migration stays within the sub-second timeframe."

This had caught my attention as well.  What you suggest is better, but still not quite there:
1) the networking component may do some pre-work to prepare for the migration (create/fill routing table, preload policies, etc.) and that needn't finish subsecond.
2) the final step, once all pages are copied, is for the orchestration system to stop the VM on the old server, then start it on the new.  Who's to say how long this takes?

> because the time required for VM migration varies widely and will be a function of the size of the VM image. A large VM image might require many seconds to migrate, with pages dirtied during the mi…

Still not quite there: see (1) above, which can be done outside the critical path (i.e., while copying VM pages).  But to tweak your statement slightly, "the network component should not become the bottleneck during (2)."

Channeling Stewart, is this worth clarifying in the charter?

Kireeti

[non-normative comment, not to be taken as endorsement of any particular solution (thanks, Rob!): VDP has a "pre-associate" message whose purpose, as far as I can glean, is to allow the network component a chance to do pre-work.  That seems a sensible approach for any solution.]

---

From: James Kempf <[email-masked]>
Date: Fri, 13 Apr 2012 13:12:38 -0400

Hi Kireeti,

"the network component 
should not become the bottleneck during (2)" sounds fine.

		jak 

> From: Kireeti Kompella [mailto:[email-masked]] 
> Sent: Friday, April 13, 2012 9:44 AM
> To: James Kempf
> Cc: [email-masked]; David Meyer; [email-masked]
> > "and provide the network functions required for the 
> migration of VMs within the network in a sub-second timeframe."
> > 
> > to:
> > "and provide the network functions required to ensure that 
> the networking component of VM migration stays within the 
> sub-second timeframe."
> 
> 1) the networking component may do some pre-work to prepare 
> for the migration (create/fill routing table, preload 
> policies, etc.) and that needn't finish subsecond.
> 2) the final step, once all pages are copied, is for the 
> orchestration system to stop the VM on the old server, then 
> start it on the new.  Who's to say how long this takes?
> 
> > because the time required for VM migration varies widely 
> and will be a function of the size of the VM image. A large 
> VM image might require many seconds to migrate, with pages 
> dirtied during the migration having to be recopied. I think 
> the important point for this working group is that the 
> networking component not require complex network functions 
> (re-arping, updating care-of addresses, etc.) so that it 
> doesn't become the bottleneck.
> 
> Still not quite there: see (1) above, which can be done 
> outside the critical path (i.e., while copying VM pages).  
> But to tweak your statement slightly, "the network component 
> should not become the bottleneck during (2)."
> 
> [non-normative comment, not to be taken as endorsement of any 
> particular solution (thanks, Rob!): VDP has a "pre-associate" 
> message whose purpose, as far as I can glean, is to allow the 
> network component a chance to do pre-work.  That seems a 
> sensible approach for any solution.]
> 
> >        jak
> > 
> >> I don't think there is space in the charter for that, 
> unless you can 
> >> propose a sentence.
> >> 
> >> Stewart

---

From: Larry Kreeger <[email-masked]>
Date: Fri, 13 Apr 2012 11:05:17 -0700

Stewart,

No, I think the concept of a data center is well enough known that it isn't
required to be in the charter.  I only sent it if you felt it was
appropriate.

 - Larry

---

From: AshwoodsmithPeter <[email-masked]>
Date: Fri, 13 Apr 2012 19:24:00 +0000

Stuart,

Entropy considerations need to be in scope as the different encapsulations have different degrees of entropy which may or may not require core changes to increase.

Eg - VxLan (LISP?) use of port to extend down into the 5-tuple to increase LAG/ECMP entropy.

So while transparency to the core is nice it can't be totally transparent or the entropy is reduced or eliminated.

We are going to see a lot of 4x40GE LAGs out of the TORs.

Peter

---

From: Stewart Bryant <[email-masked]>
Date: Mon, 16 Apr 2012 10:01:19 +0100

Peter

As someone that has worked on entropy, I an not one to push back on the 
protocol supporting it, but does it really need to be in the charter? I 
cannot see anything on the charter that precludes entropy being designed in.

Surely the entropy properties will be put on the table as part of the 
requirements discussion, along
with many other details that are not in the charter?

Stewart

---

From: AshwoodsmithPeter <[email-masked]>
Date: Mon, 16 Apr 2012 14:21:53 +0000

Stuart, Yeah ... ok, in retrospect I guess its kind of an obvious requirement that it has to work over existing LAG/ECMP hardware.

Peter

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Stewart Bryant
Sent: Monday, April 16, 2012 5:01 AM
To: AshwoodsmithPeter
Cc: Puneet Agarwal; [email-masked]; Larry Kreeger
Subject: Re: [nvo3] NVO3 charter 1530UK 12April

Peter

Stewart

---

From: Kireeti Kompella <[email-masked]>
Date: Tue, 17 Apr 2012 11:16:02 -0700

Hi Stewart,

> a slight change in WG name, but one that is consistent

I thought that both those changes really helped clarify the charter.
 However, in the email sent out by the IESG secretary, the name has
reverted, both in the subject line and in the body of the email.

Was that intended or just a oversight?

(snip)

This best captures the intent of the WG, and doesn't get into extraneous
debates on "what is an overlay?"

(snip)

> service capable of satisfying typical data center deployments.

The new wording:

NVO3 will develop an approach to multi-tenancy that uses a
Layer 3 encapsulation rather than relying on
traditional L2 isolation mechanisms ...

is better.  Was the more explicit "encapsulation over
IP or MPLS enabled packet switched networks (PSN)" discussed?

Cheers,
Kireeti

---

From: Greg Mirsky <[email-masked]>
Date: Tue, 17 Apr 2012 15:00:07 -0700

Dear Stewart, et al.,
question and proposal to the following goals for NVO3:
"NVO3 will develop operational, maintenance, troubleshooting, and security
requirements."
- should Administration of DCVPN be left outside of scope or it is
implicitly in the scope as part of usual interpretation of the OAM? If
administration of DCVPN is in scope for NVO3 it might be mentioned
explicitly either by direct reference or by substituting "operational,
maintenance" with "OAM";
- I think that survivability of DCVPN, including protection and restoration
mechanisms, is in scope for NVO3 and can be explicitly mentioned.
Thus the sentence might look like the following:
"NVO3 will develop operational, administration, maintenance,
troubleshooting, as well as, survivability and security requirements for
DCVPN."

Regards,
Greg

---

From: Stewart Bryant <[email-masked]>
Date: Wed, 18 Apr 2012 08:52:52 +0100

> "overlay" technology, namely PWE3, it seems fitting.

What makes me nervous is that in the PWE3 case we spent a lot
of time working on an PW/IP and PW/MPLS until the market
eventually decided in favour of PW/MPLS and PW/IP for most
practical purposes died out.

I do not know whether we need IP, MPLS or both in this case,
and unfortunately I am not sure how we get firm objective
evidence. However we need to be careful that on the one
hand the charter does not preempt an objective decision, and
on the other hand does not create a mechanism whereby the
WG spends a lot of time on technology to support minority
deployments.

The specific problem is with the ambiguity of the word
"or" since  it may bind us to doing both even if the
evidence supports the need for only one (of type currently
unknown), or it may force us to choose when the market
is split and we need to support both.

My hope was that "layer 3" could be taken to include
IP and MPLS in such a way as to allow us to make a
more considered decision of {IP, MPLS, IP and MPLS}
when we have more evidence.

- Stewart

---

From: John E Drake <[email-masked]>
Date: Wed, 18 Apr 2012 14:12:01 -0700

Stewart,

If you recall, last week I suggested that you strike the following paragraph from the proposed NV03 charter:

"NVO3 will develop an approach to multi-tenancy that uses a Layer 3 encapsulation rather than relying on traditional L2 isolation mechanisms (e.g., VLANs) to support multi-tenancy. The approach will provide an emulated Ethernet service capable of satisfying typical data center deployments."

because I felt it was inconsistent with the rest of the charter.  Upon further study, I think the 'inconsistent' may have been too kind.  I think a better word might be 'antithetical', because as is pointed out in this paragraph of the charter:

"Driven by the requirements and consistent with the gap analysis, the WG may request being rechartered to document solutions consisting of one or more data plane encapsulations and control plane protocols as applicable.  Any documented solutions will use existing mechanisms if suitable, or will develop new mechanisms if necessary."

development is strictly proscribed from the charter of this group.  So, you really should strike the offending paragraph, as I suggested.

Also, since we are on the topic, I am not completely happy with the paragraph I quoted just above and would suggest that it be reworded as follows:

"Driven by the requirements and consistent with the gap analysis, the WG may request to be re-chartered to document solutions consisting of one or more data plane encapsulations and control plane protocols as applicable.  Any documented solutions will use existing standards if suitable, or will develop new standards if the WG determines and documents that no existing standards can serve as an adequate basis for satisfying the requirements."

Thanks,

John

Sent from my iPhone

From: [email-masked] [mailto:[email-masked]] On Behalf Of Stewart Bryant
Sent: Wednesday, April 18, 2012 12:53 AM
To: Kireeti Kompella
Cc: Yakov Rekhter; [email-masked]
Subject: Re: [nvo3] NVO3 charter 1530UK 12April

That's even better ... and since it was the approach taken by another "overlay" technology, namely PWE3, it seems fitting.

- Stewart

---

From: Thomas Narten <[email-masked]>
Date: Wed, 18 Apr 2012 17:17:56 -0400

Hi Stewart.

> WG spends a lot of time on technology to support minority

I am very much concerned about this. I know that this point is not
shared by all, but for the DC folk I've talked to (and there are
others I've talked to that say *exactly* the same thing), MPLS/BGP is
simply a non-starter.

I know that me saying that won't convince those that disagree with
this premise. 

One of initial deliverables of the WG is to do requirements gathering
and a gap analysis. We will have the above conversation then for sure.

But I am also worried that the voices of vendors could easily drown
out the voices of operators. The IETF gets little enough direct
particpation from operators as it is.

> when we have more evidence.

IMO, saying L3 is enough. 

Thomas

---

From: John E Drake <[email-masked]>
Date: Wed, 18 Apr 2012 14:26:07 -0700

Thomas,

I would question the use of the word 'premise'.  I think 'assertion' is much more accurate.

I also have a question for clarification.  Do you consider yourself to be a 'vendor' or an 'operator'?

Thanks,

John

Sent from my iPhone

> Thomas Narten
> Sent: Wednesday, April 18, 2012 2:18 PM
> To: [email-masked]
> Cc: Yakov Rekhter; Kireeti Kompella; [email-masked]
> > I do not know whether we need IP, MPLS or both in this case, and
> > unfortunately I am not sure how we get firm objective evidence.
> > However we need to be careful that on the one hand the charter does
> > not preempt an objective decision, and on the other hand does not
> > create a mechanism whereby the WG spends a lot of time on technology
> > to support minority deployments.
> 
> shared by all, but for the DC folk I've talked to (and there are others
> I've talked to that say *exactly* the same thing), MPLS/BGP is simply a
> non-starter.
> 
> I know that me saying that won't convince those that disagree with this
> premise.
> 
> But I am also worried that the voices of vendors could easily drown out
> the voices of operators. The IETF gets little enough direct
> > The specific problem is with the ambiguity of the word "or" since  it
> > may bind us to doing both even if the evidence supports the need for
> > only one (of type currently unknown), or it may force us to choose
> > when the market is split and we need to support both.
> 
> > My hope was that "layer 3" could be taken to include IP and MPLS in
> > such a way as to allow us to make a more considered decision of {IP,
> > MPLS, IP and MPLS} when we have more evidence.
>

---

From: Larry Kreeger <[email-masked]>
Date: Wed, 18 Apr 2012 15:46:34 -0700

> particpation from operators as it is.

The vast majority of data centers in the world are run by Enterprises.  The
people who run these data centers have little time nor interest in
participating in the IETF. Speaking as a vendor who talks with these
Enterprise data center operators, they just want interoperable multi-vendor
solutions.  They also have little interest in replacing all the data center
networking equipment with MPLS capable gear.  As far as I know, they
universally use VLANs and IP.

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 19 Apr 2012 01:24:04 +0200

Hi Thomas,

> out the voices of operators.

> IMO, saying L3 is enough.

As an operator I 100% agree with above two statements ! Very well observed.

Operator's focus is to find best ways to realize new and unique services 
for their customers. Most vendors above all seek ways to protect their 
portfolio and their margins.

Clearly this list and recent ONS summit (http://opennetsummit.org/) has 
proven that those interests are unfortunately very often non-intersecting.

Rgs,
R.

---

From: Joel jaeggli <[email-masked]>
Date: Wed, 18 Apr 2012 22:31:46 -0700

> particpation from operators as it is.

It's always possible that the wg won't like what they have to say anyway.

---

From: "Bocci, Matthew (Matthew)" <[email-masked]>
Date: Thu, 19 Apr 2012 10:33:15 +0200

John,

On 18/04/2012 22:12, "John E Drake" <[email-masked]<mailto:[email-masked]>> wrote:

Stewart,

“NVO3 will develop an approach to multi-tenancy that uses a Layer 3 encapsulation rather than relying on traditional L2 isolation mechanisms (e.g., VLANs) to support multi-tenancy. The approach will provide an emulated Ethernet service capable of satisfying typical data center deployments.”

because I felt it was inconsistent with the rest of the charter.  Upon further study, I think the ‘inconsistent’ may have been too kind.  I think a better word might be ‘antithetical’, because as is pointed out in this paragraph of the charter:

“Driven by the requirements and consistent with the gap analysis, the WG may request being rechartered to document solutions consisting of one or more data plane encapsulations and control plane protocols as applicable.  Any documented solutions will use existing mechanisms if suitable, or will develop new mechanisms if necessary.”

Why not just rephrase the paragraph so that it does not appear to prescribe protocol development, but rather scopes the solutions to those that the IETF traditionally deals with and that meet the requirements/gap analysis?:

“NVO3 will consider an approach to multi-tenancy that uses a Layer 3 encapsulation rather than relying on traditional L2 isolation mechanisms (e.g., VLANs) to support multi-tenancy, and consistent with a requirements gathering and gap analysis exercise. The approach will provide an emulated Ethernet service capable of satisfying typical data center deployments.”

“Driven by the requirements and consistent with the gap analysis, the WG may request to be re-chartered to document solutions consisting of one or more data plane encapsulations and control plane protocols as applicable.  Any documented solutions will use existing standards if suitable, or will develop new standards if the WG determines and documents that no existing standards can serve as an adequate basis for satisfying the requirements.”

OK with me.

Thanks

Matthew

Thanks,

John

Sent from my iPhone

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Stewart Bryant
Sent: Wednesday, April 18, 2012 12:53 AM
To: Kireeti Kompella
Cc: Yakov Rekhter; [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] NVO3 charter 1530UK 12April

- Stewart

---

From: Randy Bush <[email-masked]>
Date: Thu, 19 Apr 2012 12:08:38 +0200

> others I've talked to that say *exactly* the same thing), MPLS/BGP is

after yesterday's goog revealing, a bunch of us are wondering if vijay 
died on the mpls/bgp sword.

randy

---

From: Randy Bush <[email-masked]>
Date: Thu, 19 Apr 2012 12:11:24 +0200

> died on the mpls/bgp sword.

sigh.  apologies.  this was not meant to be on list

randy

---

From: John E Drake <[email-masked]>
Date: Thu, 19 Apr 2012 05:05:34 -0700

Matthew,

Snipped, comment inline.

Thanks,

John

Sent from my iPhone

"NVO3 will consider an approach to multi-tenancy that uses a Layer 3 encapsulation rather than relying on traditional L2 isolation mechanisms (e.g., VLANs) to support multi-tenancy, and consistent with a requirements gathering and gap analysis exercise. The approach will provide an emulated Ethernet service capable of satisfying typical data center deployments."

[JD]   I also have a problem with the last sentence as it sounds as though NV03 will be encroaching on the charter of the L2VPN WG, which reads, in part:

"The L2VPN working group is responsible for defining and specifying a limited number of solutions for supporting provider-provisioned Layer-2 Virtual Private Networks (L2VPNs). It will also address requirements driven by cloud computing services and data centers as they apply to Layer-2 VPN services."

And:

"5. Ethernet VPN (E-VPN) - An enhanced Layer-2 service that emulates an Ethernet (V)LAN across a PSN. E-VPN supports load-sharing across multiple connections from a Layer-2 site to an L2VPN service. E-VPN is primarily targeted to support large-scale L2VPNs with resiliency requirements not satisfied by other L2VPN solutions."

---

From: <[email-masked]>
Date: Thu, 19 Apr 2012 10:16:59 -0400

John,

I think the potential "data center" overlap with L2VPN can be resolved by carefully understanding the first text extract from L2VPN charter.  I read the second sentence as implying a couple of crucial words in [square brackets]:

"The L2VPN working group is responsible for defining and specifying a limited number of solutions for supporting provider-provisioned Layer-2 Virtual Private Networks (L2VPNs). It will also address requirements driven by cloud computing services and data centers as they apply to [provider-provisioned] Layer-2 VPN services."

We can split hairs over what "provider" means, but I believe the primary distinction in initial focus is data center infrastructure vs. network carrier (provider) provisioning and operation of the overlay (or VPN if one wants to use that term).

The following definition from RFC 4664, Framework for Layer 2 Virtual Private Networks (L2VPNs), may also help in understanding the L2VPN WG's scope:

   The term "provider provisioned VPNs" refers to Virtual Private
   Networks (VPNs) for which the Service Provider (SP) participates in
   management and provisioning of the VPN.

In this context, many data centers of importance to nvo3 (e.g., enterprise data centers) are not operated by the Service Provider, as the term is used in this RFC 4664 definition.

Thanks,
--David

From: [email-masked] [mailto:[email-masked]] On Behalf Of John E Drake
Sent: Thursday, April 19, 2012 8:06 AM
To: Bocci, Matthew (Matthew); Stewart Bryant; Kireeti Kompella
Cc: Yakov Rekhter; Nitin Bahadur; [email-masked]; [email-masked]
Subject: Re: [nvo3] NVO3 charter 1530UK 12April

Matthew,

Snipped, comment inline.

Thanks,

John

Sent from my iPhone

And:

---

From: John E Drake <[email-masked]>
Date: Thu, 19 Apr 2012 08:24:56 -0700

David,

I don't think you have the authority to change the words in the charter, which reads:

"It will also address requirements driven by cloud computing services and data centers as they apply to Layer-2 VPN services."

Further, as I have told you and Thomas multiple times, a data center operator is not required to provide a certificate indicating they are a 'provider' before they are allowed to deploy L3/L2 VPN technology, and many large enterprise networks consider themselves to be service providers in their own right.

Thanks,

John

Sent from my iPhone

From: [email-masked] [mailto:[email-masked]]
Sent: Thursday, April 19, 2012 7:17 AM
To: John E Drake; [email-masked]; [email-masked]; [email-masked]
Cc: Yakov Rekhter; Nitin Bahadur; [email-masked]; [email-masked]
Subject: RE: [nvo3] NVO3 charter 1530UK 12April

John,

Thanks,
--David

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of John E Drake
Sent: Thursday, April 19, 2012 8:06 AM
To: Bocci, Matthew (Matthew); Stewart Bryant; Kireeti Kompella
Cc: Yakov Rekhter; Nitin Bahadur; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] NVO3 charter 1530UK 12April

Matthew,

Snipped, comment inline.

Thanks,

John

Sent from my iPhone

And:

---

From: AshwoodsmithPeter <[email-masked]>
Date: Thu, 19 Apr 2012 17:28:04 +0000

>others I've talked to that say *exactly* the same thing), MPLS/BGP is

While not a statistical sample by any means I've been on the receiving end of some similar comments from customers but my feeling is that they are not reacting to the data plane but more to a certain implementation of the control plane / usability. That is after all what they see and touch.

Peter

---

From: Stewart Bryant <[email-masked]>
Date: Thu, 19 Apr 2012 19:13:31 +0100

On 19/04/2012 00:24, Robert Raszuk wrote:
> Operator's focus is to find best ways to realize new and unique 
> services for their customers. Most vendors above all seek ways to 
> protect their portfolio and their margins.
>
> Clearly this list and recent ONS summit (http://opennetsummit.org/) 
> has proven that those interests are unfortunately very often 
> non-intersecting.
>
> Rgs,
> R. 
Robert

I think that there are several types of operator and I am concerned that 
they have different perspectives, different business imperatives and 
different methods of influencing the IETF.

a) Large existing DC operators - Google, Yahoo, Amazon etc
b) Communications providers that would like to get into the DC business
c) Enterprises that already have large server farms that need to scale 
them using new DC technologies
d) New entrants to the DC business

Some of these send staff to the IETF, some of these only have contact 
with the IETF through their vendors. In some cases the vendor is 
influencing the customer, in some cases the customer is influencing the 
vendor.

We obviously need to listen to the operator community in the IETF, but 
we need to recognize that operators who come to IETF may not provide a 
complete and neutral representation of the requirements, and vendor may 
be pushing their own views or may be correctly representing the needs of 
their customers.

- Stewart

---

From: Paul Unbehagen <[email-masked]>
Date: Thu, 19 Apr 2012 14:41:27 -0600

I usually hear a laugh from enterprise datacenter operators when BGP and MPLS are mentioned as a option inside the DC. 

It comes down to the point of view that BGP and MPLS are too combersome to be worth the time and effort for their environment when there are simpler ways to solve the problems they've had and need just as simple ways to solve the new generation of problems. 

--
Paul Unbehagen
Director, PLM - Avaya Networking
Office: 303-538-1219
Email: [email-masked]

Sent from my iPhone

---

From: S P <[email-masked]>
Date: Thu, 19 Apr 2012 21:00:34 -0400

Paul,

I would think that a laugh from enterprise datacenter operators
is definitely not a reason not to innovate or evolve the technology; and
more importantly, should not be a reason why certain protocol is not chosen
as an effective solution.  Unless there is a fundamental technical issue
with any protocol, that given protocol should be considered a viable option.

Now, in terms of protocols at hand, ie BGP and MPLS, - yes there was a time
many years ago, when IGP was sufficient to run the network, but as soon as
that network grew into N x networks = internetwork, we all moved to BGP as
the former was not suitable - and guess what - operators started reading
RFC 1771 and 2547.

Similarly, there was a time when we all run Spanning-tree and IGPs in data
centers.  But as we evolve DCs into complex networks of networks with scale
requirements of millions of VMs,  as charter states, we should reuse what's
available and innovate when needed, but certainly not based on the premise
of someone's comfort level, but actual technical requirement not
addressable with existing tools.  Protocols will evolve and operators will
read; and this is just a beginning.

Simplicity is a relative term.

--- Senad

On Thu, Apr 19, 2012 at 4:41 PM, Paul Unbehagen <[email-masked]> wrote:

> I usually hear a laugh from enterprise datacenter operators when BGP and
> MPLS are mentioned as a option inside the DC.
>
> It comes down to the point of view that BGP and MPLS are too combersome to
> be worth the time and effort for their environment when there are simpler
> ways to solve the problems they've had and need just as simple ways to
> solve the new generation of problems.
>
> --
> Paul Unbehagen
> > While not a statistical sample by any means I've been on the receiving
> end of some similar comments from customers but my feeling is that they are
> not reacting to the data plane but more to a certain implementation of the
> control plane / usability. That is after all what they see and touch.
> >
> > Peter
> >
> >

---

From: Igor Gashinsky <[email-masked]>
Date: Thu, 19 Apr 2012 23:35:20 -0700 (PDT)

On Thu, 19 Apr 2012, AshwoodsmithPeter wrote:

:: 
:: >I am very much concerned about this. I know that this point is not
:: >shared by all, but for the DC folk I've talked to (and there are
:: >others I've talked to that say *exactly* the same thing), MPLS/BGP is
:: >simply a non-starter.
:: 
:: While not a statistical sample by any means I've been on the receiving 
:: end of some similar comments from customers but my feeling is that they 
:: are not reacting to the data plane but more to a certain implementation 
:: of the control plane / usability. That is after all what they see and 
:: touch.

I obviously can't speak for all datacenter people, but I can tell you that 
the reason for why MPLS is a non-starter for me inside the datacenter is 
that my switches there simply don't support MPLS forwarding (and, neither 
do my server NICs for that matter), and backwards compatibility with the 
(modern) hardware that I have deployed today is a very hard requirement. I 
also know for a fact that I'm not alone in that ;)

As far as pushback to BGP, I suspect that it comes from 3 things:

1) You are absolutely right - some of it is religious (but, not much you 
can do about that)...

2) quite a bit of it is implementation related - an implementation of a 
protocol for convergence across internet-scale, with all the hacks 
and optimizations around that particular set of use-cases, may not be the 
right thing to use inside the datacenter due to those convergence 
properties..

3) (hopefully) a lot of it is coming from the fact that quite a few DC 
operators believe that full transactional semantics that can be used by 
some form of a guaranteed consistency model is absolutely mandatory for 
this mapping system. Since as far as I know, that's not something BGP has, 
then it is simply not the right tool for the job, and any number of 
distributed database/state synchronization methods are a significantly 
better fit, and should be used instead.. 

Hopefully that will provide some context for the pushback...

Thanks,
-igor

--------------------+----------------------+------------------
   Igor Gashinsky   | Network Architecture | Yahoo! Inc.
 [email-masked] |  cell 917.807.2213   | Do You... Yahoo?
--------------------+----------------------+------------------

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 20 Apr 2012 09:24:07 +0200

Igor,

I obviously can't speak for all data center people as well, but I 
completely agree with your below comments from my own perspective.

As far as pushback to BGP (which I personally like a lot) I could add 
that non of my hosts can run BGP code therefor it is a non starter for 
the deployment of end to end (physical host to physical host) over L3 
virtualization.

Considering now public announcement of OVS being part of linux 
distributions I would find it counter productive to propose 
virtualization technology which is not compatible with most common 
operating systems which resources (example: VMs) we are to really 
virtualize.

I do realize that networking people think of network as a layer to 
deliver services. This is just like transport folks were 100% convinced 
in the past that their layer as critical plane to networking people.

However as we have observed in the past that transport and networking 
people have merged now I think we need to face the requirement for 
networking side and host side to merge as well to deliver competitive 
services.

R.

> ::>I am very much concerned about this. I know that this point is not
> ::>shared by all, but for the DC folk I've talked to (and there are
> ::>others I've talked to that say *exactly* the same thing), MPLS/BGP is
> ::>simply a non-starter.
> ::

---

From: "Ivan Pepelnjak" <[email-masked]>
Date: Fri, 20 Apr 2012 10:02:24 +0200

I totally agree with Igor's second argument. Unfortunately the data center switching vendors keep forgetting that most large data centers offer services to more than one tenant (or think that VLANs and VLAN-based VRFs are good enough).

As for the first one, keep in mind that TRILL and SPB both use IS-IS, the protocol that is as religiously opposed in enterprise networks as BGP and MPLS, and yet these same enterprise engineers seem willing to deploy TRILL or SPB or a vendor-specific variant of one or the other. The "trick" is in proper packaging. TRILL and SPB use IS-IS internally ... but only a reasonable subset of IS-IS (no NSAPs, no areas) and its intricacies are never exposed to the operator (unless you have to do in-depth troubleshooting).

Likewise, every hypervisor vendor uses numerous protocols internally (see http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1012382 for a sample list) and nobody would care if BGP is added to that list ... as long as the server/virtualization/network admins don't have to configure it.

Finally, while I agree with Igor's view on BGP's lack of transactional consistency, I am far away from being as positive as he is regarding the perception of DC operators. Most people I'm speaking with either don't understand that problem or don't perceive it as being important (but admittedly their data centers are at least an order of magnitude smaller than Igor's).

Kind regards,
Ivan Pepelnjak

> Igor Gashinsky
> Sent: Friday, April 20, 2012 8:35 AM
> To: AshwoodsmithPeter
> Cc: Thomas Narten; Yakov Rekhter; [email-masked]; Kireeti Kompella;
> [email-masked]

---

From: Randy Bush <[email-masked]>
Date: Fri, 20 Apr 2012 10:14:03 +0200

> TRILL and SPB use IS-IS internally ... but only a reasonable subset of
> IS-IS (no NSAPs, no areas) and its intricacies are never exposed to
> the operator (unless you have to do in-depth troubleshooting).

is-is used to be simple until the ietf complicators decided it needed to
compete with ospf in knob and switch count.  most backbone use of is-is
is still very simple.

randy

---

From: "Bocci, Matthew (Matthew)" <[email-masked]>
Date: Fri, 20 Apr 2012 13:31:38 +0200

John,

How about the following changes to the paragraph below? The intention is to clearly phrase the charter in terms of requirements work that the WG will do, while at the same time scope the requirements we will work on, rather than predicate any ultimate solutions.

"NVO3 will consider approaches to multi-tenancy that use an
encapsulation header that resides at or above the network layer, rather than relying on
traditional L2 isolation mechanisms (e.g., VLANs) to support
multi-tenancy. The WG will determine whether an IP, and/or an emulated Ethernet service is needed to satisfy the needs of the typical data centre."

Regards,

Matthew

On 19/04/2012 16:24, "John E Drake" <[email-masked]<mailto:[email-masked]>> wrote:

David,

I don’t think you have the authority to change the words in the charter, which reads:

“It will also address requirements driven by cloud computing services and data centers as they apply to Layer-2 VPN services."

Further, as I have told you and Thomas multiple times, a data center operator is not required to provide a certificate indicating they are a ‘provider’ before they are allowed to deploy L3/L2 VPN technology, and many large enterprise networks consider themselves to be service providers in their own right.

Thanks,

John

Sent from my iPhone

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]
Sent: Thursday, April 19, 2012 7:17 AM
To: John E Drake; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Cc: Yakov Rekhter; Nitin Bahadur; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: RE: [nvo3] NVO3 charter 1530UK 12April

John,

I think the potential “data center” overlap with L2VPN can be resolved by carefully understanding the first text extract from L2VPN charter.  I read the second sentence as implying a couple of crucial words in [square brackets]:

We can split hairs over what “provider” means, but I believe the primary distinction in initial focus is data center infrastructure vs. network carrier (provider) provisioning and operation of the overlay (or VPN if one wants to use that term).

The following definition from RFC 4664, Framework for Layer 2 Virtual Private Networks (L2VPNs), may also help in understanding the L2VPN WG’s scope:

Thanks,
--David

Matthew,

Snipped, comment inline.

Thanks,

John

Sent from my iPhone

And:

---

From: AshwoodsmithPeter <[email-masked]>
Date: Fri, 20 Apr 2012 12:12:55 +0000

> better fit, and should be used instead.. 

Yes, storing the tunnel inner to outer relationships and the sets of other members of a given L2 domain should be a cake-walk for something like Hadoop/HDFS and there is a list of about 30 odd DC's already using it on impressive scales so clearly that would work.

However the challenge here is that Hadoop and others  of its ilk are not standardized at the IETF so procedurally how would that work? Or are we going to have to get into BGP to Hadoop interworking ;)

Peter

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Fri, 20 Apr 2012 15:17:52 +0200

Hi Matthew,

As a clarification, I'd suggest using the following wording in the last sentence that you suggested:

"The WG will determine whether a routed and/or bridged Ethernet service is needed to satisfy Data Center needs."

Marc
________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Bocci, Matthew (Matthew)
Sent: Friday, April 20, 2012 1:32 PM
To: John E Drake; [email-masked]; Stewart Bryant; [email-masked]
Cc: Yakov Rekhter; Nitin Bahadur; [email-masked]; [email-masked]
Subject: Re: [nvo3] NVO3 charter 1530UK 12April

John,

Regards,

Matthew

David,

Thanks,

John

Sent from my iPhone

John,

Thanks,
--David

Matthew,

Snipped, comment inline.

Thanks,

John

Sent from my iPhone

And:

---

From: Paul Unbehagen <[email-masked]>
Date: Fri, 20 Apr 2012 07:28:20 -0600

This is also the case in all SPB deployments I'm aware of as well.  The simplicity of it makes the most compelling reason for deployment. But it's not a "subset" per say, SPB only added a couple of TLVs to IS-IS, it did not remove functionality. 

Only the edge nodes where a vlan service, of which 16 million are available, needs to provisioned and the core doesn't need to be touched.  I most implementations I've seen so far this is a single cli command.  Compared to BGP configurations that are required in today's vendors  implementations it's difficult to see a equal comparison for a DC operator  

--
Paul Unbehagen

Sent from my iPad

---

From: John E Drake <[email-masked]>
Date: Fri, 20 Apr 2012 08:22:55 -0700

Matthew,

I like Marc's proposal for the paragraph's last sentence.  For the paragraph's first sentence, how about:

"NVO3 will consider approaches to multi-tenancy other than the traditional L2 isolation mechanisms (e.g., VLANs). >

Mentioning an encapsulation header seems to be suggesting  a specific solution.

Thanks,

John

Sent from my iPhone

From: LASSERRE, MARC (MARC) [mailto:[email-masked]]
Sent: Friday, April 20, 2012 6:18 AM
To: Bocci, Matthew (Matthew); John E Drake; [email-masked]; Stewart Bryant; [email-masked]
Cc: Yakov Rekhter; Nitin Bahadur; [email-masked]; [email-masked]
Subject: RE: [nvo3] NVO3 charter 1530UK 12April

Hi Matthew,

Marc
________________________________
From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Bocci, Matthew (Matthew)
Sent: Friday, April 20, 2012 1:32 PM
To: John E Drake; [email-masked]<mailto:[email-masked]>; Stewart Bryant; [email-masked]<mailto:[email-masked]>
Cc: Yakov Rekhter; Nitin Bahadur; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] NVO3 charter 1530UK 12April

John,

Regards,

Matthew

David,

Thanks,

John

Sent from my iPhone

John,

Thanks,
--David

Matthew,

Snipped, comment inline.

Thanks,

John

Sent from my iPhone

And:

---

From: "Stiliadis, Dimitrios (Dimitri)" <[email-masked]>
Date: Fri, 20 Apr 2012 11:09:47 -0500

I think Ivan makes some very good points here.

Let's forget about BGP for a moment, and let's concentrate on the functionality.
Whatever you want to call it, it is pretty much a publish/subscribe model and we can
discuss whether eventual consistency or full consistency is the requirement.

In all the cases the abstract model consists of some distribution mechanism
where an entity learns about state information (routes or routes to tunnel mappings),
and distributes this information to all interested parties (entities that participate
in the same service function). 

In the Openstack/cloud lingo, you can call this RabbitMQ servers and clients, in 
IETF lingo you can call it XMPP clients and brokers, in BGP lingo you call it 
BGP speakers and Route Reflectors (since nobody would dare to use full mesh anyway). 

In different areas you will use different mechanisms to achieve scale.

Now, what you distribute through this infrastructure and what 
type of policies need to be made available, are pretty much orthogonal to 
the pub/sub infrastructure itself. In BGPs case people have gone through a lot of 
pain to standardize those policies, headers, formats, and messages to interoperate, and thus
lots of functionality is there. In all other cases, interoperability is TBD, 
since although they have been used in solutions nobody has 
defined exact message formats and semantics that would allow interoperability. 

So, my suggestion is to forget the religious wars, and let's concentrate on the 
functional requirements as is the charter of this group in any case without
any reference to names. 

Quoting Juliet:
"What's in a name? That which we call a rose By any other name would smell as sweet."

Cheers,

Dimitri

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Ivan
> Pepelnjak
> Sent: Friday, April 20, 2012 1:02 AM
> To: 'Igor Gashinsky'; 'AshwoodsmithPeter'
> Cc: 'Thomas Narten'; 'Yakov Rekhter'; [email-masked]; 'Kireeti Kompella';
> [email-masked]
> I totally agree with Igor's second argument. Unfortunately the data center
> switching vendors keep forgetting that most large data centers offer services
> to more than one tenant (or think that VLANs and VLAN-based VRFs are good
> enough).
> 
> As for the first one, keep in mind that TRILL and SPB both use IS-IS, the
> protocol that is as religiously opposed in enterprise networks as BGP and
> MPLS, and yet these same enterprise engineers seem willing to deploy TRILL or
> SPB or a vendor-specific variant of one or the other. The "trick" is in proper
> packaging. TRILL and SPB use IS-IS internally ... but only a reasonable subset
> of IS-IS (no NSAPs, no areas) and its intricacies are never exposed to the
> operator (unless you have to do in-depth troubleshooting).
> 
> Likewise, every hypervisor vendor uses numerous protocols internally (see
> http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displ
> ayKC&externalId=1012382 for a sample list) and nobody would care if BGP is
> added to that list ... as long as the server/virtualization/network admins
> don't have to configure it.
> 
> Finally, while I agree with Igor's view on BGP's lack of transactional
> consistency, I am far away from being as positive as he is regarding the
> perception of DC operators. Most people I'm speaking with either don't
> understand that problem or don't perceive it as being important (but
> admittedly their data centers are at least an order of magnitude smaller than
> Igor's).
> 
> Kind regards,
> Ivan Pepelnjak
>

---

From: AshwoodsmithPeter <[email-masked]>
Date: Fri, 20 Apr 2012 21:25:55 +0000

> Whatever you want to call it, it is pretty much a 
> publish/subscribe model

Agreed.. Dimitri, in that spirit FWIW here is a crack at a neutral 
names/no-types/protocols 'model' based on some of the debate so far.

1) We have inner,middle,outer untyped identifiers in the millions.
    { id.in, id.mid, id.out} 

2) We have global, local untyped names for a group in the 10s of thousands.
    { group.glob, group.loc}

3) We have a way to PUT ( <key>, value ) and GET(<key>)->value
   pairs and to SUBSCRIBE to changes at all outer addressable
   devices that are members of at least one group very quickly. PUB/SUB.

Here is trival example where inner id's are {a1...c3}, outer 
id's are {A,B,C} and the middle are am1..

b1 -bm1-+                +-- am1 - a1
        |                |
b2 -bm2-+ B ---------- A +-- am2 - a2
        |  \         /   |
b3 -bm3-+   \       /    +-- am3 - a3
             \     /
              \   /
                C
             +--+--+
             |  |  |
           cm1 cm2 cm3
             |  |  |
            c1 c2  c3

3) Unicast Data Plane model supports encapsulation which is superset of:

     [id.out] [group.local] [id.mid] [id.in] 

4) Multicast is by ? serial unicast only strong requirement?
5) Loop prevention by ? - not discussed

6) Control Plane model supports a superset of:

   Membership resolution
       - PUT( < id.out, group.glob > , group.loc)
       - GET( < id.out, group.glob > ) -> group.loc
       - GET (< * , group.glob > ) -> { <group.loc, id.out> .. }

   Middle<=>Outer tunnel resolution
       - PUT ( < id.mid, group.glob> , id.out )
       - GET ( < id.mid, group.glob>) -> id.out 

   Middle<=>Inner (i.e. ARP resolution)
       - PUT ( < id.in, group.glob> , id.mid )
       - GET ( < id.in, group.glob> ) -> id.mid 

Clearly there is lots of interesting stuff to debate without getting into what it looks like with any specific types/protocols and of course the gap analysis then take the form of where an existing protocol differs from an agreed 'good' abstract model.

Peter Ashwood-Smith

---

From: Larry Kreeger <[email-masked]>
Date: Fri, 20 Apr 2012 15:48:08 -0700

I support the sentence below from Matthew which reads ³ The WG will
determine whether an IP, and/or an emulated Ethernet service is needed to
satisfy the needs of the typical data centre." because I just noticed that
the latest charter is missing the following sentence that was in the charter
Benson sent out after the BOF which read, ³This approach should provide an
Ethernet service.  It may provide an IP service; an important goal is to
develop a  "layer agnostic" framework and architecture meeting data center
requirements.".

I agreed that keeping the architecture ³layer agnostic² is a very good idea
to leave room for how DC networking may evolve in the future.  A Layer 2
service may be needed now (at least for some applications), but layer 3
service may suffice for many applications and give benefits in terms of
scale and/or simplification and/or interoperability.  Lets not decide at the
time of charter.

 - Larry

On 4/20/12 4:31 AM, "Bocci, Matthew (Matthew)"
<[email-masked]> wrote:

> John,
> 
> How about the following changes to the paragraph below? The intention is to
> clearly phrase the charter in terms of requirements work that the WG will do,
> while at the same time scope the requirements we will work on, rather than
> predicate any ultimate solutions.
> 
> multi-tenancy. The WG will determine whether an IP, and/or an emulated
> Ethernet service is needed to satisfy the needs of the typical data centre."
> 
> Regards,
> 
> Matthew
> 
> 
> On 19/04/2012 16:24, "John E Drake" <[email-masked]> wrote:
> 
>> David,
>>  
>> I don¹t think you have the authority to change the words in the charter,
>> which reads:  
>>  
>> ³It will also address requirements driven by cloud computing services and
>> data centers as they apply to Layer-2 VPN services."
>>  
>> Further, as I have told you and Thomas multiple times, a data center operator
>> is not required to provide a certificate indicating they are a Œprovider¹
>> before they are allowed to deploy L3/L2 VPN technology, and many large
>> enterprise networks consider themselves to be service providers in their own
>> right.
>>  
>> Thanks,
>>  
>> John
>>  
>> 
>> Sent from my iPhone
>>  
>> 
>> I think the potential ³data center² overlap with L2VPN can be resolved by
>> carefully understanding the first text extract from L2VPN charter.  I read
>> the second sentence as implying a couple of crucial words in [square
>> brackets]:
>>  
>> "The L2VPN working group is responsible for defining and specifying a limited
>> number of solutions for supporting provider-provisioned Layer-2 Virtual
>> Private Networks (L2VPNs). It will also address requirements driven by cloud
>> computing services and data centers as they apply to [provider-provisioned]
>> Layer-2 VPN services."
>>  
>> 
>> We can split hairs over what ³provider² means, but I believe the primary
>> distinction in initial focus is data center infrastructure vs. network
>> carrier (provider) provisioning and operation of the overlay (or VPN if one
>> wants to use that term).
>>  
>> The following definition from RFC 4664, Framework for Layer 2 Virtual Private
>> Networks (L2VPNs), may also help in understanding the L2VPN WG¹s scope:
>>  
>> In this context, many data centers of importance to nvo3 (e.g., enterprise
>> data centers) are not operated by the Service Provider, as the term is used
>> in this RFC 4664 definition.
>>  
>> Thanks,
>> --David
>>  
>> 
>> Why not just rephrase the paragraph so that it does not appear to prescribe
>> protocol development, but rather scopes the solutions to those that the IETF
>> traditionally deals with and that meet the requirements/gap analysis?:
>> 
>>  
>> 
>> ³NVO3 will consider an approach to multi-tenancy that uses a Layer 3
>> encapsulation rather than relying on traditional L2 isolation mechanisms
>> (e.g., VLANs) to support multi-tenancy, and consistent with a requirements
>> gathering and gap analysis exercise. The approach will provide an emulated
>> Ethernet service capable of satisfying typical data center deployments.²
>>  
>> [JD]   I also have a problem with the last sentence as it sounds as though
>> NV03 will be encroaching on the charter of the L2VPN WG, which reads, in
>> part:
>>  
>> "The L2VPN working group is responsible for defining and specifying a limited
>> number of solutions for supporting provider-provisioned Layer-2 Virtual
>> Private Networks (L2VPNs). It will also address requirements driven by cloud
>> computing services and data centers as they apply to Layer-2 VPN services."
>>  
>> And:
>>  
>> "5. Ethernet VPN (E-VPN) - An enhanced Layer-2 service that emulates an
>> Ethernet (V)LAN across a PSN. E-VPN supports load-sharing across multiple
>> connections from a Layer-2 site to an L2VPN service. E-VPN is primarily
>> targeted to support large-scale L2VPNs with resiliency requirements not
>> satisfied by other L2VPN solutions."
>>  
> 
>

---

From: Xuxiaohu <[email-masked]>
Date: Sat, 21 Apr 2012 01:35:11 +0000

Hi Marc,

I agree that L3 over L3 should also be considered by NVo3 besides of the L2 over L3.

However, I think the sentence you proposed “a routed and/or bridged Ethernet service” is still not clear enough and hence it may be interpreted mistakenly as follows: the routed Ethernet service means a L2VPN service with control-plane based MAC learning capabilities such as EVPN or something like that (By the way, I remembered one prior name of the EVPN is “routed VPLS” or “R-VPLS” in short), while the bridged Ethernet service means a L2VPN service with data-plane based MAC learning capabilities, such as existing BGP or LDP based VPLS.

Best regards,
Xiaohu

发件人: [email-masked] [mailto:[email-masked]] 代表 LASSERRE, MARC (MARC)
发送时间: 2012年4月20日 21:18
收件人: Bocci, Matthew (Matthew); John E Drake; [email-masked]; Stewart Bryant; [email-masked]
抄送: Yakov Rekhter; Nitin Bahadur; [email-masked]; [email-masked]
主题: Re: [nvo3] NVO3 charter 1530UK 12April

Hi Matthew,

As a clarification, I’d suggest using the following wording in the last sentence that you suggested:

“The WG will determine whether a routed and/or bridged Ethernet service is needed to satisfy Data Center needs.”

Marc

John,

Regards,

Matthew

David,

Thanks,

John

Sent from my iPhone

John,

Thanks,
--David

Matthew,

Snipped, comment inline.

Thanks,

John

Sent from my iPhone

And:

---

From: Stewart Bryant <[email-masked]>
Date: Sat, 21 Apr 2012 08:33:53 +0100

On 21/04/2012 02:35, Xuxiaohu wrote:
> However, I think the sentence you proposed "a routed and/or bridged 
> Ethernet service" is still not clear enough and hence it may be 
> interpreted mistakenly as follows: the routed Ethernet service means a 
> L2VPN service with control-plane based MAC learning capabilities such 
> as EVPN or something like that (By the way, I remembered one prior 
> name of the EVPN is "routed VPLS" or "R-VPLS" in short), while the 
> bridged Ethernet service means a L2VPN service with data-plane based 
> MAC learning capabilities, such as existing BGP or LDP based VPLS.
>
There are two problems with this

Does it mean ("routed Ethernet service" || "bridged Ethernet service") 
or does it mean ("routed service" || "bridged Ethernet service")?

There are so many flavours of Ethenet switching and transmission that we 
need to be very careful.

If we say routing meaning IP routing then we end up with issues of 
whether MPLS is in or out.

The original text "The WG will determine whether an IP, and/or an 
emulated Ethernet service is needed" avoids this minefield.

- Stewart

---

From: David Freedman <[email-masked]>
Date: Sat, 21 Apr 2012 08:52:52 +0000

"The WG will determine whether an IP, and/or an emulated Ethernet service is needed"

How about dropping the references to IP and Ethernet completely to avoid arguments like this?

"The WG will determine which types of service are needed"

If you really wanted to keep IP/Ethernet, you could provide these as examples

"The WG will determine which types of service are needed (for example, IP or Ethernet)"

Dave.

From: Stewart Bryant <[email-masked]<mailto:[email-masked]>>
Reply-To: <[email-masked]<mailto:[email-masked]>>
Date: Sat, 21 Apr 2012 08:33:53 +0100
To: Xuxiaohu <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "LASSERRE, MARC (MARC)" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [nvo3] NVO3 charter 1530UK 12April

Does it mean ("routed Ethernet service" || "bridged Ethernet service") or does it mean ("routed service" || "bridged Ethernet service")?

There are so many flavours of Ethenet switching and transmission that we need to be very careful.

If we say routing meaning IP routing then we end up with issues of whether MPLS is in or out.

The original text ""The WG will determine whether an IP, and/or an emulated Ethernet service is needed" avoids this minefield.

- Stewart
_______________________________________________ nvo3 mailing list [email-masked]<mailto:[email-masked]> https://www.ietf.org/mailman/listinfo/nvo3

---

From: Stewart Bryant <[email-masked]>
Date: Sat, 21 Apr 2012 10:11:58 +0100

Yes, that looks like a good way of expressing the problem

Stewart

On 21/04/2012 09:52, David Freedman wrote:
> From: Stewart Bryant <[email-masked] <mailto:[email-masked]>>
> Reply-To: <[email-masked] <mailto:[email-masked]>>
> To: Xuxiaohu <[email-masked] <mailto:[email-masked]>>
> Cc: "[email-masked] <mailto:[email-masked]>" <[email-masked] 
> <mailto:[email-masked]>>, "[email-masked] <mailto:[email-masked]>" 
> <[email-masked] <mailto:[email-masked]>>, "LASSERRE, MARC (MARC)" 
> <[email-masked] 
> <mailto:[email-masked]>>
>> However, I think the sentence you proposed “a routed and/or bridged 
>> Ethernet service” is still not clear enough and hence it may be 
>> interpreted mistakenly as follows: the routed Ethernet service means 
>> a L2VPN service with control-plane based MAC learning capabilities 
>> such as EVPN or something like that (By the way, I remembered one 
>> prior name of the EVPN is “routed VPLS” or “R-VPLS” in short), while 
>> the bridged Ethernet service means a L2VPN service with data-plane 
>> based MAC learning capabilities, such as existing BGP or LDP based VPLS.
>>
> The original text ""The WG will determine whether an IP, and/or an 
> _______________________________________________ nvo3 mailing list 
> [email-masked] <mailto:[email-masked]>

---

From: Xuxiaohu <[email-masked]>
Date: Sat, 21 Apr 2012 01:16:16 +0000

I agree with Larry that both L2 over L3 and L3 over L3 should be considered and be included in the charter. However, the term “layer agnostic”  seems not concrete enough and hence may cause some confusion. For example, it could be interpreted as “underlying network agnostic” (e.g., L2 over L2, L2 over nickname…)

Best regards,
Xiaohu

发件人: [email-masked] [mailto:[email-masked]] 代表 Larry Kreeger
发送时间: 2012年4月21日 6:48
收件人: Bocci, Matthew (Matthew); John E Drake; David Black; Stewart Bryant; [email-masked]
抄送: Yakov Rekhter; Nitin Bahadur; [email-masked]; [email-masked]
主题: Re: [nvo3] NVO3 charter 1530UK 12April

I support the sentence below from Matthew which reads “ The WG will determine whether an IP, and/or an emulated Ethernet service is needed to satisfy the needs of the typical data centre." because I just noticed that the latest charter is missing the following sentence that was in the charter Benson sent out after the BOF which read, “This approach should provide an Ethernet service.  It may provide an IP service; an important goal is to develop a  "layer agnostic" framework and architecture meeting data center requirements.".

I agreed that keeping the architecture “layer agnostic” is a very good idea to leave room for how DC networking may evolve in the future.  A Layer 2 service may be needed now (at least for some applications), but layer 3 service may suffice for many applications and give benefits in terms of scale and/or simplification and/or interoperability.  Lets not decide at the time of charter.

 - Larry

On 4/20/12 4:31 AM, "Bocci, Matthew (Matthew)" <[email-masked]> wrote:
John,

Regards,

Matthew

Thanks,

John

Sent from my iPhone

John,

Thanks,
--David

Matthew,

Snipped, comment inline.

Thanks,

John

Sent from my iPhone

And:

---

From: James Kempf <[email-masked]>
Date: Mon, 23 Apr 2012 17:00:01 -0400

I've talked with a Tier 1 service provider that is interested in MPLS in their data center.

People tend to ignore the distinction between the MPLS data plane protocol, which is conceptually simple and quite elegant, and the control plane protocols, which are complex and difficult to understand. 

		jak 

> Behalf Of Paul Unbehagen
> Sent: Thursday, April 19, 2012 1:41 PM
> To: AshwoodsmithPeter
> Cc: Thomas Narten; Yakov Rekhter; [email-masked]; Kireeti 
> Kompella; [email-masked]
> I usually hear a laugh from enterprise datacenter operators 
> when BGP and MPLS are mentioned as a option inside the DC. 
> 
> It comes down to the point of view that BGP and MPLS are too 
> combersome to be worth the time and effort for their 
> environment when there are simpler ways to solve the problems 
> they've had and need just as simple ways to solve the new 
> generation of problems. 
> 
> --
> Paul Unbehagen
> > While not a statistical sample by any means I've been on 
> the receiving end of some similar comments from customers but 
> my feeling is that they are not reacting to the data plane 
> but more to a certain implementation of the control plane / 
> usability. That is after all what they see and touch.
> > 
> > Peter
> > 
> >

---

From: Linda Dunbar <[email-masked]>
Date: Fri, 27 Apr 2012 21:33:34 +0000

Peter, 

Very good analysis of the needed protocols. 

Is it correct to assume the following?
  -  "id.in" refers to VM/host's IP address, 
  - "id.mid" refers to VM/host's MAC address, 
  - "id.out" refers to overlay boundary point's IP address,
  - "group.loc" is the label encoded in the data packet, like NvGRE or VxLan's key field, or MPLS ID if MPLS is used, 
  - "group.glob" is the globally unique client id, which can be same as "group.loc" if NvGRE or VxLAN encapsulation is used. 

Then, I think we should also introduce the "id.mid.VID", which is the local VLAN id or C-VLAN id. 
When hosts or VMs are on hop away from the "overlay boundary point", that is when MAC address and c-VLAN is added to the data frame. In order to enable more than 4K's client isolation, the c-VLAN has to be locally significant. 
The "PUT" action by Overlay boundary point (Hypervisor or ToR) has to indicate its local c-VLAN. When the packet reaches to the Overlay egress point, the egress node has to change the c-VLAN in the payload of the encapsulated data frame to the correct local c-VLAN. 

Linda 

> AshwoodsmithPeter
> Sent: Friday, April 20, 2012 4:26 PM
> To: Stiliadis, Dimitrios (Dimitri); Ivan Pepelnjak; 'Igor Gashinsky'
> Clearly there is lots of interesting stuff to debate without getting
> into what it looks like with any specific types/protocols and of course
> the gap analysis then take the form of where an existing protocol
> differs from an agreed 'good' abstract model.
> 
> Peter Ashwood-Smith

---

From: AshwoodsmithPeter <[email-masked]>
Date: Mon, 30 Apr 2012 13:38:04 +0000

Thanks Linda,

Actually I was just thinking that the middle identifier id.mid could actually include the vid or vid pair so from the models perspective its one big identifier or would something 'not work' if its grouped that way?

Peter

-----Original Message-----
From: Linda Dunbar 
Sent: Friday, April 27, 2012 5:34 PM
To: AshwoodsmithPeter; Stiliadis, Dimitrios (Dimitri); Ivan Pepelnjak; 'Igor Gashinsky'
Cc: 'Thomas Narten'; 'Yakov Rekhter'; [email-masked]; 'Kireeti Kompella'; [email-masked]
Subject: RE: [nvo3] NVO3 charter 1530UK 12April

Peter, 

Linda

---

From: Linda Dunbar <[email-masked]>
Date: Mon, 30 Apr 2012 22:38:13 +0000

Peter, 

If "id.mid" includes the local VID, the pair of <id.in, global.glob> will be mapped to different "id.mid" under different overlay boundary point ("id.out"). This is due the fact that VID is locally significant. 

Under this circumstance, your Middle<=>Inner (i.e. ARP resolution) has to be different for different Overlay Boundary points. i.e. need an additional attribute: 

	- PUT ( < id.in, group.glob, id.out> , id.mid )
	- GET ( < id.in, group.glob, id.out> ) -> id.mid

Basically there are two different types of "local ID": Your current "global.loc refers to the local identifier facing "underlay network" (like MPLS local TAG).

There is another local group identifier:  (for lack of better name) "group.loc-towards-Host" for local VID or other identifiers facing towards VMs, blade servers, or servers. 

Linda 

> From: AshwoodsmithPeter
> Sent: Monday, April 30, 2012 8:38 AM
> To: Linda Dunbar; Stiliadis, Dimitrios (Dimitri); Ivan Pepelnjak; 'Igor
> Gashinsky'
> Actually I was just thinking that the middle identifier id.mid could
> actually include the vid or vid pair so from the models perspective its
> one big identifier or would something 'not work' if its grouped that
> way?
> 
> Peter
> 
> 
>   - "group.loc" is the label encoded in the data packet, like NvGRE or
> VxLan's key field, or MPLS ID if MPLS is used,
>   - "group.glob" is the globally unique client id, which can be same as
> "group.loc" if NvGRE or VxLAN encapsulation is used.
> 
> When hosts or VMs are on hop away from the "overlay boundary point",
> that is when MAC address and c-VLAN is added to the data frame. In
> order to enable more than 4K's client isolation, the c-VLAN has to be
> locally significant.
> The "PUT" action by Overlay boundary point (Hypervisor or ToR) has to
> indicate its local c-VLAN. When the packet reaches to the Overlay
> egress point, the egress node has to change the c-VLAN in the payload
> of the encapsulated data frame to the correct local c-VLAN.
> 
> Linda
> 
> 
> 
> Of