Subject: [nvo3] Revised Draft Charter

From: Thomas Narten <[email-masked]>
Date: Sat, 24 Mar 2012 16:19:14 -0400

Hi.

Here is a revised draft charter for this effort. Comments welcome on
the list or during Wednesday's session.

It has been shortened by removing a bunch of the background
information. And the proposed deliverables have been crisped up.

Thomas

NVO: Network Virtualization Overlays (Version 2012-03-24)

Support for multi-tenancy has become a core requirement of data
centers (DCs), especially in the context of data centers supporting
virtualized servers known as virtual machines (VMs).  A key
multi-tenancy requirement is traffic isolation, so that a tenant's
traffic (and internal address usage) is not visible to any other
tenant and does not collide with addresses used within the data center
itself. Another key requirement is to support the placement and live
migration of VMs anywhere within a data center, without being
constrained by DC network constraints such as the IP subnet boundaries
of the underlying DC network.

This WG will develop an approach to multi-tenancy that does not rely
on any underlying L2 mechanisms to support multi-tenancy. In
particular, the WG will develop an approach where multitenancy is
provided at the IP layer using an encapsulation header that resides
above IP.

A key area of work is the development of requirements for (and to find
solutions for) control plane functions that allow an ingress to the
overlay to map the "inner" (tenant VM) address into an "outer"
(underlying network) address in order to tunnel a packet to the
destination VM. Two different approaches will be pursued, targeting
two different deployment environments.

The first approach will document a data plane learning mechanism
similar to IEEE 802.1Q learning and would be appropriate for smaller
data centers where flooding of frames to unknown destinations is not
considered a major problem. This document is expected to be relatively
short, as IEEE 802.1Q learning is well-documented and understood.

The second approach will focus on a control plane aimed at large data
centers, capable of scaling to hundreds of thousands of nodes.  This
control plane will be the major work item for this WG. A key
requirement will be to avoid the need for flooding in the case where a
mapping from an inner source VM to a target tunnel end point address
is not present at the ingress to the overlay network. A second
requirement may be that the control plane must also be applicable to
geographically dispersed data centers supporting a single overlay. The
first work item is to develop requirements for this control protocol.

Both control planes will need to handle the case where VMs dynamically
move within the network in a dynamic fashion, meaning that they will
need to support the case where the address mappings change and an
encapsulating ingress to the overlay may have stale information that
needs updating.

A third work area concerns VMs and end systems attaching to and
detaching from an overlay. When a VM is activated at a new location,
the hypervisor acting on behalf of the VM needs to register the VM
with the overlay, so that the overlay can ensure that traffic to and
from the VM is delivered properly. This WG will develop appropriate
mechanisms for registration and deregistration. One candidate for
providing this functionality may be IEEE's VDP mechanism. Should a
VDP-based approach be pursued, any activity would be closely
coordinated with the appropriate IEEE group.

A fourth area of work concerns the encapsulation format needed to
provide an overlay service. This WG will develop requirements and
desirable properties for any encapsulation format. Given the number of
already existing encapsulation formats that appear to be suitable for
overlays, it is not an explicit goal of this effort to choose exactly
one format or to develop yet another new one. Rather, it is a goal to
ensure that any existing encapsulation formats that support the
necessary requirements be supported within the overall framework.

A fifth work area may be needed to connect an overlay network with a
traditional L2 or L3 VPN (e.g., VPLS). One approach appears straight
forward, in that there is a clear boundary between a VPN device and
the edge of an overlay network. Packets forwarded across the boundary
would simply need to have the tenant identifier on the overlay side
mapped into a corresponding VPN identifier on the VPN
side. Conceptually, this would appear to be analogous to what is done
already today when interfacing between L2 VLANs and VPNs.

Finally, although the main objective of this WG is to produce a
solution that supports an L2 over L3 overlay, an important goal is to
develop a "layer agnostic" framework and architecture meeting data
center requirements.  It is not the goal of the working group to
develop more general layer-x-over-layer-y overlay solutions.

The specific deliverables for this group include:

0) Problem statement as an Informational RFC (starting point:
draft-narten-nvo3-overlay-problem-statement-01.txt)

1) Standards Track control plane document that uses learning for
distributing address mappings.

2) Control plane requirements for a scalable control plane as
informational RFC. Starting point:
draft-kreeger-nvo3-overlay-cp-00.txt

3) Control plane solution matching requirements above.

4) Mechanisms for end systems to attachment/detachment from an
overlay network. 

5) Encapsulation requirements as informational RFC.

---

From: Dino Farinacci <[email-masked]>
Date: Sat, 24 Mar 2012 14:05:28 -0700

> information. And the proposed deliverables have been crisped up.

A couple of points. See inline.

> coordinated with the appropriate IEEE group.

Work areas 1-2 and 4-5 do not talk about specific solutions but rather general approaches. Why does this 3rd work area get specific about a solution?  I suggest striking the last 2 sentences from the paragraph.

> develop more general layer-x-over-layer-y overlay solutions.

Why does a goal have a specific solution stated? Doesn't the working group want to solve the problem the best and most scalable way? If so, this early in the process one should not state in the charter that L2-over-L3 should be a solution. Otherwise, you should also state L3-over-L3, L3-over-L4, etc. And you probably don't want to enumerate all combinations.

Dino

---

From: "Guyingjie (Yingjie)" <[email-masked]>
Date: Sun, 25 Mar 2012 04:06:25 +0000

Hi  Thomas,
Does the sequence of the work areas listed in the charter  also show the sequence of the work  that will be done after the wg is set up? If yes, I would say the 3rd item, i.e. the attach/dettach mechanism should be defined before a control plane is defined, because the definition of thr former may influent the latter. 

I support VDP, but there still might be other mechanism that can also make it, could it be a notification from end-system, or from  VM Manager. Different mechanism will require different action on control plane.

________________________________________
发件人: [email-masked] [[email-masked]] 代表 Thomas Narten [[email-masked]]
发送时间: 2012年3月25日 4:19
到: [email-masked]
主题: [nvo3] Revised Draft Charter

Hi.

Thomas

_______________________________________________
nvo3 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nvo3

---

From: <[email-masked]>
Date: Sun, 25 Mar 2012 07:36:09 -0400

Hi Dino,

It looks like solution-oriented material on not only VDP, but also VPN identifier
mapping has been included in the draft charter for context.  It's not unusual to tell
a WG that it should look at certain approaches.  VDP is an important non-IETF candidate
protocol for this function that needs to be considered in addition to IETF protocols
(e.g., there's another draft that proposes XMPP for essentially the same purpose).
The text only suggests VDP - it does not mandate it's use - IMHO, it'd be ok to make
the text more inclusive of alternative approaches, but I think it's useful to draw
attention to VDP.

In contrast, there should eventually be some sort of framework draft on interaction of
NVO3 with VPNs (mea culpa - I didn't have the time to write one for Paris, and it's
probably going to require an author team that includes familiarity with all of the
interesting/relevant VPN technologies). 

The reason for mentioning L2 over L3 overlays is that there are already at least 3
such data planes (NVGRE, STT, VXLAN) that are being deployed without asking for the
IETF's permission or blessing.  The window to do something wrt these isn't going
to stay open for very long ...

Thanks,
--David

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Dino Farinacci
> Sent: Saturday, March 24, 2012 5:05 PM
> To: Thomas Narten
> Cc: [email-masked]
> Subject: Re: [nvo3] Revised Draft Charter
> 
> Work areas 1-2 and 4-5 do not talk about specific solutions but rather general approaches. Why does
> this 3rd work area get specific about a solution?  I suggest striking the last 2 sentences from the
> paragraph.
> 
> Why does a goal have a specific solution stated? Doesn't the working group want to solve the problem
> the best and most scalable way? If so, this early in the process one should not state in the charter
> that L2-over-L3 should be a solution. Otherwise, you should also state L3-over-L3, L3-over-L4, etc.
> And you probably don't want to enumerate all combinations.
> 
> Dino
> 
>

---

From: Dino Farinacci <[email-masked]>
Date: Sun, 25 Mar 2012 10:13:49 -0700

> mapping has been included in the draft charter for context.  It's not unusual to tell

VPN identifer is just a term.

> the text more inclusive of alternative approaches, but I think it's useful to draw

Then can you state that "IEEE protocols will be considered" and not single out one specific solution?

> to stay open for very long ...

Yes, but there are other proposals for L3 over L3 that have been around years before the ones you mention. I am not asking you to call them out. I am asking you to be more generic and keep an open mind to any solution that ia brought to the working group. The charter needs to convey that open mindedness in the text.

Dino

---

From: Qin Wu <[email-masked]>
Date: Sun, 25 Mar 2012 19:36:35 +0000

[Qin]: Agree, I am wondering why do we need to emphasize on L2 over L3 approaches, would you like to clarify why L2 over L3 approach is better than L3 over L3 approach? Have you done the comparison analysis Or this is just becos this piece of work only belongs to routing area or only IEEE approaches are taken into accounted? Any other work not belonging to routing area or non-IEEE approaches should not be the scope of this proposed charter?

Dino

---

From: Dino Farinacci <[email-masked]>
Date: Sun, 25 Mar 2012 12:39:02 -0700

> [Qin]: Agree, I am wondering why do we need to emphasize on L2 over L3 approaches, would you like to clarify why L2 over L3 approach is better than L3 over L3 approach? Have you done the comparison analysis Or this is just 

We can discuss that at the working group. But for now we are commenting on the charter.

> becos this piece of work only belongs to routing area or only IEEE approaches are taken into accounted? Any other work not belonging to routing area or non-IEEE approaches should not be the scope of this proposed charter?

The working group should to the analysis.

Dino

---

From: Ken Gray <[email-masked]>
Date: Sun, 25 Mar 2012 13:00:09 -0700

>two different deployment environments.

I'm curious as to what the goals of that design are ..

That the protocol be self-mapping (that one layer's address be somehow
derivable from the others)?

Failing that, that the mapping agent reuse some existing protocol for
mapping (probably by extension) rather than inventing a new one?

---

From: <[email-masked]>
Date: Sun, 25 Mar 2012 16:46:58 -0400

> [Qin]: Agree, I am wondering why do we need to emphasize on L2 over L3 approaches, would you like to
> clarify why L2 over L3 approach is better than L3 over L3 approach?  Have you done the comparison
> analysis

Running code - a lot of the deployed hypervisor networking support for virtual machines currently require L2 networking, and those are the systems that are hitting the VLAN tag size limits.  As I noted in my response to Dino, the L2 over L3 data planes are being developed **now**, so the window to engage on their control aspects is fairly short.  A debate on whether L3 over L3 is somehow better will be ignored by that running code ...

Or this is just becos this piece of work only belongs to routing area or only IEEE approaches
> are taken into accounted?  Any other work not belonging to routing area or non-IEEE approaches should
> not be the scope of this proposed charter?

I hope those weren't intended as serious questions ...

Thanks,
--David

> From: Qin Wu [mailto:[email-masked]]
> Sent: Sunday, March 25, 2012 3:37 PM
> To: Dino Farinacci; Black, David
> Cc: [email-masked]; [email-masked]
> Subject: 答复: [nvo3] Revised Draft Charter
> 
> Yes, but there are other proposals for L3 over L3 that have been around years before the ones you
> mention. I am not asking you to call them out. I am asking you to be more generic and keep an open
> mind to any solution that ia brought to the working group. The charter needs to convey that open
> mindedness in the text.
> 
> [Qin]: Agree, I am wondering why do we need to emphasize on L2 over L3 approaches, would you like to
> clarify why L2 over L3 approach is better than L3 over L3 approach? Have you done the comparison
> analysis Or this is just becos this piece of work only belongs to routing area or only IEEE approaches
> are taken into accounted? Any other work not belonging to routing area or non-IEEE approaches should
> not be the scope of this proposed charter?
> 
> Dino
> 
> >
> > Thanks,
> > --David
> >

---

From: Yiqun Cai <[email-masked]>
Date: Sun, 25 Mar 2012 22:19:34 +0000

My $0.02 inline.

--
Yiqun

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> [email-masked]
> Sent: Sunday, March 25, 2012 1:47 PM
> To: [email-masked]; [email-masked]
> Cc: [email-masked]; [email-masked]
> > [Qin]: Agree, I am wondering why do we need to emphasize on L2 over L3
> > approaches, would you like to clarify why L2 over L3 approach is
> > better than L3 over L3 approach?  Have you done the comparison
> > analysis
> 
> Running code - a lot of the deployed hypervisor networking support for
> virtual machines currently require L2 networking, and those are the systems
> that are hitting the VLAN tag size limits.  As I noted in my response to Dino,
> the L2 over L3 data planes are being developed **now**, so the window to
> engage on their control aspects is fairly short

This hasn't shown to be a problem with the IETF standardization process, at least not recently (in the last 15-18 years).  I personally went through a couple of those situations while I was with a different company.  In one case, we had more than running code.  There was real-life deployment with multi-vendor implementations.   I was reminded multiple times that "IETF is not there to rubber stamp and cater to individual vendor, or vendors' need".

In short,  having running code out there is a consideration, but not a deciding factor when the working group charter is defined.
 
> A debate on whether L3 over
> L3 is somehow better will be ignored by that running code ...
> 

A vendor can choose to ignore whatever IETF produces. Nobody, other than its customers, will care. 

But we are discussing the charter. So  I would rather see a technical analysis that an L3 over L3 approach is inferior so that it is excluded.  In the absence of such an analysis, it would be more appropriate to leave some room for the future, or redefine/rename the WG.

Thanks.

---

From: Qin Wu <[email-masked]>
Date: Mon, 26 Mar 2012 05:07:41 +0000

-----邮件原件-----
发件人: [email-masked] [mailto:[email-masked]] 
发送时间: 2012年3月26日 4:47
收件人: Qin Wu; [email-masked]
抄送: [email-masked]; [email-masked]
主题: RE: [nvo3] Revised Draft Charter

[Qin]: Are you assuming L3 over L3 approach has no running code? No?
It seems charter claims to try to reuse existing mechanisms? So why not reuse the existing
Protocol IETF have already had?
I agree VLAN tag size is going to running out limits, however I am wondering what kind of special requirements you want to focus on?
Since I still don't understand why L3 over L3 (the existing protocol) approach should be excluded?
Maybe I miss something in the past discussion.

[Qin]: No, but I am just curious to know the answer since I am not so much familiar what IEEE are doing in this aspect.

Thanks,
--David

---

From: <[email-masked]>
Date: Mon, 26 Mar 2012 03:38:21 -0400

> requirements you want to focus on?

The networking service provided by hypervisors to VMs is an L2 service not L3 (VMs send L2 frames and expect L2 functionality).  This is part of the *problem* definition (VLAN tag isn't large enough, need something larger to functionally replace it), and the running code in hypervisors is L2 (e.g., Ethernet softswitches), and I take that as a given (any proposal to use L3 over L3 that requires major hypervisor networking modifications, in contrast to just softswitch modifications, is not going to get the job done).

The *solution* has to provide a full L2 service, including support for L2 flooding for data plane learning (or a functionally equivalent more scalable mechanism - see the draft nvo3 charter) and L2 multicast.  There are current applications in VMs that depend on L2 multicast (and that we cannot assume will be rewritten), although they're a distinct minority - load balancers and high availability cluster software are the most common sources of examples, although not all load balancers and not all HA clusters have that L2 dependency.

Which existing protocol(s) do you have in mind?  I've seen a proposed L3 BGP VPN approach that won't work (yet?) because it does not support L2 multicast or data plane learning.  Off line, Dino's pointed me to a LISP multicast draft that I need to go read in detail.  At the end of the day, if there are functionally sufficient L3 over L3 solutions, the attach/detach protocol (item 3 in draft nvo3 charter) should be designed to be general enough to support them.  OTOH, I read your email as attempting to exclude L2 over L3 from the outset, which is even worse than the allegation of no interest in L3 over L3, courtesy of how hypervisors work today - if that was not your intent, I apologize.

> [Qin]: No, but I am just curious to know the answer since I am not so much familiar what IEEE are

For IEEE, Service Provider Bridging (802.1aq) is in this space, although that tends to be carrier technology (SP = Service Provider) rather than data center technology (predominant usage in both cases).

Thanks,
--David

> Sent: Monday, March 26, 2012 1:08 AM
> To: Black, David; [email-masked]
> Cc: [email-masked]
> Running code - a lot of the deployed hypervisor networking support for virtual machines currently
> require L2 networking, and those are the systems that are hitting the VLAN tag size limits.  As I
> noted in my response to Dino, the L2 over L3 data planes are being developed **now**, so the window to
> engage on their control aspects is fairly short.  A debate on whether L3 over L3 is somehow better
> will be ignored by that running code ...
>

---

From: David Allan I <[email-masked]>
Date: Mon, 26 Mar 2012 03:58:27 -0400

HI David:

Why do you think 802.1aq applicability is specicially carrier. IMO a multi-tenant cloud and a carrier network is just about exactly the same problem space....

Dave 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Monday, March 26, 2012 12:38 AM
To: [email-masked]; [email-masked]
Cc: [email-masked]; [email-masked]
Subject: Re: [nvo3] Revised Draft Charter

> It seems charter claims to try to reuse existing mechanisms? So why 
> not reuse the existing Protocol IETF have already had?
> I agree VLAN tag size is going to running out limits, however I am 
> wondering what kind of special requirements you want to focus on?

> [Qin]: No, but I am just curious to know the answer since I am not so 
> much familiar what IEEE are doing in this aspect.

Thanks,
--David

> > [Qin]: Agree, I am wondering why do we need to emphasize on L2 over 
> > L3 approaches, would you like to clarify why L2 over L3 approach is 
> virtual machines currently require L2 networking, and those are the 
> systems that are hitting the VLAN tag size limits.  As I noted in my 
> response to Dino, the L2 over L3 data planes are being developed 
> **now**, so the window to engage on their control aspects is fairly short.  A debate on whether L3 over L3 is somehow better will be ignored by that running code ...
> 
> It seems charter claims to try to reuse existing mechanisms? So why 
> not reuse the existing Protocol IETF have already had?
> I agree VLAN tag size is going to running out limits, however I am 
> wondering what kind of special requirements you want to focus on?
> >Or this is just becos this piece of work only belongs to routing area 
> >or only IEEE approaches  are taken into accounted?  Any other work 
> >not belonging to routing area or non-IEEE approaches
> should
> [Qin]: No, but I am just curious to know the answer since I am not so 
> much familiar what IEEE are doing in this aspect.
> 
> Thanks,
> --David
> 
> > > NVO3 with VPNs (mea culpa - I didn't have the time to write one 
> > > for Paris, and it's probably going to require an author team that 
> > > includes familiarity with all of the interesting/relevant VPN technologies).
> > >
> > > The reason for mentioning L2 over L3 overlays is that there are 
> > > already at least 3 such data planes (NVGRE, STT, VXLAN) that are 
> > > being deployed without asking for the IETF's permission or 
> > > blessing.  The window to do something wrt these isn't going to stay open for very long ...
> >
> > Yes, but there are other proposals for L3 over L3 that have been 
> > around years before the ones you mention. I am not asking you to 
> > call them out. I am asking you to be more generic and keep an open 
> > mind to any solution that ia brought to the working group. The charter needs to convey that open mindedness in the text.
> >
> > [Qin]: Agree, I am wondering why do we need to emphasize on L2 over 
> > L3 approaches, would you like to clarify why L2 over L3 approach is 
> > analysis Or this is just becos this piece of work only belongs to 
> > routing area or only IEEE
> approaches
> > are taken into accounted? Any other work not belonging to routing 
> > area or non-IEEE approaches should not be the scope of this proposed charter?
> >
> > Dino
> >
> > >
> > > Thanks,
> > > --David
> > >
> > >>> Here is a revised draft charter for this effort. Comments 
> > >>> welcome on the list or during Wednesday's session.
> > >>>
> > >>> detaching from an overlay. When a VM is activated at a new 
> > >>> location, the hypervisor acting on behalf of the VM needs to 
> > >>> register the VM with the overlay, so that the overlay can ensure 
> > >>> that traffic to and from the VM is delivered properly. This WG 
> > >>> will develop appropriate mechanisms for registration and 
> > >>> deregistration. One candidate for providing this functionality 
> > >>> may be IEEE's VDP mechanism. Should a VDP-based approach be 
> > >>> pursued, any activity would be closely coordinated with the appropriate IEEE group.
> > >>
> > >> Work areas 1-2 and 4-5 do not talk about specific solutions but 
> > >> rather general approaches. Why
> does
> > >> this 3rd work area get specific about a solution?  I suggest 
> > >> striking the last 2 sentences from
> the
> > >> paragraph.
> > >>
> > >>> solution that supports an L2 over L3 overlay, an important goal 
> > >>> is to develop a "layer agnostic" framework and architecture 
> > >>> meeting data center requirements.  It is not the goal of the 
> > >>> working group to develop more general layer-x-over-layer-y overlay solutions.
> > >>
> > >> Why does a goal have a specific solution stated? Doesn't the 
> > >> working group want to solve the
> > problem
> > >> the best and most scalable way? If so, this early in the process 
> > >> one should not state in the
> > charter
> > >> that L2-over-L3 should be a solution. Otherwise, you should also 
> > >> state L3-over-L3, L3-over-L4,
> etc.

---

From: <[email-masked]>
Date: Mon, 26 Mar 2012 04:32:01 -0400

I wouldn't say "specifically carrier" as a fundamental limitation of applicability - as indicated by the first two letters in the SPB acronym, the predominant usage of 802.1aq is by carriers.  My experience is that data center operators have diverse views of carrier networking technology - there are some for which it makes sense to view the data center as an appendage of the WAN and plumb the WAN technology into the data center as close to the servers as possible.  There are others who believe that one of the functions of the CPE box is to keep carrier networking technology on the carrier side of that box, and mixes are possible (e.g., plumb carrier technology into core switches, but not access switches).

Thanks,
--David

> From: [email-masked] [mailto:[email-masked]] On Behalf Of David Allan I
> Sent: Monday, March 26, 2012 3:58 AM
> To: Black, David; [email-masked]; [email-masked]
> Cc: [email-masked]
> Why do you think 802.1aq applicability is specicially carrier. IMO a multi-tenant cloud and a carrier
> network is just about exactly the same problem space....
> 
> Dave
> 
> The networking service provided by hypervisors to VMs is an L2 service not L3 (VMs send L2 frames and
> expect L2 functionality).  This is part of the *problem* definition (VLAN tag isn't large enough, need
> something larger to functionally replace it), and the running code in hypervisors is L2 (e.g.,
> Ethernet softswitches), and I take that as a given (any proposal to use L3 over L3 that requires major
> hypervisor networking modifications, in contrast to just softswitch modifications, is not going to get
> the job done).
> 
> The *solution* has to provide a full L2 service, including support for L2 flooding for data plane
> learning (or a functionally equivalent more scalable mechanism - see the draft nvo3 charter) and L2
> multicast.  There are current applications in VMs that depend on L2 multicast (and that we cannot
> assume will be rewritten), although they're a distinct minority - load balancers and high availability
> cluster software are the most common sources of examples, although not all load balancers and not all
> HA clusters have that L2 dependency.
> 
> Which existing protocol(s) do you have in mind?  I've seen a proposed L3 BGP VPN approach that won't
> work (yet?) because it does not support L2 multicast or data plane learning.  Off line, Dino's pointed
> me to a LISP multicast draft that I need to go read in detail.  At the end of the day, if there are
> functionally sufficient L3 over L3 solutions, the attach/detach protocol (item 3 in draft nvo3
> charter) should be designed to be general enough to support them.  OTOH, I read your email as
> attempting to exclude L2 over L3 from the outset, which is even worse than the allegation of no
> interest in L3 over L3, courtesy of how hypervisors work today - if that was not your intent, I
> apologize.
> 
> For IEEE, Service Provider Bridging (802.1aq) is in this space, although that tends to be carrier
> technology (SP = Service Provider) rather than data center technology (predominant usage in both
> cases).
> 
> Thanks,
> --David
> 
> 
> > **now**, so the window to engage on their control aspects is fairly short.  A debate on whether L3
> over L3 is somehow better will be ignored by that running code ...
> >

---

From: <[email-masked]>
Date: Mon, 26 Mar 2012 04:39:06 -0400

SPB = Shortest Path Bridging, for some reason, I had it mapped to Service Provider Bridging, oops (mea culpa).  I'll leave it to others to comment on deployments ...

Thanks,
--David

> Sent: Monday, March 26, 2012 4:32 AM
> To: [email-masked]
> Cc: [email-masked]
> I wouldn't say "specifically carrier" as a fundamental limitation of applicability - as indicated by
> the first two letters in the SPB acronym, the predominant usage of 802.1aq is by carriers.  My
> experience is that data center operators have diverse views of carrier networking technology - there
> are some for which it makes sense to view the data center as an appendage of the WAN and plumb the WAN
> technology into the data center as close to the servers as possible.  There are others who believe
> that one of the functions of the CPE box is to keep carrier networking technology on the carrier side
> of that box, and mixes are possible (e.g., plumb carrier technology into core switches, but not access
> switches).
> 
> Thanks,
> --David
> 
>

---

From: David Allan I <[email-masked]>
Date: Mon, 26 Mar 2012 04:41:03 -0400

That saves me an email.... ;-)

Cheers
Dave 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Monday, March 26, 2012 1:39 AM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [nvo3] Revised Draft Charter

Thanks,
--David

> I wouldn't say "specifically carrier" as a fundamental limitation of 
> applicability - as indicated by the first two letters in the SPB 
> acronym, the predominant usage of 802.1aq is by carriers.  My 
> experience is that data center operators have diverse views of carrier 
> networking technology - there are some for which it makes sense to 
> view the data center as an appendage of the WAN and plumb the WAN 
> technology into the data center as close to the servers as possible.  
> There are others who believe that one of the functions of the CPE box is to keep carrier networking technology on the carrier side of that box, and mixes are possible (e.g., plumb carrier technology into core switches, but not access switches).
> 
> Thanks,
> --David
> 
> 
> > Why do you think 802.1aq applicability is specicially carrier. IMO a 
> > multi-tenant cloud and a
> carrier
> > > It seems charter claims to try to reuse existing mechanisms? So 
> > > why not reuse the existing Protocol IETF have already had?
> > The networking service provided by hypervisors to VMs is an L2 
> > service not L3 (VMs send L2 frames
> and
> > expect L2 functionality).  This is part of the *problem* definition 
> > (VLAN tag isn't large enough,
> need
> > something larger to functionally replace it), and the running code 
> > in hypervisors is L2 (e.g., Ethernet softswitches), and I take that 
> > as a given (any proposal to use L3 over L3 that requires
> major
> > hypervisor networking modifications, in contrast to just softswitch 
> > modifications, is not going to
> get
> > the job done).
> >
> > The *solution* has to provide a full L2 service, including support 
> > for L2 flooding for data plane learning (or a functionally 
> > equivalent more scalable mechanism - see the draft nvo3 charter) and 
> > L2 multicast.  There are current applications in VMs that depend on 
> > L2 multicast (and that we cannot assume will be rewritten), although 
> > they're a distinct minority - load balancers and high
> availability
> > cluster software are the most common sources of examples, although 
> > not all load balancers and not
> all
> > Which existing protocol(s) do you have in mind?  I've seen a 
> > proposed L3 BGP VPN approach that won't work (yet?) because it does 
> > not support L2 multicast or data plane learning.  Off line, Dino's
> pointed
> > me to a LISP multicast draft that I need to go read in detail.  At 
> > the end of the day, if there are functionally sufficient L3 over L3 
> > solutions, the attach/detach protocol (item 3 in draft nvo3
> > charter) should be designed to be general enough to support them.  
> > OTOH, I read your email as attempting to exclude L2 over L3 from the 
> > outset, which is even worse than the allegation of no interest in L3 
> > over L3, courtesy of how hypervisors work today - if that was not your intent, I apologize.
> >
> > > [Qin]: No, but I am just curious to know the answer since I am not 
> > > so much familiar what IEEE are doing in this aspect.
> >
> > For IEEE, Service Provider Bridging (802.1aq) is in this space, 
> > although that tends to be carrier technology (SP = Service Provider) 
> > rather than data center technology (predominant usage in both cases).
> >
> > Thanks,
> > --David
> >
> >
> > > > L3 approaches, would you like to clarify why L2 over L3 approach 
> > > > is better than L3 over L3 approach?  Have you done the 
> > > > comparison analysis
> > >
> > > Running code - a lot of the deployed hypervisor networking support 
> > > for virtual machines currently require L2 networking, and those 
> > > are the systems that are hitting the VLAN tag size limits.  As I 
> > > noted in my response to Dino, the L2 over L3 data planes are being 
> > > developed **now**, so the window to engage on their control 
> > > aspects is fairly short.  A debate on whether L3
> > > It seems charter claims to try to reuse existing mechanisms? So 
> > > why not reuse the existing Protocol IETF have already had?
> > > >Or this is just becos this piece of work only belongs to routing 
> > > >area or only IEEE approaches  are taken into accounted?  Any 
> > > >other work not belonging to routing area or non-IEEE approaches
> > > should
> > > [Qin]: No, but I am just curious to know the answer since I am not 
> > > so much familiar what IEEE are doing in this aspect.
> > >
> > > Thanks,
> > > --David
> > >
> > > > > NVO3 with VPNs (mea culpa - I didn't have the time to write 
> > > > > one for Paris, and it's probably going to require an author 
> > > > > team that includes familiarity with all of the interesting/relevant VPN technologies).
> > > > >
> > > > > The reason for mentioning L2 over L3 overlays is that there 
> > > > > are already at least 3 such data planes (NVGRE, STT, VXLAN) 
> > > > > that are being deployed without asking for the IETF's 
> > > > > permission or blessing.  The window to do something wrt these isn't going to stay open for very long ...
> > > >
> > > > call them out. I am asking you to be more generic and keep an 
> > > > open mind to any solution that ia brought to the working group. 
> > > > The charter needs to convey that open
> > mindedness in the text.
> > > >
> > > > L3 approaches, would you like to clarify why L2 over L3 approach 
> > > > is better than L3 over L3 approach? Have you done the comparison 
> > > > analysis Or this is just becos this piece of work only belongs 
> > > > to routing area or only IEEE
> > > approaches
> > > > are taken into accounted? Any other work not belonging to 
> > > > routing area or non-IEEE approaches should not be the scope of this proposed charter?
> > > >
> > > > Dino
> > > >
> > > > >
> > > > > Thanks,
> > > > > --David
> > > > >
> > > > >>> A third work area concerns VMs and end systems attaching to 
> > > > >>> and detaching from an overlay. When a VM is activated at a 
> > > > >>> new location, the hypervisor acting on behalf of the VM 
> > > > >>> needs to register the VM with the overlay, so that the 
> > > > >>> overlay can ensure that traffic to and from the VM is 
> > > > >>> delivered properly. This WG will develop appropriate 
> > > > >>> mechanisms for registration and deregistration. One 
> > > > >>> candidate for providing this functionality may be IEEE's VDP 
> > > > >>> mechanism. Should a VDP-based approach be pursued, any activity would be closely coordinated with the appropriate IEEE group.
> > > > >>
> > > > >> Work areas 1-2 and 4-5 do not talk about specific solutions 
> > > > >> but rather general approaches. Why
> > > does
> > > > >>> Finally, although the main objective of this WG is to 
> > > > >>> produce a solution that supports an L2 over L3 overlay, an 
> > > > >>> important goal is to develop a "layer agnostic" framework 
> > > > >>> and architecture meeting data center requirements.  It is 
> > > > >>> not the goal of the working group to develop more general layer-x-over-layer-y overlay solutions.
> > > > >>
> > > > >> the best and most scalable way? If so, this early in the 
> > > > >> process one should not state in the
> > > > charter
> > > > >> that L2-over-L3 should be a solution. Otherwise, you should 
> > > > >> also state L3-over-L3, L3-over-L4,
> > > etc.

---

From: Paul Unbehagen <[email-masked]>
Date: Mon, 26 Mar 2012 10:41:47 +0200

Hi David,

Actually the first two letters of SPB stand for Shortest Path not Service Provider.  

Also, we're presenting the deployments considerations draft (http://tools.ietf.org/html/draft-lapuh-spb-deployment-01) in isis wg and so far we've seen deployments of SPB in enterprise campus and data centers for both single and multi-tenant networks. 

--
Paul Unbehagen

Sent from my iPad

---

From: "Pat Thaler" <[email-masked]>
Date: Mon, 26 Mar 2012 09:00:29 +0000

SP in SPB actually stands for Shortest Path, not Service Provider. IEEE 802 uses Provider in the names Provider Bridging and Provider Backbone Bridging.

"Provider" doesn't have to equal "carrier". Provider is applicable to large data centers that host multiple tenant networks on the same physical network. 

However, I don't see what that has to do with L2 over L3 vs. L3 over L3. 

Regards, 
Pat

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Monday, March 26, 2012 1:32 AM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [nvo3] Revised Draft Charter

Thanks,
--David

---

From: Larry Kreeger <[email-masked]>
Date: Mon, 26 Mar 2012 11:10:25 -0700

On 3/24/12 1:19 PM, "Thomas Narten" <[email-masked]> wrote:

snip

> provided at the IP layer using an encapsulation header that resides

snip

> develop more general layer-x-over-layer-y overlay solutions.

snip 

Hi Thomas,

I have seen several comments related to the above two paragraphs I've pulled
from the latest draft charter.

IMO, both draft-narten-nvo3-overlay-problem-statement-01 and
draft-kreeger-nvo3-overlay-cp-00 talk generally about developing
protocols/solutions that are general/extensible enough to support any
address family (e.g. L2 or L3) over any other address family (L2 or L3), and
people are objecting to the seeming narrowing of the scope of the WG charter
to only L2 over L3.

I also share this concern.  While I agree that there are existing industry
solutions that specifically address L2 over L3, I think we would do a
disservice to essentially narrow the WG charter to only L2 over L3.  Maybe
if the charter specifically keeps the scope wider to Lx over Lx and simply
add a statement that the priority can be placed on L2 over L3 is would be a
good compromise.

Thanks, Larry

---

From: David Allan I <[email-masked]>
Date: Mon, 26 Mar 2012 05:18:08 -0400

Hi Pat:

Backtracking to Dave's original post to close the loop:

> > for virtual machines currently require L2 networking, and those are 
> > the systems that are hitting the VLAN tag size limits.  As I noted 
> > in my response to Dino, the L2 over L3 data planes are being 
> > developed **now**, so the window to engage on their control aspects 
> > is fairly short.  A debate on whether L3

I would agree on the L3 over L3 observation Dave made, and observe that 802.1aq SPBM offers a solution to the 4K VLAN limit as an L2 over L2 overlay. The 24 bit I-tag permitting in theory 16M VLANs....

I was objecting to the characterization and/or potential dismissing of SPBM as a carrier technology and therefore not applicable to datacenters in offering an "L2 over foo" model. I think it plays to the requirements rather nicely ;-)

Cheers
Dave

-----Original Message-----
From: Pat Thaler [mailto:[email-masked]] 
Sent: Monday, March 26, 2012 2:00 AM
To: [email-masked]; David Allan I
Cc: [email-masked]
Subject: RE: [nvo3] Revised Draft Charter

Regards,
Pat

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Monday, March 26, 2012 1:32 AM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [nvo3] Revised Draft Charter

Thanks,
--David

> multi-tenant cloud and a carrier network is just about exactly the same problem space....
> 
> Dave
> 
> The networking service provided by hypervisors to VMs is an L2 service 
> not L3 (VMs send L2 frames and expect L2 functionality).  This is part 
> of the *problem* definition (VLAN tag isn't large enough, need 
> something larger to functionally replace it), and the running code in 
> hypervisors is L2 (e.g., Ethernet softswitches), and I take that as a 
> given (any proposal to use L3 over L3 that requires major hypervisor networking modifications, in contrast to just softswitch modifications, is not going to get the job done).
> 
> The *solution* has to provide a full L2 service, including support for 
> L2 flooding for data plane learning (or a functionally equivalent more 
> scalable mechanism - see the draft nvo3 charter) and L2 multicast.  
> There are current applications in VMs that depend on L2 multicast (and 
> that we cannot assume will be rewritten), although they're a distinct 
> minority - load balancers and high availability cluster software are the most common sources of examples, although not all load balancers and not all HA clusters have that L2 dependency.
> 
> Which existing protocol(s) do you have in mind?  I've seen a proposed 
> L3 BGP VPN approach that won't work (yet?) because it does not support 
> L2 multicast or data plane learning.  Off line, Dino's pointed me to a 
> LISP multicast draft that I need to go read in detail.  At the end of 
> the day, if there are functionally sufficient L3 over L3 solutions, 
> the attach/detach protocol (item 3 in draft nvo3
> > for virtual machines currently require L2 networking, and those are 
> > the systems that are hitting the VLAN tag size limits.  As I noted 
> > in my response to Dino, the L2 over L3 data planes are being 
> > developed **now**, so the window to engage on their control aspects 
> > is fairly short.  A debate on whether L3
> > >area or only IEEE approaches  are taken into accounted?  Any other 
> > >work not belonging to routing area or non-IEEE approaches
> > should
> > > > for Paris, and it's probably going to require an author team 
> > > > that includes familiarity with all of the interesting/relevant VPN technologies).
> > > >
> > > >>> and detaching from an overlay. When a VM is activated at a new 
> > > >>> register the VM with the overlay, so that the overlay can 
> > > >>> ensure that traffic to and from the VM is delivered properly. 
> > > >>> This WG will develop appropriate mechanisms for registration 
> > > >>> and deregistration. One candidate for providing this 
> > > >>> functionality may be IEEE's VDP mechanism. Should a VDP-based 
> > > >>> approach be pursued, any activity would be closely coordinated with the appropriate IEEE group.
> > > >>
> > > >>> Finally, although the main objective of this WG is to produce 
> > > >>> a solution that supports an L2 over L3 overlay, an important 
> > > >>> goal is to develop a "layer agnostic" framework and 
> > > >>> architecture meeting data center requirements.  It is not the 
> > > >>> goal of the working group to develop more general layer-x-over-layer-y overlay solutions.
> > > >>

---

From: Qin Wu <[email-masked]>
Date: Mon, 26 Mar 2012 16:10:33 +0000

>The networking service provided by hypervisors to VMs is an L2 service not L3 (VMs send L2 frames and expect L2 functionality).  This is part of the *problem* definition (VLAN tag isn't large enough, need something larger to 
>functionally replace it), and the running code in hypervisors is L2 (e.g., Ethernet softswitches), and I take that as a given (any proposal to use L3 over L3 that requires major hypervisor networking modifications, in contrast to just
>softswitch modifications, is not going to get the job done).

>The *solution* has to provide a full L2 service, including support for L2 flooding for data plane learning (or a functionally equivalent more scalable mechanism - see the draft nvo3 charter) and L2 multicast.  There are current 
>applications in VMs that depend on L2 multicast (and that we cannot assume will be rewritten), although they're a distinct minority - load balancers and high availability cluster software are the most common sources of examples,
>although not all load balancers and not all HA clusters have that L2 dependency.

[Qin]: Thank for your clarification. what about the scalability of such solution?

>Which existing protocol(s) do you have in mind?  I've seen a proposed L3 BGP VPN approach that won't work (yet?) because it does not support L2 multicast or data plane learning.  Off line, Dino's pointed me to a LISP multicast
>draft that I need to go read in detail.  At the end of the day, if there are functionally sufficient L3 over L3 solutions, the attach/detach protocol (item 3 in draft nvo3 charter) should be designed to be general enough to support them. 

[Qin]: Yeah, LISP is one of candidate protocols in my mind.

>OTOH, I read your email as attempting to exclude L2 over L3 from the outset, which is even worse than the allegation of no interest in L3 over L3, courtesy of how hypervisors work today - if that was not your intent, I apologize.

[Qin]: Not my intent.

---

From: <[email-masked]>
Date: Mon, 26 Mar 2012 12:23:57 -0400

Qin,

> > >The *solution* has to provide a full L2 service ...

> [Qin]: Thank for your clarification. what about the scalability of such solution?

Doing something about the first of those challenges, scalability of data plane learning, is in the nvo3 draft charter.  That's a start.

> [Qin]: Yeah, LISP is one of candidate protocols in my mind.

Others?

Thanks,
--David

> Sent: Monday, March 26, 2012 12:11 PM
> To: David Allan I; Black, David; [email-masked]
> Cc: [email-masked]
> something larger to
> >functionally replace it), and the running code in hypervisors is L2 (e.g., Ethernet softswitches),
> and I take that as a given (any proposal to use L3 over L3 that requires major hypervisor networking
> modifications, in contrast to just
> multicast.  There are current
> >applications in VMs that depend on L2 multicast (and that we cannot assume will be rewritten),
> although they're a distinct minority - load balancers and high availability cluster software are the
> most common sources of examples,
> me to a LISP multicast
> >draft that I need to go read in detail.  At the end of the day, if there are functionally sufficient
> L3 over L3 solutions, the attach/detach protocol (item 3 in draft nvo3 charter) should be designed to
> be general enough to support them.
> 
> >OTOH, I read your email as attempting to exclude L2 over L3 from the outset, which is even worse than
> the allegation of no interest in L3 over L3, courtesy of how hypervisors work today - if that was not
> your intent, I apologize.
> 
> [Qin]: Not my intent.

---

From: Thomas Narten <[email-masked]>
Date: Tue, 27 Mar 2012 01:31:48 +0200

Hi Larry.

> add a statement that the priority can be placed on L2 over L3 is would be a

I agree completely. :-)

Personally, I think having this WG do L2overL3 and L3overL3 would be
desirable. (Both were in an earlier version of the charter, and I did
not mean to remove the latter as an indication we shouldn't do
L3overL3.)

I believe supporting L2 over L3 is essential. If we don't do L2 over
L3, our output will be largely irrelevant to data centers.

One can also make the observation that many DCs run a lot of IP, and
why not just support L3 over L3 as a sort of optimization of the L2
over L3 case. But given that you can (obviously) support IP via an L2
over L3 overlay, it's arguably not critical to include this up
front. So I personally could go either way.

What I do not think we should do is have this WG attempt to pursue
arbitrary Lx over Ly approaches. This could easily lead to boiling the
ocean and developing solutions in search of a problem.

For example, do we need another L2 over L2 overlay? I don't think
so. We already have SPB and I don't hear people saying we need
something else in this space. 

So the balancing act I think we need to do is:

1) have deliverables that match what is needed (IMO, L2oL3 is
essential and L3oL3 would be OK as well)

2) Make clear that we intend the NVO3 framework to be general and
allow it to support other Lx over Ly combinations, but not actually
have it be in-scope for this WG to do such work. If someone wants to
propose such work, they need to first make the case that such an
approach needs a solution.

Thomas

---

From: Thomas Narten <[email-masked]>
Date: Tue, 27 Mar 2012 02:09:02 +0200

Dino,

> rather general approaches. Why does this 3rd work area get specific
> about a solution?  I suggest striking the last 2 sentences from the
> paragraph.

I put in those two sentences specifically to alert readers (and the
IESG) that one of the potential solutions involves an IEEE
protocol. If the WG goes that route, we need to coordinate with IEEE
properly.

Thus, the proposed wording is just a heads up about this.

If I take those sentences out, I suspect somebody will ask "hey, don't
we need to coordinate with IEEE, VDP could be used and that's their
protocol..." and my response would be "OK, I'll add a couple of
sentences". :-)

> Why does a goal have a specific solution stated? Doesn't the working
> group want to solve the problem the best and most scalable way? If
> so, this early in the process one should not state in the charter
> state L3-over-L3, L3-over-L4, etc. And you probably don't want to
> enumerate all combinations.

Now that I've read some of the related messages, I notice that the
charter itself never says the main focus of the WG would be the L2
over L3 case. That should probably be stated up front as it is an
assumption. I can see how the above reads a little odd, if it's the
first time its implied that the L2 over L3 case is a key one.

Thomas

---

From: Thomas Narten <[email-masked]>
Date: Tue, 27 Mar 2012 02:15:21 +0200

Hi  Yingjie.

> Does the sequence of the work areas listed in the charter also show
>  the sequence of the work that will be done after the wg is set up?

Not exactly. 

>  If yes, I would say the 3rd item, i.e. the attach/dettach mechanism
>  should be defined before a control plane is defined, because the
>  definition of thr former may influent the latter.

I would expect they'd be worked on in parallel. I would expect that
neither document/mechanism could be finalized until we were sure the
other one was stable or didn't have any interdependencies. (This would
be a general principle that would apply to any outputs of the WG.)

Thomas

---

From: Thomas Narten <[email-masked]>
Date: Tue, 27 Mar 2012 02:23:44 +0200

Ken,

> derivable from the others)?

I'm not entirely sure what you are suggesting here, but I think the
answer is no. In order to tunnel a packet over an overlay, you have to
be able to map the "inner" address of the tunneled packet into an
"outer" address that is the egress of the tunnel.

I don't see how we can assume one of those addresses can be
(algorithmically?) derived from the other (if that is what you are
asking).

> mapping (probably by extension) rather than inventing a new one?

Yes.

So reading the wording now, I don't think it is as clear as it could
have been. The intention of "find solutions for" was not to say we
already knew we had to develop new solutions. Rather it was intended
to to mean we had to "find solutions to the requirements". That
includes looking at existing protocols.

Thomas

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Tue, 27 Mar 2012 09:51:16 +0200

draft-lasserre-nvo3-framework is generic enough so as to allow for various combinations of Lx over Ly.

Clearly, some combinations (e.g. Ethernet over IP/GRE) are desired, and hence have been described in various drafts.

As long as other options can fit within the framework and satisfy the corresponding requirements, it is likely that other solutions might be proposed as dictated by market needs.

Marc

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Narten
Sent: Tuesday, March 27, 2012 1:32 AM
To: Larry Kreeger
Cc: [email-masked]
Subject: [nvo3] Lx over Ly [was Re: Revised Draft Charter]

Hi Larry.

I agree completely. :-)

Thomas

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 27 Mar 2012 01:39:26 -0700

> protocol..." and my response would be "OK, I'll add a couple of

Just remove "VDP" and indicate that IEEE coordination is necessary and you could resolve my comment.

> first time its implied that the L2 over L3 case is a key one.

Sounds good. Thanks for the reply.

Dino

---

From: Kireeti Kompella <[email-masked]>
Date: Wed, 28 Mar 2012 02:07:57 -0700

>> clarify why L2 over L3 approach is better than L3 over L3 approach?  Have you done the comparison

Perhaps the question to answer is, what aspects of "Layer 2" do we really need?  Do we need to reply truthfully to ARP requests?  Do we need to forward on MAC addresses if all traffic is IP?  Is it enough to emulate subnet broadcasts and link-local multicasts?

One way of looking at this is that we've done too good a job of emulating LANs with VPLS; we're scaling that back a bit with TRILL and E-VPN.  In the next iteration, can we scale that way back?  How far can we go without breaking things?

> Running code - a lot of the deployed hypervisor networking support for virtual machines currently require L2 networking, and those are the systems that are hitting the VLAN tag size limits. 

The running code I worry about is existing applications, guest OSs and such that expect to run on a LAN. That's not to minimize work done in hypervisor networking support, but the relative volume (and malleability) of code is orders of magnitude apart.  That might be what ends up keeping us in the Dark Age of L2 :-(  But not starting this conversation will certainly keep us there.

Kireeti