Subject: Re: [nvo3] inter-CUG traffic [was Re: call for adoption: draft-narten-nvo3-overlay-problem-statement-02]

From: Robert Raszuk <[email-masked]>
Date: Mon, 02 Jul 2012 16:52:19 +0200

Hi Joel,

> So I was urging that we not mandate optimal inter-subnet routing as part
> of the NVO3 requirements.

If my customers who happen to try my standards based offering to run 
their 3-tier applications will perform their application level 
measurements (and trust me - all of them do it these days) and find that 
my competition offers more optimal data plane results via non standard 
solution - I will likely loose those customers.

While in L2 VNs going via some randomly placed gateway may be ok in L3 
services I am afraid the bar is already much higher today. That actually 
is one good reason to decouple NVO3 requirements for L2 and L3 VN 
services and address them separately.

Thx,
R.

---

From: Thomas Narten <[email-masked]>
Date: Mon, 02 Jul 2012 11:05:12 -0400

Pedro Roque Marques <[email-masked]> writes:

> On Jun 29, 2012, at 2:52 PM, Thomas Narten wrote:

> > Pedro Roque Marques <[email-masked]> writes:
> > 
> >> I object to the document on the following points:
> > 
> >> 1) It doesn't appear to recognize that there are multiple type of 
> >> data-center deployments with different requirements.
> > 
> > I'd like to hear more from others on this point.
> > 
> > Please provide some examples of differences in DC deployments and
> > *how* they impact NVO3, either from a problem statement perspective or
> > from a framework perspective.

> 1) Whether L2 headers are preserved (including for instance VLAN
>  tags) or L2 headers are rewritten an only IP packets are delivered.

The goal all along with L2/ethernet service emulation has been to
provide a simple ethernet service. It has never been a goal (as far as
I can tell) to preserve, e.g., VLAN tags, or other specialized L2
features. The intention has been to carry L2 frames that have a
src/dest/Ethertype and that's about it.

One of the tasks for this WG is to flesh out the above in more precise
detail (I know this came up during chartering and possibly the BOF as
well).

But, IMO we should keep it simple, covering common case
deployments. 

In the case where a workload is using multiple VLANs, the logical
starting point would be to map each VLAN into a seperate VN.

> 2) Whether the problem statement is about building a "bigger vlan"
>  or about routing between IP "subnets" where there is an ACL that
>  controls traffic exchanges between subnets.

What is meant by "bigger VLAN"? More nodes attached to an indvidual
VLAN? I'm not sure this is a goal per se. I.e., how many individual
nodes can you put on a single VLAN today? Is that considered a
limitation?

One thing we do need to do is support many many more VLANs than the
current 4096 limit.

> 3) Whether the underlying network can be assumed to support
>  broadcast/multicast services.

It cannot. At least, this is what everyone I've asked has said. I.e.,
use it if available and makes sense, but one cannot assume it's
present. There are numerous DCs where multicast is not supported.

> 4) Whether it is assumed that the hypervisor switch can or not be
> modified.

It can. The hypervisor will need to support NVE functionality.

> 5) Whether "unknown unicast" frames are broadcasted or communication
>  requires the end-points to be known a-priori.

I think both models can be supported. For smaller DCs (and I'm not
defining smaller precisely for obvious reasons), flooding unknown
would appear to be a viable approach. I.e, that is the approach VXLAN
seems to be taking.

But my sense in talking with various folk is that there is a strong
sense that flooding per above is not sufficient for larger DCs and
that an alternate model needs to be pursued. One where "unknown
unicast" frames trigger a request to an oracle to provide the
necessary mapping.

> > I agree that support for multicast in the underlay cannot be
> > assumed. Pretty much everyone I've asked on this point says the same
> > thing.

> My understanding is that there is a significant number of proposals
> in these space that are based on data-plane learning. These tend to
> assume an underlying multicast service.  Are you suggesting that
> these need to also support ingress replication ? How is the
> membership discovered ? How does ingress replication "fit" with
> scaling requirements ?

Yes, ingres replication (or equivalent) will IMO need to be
supported. How else will you deliver broadcast/multicast if the
underlay doesn't provide such functionality?

As mentioned above, I think there are two different target
environments we should support. Smaller and larger DCs.

> The problem space that i'm personally interested in is not one where
>  the mac-in-mac style solutions is applicable. There are however DC
>  designs where this type of solutions is valid and i suspect that
>  the proponents of these style of solutions will disagree with your
>  statement.

Available support for multicast/broadcast in the DC is NOT just a
technology statement. It's also about whether the DC operator is
willing to enable the functionality.  There are plenty of DCs where
multicast is not enabled on the underlay (beyond link-local), even if
the deployed hardware is capable of supporting it.

> > So I think one of the requirements will need to be emulation of
> > multicast (at the VN level) across underlays that do not support (or
> > want to enable) multicast. But that is a requirements discussion, not
> > a problem statement issue.

> I'm not sure i agree. If you define the problem as supporting
> reasonable small scale DCs that follow a rather conventional
> architecture, which some participants in this WG do (reasonably so)
> then emulation of multicast is no longer an issue.

Not sure what you are saying here. If you are saying some DCs have
multicast enabled on the underlay, I agree. But it's also the case
that some do not and we need an approach that works in such
deployments as well. So we can't just say "the underlay is assumed to
support multicast".

> > I believe it is perfectly reasonably for an overlay to provide two
> > almost identical services. L2 services (which preserves ethernet
> > frames at an e2e level) and L3 service, for those cases where a
> > deployment is happy with IP only emulation.

> These are very different problems. An L2 service is expected to
> solve complex issues relating with interaction with IEEE bridging
> semantics.  Solutions such as TRILL, etc spend a very considerable
> effort in dealing with this interaction.  An L3 service on the other
> hand is mostly about dealing with traffic that crosses subnets.

As said above, it is absolutely not a goal to perfectly emulate all L2
features. TRILL is operating in a very different design space than NVO3.

> My understanding is that DC operators that are focused on an L3
>  service have no need for an L2 service and its associated
>  complexity. Saddling a possible solution with the need to provide
>  an L2 service is imho highly counter-productive.

I keep hearing that *both* are needed. L3 when you can. L2 as a fall
back when for some reason L3 only emulation isn't good enough.

> > While there clearly are some differences in those two service models,
> > I think both can be accomodated by the same NVO3 framework (and
> > problem statement).

> I completely disagree. You can write a problem statement that
>  contains both but it will impose unnecessary complexity in the
>  solutions.

The frameworks and core operations for both are 90% the same. The
differences will be in packet format, address formats, etc. I.e.,
minor details, not in the concepts.

And a solution doesn't have to do both. If you think L3 only is good
enough for your product, then implement only that. Likewise for L2
only.

On the other hand, doing both in a completely independent manner,
without using a common framework seems like a missed opportunity or
worse.

> > Earlier versions of the charter had explicit text about this, that
> > was later removed. But I see no reason why it should not continue to
> > be a goal of this WG, even though not explicitly called out in the
> > current charter.

> Is it possible that it was removed because it is actually
> counter-productive to attempt to build both an L2 overlay and an L3
> overlay with the same solution ?

No. It was removed for other reasons, mostly having to do with keeping
the charter focussed and simple.

Thomas

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 02 Jul 2012 12:27:20 -0400

There are clear data center architectures, and data center deployments, 
where optimal inter-subnet routing is important.  Equally clearly, there 
are cases where bouncing everything off the gateway is sufficient.  And 
cases in between.

It seems to me that the framework and problem statement efforts should 
not mandate that one particular point on that spectrum is a MUST for all 
NVO3 solutions.

Yours,
Joel

On 7/2/2012 10:52 AM, Robert Raszuk wrote:
> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3
>

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 02 Jul 2012 18:45:15 +0200

To me it seems rather clear that going for _single_ NVO3 solution to 
accommodate partial pieces of various (apparently very different) 
requirements to only enforce that we all must be on the same single bus 
is rather a poor choice.

Best,
R.

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Mon, 2 Jul 2012 16:59:19 +0000

Very much agree with Joel and Rob.
A few of us had the same concern for a long time, diff. type of DC focus drives diff. problems statement, diff. req.
It does not scale, nor efficient to progress when trying to pack all very diff. approached into the same doc, be it PB statement, framework, requirement...
We discussed this concerns off-line with some folks before and after the Taiwan meeting where nvo3 and vpn4dc were first discussed.

Luyuan

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Robert Raszuk
> Sent: Monday, July 02, 2012 12:45 PM
> To: Joel M. Halpern
> Cc: Pedro Roque Marques; [email-masked]
> Subject: Re: [nvo3] inter-CUG traffic [was Re: call for adoption:
> draft-narten-nvo3-overlay-problem-statement-02]
> 
>

---

From: Linda Dunbar <[email-masked]>
Date: Mon, 2 Jul 2012 18:56:59 +0000

Maria, LuYuan, and Dennis

Your draft (http://www.ietf.org/id/draft-fang-vpn4dc-problem-statement-01.txt ) demonstrates the approaches on how to extend L3VPN to end systems in data center.  "draft-narten-nvo3-problem-statement" describes the issues facing data centers with massive number of VMs interconnected by virtualized switches.  

Majority of text in your draft is on how L3VPN can be extended to Data Center. It would be good solution draft, because some tenants do want to extend their VPNs (L2 or L3) into data centers. 

If you want to merge your draft to general NVo3 problem statement, a lot of sections in your draft need to be removed. For example, Section 7 on how MAC addresses should not be used, etc. MAC addresses don't have to be used in the solution described in your draft. But not all data centers will use L3VPN. Many data centers don't have any end systems which are part of any L3VPN; Many data centers only allow IPSec tunnels terminated at their gateways; Some multi-tenant data centers have to use L2 among VMs due to their tenant's requirement.  

Since NVo3 current charter is only on studying problems and identify requirement, it might be worthwhile to revise your draft focusing only on the problems facing data centers when some tenants need to extend their L3VPN to VMs within data centers. 

Linda Dunbar

> Yakov Rekhter
> Sent: Thursday, June 21, 2012 2:44 AM
> To: Benson Schliesser
> Cc: Matthew (Matthew) Bocci; Luyuan Fang (lufang); [email-masked];
> Dennis Cai (dcai); Pedro Marques; draft-narten-nvo3-overlay-problem-
> [email-masked]; [email-masked]
> Subject: Re: [nvo3] call for adoption: draft-narten-nvo3-overlay-
> problem-statement-02
> 
> Benson,
> 
> > Dear NVO3 Participants -
> >
> > This message begins a two week Call for Adoption of
> > http://tools.ietf.org/html/draft-narten-nvo3-overlay-problem-
> statement-02
> > by the NVO3 working group, ending on 30-June-2012.
> >
> > Please respond to the NVO3 mailing list with any statements of
> approval
> > or disapproval, along with any additional comments that might explain
> > your position. Also, if any NVO3 participant is aware of IPR
> associated
> > with this draft, please inform the mailing list and/or the NVO3
> chairs.
> 
> Before considering adoption of draft-narten-nvo3-overlay-problem-
> statement,
> we should consider how this draft is related to
> draft-fang-vpn4dc-problem-statement (see below) ? Do these two drafts
> cover the same topic ? Are they complementary with each other ?
> Should both of these drafts be merged into a single draft, before
> being adopted by the NVO3 working group ?
> 
> Yakov.
> 
> -----------------------------------------------------------------------
> Date:    Wed, 20 Jun 2012 10:50:53 CDT
> To:      <[email-masked]>
> cc:      Pedro Marques <[email-masked]>, [email-masked],
> 	 "Dennis Cai \(dcai\)" <[email-masked]>
> From:    "Luyuan Fang (lufang)" <[email-masked]>
> Subject: [nvo3] FW: New Version Notification fordraft-fang-vpn4dc-
> problem-sta
>      ***tement-01.txt
> 
> Dear colleagues,
> 
> We have uploaded a new updated version of draft-fang-vpn4dc-problem-
> statement
> -01.txt (on 6/12), by Maria Napierala, Dennis Cai, and myself.
> Appreciate your review and comments.
> 
> Thanks,
> Luyuan
> 
> From: [email-masked] [mailto:[email-masked]]
> Sent: Tuesday, June 12, 2012 10:48 PM
> To: Luyuan Fang (lufang)
> Cc: [email-masked]; Dennis Cai (dcai)
> Subject: New Version Notification fordraft-fang-vpn4dc-problem-
> statement-01.t
> xt
> 
> 
> A new version of I-D, draft-fang-vpn4dc-problem-statement-01.txt
> has been successfully submitted by Luyuan Fang and posted to the
> IETF repository.
> 
> Filename:	 draft-fang-vpn4dc-problem-statement
> Revision:	 01
> Title:		 IP-VPN Data Center Problem Statement and
> Requirements
> Creation date:	 2012-06-12
> WG ID:		 Individual Submission
> Number of pages: 18
> URL:             http://www.ietf.org/internet-drafts/draft-fang-vpn4dc-
> proble
> m-statement-01.txt
> Status:          http://datatracker.ietf.org/doc/draft-fang-vpn4dc-
> problem-st
> atement
> Htmlized:        http://tools.ietf.org/html/submission.filename }}-01
> Diff:            http://tools.ietf.org/rfcdiff?url2=draft-fang-vpn4dc-
> problem
> -statement-01
> 
> Abstract:
>    Network Service Providers commonly use BGP/MPLS VPNs [RFC 4364] as
>    the control plane for virtual networks. This technology has proven
>    to scale to a large number of VPNs and attachment points, and it is
>    well suited for Data Center connectivity, especially when
>    supporting all IP applications.
> 
>    The Data Center environment presents new challenges and imposes
>    additional requirements to IP VPN technologies, including multi-
>    tenancy support, high scalability, VM mobility, security, and
>    orchestration. This document describes the problems and defines the
>    new requirements.
> 
> 
> 
> 
> 
> 
> The IETF Secretariat

---

From: Melinda Shore <[email-masked]>
Date: Mon, 02 Jul 2012 11:21:46 -0800

On 7/2/12 10:56 AM, Linda Dunbar wrote:
> But not all data centers will use L3VPN. Many data centers don't have
> any end systems which are part of any L3VPN; Many data centers only
> allow IPSec tunnels terminated at their gateways; Some multi-tenant
> data centers have to use L2 among VMs due to their tenant's
> requirement.

I keep drafting email and then throwing it out rather than sending it,
because I think this may be the earliest I've seen a working group go
off the rails, and I have to say I've been quite confused by some of
the discussion around scoping.  Part of it is, I think, tied to a lack
of clarity in IP networks about which layer is which (and that's fine -
layering for its own sake doesn't buy you much), but at any rate I
don't think the work at hand is that NVO3 is looking at every single
network virtualization technology that could possibly be used in a data
center.

The charter says this: "NVO3 will consider approaches to
multi-tenancy that reside at the network layer rather than using
traditional isolation mechanisms that rely on the underlying layer 2
technology (e.g., VLANs)" which suggests to me that while layer 2
technologies may (and, in fact, are) used in data centers that doesn't
mean that solving problems around layer 2 technologies is something
that this group has been chartered to work on.

I like that this draft has a specific focus on overlay networks.  It's
clear, unambiguous, and doesn't suffer from scope creep.  I would very
much like to see it adopted by the working group.

Melinda

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Mon, 2 Jul 2012 19:22:08 +0000

Linda,

Thanks for the comments.

> 
> Your draft (http://www.ietf.org/id/draft-fang-vpn4dc-problem-statement-
> 01.txt ) demonstrates the approaches on how to extend L3VPN to end
> systems in data center.  "draft-narten-nvo3-problem-statement"
> describes the issues facing data centers with massive number of VMs
> interconnected by virtualized switches.
> 
> Majority of text in your draft is on how L3VPN can be extended to Data
> Center. It would be good solution draft, because some tenants do want
> to extend their VPNs (L2 or L3) into data centers.

Our draft is not about "extending" tenant's L3VPNs to data centers but using L3VPNs as a data center virtualized solution. Is a combined problem statement/requirements draft for a data center virtualization using L3VPNs, including large scale data centers with massive number of VMs. It is not a solution draft. The solution draft is draft-marques-l3vpn-end-system, for example. 

> Since NVo3 current charter is only on studying problems and identify
> requirement, it might be worthwhile to revise your draft focusing only
> on the problems facing data centers when some tenants need to extend
> their L3VPN to VMs within data centers.

Again, this is not what our draft is about, namely it is not about tenants wanting to "extend" their L3VPN to SP or cloud provider data centers. 

As Luyuan explained before, the draft has started a problem statement for layer 3 VPNs as a data center virtualization solution, and then we extended it with the requirements for such a solution. 

Thanks!
Maria

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Mon, 2 Jul 2012 22:21:08 +0000

> > > from a framework perspective.

<snip>
 
> present. There are numerous DCs where multicast is not supported.

Does this mean that a solution such as proposed in http://tools.ietf.org/html/draft-mahalingam-dutt-dcops-vxlan-01 is out of scope of nvo3?

Maria

---

From: Thomas Narten <[email-masked]>
Date: Mon, 02 Jul 2012 18:33:06 -0400

> Does this mean that a solution such as proposed in
>  http://tools.ietf.org/html/draft-mahalingam-dutt-dcops-vxlan-01 is
>  out of scope of nvo3?

That's a pretty broadly worded question... :-)

That document does assume that IP multicast is enabled in the
underlay, and to send traffic to all members of a given VNI, you would
would send to a corresponding IP multicast address on the underlay.

Obviously, that part can't be counted on if the underlay doesn't
support multicast.

So yes, that solution (IMO) cannot be assumed to be sufficient for
NVO3.

Thomas

---

From: Thomas Narten <[email-masked]>
Date: Tue, 03 Jul 2012 09:44:30 -0400

"Stiliadis, Dimitrios (Dimitri)" <[email-masked]> writes:

> I believe that Section 3.1 needs to be significantly revised since it
> has some confusing statements. I cannot support this draft without a
> revision of this Section:

I'll assume that since you didn't comment about the first two
paragraphs (L2 VLAN stuff), you are OK with that text.

>     VRF's are a pure routing construct and do not have end-to-end
>     significance in the sense that the data plane carries a VRF
>     indicator on an end-to-end basis.

You're right. When I wrote this text, I didn't realize that MPLS had
adopted the VRF terminology. What the text refers to is what others
have pointed out is known as VRF lite.

Now I understand why this text produced such a backlash. :-)

Proposed new text:

         <t>
 	  In the case of IP networks, many routers provide a Virtual
          Routing and Forwarding (VRF) service commonly known as "VRF
          Lite".  The same router operates multiple instances of
          forwarding tables, one for each tenant. Each forwarding
          table instance is populated separately via routing
          protocols, and adjacent routers encapsulate traffic in a way
          that identifies the tenant (e.g., using a VLAN tag). Each
          VRF Lite instance provides address and traffic isolation.
          The VRF Lite instance for each frame is selected based on
          that tenant identification.
        </t>

        <t>
          VRF's are a pure routing construct and in VRF Lite do not
          have end-to-end significance in the sense that the data
          plane carries a VRF Lite instance selector on an end-to-end
          basis. Instead, the VRF Lite instance to be used is
          determined at each hop using a combination of incoming
          interface and some information in the frame (e.g., local
          VLAN tag).  Furthermore, the VRF Lite model has typically
          assumed that a separate control plane (e.g., based on a
          routing protocol) governs the population of each forwarding
          table. Thus, the VRF Lite model assumes multiple, logically
          independent control plane instances and has no specific tag
          within a data frame to identify the VRF Lite instance for
          that frame.
        </t>
	
	<t>
	  MPLS VRFs <xref target="RFC4364"></xref> place the VRF
	  functionality on the CE and PE devices, using MPLS
	  encapsulation to preserve tenant separation between the CE
	  and PE devices. Control plane protocols (e.g.,
	  LDP<xref target="RFC5036"></xref>) are used to set up the
	  data path between PE and CE.
	</t>

Does that work?

Thomas

---

From: <[email-masked]>
Date: Tue, 3 Jul 2012 14:23:39 +0000

Hi Thomas,

Thomas Narten :
[..snip...]
> MPLS VRFs [RFC4364] place the  VRF functionality on the CE and PE
 > devices, using MPLS encapsulation to preserve tenant separation
 > between the CE and PE devices. Control plane protocols (e.g., LDP
 > [RFC5036]) are used to set up the data path between PE and CE.
 >
 >Does that work?

The above paragraph has a lot of issues:
- "MPLS VPNs" designate the service, "VRF" is the VPN-specific routing table
- the VRFs are not on CE devices, not on PEs
- there is no MPLS encapsulation between PE and CE, MPLS encapsulation 
is used between PE routers
- LDP is not run between CE and PEs (is running between PE-P, P-P, but 
not always, e.g. not when MPLS/GRE is used)
- a non-VPN IP routing protocol may be run between CEs and PEs (such as 
BGP or OSPF), static routing is another possibility
(in the case where a carrier's-supporting-carrier approach is used, 
things differ from the above, but this is not the typical deployment)

Thanks,

-Thomas M.

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
France Telecom - Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, France Telecom - Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: "Stiliadis, Dimitrios (Dimitri)" <[email-masked]>
Date: Tue, 3 Jul 2012 10:46:27 -0500

I think the VRF lite text is ok. But, I agree with Thomas M. comments below
about the MPLS text.

> things differ from the above, but this is not the typical deployment)

In addition, the text does not make clear that unlike VRF lite, BGP MPLS
VPNs do not rely on multiple instances of routing protocols, but instead
utilize a single control plane to advertise routes for all tenants while
at the same time enabling overlapping IP addresses and tenant separation.

>From the data path perspective, the most significant difference between
the NVGRE/VXLAN type proposals and MPLSoverGRE is that the tenant identifier
is global in the former and local in the latter. (ie in MPLSoGRE there
is a control plane (BGP) for distributing labels (ie tenant separators) between
service hops, while in the other proposals it is distributed globally and in the 
management plane). 

Dimitri

> Thanks,
> 
> -Thomas M.
> 
> ______________________________________________________________________________
> ___________________________________________
> 
> Ce message et ses pieces jointes peuvent contenir des informations
> confidentielles ou privilegiees et ne doivent donc
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
> electroniques etant susceptibles d'alteration,
> This message and its attachments may contain confidential or privileged
> information that may be protected by law;
> As emails may be altered, France Telecom - Orange is not liable for messages
> that have been modified, changed or falsified.
> Thank you.
>

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Tue, 3 Jul 2012 16:57:43 +0000

Linda,

See-line.

> From: NAPIERALA, MARIA H [mailto:[email-masked]]
> Sent: Monday, July 02, 2012 3:22 PM
> To: Linda Dunbar; Yakov Rekhter; Benson Schliesser (bschlies)
> [email-masked]
> Subject: RE: [nvo3] call for adoption: draft-narten-nvo3-overlay-
> problem-statement-02
> 
> Linda,
> 
> Thanks for the comments.
> 
> >

[lf] Good point to bring up - "massive number of VMs...", scaling is one of our key motivation to focus on l3 for its proven scalability.

> > Center. It would be good solution draft, 

[lf] No. Can people implement equipments with this draft and have inter-operability? if not, then it is not solution draft.
It is pb statement and req. as titled. 

> > because some tenants do want
> Our draft is not about "extending" tenant's L3VPNs to data centers but
> using L3VPNs as a data center virtualized solution. Is a combined
> problem statement/requirements draft for a data center virtualization
> using L3VPNs, including large scale data centers with massive number of
> VMs. It is not a solution draft. The solution draft is draft-marques-
> l3vpn-end-system, for example.
> 
[lf] We are addressing both intra-DC and inter-DC, it is about DC virtualization as Maria pointed out.

> > If you want to merge your draft to general NVo3 problem statement, a
> > lot of sections in your draft need to be removed. For example,
> Section
> > 7 on how MAC addresses should not be used, etc. MAC addresses don't
> > have to be used in the solution described in your draft. 

[lf] The authors have not discussed merge yet. We are open to discussion. But as discussed by Joel and Rob too, how practical to express very different problems, requirements, and framework in one doc is a good question. 

> > But not all
> > data centers will use L3VPN. 
[lf] Correct, but it is true for any technologies regardless dc or not.

> > Many data centers don't have any end
> > systems which are part of any L3VPN; 
[lf] that is why the pb statement and requirement effort. 

Many data centers only allow
> IPSec
> > tunnels terminated at their gateways; Some multi-tenant data centers
> > have to use L2 among VMs due to their tenant's requirement.

[lf] We don't go against l2, it is needed and there are plenty people working on it. But nothing prevent developing l3. A few modern large scale DCs already deployed all l3 solutions with their own implementation. Providers are anxious to get standardized l3 solutions. So we start with pb and req.

> > requirement, 
[lf] that is exactly what we have. Could you point out where in the charter indicate we are out if you are still not convinced?

> > it might be worthwhile to revise your draft focusing
> > only
> Again, this is not what our draft is about, namely it is not about
> tenants wanting to "extend" their L3VPN to SP or cloud provider data
> centers.
> 
> As Luyuan explained before, the draft has started a problem statement
> for layer 3 VPNs as a data center virtualization solution, and then we
> extended it with the requirements for such a solution.
> 
> 
> Thanks!
> Maria
> 
Thanks,
Luyuan

> > > From: [email-masked] [mailto:[email-masked]] On
> Behalf
> > Of
> > > Yakov Rekhter
> > > -------------------------------------------------------------------
> --
> > --

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Tue, 3 Jul 2012 18:32:10 +0000

Thomas,

> So yes, that solution (IMO) cannot be assumed to be sufficient for

Thanks for clarifying. The section 4.7 would have to be updated accordingly.

I think it is important to state the assumptions about the properties (e.g., multicast support or, better, its lack of) of the underlying network in the problem statement draft.

Maria

---

From: <[email-masked]>
Date: Tue, 3 Jul 2012 14:47:40 -0400

Luyuan

> > tenants wanting to "extend" their L3VPN to SP or cloud provider data

While that may have been the intention, Section 12 ("IP-VPN Data Center
Use Case: Virtualization of Mobile Network") is not about data centers;
it's about mobile networks.

Can I suggest replacing that section with an actual Data Center use case
that doesn't extend beyond the data center?

> > problem statement

Let's be precise - a specific solution has been chosen, namely BGP/MPLS
L3VPNs, as indicated in the first sentence of the abstract as well as the
entirety of Section 10 and its subsections.

I agree that this could be a problem statement and requirements draft, but
it needs an accurate title, e.g.:

	BGP/MPLS VPN Data Center Problem Statement and Requirements

IMHO, IP-VPN is too broad, as the draft is considerably narrower than would
be required to cover all IP VPNs (e.g., IPsec VPNs as Linda mentioned).

I'm not suggesting that IPsec VPNs are a good nvo3 solution for data centers
(they may play a complementary role in some cases), rather I'm using IPsec
VPNs as an example to ask that the draft have a title that reflects its
actual content.

Thanks,
--David

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Luyuan
> Fang (lufang)
> Sent: Tuesday, July 03, 2012 12:58 PM
> To: NAPIERALA, MARIA H; Linda Dunbar; Yakov Rekhter; Benson Schliesser
> (bschlies)
> Cc: Pedro Marques; Matthew (Matthew) Bocci; draft-narten-nvo3-overlay-problem-
> [email-masked]; [email-masked]; Dennis Cai (dcai)
> Subject: Re: [nvo3] call for adoption: draft-narten-nvo3-overlay-problem-
> statement-02
> 
> Linda,
> 
> See-line.
> 
> [lf] Good point to bring up - "massive number of VMs...", scaling is one of
> our key motivation to focus on l3 for its proven scalability.
> 
> > >
> [lf] No. Can people implement equipments with this draft and have inter-
> operability? if not, then it is not solution draft.
> [lf] The authors have not discussed merge yet. We are open to discussion. But
> as discussed by Joel and Rob too, how practical to express very different
> problems, requirements, and framework in one doc is a good question.
> 
> > > But not all
> [lf] We don't go against l2, it is needed and there are plenty people working
> on it. But nothing prevent developing l3. A few modern large scale DCs already
> deployed all l3 solutions with their own implementation. Providers are anxious
> to get standardized l3 solutions. So we start with pb and req.
> 
> [lf] that is exactly what we have. Could you point out where in the charter
> indicate we are out if you are still not convinced?
>

---

From: Thomas Narten <[email-masked]>
Date: Tue, 03 Jul 2012 15:30:20 -0400

> I think it is important to state the assumptions about the
> properties (e.g., multicast support or, better, its lack of) of the
> underlying network in the problem statement draft.

Section 4.7 currently says:

   4.7.  Individual Submissions

   Many individual submissions also look to addressing some or all of
   the issues addressed in this draft.  Examples of such drafts are
   VXLAN [I-D.mahalingam-dutt-dcops-vxlan], NVGRE
   [I-D.sridharan-virtualization-nvgre] and Virtual Machine Mobility in
   L3 networks[I-D.wkumari-dcops-l3-vmmobility].

I have purposefully tried not made statements as to how well any
specific solution does or does not meet any requirements. That seems
more appropriate for the gap analysis.

IMO, text about how best to support tenant multicast, etc. doesn't
belong in the problem statement. Indeed, there was text on how to
support tenant multicast in earlier versions of the document that was
removed in the current version in favor of not duplicating what was in
the framework doc.

I'd like to hear from others before adding additional text on this to
the problem statement.

Thomas

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Tue, 3 Jul 2012 19:40:40 +0000

David,

> From: [email-masked] [mailto:[email-masked]]
> Sent: Tuesday, July 03, 2012 2:48 PM
> To: Luyuan Fang (lufang); [email-masked]; Benson Schliesser (bschlies)
> Cc: [email-masked]
> Subject: VPN4DC draft scope
> 
> Luyuan
> 

[lf] It is for Mobility DC virtualization. It was raised from multiple real cases we were asked to address.
Agree, things not about DC are out of scope. 
We can re-work the text to be clear it is all about Mobility DC. Mobile infra is not the focus, but it is the environment the mobile DCs living in, can be brief.

Some text in this subsection is clear about dc, such as the text below. Other may need rewording and revising. It is a good point.

... Using the same VPN technology in the service provider Data
   Center network (or in a public Data Center network) is a natural
   extension.

> VPNs as an example to ask that the draft have a title that reflects its

[lf] Agree with your comments on making it explicit in the title, we will do so and will clarify on the terms we use in this doc as well in the next revision.

If you followed the Taiwan meeting and all the activities going on in Routing area around DC, this is the continued work for vpn4dc. In the last IETF in Paris, it was said that nvo3 would expand its original scope to include vpn, so we brought the work here. We do not pretend to be something else. We do not claim to cover all issues in nvo3, and we do not believe other drafts covered them all either. Anything we need to clarify/improve, not an issue. We welcome all comments to shape it up.

Thanks,
Luyuan

> problem-
> > > > > URL:             http://www.ietf.org/internet-drafts/draft-
> fang-
> > > > vpn4dc-
> > > > > proble
> > > > > m-statement-01.txt
> > > > > Htmlized:        http://tools.ietf.org/html/submission.filename
> }}-
> > > 01
> > > > >    Network Service Providers commonly use BGP/MPLS VPNs [RFC
> 4364]
> > > as
> > > > >    to scale to a large number of VPNs and attachment points,
> and it
> > > > is
> > > > >    orchestration. This document describes the problems and
> defines
> > > > the
> > > > >    new requirements.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > The IETF Secretariat

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Tue, 3 Jul 2012 21:14:09 +0000

Thomas,

> From: Thomas Narten [mailto:[email-masked]]
> Sent: Tuesday, July 03, 2012 3:30 PM
> To: NAPIERALA, MARIA H
> Cc: [email-masked]
> Subject: Re: [nvo3] Data Center Types [was Re: call for adoption:

This is not about tenant's multicast. It is about whether a DC virtualization solution requires multicast transport in data center core (also referred to as underlying network) to operate.  This definitely belongs to a problem statement draft. 

Thanks!
Maria

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Tue, 3 Jul 2012 23:07:24 +0000

Thomas,

The proposed new text does not work, most talk on VRF/VPN function/operation are not correct.
Here are some points.

- VRF is not a service - it stands for Virtual Routing and Forwarding table. It is a table.
- VPN is the service assuming you are talking about. VPN - Virtual Private Network. Yes, it is Virtual Network (VN) with routing isolation - so "Private".
- VRF-Lite is a feature extended from BGP VPN base function to allow one CE supports multiple VPNs.
- VRF-Lite only modify CE, and there is no "each hop" things performed as described in your text.
- VRF-lite does not support all MPLS-VRF functionality: label exchange, LDP adjacency, or labeled packets.
- To PE, there is no difference between using VRF-lite or using multiple CEs.

Thomas Morin made more good comments to you already. 

BTW, you mentioned that "...I didn't realize that MPLS had adopted the VRF terminology...". This does not make any sense. 

VRF is part of MPLS/BGP VPN technology, it did not exist before.
VRF-Lite is one of the ways to configure CE.

Luyuan

> Thomas Narten
> Sent: Tuesday, July 03, 2012 9:44 AM
> To: Stiliadis, Dimitrios (Dimitri)
> Cc: [email-masked]
> Subject: [nvo3] VRFs [was Re: call for adoption: draft-narten-nvo3-
> overlay-problem-statement-02]
>

---

From: Truman Boyes <[email-masked]>
Date: Wed, 4 Jul 2012 11:24:03 -0400

Hi,

In terms of terminology, while most of us are aware of the functionality of
VRF-Lite; I am not aware that the term is standardized. VRF-Lite refers to
a particular implementation terminology, while "Multi-VRF CE" is term used
most frequently in IETF.

Kind regards,
Truman

On Tue, Jul 3, 2012 at 7:07 PM, Luyuan Fang (lufang) <[email-masked]>wrote:

> - VPN is the service assuming you are talking about. VPN - Virtual Private
> Network. Yes, it is Virtual Network (VN) with routing isolation - so
> "Private".
> BTW, you mentioned that "...I didn't realize that MPLS had adopted the VRF
> terminology...". This does not make any sense.
>

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Wed, 4 Jul 2012 17:54:08 +0000

Truman,

Good point, VRF-Lite is not in RFC 4364. It is just one way to config CE to support multiple VPNs in the implementation.
We should refer to RFC 4364 BGP VPN when talk about this technology as existing standards,  as we touched the point last week.
Thx,
Luyuan

From: Truman Boyes [mailto:[email-masked]]
Sent: Wednesday, July 04, 2012 11:24 AM
To: Luyuan Fang (lufang)
Cc: Thomas Narten; Stiliadis, Dimitrios (Dimitri); [email-masked]
Subject: Re: [nvo3] VRFs [was Re: call for adoption: draft-narten-nvo3-overlay-problem-statement-02]

Hi,

In terms of terminology, while most of us are aware of the functionality of VRF-Lite; I am not aware that the term is standardized. VRF-Lite refers to a particular implementation terminology, while "Multi-VRF CE" is term used most frequently in IETF.

Kind regards,
Truman

On Tue, Jul 3, 2012 at 7:07 PM, Luyuan Fang (lufang) <[email-masked]<mailto:[email-masked]>> wrote:
Thomas,

Luyuan

> From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of
> Thomas Narten
> Cc: [email-masked]<mailto:[email-masked]>
> "Stiliadis, Dimitrios (Dimitri)" <[email-masked]<mailto:[email-masked]>>
> writes:
>
> [email-masked]<mailto:[email-masked]>
_______________________________________________
nvo3 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nvo3

---

From: <[email-masked]>
Date: Thu, 5 Jul 2012 13:43:17 +0000

Hi Robert,

Robert Raszuk :
> requirements to only enforce that we all must be on the same single 
> bus is rather a poor choice.

On the other hand, we are working under the implicit assumption that 
having standards can be useful.
We can go beyond saying that the requirements are "different" and 
observe that some use cases are particular cases of more general use 
cases. Identifying one generic solution that would be able to also 
address the subset use caess can be beneficial to everybody.

-Thomas

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Thu, 5 Jul 2012 15:06:07 +0000

Robert is making a good point, and I agree with it. 

Developing standards does not mean that there can be only solution, especially to a network design ;-). Take, for example, Service Providers WAN networks. Each is using a different solution and different standard protocols. This is because different organizations have different business requirements, different IT approaches, different network operations approaches, etc. The same is true for Data Centers. Why would we think that there has to be one Data Center Virtualization solution? There is no reason that it has to be, even more than in core network space. 

Specifically, if a problem statement document for a data center virtualization assumes that there is a bridge domain then the requirements (and of course, the solution) will be different then if such an assumption doesn't exist. So, either we accommodate all the above in one problem statement draft draft or we separate them into different documents. Or we narrow this group's scope.

Maria

> [email-masked]
> Sent: Thursday, July 05, 2012 9:43 AM
> To: [email-masked]
> Subject: Re: [nvo3] inter-CUG traffic
> 
> Hi Robert,
> 
> Robert Raszuk :
> >>>> So I was urging that we not mandate optimal inter-subnet routing
> as
> >>>> part
> _______________________________________________________________________
> __________________________________________________
> 
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez
> recu ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les
> messages electroniques etant susceptibles d'alteration,
> As emails may be altered, France Telecom - Orange is not liable for
> messages that have been modified, changed or falsified.
> Thank you.
>

---

From: <[email-masked]>
Date: Thu, 5 Jul 2012 15:41:09 +0000

Hi all,

> 3.2.  Benefits of Network Overlays
>
> [...]  Some examples of network overlays are tunnels such
>    as IP GRE [RFC2784], LISP [I-D.ietf-lisp] or TRILL [RFC6325].

Three comments things on the above:
- I don't think the "tunnel" term is helpful to designate LISP or TRILL, 
and some possible uses of GRE
- the encapsulation in itself does not define how you build an overlay, 
so I don't think the example of GRE RFC2784 is not an appropriate 
example; first of all, you can put different payloads in GRE (e.g. IP, 
Ethernet, MPLS, and of course many more), in some cases (e.g MPLS in 
GRE) there are even other possible payloads; second, the payload carried 
is also not enough to describe how the overlay is built (e.g. you can 
use GRE keys to separate tenants, or you can use something else; for 
MPLS-in-GRE with an Ethernet payload, you could build the overlay with 
VPLS (different flavors) or with E-VPN ).
  - maybe the wiser is to not  rush into the  gap analysis, and not use 
some approaches as examples, rather than others.

>   The use of a large (e.g., 24-bit) VNID would allow 16 million
>    distinct virtual networks within a single data center, eliminating
>    current VLAN size limitations.  This VNID needs to be carried in the
>    data plane along with the packet.  Adding an overlay header provides
>    a place to carry this VNID.

I find the above very misleading, since you can very much achieve the 
same result without having a "large" 24-bit VNID in the dataplane.

>    External communications (from a VM within a virtual network instance
>    to a machine outside of any virtual network instance, e.g. on the
>    Internet) is handled by having an ingress switch forward traffic to
>    an external router, where an egress switch decapsulates a tunneled
>    packet and delivers it to the router for normal processing. This
>    router is external to the overlay, and behaves much like existing
>    external facing routers in data centers today.

If this is all we'll achieve with NVO3, then I certainly wouldn't put it 
in a section called "benefits of overlays", but rather in a section 
called "Drawbacks of NVO3"... ;-)

More seriously, beyond the fact that the paragraph above looks misplaced 
in this section, I think the problem statement should insist on the 
feasibility of efficient interworking with external networks. Being 
limited to an architecture where a box "decapsulates NVO3" to some VLAN 
toward another box which then has to map this VLAN in the proper 
context, will actually be a pain to manage. The provisioning efficiency 
brought by NVO3 is also needed for these interconnects, and the problem 
statement should I think reflect this: the problem statement should 
include the ability to terminate NVO3 directly on a router.

>    Overlays are designed to allow a set of VMs to be placed within a
>    single virtual network instance, whether that virtual network
>    provides the bridged network or a routed network.

(typo above: s/the bridged network/a bridged network/ ? )

-Thomas

---

From: <[email-masked]>
Date: Thu, 5 Jul 2012 16:09:52 +0000

Hi,

draft-narten-nvo3-overlay-problem-statement-02 / section 3.3 :
> [...] routing protocols (e.g., IS-IS, BGP) may have scaling 
> difficulties if implemented directly in
>    all NVEs, based on both flooding and convergence time concerns.  This
>    suggests that use of a standard lookup protocol between NVEs and a
>    smaller number of network nodes that implement the actual routing
>    protocol (or the directory-based "oracle") is a more promising
>    approach at larger scale.

I think that the text above really goes far beyond the scope a problem 
statement.

>    In summary, there are three areas of potential work.  The first area
>    concerns the oracle itself and any on-the-wire protocols it needs.  A
>    second area concerns the interaction between the oracle and NVEs.
>    The third work area concerns protocols associated with attaching and
>    detaching a VM from a particular virtual network instance.  The
>    latter two items are the priority work areas and can be done largely
>    independent of any oracle-related work.

I would infer from the above that the first work area, concerning 
protocols internal to the "oracle", would not be a priority.
I think this is not a good idea.

Indeed, redundancy, and very likely the ability to distribute the load, 
will be needed for the "oracle", not addressing the topic will lead to a 
situation where possibly you wouldn't have a standard to rely upon to do 
a real deployment.

Moreover, it's not certain for me that the exchanges between the 
entities forming the "oracle" can be designed independently of the 
protocol used to query the oracle; at the very least it seems that the 
redundancy aspect will require describing how an NVE processes 
information from/to more than one "oracle" entity (ie. does it query all 
of them, only some of them) and this may depend a lot how the different 
entities forming the oracle will exchange mapping information internally.

By and large I don't think we should downplay the importance of any of 
those three work areas.

-Thomas

---

From: Thomas Narten <[email-masked]>
Date: Thu, 05 Jul 2012 16:22:56 -0400

Here is another cut at the VRF text. Thanks to both the on-list and
off-list comments/discussion. Hopefully third time's the charm! :-)

         <t>
 	  In the case of IP networks, many routers provide a virtual
          routing and forwarding capability whereby a single
          router supports multiple "virtual routers", each using its
          own forwarding table, i.e., one tied to a specific tenant or
          VPN. Each forwarding table instance is populated separately
          via routing protocols, and adjacent routers encapsulate
          traffic in such a way that the data plane identifies the
          tenant or VPN that traffic belongs to. The combination of
          virtual router functionality and data plane separation
          provides address and traffic isolation for individual
          tenants.
        </t>

	<t>
	  Virtual routing and forwarding is also used on PEs as part
	  of providing BGP/MPLS VPN
	  service <xref target="RFC4364"></xref>. With BGP/MPLS VPNs,
	  MPLS encapsulation is used to provide tenant separation
	  across the transport "underlay" network between PEs. When
	  PEs are connected by MPLS paths, control plane protocols
	  (e.g., LDP <xref target="RFC5036"></xref>) are used to set
	  up the data path between PEs. Whether native MPLS paths or
	  MPLs over GRE encapsulation is
	  used <xref target="RFC4023"></xref>, BGP distributes the
	  necessary labels among PEs for tenant separation.
	</t>

Thomas

---

From: Lucy yong <[email-masked]>
Date: Thu, 5 Jul 2012 21:19:12 +0000

Following text is more clear to me. The question is: does nvo3 need control plane segregation per a tenant?
BGP/MPLS VPN does not support that. Should we explicitly mention it?

Lucy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Narten
Sent: Thursday, July 05, 2012 3:23 PM
To: [email-masked]
Subject: [nvo3] VRF text (take 3) in draft-narten-nvo3-overlay-problem-statement-02.txt

         <t>

	<t>

Thomas

---

From: Thomas Narten <[email-masked]>
Date: Thu, 05 Jul 2012 17:32:39 -0400

Hi Thomas.

> some approaches as examples, rather than others.

You make some good points, and the cited sentence needs
tightening. How about I replace it with:

	  Examples of architectures based on network overlays include LISP
	  <xref target="I-D.ietf-lisp"></xref>,
	  TRILL <xref target="RFC6325"></xref>, and Shortest Path
	  Bridging <xref target="SPB"></xref>.

I've dropped the reference to GRE because it can be used to build
overlays (as NVGRE does), but that depends on how it is used. You
could say the same about any tunneling technology for that matter...

> same result without having a "large" 24-bit VNID in the dataplane.

how about s/needs to be carried/can be carried/

I expect we agree that a globally significant VNID could be carried
end to end in this manner. And I say "can" not "must", as I agree
there are other possible approaches (i.e.,  a locally significant
one could be carried as well).

> called "Drawbacks of NVO3"... ;-)

Well, the context of the text you quote was explaining how things
work. I.e., having an isolated VN unable to talk to the outside world
doesn't seem particularly useful...

That said, I can move this all into 2.7 " Support Communication
Between Virtual and Traditional Networks"

> include the ability to terminate NVO3 directly on a router.

Re your last point, how about the following text for 2.7:

      <section title="Support Communication Between Virtual and
	Traditional Networks">
	<t>
	  Not all communication will be between devices connected to
	  virtualized networks.  Devices using overlays will continue
	  to access devices and make use of services on traditional,
	  non-virtualized networks, whether in the data center, the
	  public Internet, or at remote/branch campuses.  Any virtual
	  network solution must be capable of interoperating with
	  existing routers, VPN services, load balancers, intrusion
	  detection servics, firewalls, etc.
	</t>
	
	<t>
	  Communication between devices attached to a virtual network
	  and devices connected to non-virtualized networks is handled
	  architecturally by having specialized gateway devices that
	  receive packets from a virtualized network, process them as
	  appropriate, and forward them on to non-virtualized networks
	  (and vice versa). A simple implementation could do as little
	  as forward packets to/from a virtual network, while more
	  sophisticated gateways could include full router
	  functionality, load balancing or firewall support, etc., or
	  have the overlay encapsulation/decapsulation functionality
	  implemented with hardware assist for efficiency.
	</t>

> (typo above: s/the bridged network/a bridged network/ ? )

Thanks.

Thomas

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Thu, 5 Jul 2012 22:48:33 +0000

Thomas,

It is still not correct, sorry have to say it again. 

> "... a single router supports multiple "virtual routers", each using its own forwarding table, i.e., one tied to a specific tenant or VPN."
- No. VPN with multiple VRFs on a PE does not imply "virtual router" implementation. "virtual router" means different kind of partition. VRFs are not required to be on separate virtual routers.

> "The combination of virtual router functionality and data plane separation provides address and traffic isolation for individual tenants."
- No, VPN does not provide traffic separation within the network (besides forwarding to the right CE/tenant), it only provides route isolation.

> "With BGP/MPLS VPNs, MPLS encapsulation is used to provide tenant separation across the transport "underlay" network between PEs."
- No. VPN label (inner label) is used for VPN/tenant separation, not the MPLS encap (outer label).

I think we need to discuss if we should keep separate docs., we can take care this topic; or merge if WG thinks better that way. 
Really appreciate your effort in trying... But we need to get it right, in a more efficient way. 

Luyuan

> Sent: Thursday, July 05, 2012 4:23 PM
> To: [email-masked]

---

From: <[email-masked]>
Date: Thu, 5 Jul 2012 21:06:23 -0400

Luyuan,

I'm sorry but you're seriously misreading Thomas's new text ...

> > "... a single router supports multiple "virtual routers", each using its own
> forwarding table, i.e., one tied to a specific tenant or VPN."
> - No. VPN with multiple VRFs on a PE does not imply "virtual router"

That statement is correct as written, because it describes widely available
functionality on L2/L3 (Ethernet/IP) data center switches.  The assumption
that the statement is about "VRFs" and "PE" in BFGP/MPLS VPNS is incorrect.

> > "The combination of virtual router functionality and data plane separation
> provides address and traffic isolation for individual tenants."
> - No, VPN does not provide traffic separation within the network (besides
> forwarding to the right CE/tenant), it only provides route isolation.

That's likewise incorrect; the VPN acronym was deliberately not used in
this new text because it concerns data center network switches (routers),
not VPNs.

> > "With BGP/MPLS VPNs, MPLS encapsulation is used to provide tenant separation
> across the transport "underlay" network between PEs."

That "VPN label" is part of the MPLS label stack, unless I've seriously missed
something.

Thanks,
--David

> Sent: Thursday, July 05, 2012 6:49 PM
> To: Thomas Narten; [email-masked]
> Subject: Re: [nvo3] VRF text (take 3) in draft-narten-nvo3-overlay-problem-
> statement-02.txt
> 
> Thomas,
> 
> > "... a single router supports multiple "virtual routers", each using its own
> forwarding table, i.e., one tied to a specific tenant or VPN."
> - No. VPN with multiple VRFs on a PE does not imply "virtual router"
> implementation. "virtual router" means different kind of partition. VRFs are
> not required to be on separate virtual routers.
> 
> > "The combination of virtual router functionality and data plane separation
> provides address and traffic isolation for individual tenants."
> - No, VPN does not provide traffic separation within the network (besides
> forwarding to the right CE/tenant), it only provides route isolation.
> 
> > "With BGP/MPLS VPNs, MPLS encapsulation is used to provide tenant separation
> across the transport "underlay" network between PEs."
> I think we need to discuss if we should keep separate docs., we can take care
> this topic; or merge if WG thinks better that way.

---

From: <[email-masked]>
Date: Thu, 5 Jul 2012 21:31:10 -0400

Luyuan,

> If you followed the Taiwan meeting and all the activities going on in Routing area around DC,
> this is the continued work for vpn4dc. In the last IETF in Paris, it was said that nvo3
> would expand its original scope to include vpn, so we brought the work here.

I did follow the Taiwan meeting, and I'm aware of all of this, although I was a bit surprised
to see the VPN4DC draft turn up in nvo3 rather than go back to l3vpn, as I have a somewhat
different perspective, which dates back to before the Taipei meetings.

The nvo3 effort started out as overlay-focused, and it was pointed out that overlays are
a type of VPN.  That observation is correct, as all three of "virtual", "private" and
"network" apply to the overlay VNs in nvo3.  One of the results was that the first IETF
session devoted to nvo3 was held as the second l2vpn WG session in Taipei, and another
result was use of the VPN acronym in the nvo3 WG charter.

I think it's a bit of a stretch to jump from there to a conclusion that all VPN work in
data centers is therefore within the scope of nvo3.  To take an obvious example, if there
was interest in data-center-specific work for IPsec VPNs, that work would be done in the
Security Area's ipsecme WG, and attempts to do it in nvo3 would result in long versions
of the work "No" from the Security ADs :-).

I agree that the vpn4dc work is potentially in the scope of nvo3 (and I believe I effectively
said so at the microphone in Paris in response to a question from Thomas Morin), but I would
really like to better understand how and why this work moved to nvo3 as the WG of primary
focus instead of where it started in the l3vpn WG back in Taipei.

Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Fri, 6 Jul 2012 02:52:47 +0000

- "virtual routers" <> multiple VRFs on a router. When SPs request "virtual router" capability of an equipment, they are not asking about VPN. Could you help us with the IETF reference if you think your "virtual router" definition is correct?

- "The combination of virtual router functionality and data plane separation provides address and traffic isolation for individual tenants." 
now you are saying Thomas switched out from talking VPN, even though that paragraph was talking VPN, and the paragraph after, and the subject... Regardless, let's assume so, my point still stands that there is no traffic separation within the DC network infrastructure other than the last hop distribution. You can share with us what technology you are using to separate the tenant traffic through the network infra if this is not true.

- On the MPLS label, good point, agree that VPN label is part of MPLS label stack. But there is big difference between the VPN label which is the inner label, and MPLS label for forwarding (LDP, RSVP...) which is the outer label. Now this helps me to understand why people thought that MPLS VPN approach was not an overlay, it was hop by hop, network infra aware, etc., more likely it causes confusion when using the general term of mpls label. We have done many years of VPN design/deployment work in SP, we generally call it VPN label explicitly to distinguish from MPLS labels which are used for forwarding purpose. Say MPLS label encap to separate tenant is not accurate enough at least IMHO. 

Luyuan

> Sent: Thursday, July 05, 2012 9:06 PM
> To: Luyuan Fang (lufang); [email-masked]
> Subject: RE: [nvo3] VRF text (take 3) in draft-narten-nvo3-overlay-
> problem-statement-02.txt
> 
> Luyuan,
> 
> > - No. VPN label (inner label) is used for VPN/tenant separation, not
> the MPLS
> > encap (outer label).
> 
> > - No. VPN label (inner label) is used for VPN/tenant separation, not
> the MPLS
> > encap (outer label).
> >
> > Really appreciate your effort in trying... But we need to get it
> right, in a
> > more efficient way.
> >
> > Luyuan
> >
> Behalf Of
> > > Subject: [nvo3] VRF text (take 3) in draft-narten-nvo3-overlay-
> problem-
> > > statement-02.txt
> > >

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Fri, 6 Jul 2012 03:32:54 +0000

David,

Thanks for sharing your thought.

Regarding overlay and VPN, Adrian posted the question asking the difference during the charter discussion, at the end, no good distinction could be made, therefore the term "DCVPN" was adopted in the charter. 

Regarding scope, I don't have much agree or disagree with you, we follow IETF/IESG directions. We would be happy to take the vpn4dc work in a separate new or existing group and progress it too. From the current charter write-up, nvo3 is indeed the place. We share the same concerns on scaling, you saw a few mails on the list too, from Pedro, Rob, Maria...  We have several SPs really interested in the vpn solutions for DC, we started the initiative last year without knowing nvo3 effort was running in parallel. We need to continue progress it.

Below is some quote from the current nvo3 charter. It seems to me: "NV03 = DCVPN". Next, what is the difference between DCVPN to VPN4DC? :-). OK, would not go there. :-)

"An NVO3 solution (known here as a Data Center Virtual Private 
Network (DCVPN)) is a VPN that is viable across a scaling 
range of a few thousand VMs to several million VMs running on 
greater than one hundred thousand physical servers. It thus has 
good scaling properties from relatively small networks to 
networks with several million DCVPN endpoints and hundreds of 
thousands of DCVPNs within a single administrative domain.

A DCVPN also supports VM migration between physical servers 
in a sub-second timeframe."

Thanks,
Luyuan

> Sent: Thursday, July 05, 2012 9:31 PM
> To: Luyuan Fang (lufang)
> Cc: [email-masked]
> Subject: NVO3 scope - VPN4DC
> 
> Luyuan,
>

---

From: Thomas Narten <[email-masked]>
Date: Fri, 06 Jul 2012 07:25:29 -0400

Hi Luyuan.

> > "... a single router supports multiple "virtual routers", each
> >  using its own forwarding table, i.e., one tied to a specific
> >  tenant or VPN."

>    implementation. "virtual router" means different kind of
>    partition. VRFs are not required to be on separate virtual
>    routers.

I don't understand the issue here. "virtual router" is a fairly
general term that has long been in use to describe the case where you
have a single router device that actualy has multiple separate
"virtual" instances that individually behave like separate
routers. E.g., see RFC 2917.

To make this more clear, I'd be happy to add the word "conceptually"
to the sentence as the intent is to describe the conceptual
functionality, not a specific implementation. E.g.:

      In the case of IP networks, many routers provide a virtual
      routing and forwarding capability whereby a single router
      conceptually supports multiple "virtual routers", each using its
      own forwarding table, i.e., one tied to a specific tenant or
      VPN.

> > "The combination of virtual router functionality and data plane
> > separation provides address and traffic isolation for individual
> > tenants."

> - No, VPN does not provide traffic separation within the network
>   (besides forwarding to the right CE/tenant), it only provides
>   route isolation.

Are we having a terminology issue? I.e, is the issue here the word
"traffic separation"? The term as used here (and in other places in
the document) is for the purpose of logically separating traffic of
different tenants and to keep one tenant from having access to another
tenant's traffic. i.e., "tenant separation".

Are you using the term in the sense of traffic needing be kept on
separate links as opposed to allowing traffic from different VPNs to
traverse over shared links?

> > "With BGP/MPLS VPNs, MPLS encapsulation is used to provide tenant
> > separation across the transport "underlay" network between PEs."

>   the MPLS encap (outer label).

So thanks to some offline discussion, I have a much better
understanding of this point now. But it seems to me to be a bit of
splitting hairs.

With BGP/MPLS VPNs, tenant separation is achieved by having an MPLS
label that has significance to the ingress and egress PE. The egress
PE picks the label it wants to use for a given Route Target/Route
Distinguisher. BGP distributes this info back to the ingress PE, which
then knows how to encap VPN traffic for a specific VPN to the egress
PE. If there is an MPLS path as well, the VPN label is stacked, so the
the VPN label is independent of whether an actual MPLS LSR path is
used or whether MPLSoGRE is used.

Now, I think you are arguing that the VPN label is not MPLS. Well,
IMO, that is splitting hairs. It has the syntax of an MPLS label, and
is encoded on the data plane in an MPLSoGRE packet. I would agree that
this is not normal MPLS because in essence, the label is just a
demultiplexor that the ingress/egress PE have agreed to use to denote
a particular VPN. But given that the label is used with MPLS encoding
on the data plane, I think the quoted text is correct and it's a bit
misleading to argue that this is not MPLS...

Thomas

---

From: <[email-masked]>
Date: Fri, 6 Jul 2012 09:35:13 -0400

Luyuan,

> - "virtual routers" <> multiple VRFs on a router ... Could you
> help us with the IETF reference if you think your "virtual router"
> definition is correct?

Sure, "virtual router" is the correct term, a "virtual router" is definitely
not a VRF for a BGP/MPLS VPN and the term "virtual router" has been in use in
the IETF for well over a decade.

The two paragraphs in question were always intended to refer to the concept
of a "virtual router" as that term is used with VRRP, see RFC 5798, and the
use of "virtual router" dates back to at least the first version of VRRP,
RFC 2338 (1998).  In 20/20 hindsight, the use of the VRF acronym in those
two paragraphs was a mistake that we are now correcting - that mistake is
at the root of this confusion (mea culpa, as I'm a co-author of that original
text).  Do we need to cite RFC 5798 to make this clearer?

> Say MPLS label encap to separate tenant is not accurate enough at
> least IMHO.

I'm sorry, but not only did Thomas not use the word "label", but the words
that he did use are consistent with the second paragraph of Section 1 of
RFC 4364 where both labels are referred to as MPLS labels and both are
described as being used for encapsulation.  I do hope that settles this
issue ;-).

Thanks,
--David

> From: Luyuan Fang (lufang) [mailto:[email-masked]]
> Sent: Thursday, July 05, 2012 10:53 PM
> To: Black, David; [email-masked]
> Subject: RE: [nvo3] VRF text (take 3) in draft-narten-nvo3-overlay-problem-
> statement-02.txt
> 
> - "virtual routers" <> multiple VRFs on a router. When SPs request "virtual
> router" capability of an equipment, they are not asking about VPN. Could you
> help us with the IETF reference if you think your "virtual router" definition
> is correct?
> 
> - "The combination of virtual router functionality and data plane separation
> now you are saying Thomas switched out from talking VPN, even though that
> paragraph was talking VPN, and the paragraph after, and the subject...
> Regardless, let's assume so, my point still stands that there is no traffic
> separation within the DC network infrastructure other than the last hop
> distribution. You can share with us what technology you are using to separate
> the tenant traffic through the network infra if this is not true.
> 
> - On the MPLS label, good point, agree that VPN label is part of MPLS label
> stack. But there is big difference between the VPN label which is the inner
> label, and MPLS label for forwarding (LDP, RSVP...) which is the outer label.
> Now this helps me to understand why people thought that MPLS VPN approach was
> not an overlay, it was hop by hop, network infra aware, etc., more likely it
> causes confusion when using the general term of mpls label. We have done many
> years of VPN design/deployment work in SP, we generally call it VPN label
> explicitly to distinguish from MPLS labels which are used for forwarding
> purpose. Say MPLS label encap to separate tenant is not accurate enough at
> least IMHO.
> 
> Luyuan
> 
> > using its own
> > separation
> > (besides
> > separation
> > using its own
> > separation
> > (besides
> > separation

---

From: <[email-masked]>
Date: Fri, 6 Jul 2012 10:06:11 -0400

Luyuan,

I think I'm going to have to agree to disagree, as WG charter text is
malleable - the IESG can change it, and I have plenty of experience with
that as a WG chair.  As a consequence, I don't regard the use of the VPN
acronym in the current nvo3 charter as definitively settling the location
of all data-center-related VPN work for all time.

With you and a number of others suggesting that we have at least two
separate problems that need separate attention, I will observe that it
is possible for a single WG to accommodate multiple significant work
areas that have limited interaction/overlap/commonality, as I've chaired
a WG that did exactly that for three major areas of work:
	http://www.ietf.org/wg/concluded/ips.html

OTOH, it wasn't easy - the material on technical coordinators at the end of
the WG description in that charter is an initial hint to what had to be put
in place to handle the work.  There was a period of time where there were 3
WG co-chairs in addition to the 3 technical coordinators in order to stay on
top of everything ... and even so, it still felt like a 3-ring circus from
time to time :-).  Nonetheless, I still think that keeping everything in one
WG was the right decision at the time because there were overriding transport
and security issues that needed common attention, and the list of RFCs at the
bottom of that charter indicates that the WG was effective.

Time will tell whether nvo3 has analogous common issues that favor keeping
everything in one place.  Beyond the current framework and problem statement
adventures, there clearly needs to be exactly one virtual interface attach/
detach protocol (which might be VDP or XMPP or something else), and that
protocol has to carry interface and network identifiers that are sufficiently
general to support use by any of the approaches to overlays/VPNs/etc.  In
contrast, if the VPN4DC work on BGP/MPLS VPNs requires BGP extensions, the
L3VPN WG seems like a better place for those because the BGP expertise is
already there, and any such BGP extensions will have to be coordinated with
the existing use of BGP for L3VPNs.

What's the current state of the VPN4DC draft wrt the L3VPN WG?

Thanks,
--David

> Sent: Thursday, July 05, 2012 11:33 PM
> To: Black, David
> Cc: [email-masked]
> Subject: RE: NVO3 scope - VPN4DC
> 
> David,
> 
> Regarding overlay and VPN, Adrian posted the question asking the difference
> during the charter discussion, at the end, no good distinction could be made,
> therefore the term "DCVPN" was adopted in the charter.
> 
> Regarding scope, I don't have much agree or disagree with you, we follow
> IETF/IESG directions. We would be happy to take the vpn4dc work in a separate
> new or existing group and progress it too. From the current charter write-up,
> nvo3 is indeed the place. We share the same concerns on scaling, you saw a few
> mails on the list too, from Pedro, Rob, Maria...  We have several SPs really
> interested in the vpn solutions for DC, we started the initiative last year
> without knowing nvo3 effort was running in parallel. We need to continue
> progress it.
> 
> Below is some quote from the current nvo3 charter. It seems to me: "NV03 =
> DCVPN". Next, what is the difference between DCVPN to VPN4DC? :-). OK, would
> not go there. :-)
>

---

From: <[email-masked]>
Date: Mon, 9 Jul 2012 16:17:19 +0000

Hi Thomas,

> one could be carried as well).

By and large, this idea of a global VNID carried in the dataplane looks 
to me as quite orthogonal to the explanation of the benefits of an 
overlay solution.

I think the text should go straight to the fact that using an overlay 
allows to build a large enough number of virtual network instances 
independently of current VLAN size limitation.  This is true whether a 
globally significant or locally significant id is used, so that aspect 
is I think better to kept out of that section.

>> called "Drawbacks of NVO3"...
>

I would agree with this change.

>> I think the problem statement should insist on the

The above does not reflect the fact that external networks can also be 
virtualized, and that part of the problem of designing NVO3 is so that 
mapping an NVO3 virtual network to an external virtual network can be 
efficient, ie. better than a VLAN-based intermediate handoff.

-Thomas

---

From: Thomas Narten <[email-masked]>
Date: Mon, 09 Jul 2012 13:54:19 -0400

Hi Thomas.

> to me as quite orthogonal to the explanation of the benefits of an 

OK.

> is I think better to kept out of that section.

OK.

How about:

    The use of a sufficiently large VNID would eliminate current VLAN
    size limitations.  This VNID can be carried in the data plane
    along with the packet.  Adding an overlay header provides a place
    to carry this VNID.

> efficient, ie. better than a VLAN-based intermediate handoff.

Are you talking specifically about the case of having a single device
take NVO3 traffic on (say) one interface and send it out on another
(different) virtualized network (e.g., a VPN), all in one device?

That's (IMO) clearly allowed and expected to be supported. Are you
asking that this example be made explicitely in the text?

I.e., you don't want to require the use of an L2 hop using VLANs. I
agree, but also think that is clearly allowed by the sentence you
quote.

Thomas

---

From: Lucy yong <[email-masked]>
Date: Mon, 9 Jul 2012 19:34:55 +0000

Snip..

[[LY]] The nvo3 focuses on the overlay and virtualized network. IMO, it is important to state that the interconnection of VNs can be virtualized on a single device.

Lucy

Thomas

---

From: Linda Dunbar <[email-masked]>
Date: Mon, 9 Jul 2012 19:51:16 +0000

Joel, 

I agree with you that VPN related WG (e.g. L2VPN, L3VPN) may not need to address too much on across VN communications. 

But most traffic in Data Center are across subnets. Given NVo3 is for identifying issues associated with data centers, I think that cross Subnet traffic should not be ignored. 

Linda

> Joel M. Halpern
> Sent: Friday, June 29, 2012 9:32 PM
> To: Pedro Roque Marques
> Cc: [email-masked]; [email-masked]; [email-masked]
> I would not be so bold as to insist that all deployments can safely
> ignore inter-VPN intra-data-center traffic.  But there are MANY cases
> where that is not an important part of the traffic mix.
> I would not want to prohibit it either, as there are definitely cases
> where it matters, some along the lines you alude to.
> 
> Yours,
> Joel
> 
> On 6/29/2012 9:40 PM, Pedro Roque Marques wrote:
> > Joel,
> > A very common model currently is to have a 3 tier app where each tier
> is in its VLAN. You will find that web-servers for instance don't
> actually talk much to each other... although they are on the same VLAN
> 100% of their traffic goes outside VLAN. Very similar story applies to
> app logic tier. The database tier may have some replication traffic
> within its VLAN but hopefully than is less than the requests that it
> serves.
> >
> > There isn't a whole lot of intra-CUG/subnet traffic under that
> deployment model. A problem statement that assumes (implicitly) that
> most or a significant part of the traffic stays local to a
> VLAN/subnet/CUG is not a good match for the common 3-tier application
> model. Even if you assume that web and app tiers use a  VLAN/subnet/CUG
> per tenant (which really is an application in enterprise) the database
> is typically common for a large number of apps/tenants.
> >
> >    Pedro.
> >
> > On Jun 29, 2012, at 5:26 PM, Joel M. Halpern wrote:
> >
> >> Depending upon what portion of the traffic needs inter-region
> handling (inter vpn, inter-vlan, ...) it is not obvious that "optimal"
> is an important goal.  As a general rule, perfect is the enemy of good.
> >>
> >> Yours,
> >> Joel
> >>
> >> On 6/29/2012 7:54 PM, [email-masked] wrote:
> >>> Pedro,
> >>>
> >>>> Can you please describe an example of how you could set up such
> >>>> straightforward routing, assuming two Hosts belong to different
> "CUGs" such
> >>>> that these can be randomly spread across the DC  ? My question is
> where is the
> >>>> "gateway", how is it provisioned and  how can traffic paths be
> guaranteed to
> >>>> be optimal.
> >>>
> >>> Ok, I see your point - the routing functionality is straightforward
> to move over,
> >>> but ensuring optimal pathing is significantly more work, as noted
> in another one
> >>> of your messages:
> >>>
> >>>> Conceptually, that means that the functionality of the "gateway"
> should be
> >>>> implemented at the overlay ingress and egress points, rather than
> requiring
> >>>> a mid-box.
> >>>
> >>> Thanks,
> >>> --David
> >>>
> >>>
> >>>> From: Pedro Roque Marques [mailto:[email-masked]]
> >>>> Sent: Friday, June 29, 2012 7:38 PM
> >>>> To: Black, David
> >>>> Cc: [email-masked]; [email-masked]
> draft-
> >>>> narten-nvo3-overlay-problem-statement-02]
> >>>>
> >>>>
> >>>> On Jun 29, 2012, at 4:02 PM, <[email-masked]> wrote:
> >>>>
> >>>>>> There is an underlying assumption in NVO3 that isolating tenants
> from
> >>>>>> each other is a key reason to use overlays. If 90% of the
> traffic is
> >>>>>> actually between different tenants, it is not immediately clear
> to me
> >>>>>> why one has set up a system with a lot of "inter tenant" traffic.
> Is
> >>>>>> this is a case we need to focus on optimizing?
> >>>>>
> >>>>> A single tenant may have multiple virtual networks with routing
> used to
> >>>>> provide/control access among them.  The crucial thing is to avoid
> assuming
> >>>>> that a tenant or other administrative entity has a single virtual
> network
> >>>>> (or CUG in Pedro's email).  For example, consider moving a
> portion of
> >>>>> a single data center that uses multiple VLANs and routers to
> selectively
> >>>>> connect them into an nvo3 environment - each VLAN gets turned
> into a virtual
> >>>>> network, and the routers now route among virtual networks instead
> of VLANs.
> >>>>>
> >>>>> One of the things that's been pointed out to me in private is
> that the level
> >>>>> of importance that one places on routing across virtual networks
> may depend
> >>>>> on one's background.  If one is familiar with VLANs and views
> nvo3 overlays
> >>>>> providing VLAN-like functionality, IP routing among virtual
> networks is a
> >>>>> straightforward application of IP routing among VLANs (e.g., the
> previous
> >>>>> mention of L2/L3 IRB functionality that is common in data center
> network
> >>>>> switches).
> >>>>
> >>>> Can you please describe an example of how you could set up such
> >>>> straightforward routing, assuming two Hosts belong to different
> "CUGs" such
> >>>> that these can be randomly spread across the DC  ? My question is
> where is the
> >>>> "gateway", how is it provisioned and  how can traffic paths be
> guaranteed to
> >>>> be optimal.
> >>>>
> >>>>>   OTOH, if one is familiar with VPNs where access among
> >>>>> otherwise-closed groups has to be explicitly configured,
> particularly
> >>>>> L3 VPNs where one cannot look to L2 to help with grouping the end
> systems,
> >>>>> this sort of cross-group access can be a significant area of
> functionality.
> >>>>
> >>>> Considering that in a VPN one can achieve inter-CUG traffic
> exchange without
> >>>> an gateway in the middle via policy, it is unclear why you suggest
> that "look
> >>>> to L2" would help.
> >>>>
> >>>>>
> >>>>> Thanks,
> >>>>> --David
> >>>>>
> >>>>>> Narten
> >>>>>> Sent: Friday, June 29, 2012 5:56 PM
> >>>>>> To: Pedro Roque Marques
> >>>>>> Cc: [email-masked]
> >>>>>> Subject: [nvo3] inter-CUG traffic [was Re: call for adoption:
> draft-narten-
> >>>>>>> 3) Does not discuss the requirements for inter-CUG traffic.
> >>>>>>
> >>>>>> Given that the problem statement is not supposed to be the
> >>>>>> requirements document,, what exactly should the problem
> statement say
> >>>>>> about this topic?
> >>>>>>
> >>>>>> <[email-masked]> writes:
> >>>>>>
> >>>>>>> Inter-VN traffic (what you refer to as inter-CUG traffic) is
> handled
> >>>>>>> by a straightforward application of IP routing to the inner IP
> >>>>>>> headers; this is similar to the well-understood application of
> IP
> >>>>>>> routing to forward traffic across VLANs.  We should talk about
> VRFs
> >>>>>>> as something other than a limitation of current approaches -
> for
> >>>>>>> VLANs, VRFs (separate instances of routing) are definitely a
> >>>>>>> feature, and I expect this to carry forward to nvo3 VNs.  In
> >>>>>>> addition, we need to make changes to address Dimitri's comments
> >>>>>>> about problems with the current VRF text.
> >>>>>>
> >>>>>>> That is where again the differences between different types of
> >>>>>>> data-centers do play in. If for instance 90% of a VMs traffic
> >>>>>>> happens to be between the Host OS and a network attached
> storage
> >>>>>>> file system run as-a-Service (with the appropriate multi-tenent
> >>>>>>> support) then the question of where are the routers becomes a
> very
> >>>>>>> important issue. In a large scale data-center where the Host VM
> and
> >>>>>>> the CPU that hosts the filesystem block can be randomly spread
> >>>>>>> where is the router ?
> >>>>>>
> >>>>>> Where is what router? Are you assuming the Host OS and NAS are
> in the
> >>>>>> different VNs? And hence, traffic has to (at least conceptually)
> exit
> >>>>>> one VN and reenter another whenever there is HostOS - NAS
> traffic?
> >>>>>>
> >>>>>>> Is every switch a router ? Does it have all the CUGs present ?
> >>>>>>
> >>>>>> The underlay can be a mixture of switches and routers... that is
> not
> >>>>>> our concern. So long as the underlay delivers traffic sourced by
> an
> >>>>>> ingress NVE to the appropriate egress NVE, we are good.
> >>>>>>
> >>>>>> If there are issues with the actual path taken being suboptimal
> in
> >>>>>> some sense, that is an underlay problem to solve, not for the
> overlay.
> >>>>>>
> >>>>>>> In some DC designs the problem to solve is the inter-CUG
> >>>>>>> traffic. With L2 headers being totally irrelevant.
> >>>>>>
> >>>>>> There is an underlying assumption in NVO3 that isolating tenants
> from
> >>>>>> each other is a key reason to use overlays. If 90% of the
> traffic is
> >>>>>> actually between different tenants, it is not immediately clear
> to me
> >>>>>> why one has set up a system with a lot of "inter tenant" traffic.
> Is
> >>>>>> this is a case we need to focus on optimizing?
> >>>>>>
> >>>>>> But in any case, if one does have inter-VN traffic, that will
> have to
> >>>>>> get funneled through a "gateway" between VNs, at least
> conceptually, I
> >>>>>> would assume that an implementation of overlays would provide at
> least
> >>>>>> one, and likely more such gateways on each VN. How many and
> where to
> >>>>>> place them will presumably depend on many factors but would be
> done
> >>>>>> based on traffic patterns and network layout. I would not think
> every
> >>>>>> NVE has to provide such functionality.
> >>>>>>
> >>>>>> What do you propose needs saying in the problem statement about
> that?
> >>>>>>
> >>>>>> Thomas
> >>>>>>

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 09 Jul 2012 16:00:17 -0400

I have not seen data to suggest "most".  Do you have a source for that? 
  I have seen descriptions of data centers for which it is almost "all", 
and other data centers for which it is negligible.
Yours,
Joel

---

From: Benson Schliesser <[email-masked]>
Date: Mon, 09 Jul 2012 15:28:18 -0500

> text).  Do we need to cite RFC 5798 to make this clearer?

FYI, per http://tools.ietf.org/html/rfc4110#section-1.5 a Virtual Router 
is defined as:

    Virtual Router (VR): An instance of one of a number of logical
    routers located within a single physical router.  Each logical router
    emulates a physical router using existing mechanisms and tools for
    configuration, operation, accounting, and maintenance.

For more context, one might also wish to read 
http://tools.ietf.org/html/draft-ietf-l3vpn-as-vr-02 and some of its 
references.

Cheers,
-Benson

---

From: David Allan I <[email-masked]>
Date: Mon, 9 Jul 2012 16:36:22 -0400

Can we refer to the "limitations of a single 12 bit VLAN tag" if we find it necessary to discuss it? That would make clear what limitations we were discussing.

I would be much happier ;-)
Dave

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Narten
Sent: Monday, July 09, 2012 10:54 AM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [nvo3] overlay benefits section: [was: Re: call for adoption: draft-narten-nvo3-overlay-problem-statement-02 / section 3.2]

Hi Thomas.

> By and large, this idea of a global VNID carried in the dataplane 
> looks to me as quite orthogonal to the explanation of the benefits of 
> an overlay solution.

OK.

OK.

How about:

> efficient, ie. better than a VLAN-based intermediate handoff.

Are you talking specifically about the case of having a single device take NVO3 traffic on (say) one interface and send it out on another
(different) virtualized network (e.g., a VPN), all in one device?

That's (IMO) clearly allowed and expected to be supported. Are you asking that this example be made explicitely in the text?

I.e., you don't want to require the use of an L2 hop using VLANs. I agree, but also think that is clearly allowed by the sentence you quote.

Thomas

---

From: Linda Dunbar <[email-masked]>
Date: Mon, 9 Jul 2012 20:56:15 +0000

Joel, 

Many data centers' web servers are in different subnet than other applications (e.g. DB servers). The web server applications frequently communicate with peers in the same DC which are usually in different subnets. 

What kind of DCs which only have intra subnet communication? 

Linda
> From: Joel M. Halpern [mailto:[email-masked]]
> Sent: Monday, July 09, 2012 3:00 PM
> To: Linda Dunbar
> Cc: Pedro Roque Marques; [email-masked]; [email-masked];
> [email-masked]
> > I agree with you that VPN related WG (e.g. L2VPN, L3VPN) may not need
> to address too much on across VN communications.
> >
> > But most traffic in Data Center are across subnets. Given NVo3 is for
> identifying issues associated with data centers, I think that cross
> Subnet traffic should not be ignored.
> >
> > Linda
> >

---

From: "Stiliadis, Dimitrios (Dimitri)" <[email-masked]>
Date: Mon, 9 Jul 2012 16:13:40 -0500

>
> Many data centers' web servers are in different subnet than other applications
> (e.g. DB servers). The web server applications frequently communicate with
> peers in the same DC which are usually in different subnets.
>

A large hadoop/mapreduce cluster will not bother about different subnets,
and the traffic requirements can be huge. Thousands of enterprise applications
will also do the same, since they assume they are in a protected environment.

Dimitri

---

From: Linda Dunbar <[email-masked]>
Date: Mon, 9 Jul 2012 21:58:53 +0000

Dimitrios, 

Do you mean that all applications in "a large hddop/mapredure cluster" are in one subnet? 
And the applications in  "one cluster" never communicate with another "cluster" in the same data center?  Really? 
I would think applications in DMZ would be at least put in a different subnet. 

Linda 

> From: Stiliadis, Dimitrios (Dimitri) [mailto:dimitri.stiliadis@alcatel-
> lucent.com]
> > > >>> A very common model currently is to have a 3 tier app where
> each
> > > tier
> > > >> actually talk much to each other... although they are on the
> same
> > > VLAN
> > > >> 100% of their traffic goes outside VLAN. Very similar story
> applies
> > > to
> > > >> is an important goal.  As a general rule, perfect is the enemy
> of
> > > good.
> > > >>>>
> > > >>>> Yours,
> > > >>>> Joel
> > > >>>>
> > > >>>>>> that these can be randomly spread across the DC  ? My
> question
> > > is
> > > >> where is the
> > > >>>>>> that these can be randomly spread across the DC  ? My
> question
> > > is
> > > >> where is the
> > > >>>>>>> L3 VPNs where one cannot look to L2 to help with grouping
> the
> > > end
> > > >> systems,
> > > >>>>>>>>> by a straightforward application of IP routing to the
> inner
> > > IP
> > > >>>>>>>>> headers; this is similar to the well-understood
> application
> > > of
> > > >> IP
> > > >>>>>>>>> That is where again the differences between different
> types
> > > of
> > > >>>>>>>>> important issue. In a large scale data-center where the
> Host
> > > VM
> > > >> and
> > > >>>>>>>> The underlay can be a mixture of switches and routers...
> that
> > > is
> > > >> not
> > > >>>>>>>> our concern. So long as the underlay delivers traffic
> sourced
> > > by
> > > >> an
> > > >>>>>>>> would assume that an implementation of overlays would
> provide
> > > at
> > > >> least

---

From: "Stiliadis, Dimitrios (Dimitri)" <[email-masked]>
Date: Mon, 9 Jul 2012 17:40:50 -0500

There is no DMZ in such applications. They don't reside on the DMZ.

Applications maintain Tbytes of data in a hadoop cluster and running mapreduce
on them. There is a single application and single tenant on this whole cluster.
mapreduce nodes will send lots  of data between the map and reduce phase
and there is no need for any L3 separation in these cases.

This is just one example. We can't possibly make the assumption that any
app in the DC is related to a web-service on the DMZ.

Dimitri

> From: Linda Dunbar [mailto:[email-masked]]
> Sent: Monday, July 09, 2012 2:59 PM
> To: Stiliadis, Dimitrios (Dimitri); Joel M. Halpern
> Cc: Pedro Roque Marques; [email-masked]; [email-masked]; [email-masked]
> Subject: RE: [nvo3] inter-CUG traffic [was Re: call for adoption: draft-
> And the applications in  "one cluster" never communicate with another
> "cluster" in the same data center?  Really?

---

From: <[email-masked]>
Date: Tue, 10 Jul 2012 15:45:14 +0000

Hi David, Luyuan, Thomas,

> text).  Do we need to cite RFC 5798 to make this clearer?

I think citing RFC5798 may increase confusion.
As Benson noted, "virtual router" is a term that was (also) used in 
RFC4110; this approach (where a full router is virtualized) was not 
actively pursued, and instead RFC2547 BGP/MPLS where actively worked on 
(where only the routing and forwarding tables are virtualized, but not 
the other parts of the router, namely not the routing protocols, nor the 
"configuration, operation, accounting, and maintenance" parts (or at 
least not all of them). Some confusion comes from considering that an 
entity is a router as soon as it forwards IP packets, or considering 
that the entity has to run its own *routing* protocols to be a router...

In the context of the first paragraph, which really talks about separate 
routing instances (also called "VRF lite"), saying "virtual router" does 
not look problematic to me.
It would be problematic in the context of the second paragraph which is 
about BGP/MPLS.

> described as being used for encapsulation.  I do hope that settles this

If I may attempt at a pacifying interpretation (;-): Luyuan sees a 
possible source of confusion in calling the inner label an 
"encapsulation", since, in the context of NVO3, it plays the role of the 
overlay header rather than the role of the IP encapsulation. If my 
interpretation is correct, then maybe saying...

           With BGP/MPLS VPNs, an MPLS ***label*** is used to provide tenant separation
	  across the transport "underlay" network between PEs.

...would resolve the issue.

-Thomas

---

From: Linda Dunbar <[email-masked]>
Date: Tue, 10 Jul 2012 15:55:38 +0000

Dimitrios, 

I thought that we are discussing if it is necessary for NVo3 to consider cross subnet communications. For the example you give where all applications in the DC are all in one subnet and those applications don't communicate with peers in other subnets, I would think they are pretty small DC, which is not really what NVo3 is targeted for, is it?

Linda

> Sent: Monday, July 09, 2012 5:41 PM
> To: Linda Dunbar; Joel M. Halpern
> Subject: RE: [nvo3] inter-CUG traffic [was Re: call for adoption:
> > Do you mean that all applications in "a large hddop/mapredure
> cluster" are in
> > one subnet?
> > > > > I have not seen data to suggest "most".  Do you have a source
> for
> > > that?
> > > > >   I have seen descriptions of data centers for which it is
> almost
> > > "all",
> > > > > > I agree with you that VPN related WG (e.g. L2VPN, L3VPN) may
> not
> > > need
> > > > > > But most traffic in Data Center are across subnets. Given
> NVo3 is
> > > for
> > > > > >> ignore inter-VPN intra-data-center traffic.  But there are
> MANY
> > > > > cases
> > > > > >> I would not want to prohibit it either, as there are
> definitely
> > > > > cases
> > > > > >> deployment model. A problem statement that assumes
> (implicitly)
> > > that
> > > > > >> per tenant (which really is an application in enterprise)
> the
> > > > > database
> > > > > >>>>>> Can you please describe an example of how you could set
> up
> > > such
> > > > > >>>>>> "gateway", how is it provisioned and  how can traffic
> paths
> > > be
> > > > > >> guaranteed to
> > > > > >>>>>> be optimal.
> > > > > >>>>>
> > > > > >>>>> but ensuring optimal pathing is significantly more work,
> as
> > > noted
> > > > > >> in another one
> > > > > >>>>> of your messages:
> > > > > >>>>>
> > > > > >>>>>> implemented at the overlay ingress and egress points,
> rather
> > > > > than
> > > > > >> requiring
> > > > > >>>>>> a mid-box.
> > > > > >>>>>
> > > > > >>>>> Thanks,
> > > > > >>>>> --David
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>>>>> There is an underlying assumption in NVO3 that
> isolating
> > > > > tenants
> > > > > >> from
> > > > > >>>>>>>> actually between different tenants, it is not
> immediately
> > > > > clear
> > > > > >> to me
> > > > > >>>>>>>> why one has set up a system with a lot of "inter
> tenant"
> > > > > traffic.
> > > > > >> Is
> > > > > >>>>>>> provide/control access among them.  The crucial thing
> is to
> > > > > avoid
> > > > > >> assuming
> > > > > >>>>>>> that a tenant or other administrative entity has a
> single
> > > > > virtual
> > > > > >> network
> > > > > >>>>>>> a single data center that uses multiple VLANs and
> routers
> > > to
> > > > > >> selectively
> > > > > >>>>>>> network, and the routers now route among virtual
> networks
> > > > > instead
> > > > > >> of VLANs.
> > > > > >>>>>>>
> > > > > >>>>>>> One of the things that's been pointed out to me in
> private
> > > is
> > > > > >> that the level
> > > > > >>>>>>> straightforward application of IP routing among VLANs
> (e.g.,
> > > > > the
> > > > > >> previous
> > > > > >>>>>>> mention of L2/L3 IRB functionality that is common in
> data
> > > > > center
> > > > > >> network
> > > > > >>>>>>> switches).
> > > > > >>>>>>
> > > > > >>>>>> Can you please describe an example of how you could set
> up
> > > such
> > > > > >>>>>> "gateway", how is it provisioned and  how can traffic
> paths
> > > be
> > > > > >> guaranteed to
> > > > > >>>>>> be optimal.
> > > > > >>>>>>
> > > > > >>>>>>> this sort of cross-group access can be a significant
> area
> > > of
> > > > > >> functionality.
> > > > > >>>>>>
> > > > > >>>>>> an gateway in the middle via policy, it is unclear why
> you
> > > > > suggest
> > > > > >> that "look
> > > > > >>>>>> to L2" would help.
> > > > > >>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> Thanks,
> > > > > >>>>>>> --David
> > > > > >>>>>>>
> > > > > >>>>>>>> From: [email-masked] [mailto:nvo3-
> [email-masked]]
> > > On
> > > > > >> Behalf Of
> > > > > >>>>>> Thomas
> > > > > >>>>>>>> Narten
> > > > > >>>>>>>>> Inter-VN traffic (what you refer to as inter-CUG
> traffic)
> > > is
> > > > > >> handled
> > > > > >>>>>>>>> routing to forward traffic across VLANs.  We should
> talk
> > > > > about
> > > > > >> VRFs
> > > > > >>>>>>>>> feature, and I expect this to carry forward to nvo3
> VNs.
> > > In
> > > > > >>>>>>>>> addition, we need to make changes to address
> Dimitri's
> > > > > comments
> > > > > >>>>>>>>> file system run as-a-Service (with the appropriate
> multi-
> > > > > tenent
> > > > > >>>>>>>>> the CPU that hosts the filesystem block can be
> randomly
> > > > > spread
> > > > > >>>>>>>>> where is the router ?
> > > > > >>>>>>>>
> > > > > >>>>>>>> Where is what router? Are you assuming the Host OS and
> NAS
> > > are
> > > > > >> in the
> > > > > >>>>>>>> some sense, that is an underlay problem to solve, not
> for
> > > the
> > > > > >> overlay.
> > > > > >>>>>>>>
> > > > > >>>>>>>> There is an underlying assumption in NVO3 that
> isolating
> > > > > tenants
> > > > > >> from
> > > > > >>>>>>>> actually between different tenants, it is not
> immediately
> > > > > clear
> > > > > >> to me
> > > > > >>>>>>>> why one has set up a system with a lot of "inter
> tenant"
> > > > > traffic.
> > > > > >> Is
> > > > > >>>>>>>> But in any case, if one does have inter-VN traffic,
> that
> > > will
> > > > > >> have to
> > > > > >>>>>>>> one, and likely more such gateways on each VN. How
> many
> > > and
> > > > > >> where to
> > > > > >>>>>>>> based on traffic patterns and network layout. I would
> not
> > > > > think
> > > > > >> every
> > > > > >>>>>>>> What do you propose needs saying in the problem
> statement
> > > > > about
> > > > > >> that?
> > > > > >>>>>>>>
> > > > > >>>>>>>> Thomas
> > > > > >>>>>>>>

---

From: <[email-masked]>
Date: Tue, 10 Jul 2012 15:59:58 +0000

Hi Thomas,

>      along with the packet.  Adding an overlay header provides a place

I still think that we introduce something (global id in the dataplane) 
that does not relate directly to the benefits of an overlay approach.

Strictly speaking, if one server never has more than 2**12 virtual 
networks bound onto it, then a 12 bit demultiplexor in the dataplane 
would be enough.

The size of the "VNID in the control plane" is actually the factor that 
will make the difference.

What about:

       The use of a sufficiently large VNID, present in the overlay
       control plane and possibly also in the dataplane would eliminate
       current VLAN size limitations.

> (different) virtualized network (e.g., a VPN), all in one device?

A VPN is one example.
Virtualized L4-L7 service instances would be another.

> asking that this example be made explicitely in the text?

Yes, because so this efficiently is I think part of the problem to solve.

-Thomas

---

From: <[email-masked]>
Date: Tue, 10 Jul 2012 14:01:37 -0400

And I made a paragraph count mistake (mea culpa, sorry), as I confused
this w/a private draft that had split the first paragraph into two.  The
second paragraph in this thread is clearly about BGP/MPLS VPNs, and has
no need to use the VRF acronym.  "virtual router" is still the correct
term for the first paragraph to refer to multiple instances of routing
functionality in a single IP router (e.g., one box).

If introducing the word "label" will clarify the BGP/MPLS VPN paragraph,
that works for me.

Thanks,
--David

> Sent: Tuesday, July 10, 2012 11:45 AM
> To: [email-masked]
> definition

---

From: "Stiliadis, Dimitrios (Dimitri)" <[email-masked]>
Date: Tue, 10 Jul 2012 13:39:33 -0500

\>
> I thought that we are discussing if it is necessary for NVo3 to consider cross
> subnet communications. For the example you give where all applications in the
> DC are all in one subnet and those applications don't communicate with peers
> in other subnets, I would think they are pretty small DC, which is not really
> what NVo3 is targeted for, is it?
>

On the contrary. Lots of very large DCs are there for such applications.

(most Big Data related data centers are like that).

Dimitri

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Tue, 10 Jul 2012 21:17:20 +0000

Benson, Thomas, and David,

As some of us have gone through the ppvpn/l3vpn development in the last decade are familiar with the history around VR development in ppvpn/l3vpn, others may benefit with more clarification.

Two approaches were proposed initially: rfc 2547 BGP/MPLS VPN, and Virtual Router (VR) VPN. 2547 has been development and widely deployed, VR VPN solution did not progress to standard RFCs.
Thomas Morin has provided good summary of the differences between the two, and the status for each.

The base VR solution drafts and the VR applicability statement draft Benson referenced did not become RFCs. As IESG write-ups in 2006 indicated: "Active participation in the L3VPN WG on Virtual Routers has not existed for some time. There is little energy to modify these documents further,..."

So the fact remains there is no Virtual Router solution for l3vpn/l2vpn as Standard track RFCs, even though the informational RFC 4110 on Framework talked virtual router as it was published in 2005 before VR was ended.

Virtual Router term is often used in other context as general term. You can virtualize a physical router into multiple virtual routers with resource partition, some only has control plane partition, e.g. each can run separate IGP, BGP..., others may have HW resource partition as well. There are can be various levels of virtualization. We were discussing with vendors for virtual router/logical router starting 2000 when I was working in SP. E.g. virtual router/logical router could be considered to separate L3 VPN services and Internet services through resource partition, or make separate P and PE routers with resource partition using the same box, etc. all not much to do VPNs, it is general device virtualization. Many devices today can support pretty sophisticated virtualization.

I'm OK if you use the virtual router as generic term and explain the use case, and not to reference back to the VR VPN approach which is not IETF standards.

Luyuan

From: Benson Schliesser [mailto:[email-masked]]
Sent: Monday, July 09, 2012 4:28 PM
To: [email-masked]
Cc: Luyuan Fang (lufang); [email-masked]
Subject: Re: [nvo3] VRF text (take 3) in draft-narten-nvo3-overlay-problem-statement-02.txt

On 2012-07-06 8:35 AM, [email-masked]<mailto:[email-masked]> wrote:

Luyuan,

definition is correct?

FYI, per http://tools.ietf.org/html/rfc4110#section-1.5 a Virtual Router is defined as:

For more context, one might also wish to read http://tools.ietf.org/html/draft-ietf-l3vpn-as-vr-02 and some of its references.

Cheers,
-Benson

---

From: John E Drake <[email-masked]>
Date: Tue, 10 Jul 2012 14:15:52 -0700

Thomas,

As usual, very well said.  One slight tweak on your proposed text revision:  "With BGP/MPLS VPNs, an MPLS label is used to provide tenant separation between PEs."

Thanks,

John

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Tuesday, July 10, 2012 8:45 AM
To: [email-masked]
Subject: Re: [nvo3] VRF text (take 3) in draft-narten-nvo3-overlay-problem-statement-02.txt

[email-masked] :
>> - "virtual routers" <> multiple VRFs on a router ... Could you help 
>> us with the IETF reference if you think your "virtual router"
>> definition is correct?
> Sure, "virtual router" is the correct term, a "virtual router" is 
> definitely not a VRF for a BGP/MPLS VPN and the term "virtual router" 
> has been in use in the IETF for well over a decade.
>
> The two paragraphs in question were always intended to refer to the 
> concept of a "virtual router" as that term is used with VRRP, see RFC 
> 5798, and the use of "virtual router" dates back to at least the first 
> version of VRRP, RFC 2338 (1998).  In 20/20 hindsight, the use of the 
> VRF acronym in those two paragraphs was a mistake that we are now 
> correcting - that mistake is at the root of this confusion (mea culpa, 
> as I'm a co-author of that original text).  Do we need to cite RFC 5798 to make this clearer?

I think citing RFC5798 may increase confusion.
As Benson noted, "virtual router" is a term that was (also) used in RFC4110; this approach (where a full router is virtualized) was not actively pursued, and instead RFC2547 BGP/MPLS where actively worked on (where only the routing and forwarding tables are virtualized, but not the other parts of the router, namely not the routing protocols, nor the "configuration, operation, accounting, and maintenance" parts (or at least not all of them). Some confusion comes from considering that an entity is a router as soon as it forwards IP packets, or considering that the entity has to run its own *routing* protocols to be a router...

In the context of the first paragraph, which really talks about separate routing instances (also called "VRF lite"), saying "virtual router" does not look problematic to me.
It would be problematic in the context of the second paragraph which is about BGP/MPLS.

> I'm sorry, but not only did Thomas not use the word "label", but the 
> words that he did use are consistent with the second paragraph of 
> Section 1 of RFC 4364 where both labels are referred to as MPLS labels 
> and both are described as being used for encapsulation.  I do hope 
> that settles this issue ;-).

If I may attempt at a pacifying interpretation (;-): Luyuan sees a possible source of confusion in calling the inner label an "encapsulation", since, in the context of NVO3, it plays the role of the overlay header rather than the role of the IP encapsulation. If my interpretation is correct, then maybe saying...

-Thomas

>> - "virtual routers" <> multiple VRFs on a router. When SPs request 
>> "virtual router" capability of an equipment, they are not asking 
>> about VPN. Could you help us with the IETF reference if you think 
>> your "virtual router" definition is correct?
>>
>> - "The combination of virtual router functionality and data plane 
>> separation provides address and traffic isolation for individual tenants."
>> now you are saying Thomas switched out from talking VPN, even though 
>> that paragraph was talking VPN, and the paragraph after, and the subject...
>> Regardless, let's assume so, my point still stands that there is no 
>> traffic separation within the DC network infrastructure other than 
>> the last hop distribution. You can share with us what technology you 
>> are using to separate the tenant traffic through the network infra if this is not true.
>>
>> - On the MPLS label, good point, agree that VPN label is part of MPLS 
>> label stack. But there is big difference between the VPN label which 
>> is the inner label, and MPLS label for forwarding (LDP, RSVP...) which is the outer label.
>> Now this helps me to understand why people thought that MPLS VPN 
>> approach was not an overlay, it was hop by hop, network infra aware, 
>> etc., more likely it causes confusion when using the general term of 
>> mpls label. We have done many years of VPN design/deployment work in 
>> SP, we generally call it VPN label explicitly to distinguish from 
>> MPLS labels which are used for forwarding purpose. Say MPLS label 
>> encap to separate tenant is not accurate enough at least IMHO.
>>
>> Luyuan
>>
>>> That statement is correct as written, because it describes widely 
>>> available functionality on L2/L3 (Ethernet/IP) data center switches.  
>>> The assumption that the statement is about "VRFs" and "PE" in 
>>> BFGP/MPLS VPNS is incorrect.
>>>
>>> That's likewise incorrect; the VPN acronym was deliberately not used 
>>> in this new text because it concerns data center network switches 
>>> (routers), not VPNs.
>>>
>>> That "VPN label" is part of the MPLS label stack, unless I've 
>>> seriously missed something.
>>>
>>> Thanks,
>>> --David
>>>
>>>
>>> Of Luyuan
>>>>> Here is another cut at the VRF text. Thanks to both the on-list 
>>>>> and off-list comments/discussion. Hopefully third time's the 
>>>>> charm! :-)
>>>>>
>>>>>           <t>
>>>>>            own forwarding table, i.e., one tied to a specific 
>>>>> tenant
>>> or

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, France Telecom - Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, France Telecom - Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: John E Drake <[email-masked]>
Date: Tue, 10 Jul 2012 14:22:57 -0700

The problem is that the juxtaposition of the two paragraphs implies that the term 'virtual router' has some meaning in the context of RFC 4364 and it doesn't. 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Tuesday, July 10, 2012 11:02 AM
To: [email-masked]; [email-masked]
Subject: Re: [nvo3] VRF text (take 3) in draft-narten-nvo3-overlay-problem-statement-02.txt

And I made a paragraph count mistake (mea culpa, sorry), as I confused this w/a private draft that had split the first paragraph into two.  The second paragraph in this thread is clearly about BGP/MPLS VPNs, and has no need to use the VRF acronym.  "virtual router" is still the correct term for the first paragraph to refer to multiple instances of routing functionality in a single IP router (e.g., one box).

If introducing the word "label" will clarify the BGP/MPLS VPN paragraph, that works for me.

Thanks,
--David

> > definitely not a VRF for a BGP/MPLS VPN and the term "virtual 
> > router" has been in use
> in
> > concept of a "virtual router" as that term is used with VRRP, see 
> > RFC 5798, and the use of "virtual router" dates back to at least the 
> > first version of VRRP, RFC 2338 (1998).  In 20/20 hindsight, the use 
> > of the VRF acronym in those two paragraphs was a mistake that we are 
> > now correcting - that mistake is at the root of this confusion (mea 
> > culpa, as I'm a co-author of that
> original
> actively pursued, and instead RFC2547 BGP/MPLS where actively worked 
> on (where only the routing and forwarding tables are virtualized, but 
> not the other parts of the router, namely not the routing protocols, 
> nor the "configuration, operation, accounting, and maintenance" parts 
> (or at least not all of them). Some confusion comes from considering 
> that an entity is a router as soon as it forwards IP packets, or 
> considering that the entity has to run its own *routing* protocols to be a router...
> 
> In the context of the first paragraph, which really talks about 
> separate routing instances (also called "VRF lite"), saying "virtual 
> router" does not look problematic to me.
> > Section 1 of RFC 4364 where both labels are referred to as MPLS 
> > labels and both are described as being used for encapsulation.  I do 
> > hope that settles this issue ;-).
> 
> "encapsulation", since, in the context of NVO3, it plays the role of 
> the overlay header rather than the role of the IP encapsulation. If my 
> >> about VPN. Could
> >> now you are saying Thomas switched out from talking VPN, even 
> >> though that paragraph was talking VPN, and the paragraph after, and the subject...
> >> the last hop distribution. You can share with us what technology 
> >> you are using to
> separate
> >> - On the MPLS label, good point, agree that VPN label is part of 
> >> MPLS label stack. But there is big difference between the VPN label 
> >> which is the inner label, and MPLS label for forwarding (LDP, 
> >> RSVP...) which is the outer
> label.
> >> approach
> was
> >> not an overlay, it was hop by hop, network infra aware, etc., more 
> >> likely
> it
> >> causes confusion when using the general term of mpls label. We have 
> >> done
> many
> >> years of VPN design/deployment work in SP, we generally call it VPN 
> >> label explicitly to distinguish from MPLS labels which are used for 
> >> forwarding purpose. Say MPLS label encap to separate tenant is not 
> >> accurate enough at least IMHO.
> >>
> >> Luyuan
> >>
> >>> Subject: RE: [nvo3] VRF text (take 3) in 
> >>> draft-narten-nvo3-overlay- problem-statement-02.txt
> >>>
> >>> Luyuan,
> >>>
> >>> available functionality on L2/L3 (Ethernet/IP) data center 
> >>> switches.  The assumption that the statement is about "VRFs" and 
> >>> "PE" in BFGP/MPLS VPNS is incorrect.
> >>>
> >>> That's likewise incorrect; the VPN acronym was deliberately not 
> >>> used in this new text because it concerns data center network 
> >>> switches (routers), not VPNs.
> >>>
> >>>> Subject: Re: [nvo3] VRF text (take 3) in 
> >>>> draft-narten-nvo3-overlay-
> >>> problem-
> >>>> statement-02.txt
> >>>>
> >>>> Thomas,
> >>>>
> >>>> I think we need to discuss if we should keep separate docs., we 
> >>>> can
> >>> take care
> >>>>>            router supports multiple "virtual routers", each 
> >>>>> using
> >>> its
> ______________________________________________________________________
> ________ ___________________________________________
> 
> confidentielles ou privilegiees et ne doivent donc pas etre diffuses, 
> exploites ou copies sans autorisation. Si vous avez recu ce message 
> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi 
> que les pieces jointes. Les messages electroniques etant susceptibles 
> d'alteration, France Telecom - Orange decline toute responsabilite si 
> ce message a ete altere, deforme ou falsifie. Merci.
> 
> This message and its attachments may contain confidential or 
> privileged information that may be protected by law; they should not 
> be distributed, used or copied without authorisation.

---

From: <[email-masked]>
Date: Tue, 10 Jul 2012 17:46:01 -0400

> The problem is that the juxtaposition of the two paragraphs implies that the
> term 'virtual router' has some meaning in the context of RFC 4364 and it
> doesn't.

We agree on this, please suggest specific text changes to correct what you believe is wrong.

Thanks,
--David

> From: John E Drake [mailto:[email-masked]]
> Sent: Tuesday, July 10, 2012 5:23 PM
> To: Black, David; [email-masked]; [email-masked]
> The problem is that the juxtaposition of the two paragraphs implies that the
> term 'virtual router' has some meaning in the context of RFC 4364 and it
> doesn't.
>
> [email-masked]
> And I made a paragraph count mistake (mea culpa, sorry), as I confused this
> w/a private draft that had split the first paragraph into two.  The second
> paragraph in this thread is clearly about BGP/MPLS VPNs, and has no need to
> use the VRF acronym.  "virtual router" is still the correct term for the first
> paragraph to refer to multiple instances of routing functionality in a single
> IP router (e.g., one box).
>
> > draft-narten-nvo3-overlay-problem-
> > statement-02.txt
> >
> > >> draft-narten-nvo3-overlay-problem-
> > >> statement-02.txt
> > >>

---

From: John E Drake <[email-masked]>
Date: Tue, 10 Jul 2012 15:37:39 -0700

David,

I would be happy to help.  As the text of the two paragraphs has been in flux, would you please send me what you consider to be the latest text?

Thanks,

John

Sent from my iPhone

> Sent: Tuesday, July 10, 2012 2:46 PM
> To: John E Drake; [email-masked]
> > The problem is that the juxtaposition of the two paragraphs implies
> > that the term 'virtual router' has some meaning in the context of RFC
> > 4364 and it doesn't.
>
> > The problem is that the juxtaposition of the two paragraphs implies
> > that the term 'virtual router' has some meaning in the context of RFC
> > 4364 and it doesn't.
> >
> > this w/a private draft that had split the first paragraph into two.
> > The second paragraph in this thread is clearly about BGP/MPLS VPNs,
> > and has no need to use the VRF acronym.  "virtual router" is still
> the
> > correct term for the first paragraph to refer to multiple instances
> of
> > routing functionality in a single IP router (e.g., one box).
> >
> > > Of [email-masked]
> > > > The two paragraphs in question were always intended to refer to
> > > > the concept of a "virtual router" as that term is used with VRRP,
> > > > see RFC 5798, and the use of "virtual router" dates back to at
> > > > least the first version of VRRP, RFC 2338 (1998).  In 20/20
> > > > hindsight, the use of the VRF acronym in those two paragraphs was
> > > > a mistake that we are now correcting - that mistake is at the
> root
> > > > of this confusion (mea culpa, as I'm a co-author of that
> > > original
> > > on (where only the routing and forwarding tables are virtualized,
> > > but not the other parts of the router, namely not the routing
> > > protocols, nor the "configuration, operation, accounting, and
> > > maintenance" parts (or at least not all of them). Some confusion
> > > comes from considering that an entity is a router as soon as it
> > > forwards IP packets, or considering that the entity has to run its
> > > own *routing* protocols to be a
> > router...
> > >
> > > It would be problematic in the context of the second paragraph
> which
> > > is about BGP/MPLS.
> > >
> > > > I'm sorry, but not only did Thomas not use the word "label", but
> > > > the words that he did use are consistent with the second
> paragraph
> > > > of Section 1 of RFC 4364 where both labels are referred to as
> MPLS
> > > > labels and both are described as being used for encapsulation.  I
> > > > do hope that settles this issue ;-).
> > >
> > > the overlay header rather than the role of the IP encapsulation. If
> > > my interpretation is correct, then maybe saying...
> > >
> > > >> - "virtual routers" <> multiple VRFs on a router. When SPs
> > > >> request "virtual router" capability of an equipment, they are
> not
> > > >> asking about VPN. Could
> > > you
> > > >> though that paragraph was talking VPN, and the paragraph after,
> > > >> and the
> > subject...
> > > >> Regardless, let's assume so, my point still stands that there is
> > > >> no traffic separation within the DC network infrastructure other
> > > >> than the last hop distribution. You can share with us what
> > > >> technology you are using to
> > > separate
> > > >> MPLS label stack. But there is big difference between the VPN
> > > >> label which is the inner label, and MPLS label for forwarding
> > > >> (LDP,
> > > >> not an overlay, it was hop by hop, network infra aware, etc.,
> > > >> more likely
> > > it
> > > >> causes confusion when using the general term of mpls label. We
> > > >> have done
> > > many
> > > >> years of VPN design/deployment work in SP, we generally call it
> > > >> VPN label explicitly to distinguish from MPLS labels which are
> > > >> used for forwarding purpose. Say MPLS label encap to separate
> > > >> tenant is not accurate enough at least IMHO.
> > > >>
> > > >> Luyuan
> > > >>
> > > >>> That statement is correct as written, because it describes
> > > >>> widely available functionality on L2/L3 (Ethernet/IP) data
> > > >>> center switches.  The assumption that the statement is about
> > > >>> "VRFs" and "PE" in BFGP/MPLS VPNS is incorrect.
> > > >>>
> partition.
> > > >>> VRFs are
> > > >>>>> Here is another cut at the VRF text. Thanks to both the
> > > >>>>> on-list and off-list comments/discussion. Hopefully third
> > > >>>>> time's the charm! :-)
> > > >>>>>
> > > >>>>>           <t>
> When
> ____________________________________________________________________
> > > __ ________ ___________________________________________
> > >
> > > confidentielles ou privilegiees et ne doivent donc pas etre
> > > diffuses, exploites ou copies sans autorisation. Si vous avez recu
> > > ce message par erreur, veuillez le signaler a l'expediteur et le
> > > detruire ainsi que les pieces jointes. Les messages electroniques
> > > etant susceptibles d'alteration, France Telecom - Orange decline
> > > toute responsabilite si ce message a ete altere, deforme ou
> falsifie. Merci.
> > >
> > > If you have received this email in error, please notify the sender
> > > and delete this message and its attachments.

---

From: Thomas Narten <[email-masked]>
Date: Tue, 10 Jul 2012 20:09:48 -0400

> I would be happy to help.  As the text of the two paragraphs has
> been in flux, would you please send me what you consider to be the
> latest text?

Here is what I currently have. Paragraph's 3-4 are what we've been
going back and forth on.

      <section title="Limitations of Existing Virtual Network Models">
        <t>
          Virtual networks are not new to networking. For example,
          VLANs are a well known construct in the networking industry.
          A VLAN is an L2 bridging construct that provides some of the
          semantics of virtual networks mentioned above: a MAC address
          is unique within a VLAN, but not necessarily across
          VLANs. Traffic sourced within a VLAN (including broadcast
          and multicast traffic) remains within the VLAN it originates
          from. Traffic forwarded from one VLAN to another typically
          involves router (L3) processing. The forwarding table look
          up operation is keyed on {VLAN, MAC address} tuples.
	</t>

        <t>
          But there are problems and limitations with L2 VLANs. VLANs
          are a pure L2 bridging construct and VLAN identifiers are
          carried along with data frames to allow each forwarding
          point to know what VLAN the frame belongs to. A VLAN today
          is defined as a 12 bit number, limiting the total number of
          VLANs to 4096 (though typically, this number is 4094 since 0
          and 4095 are reserved). Due to the large number of tenants
          that a cloud provider might service, the 4094 VLAN limit is
          often inadequate.  In addition, there is often a need for
          multiple VLANs per tenant, which exacerbates the issue.
        </t>

         <t>
 	  In the case of IP networks, many routers provide a virtual
          routing and forwarding capability whereby a single router
          conceptually supports multiple "virtual routers", each using
          its own forwarding table, i.e., one tied to a specific
          tenant or VPN. Each forwarding table instance is populated
          separately via routing protocols, and adjacent routers
          encapsulate traffic in such a way that the data plane
          identifies the tenant or VPN that traffic belongs to. The
          combination of virtual router functionality and data plane
          separation provides address and traffic isolation for
          individual tenants.
        </t>

	<t>

	<t>
	  There are a number of VPN approaches that provide some if
	  not all of the desired semantics of virtual networks. A gap
	  analysis will be needed to assess how well existing
	  approaches satisfy the requirements.
	</t>

Thomas

---

From: Truman Boyes <[email-masked]>
Date: Tue, 10 Jul 2012 22:16:40 -0400

Hi, when making the case that VLANS are inadequate, it may be wise to also
indicate that QinQ (double tagged VLANs) as well as PBB-style approaches
are also not appropriate in large scale cloud networks. While double tagged
VLANS work in broadband access networks, the relationship of S,C vlan
constructs in DC is not entirely flexible to cloud services.

Kind regards,
Truman

---

From: Aldrin Isaac <[email-masked]>
Date: Tue, 10 Jul 2012 23:43:16 -0400

How are these very large DCs with one large flat L2 subnet dealing with the issue of broadcast/arp today?

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Wed, 11 Jul 2012 03:53:40 +0000

That is a very good point, Truman.

I thought the same when reading several drafts, people often use 4K VLAN as the scale limitation, but the industry has already moved forward with more scalable Ethernet solutions, e.g., QinQ, and especially PBB which can support 16M (I-SID) service instances should be discussed, the reason you gave sounds like a good one, similarly, e.g. PBB's B-VLAN/S-VLAN/C-VLAN structure could be a good fit for some SP Metro-E infra networks but not so for the rather flat DC (especially overlay) structure. Then there are also different flavor of L2VPN solutions, 4K VLAN limit may or may not apply depending on the particular technology used, it would be good if each could be discussed on why or why not they fit in.  e.g. some DC operators prefer not to use L2VPN solutions due to operators skill set could be a valid reason, among others?

Luyuan

From: [email-masked] [mailto:[email-masked]] On Behalf Of Truman Boyes
Sent: Tuesday, July 10, 2012 10:17 PM
To: Thomas Narten
Cc: John E Drake; [email-masked]; [email-masked]
Subject: Re: [nvo3] VRF text (take 3) in draft-narten-nvo3-overlay-problem-statement-02.txt

Hi, when making the case that VLANS are inadequate, it may be wise to also indicate that QinQ (double tagged VLANs) as well as PBB-style approaches are also not appropriate in large scale cloud networks. While double tagged VLANS work in broadband access networks, the relationship of S,C vlan constructs in DC is not entirely flexible to cloud services.

Kind regards,
Truman

        <t>

         <t>

        <t>

Thomas

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 11 Jul 2012 05:13:25 +0000

IMHO, very larger DC doesn't mean the subnets there must be very large in the number of hosts within each subnet. Take the Iaas-based public cloud data center as an example, most tenants are SMEs which usually don't require too many VMs and those VMs of each tenant could be located within a single subnet for simplicity. Hence most subnets there would not be very large.

Best regards,
Xiaohu

> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代表 Aldrin
> Isaac
> 发送时间: 2012年7月11日 11:43
> 收件人: Stiliadis, Dimitrios (Dimitri)
> 抄送: [email-masked]; [email-masked]; [email-masked]; Linda
> Dunbar; Pedro Roque Marques; Joel M. Halpern
> 主题: Re: [nvo3] inter-CUG traffic [was Re: call for adoption:

---

From: Joel jaeggli <[email-masked]>
Date: Tue, 10 Jul 2012 23:22:19 -0700

> How are these very large DCs with one large flat L2 subnet dealing with the issue of broadcast/arp today?

DC's that I think of as large don't have that topology.

not withstanding the fitness of the control plane for such systems,
common 1 and 10Gb tors have 16k 32k 128k mac limits

---

From: Lucy yong <[email-masked]>
Date: Wed, 11 Jul 2012 15:50:37 +0000

Tom,

Regarding to the limitation of existing virtual network models, we should point out that the existing models  make that the network edge and client edge reside on different hardware, so the communication between two is over a wire. In NVO3, NVE and TES may be on the same physical device.

For the case of L3VPN, we should point out that CE in [RFC4364] is a network site while the TES in NVO3 may be an end system (VM).

IMO: these differences may lead to a new solution or the expansion/simplification of existing solutions. So it is important to state out in the problem statement.

Regards,
Lucy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Narten
Sent: Tuesday, July 10, 2012 7:10 PM
To: John E Drake
Cc: [email-masked]; [email-masked]
Subject: Re: [nvo3] VRF text (take 3) in draft-narten-nvo3-overlay-problem-statement-02.txt

        <t>

         <t>

	<t>

	<t>

Thomas

---

From: Thomas Narten <[email-masked]>
Date: Fri, 27 Jul 2012 15:57:06 -0400

WG:

To circle back to this thread:

> been in flux, would you please send me what you consider to be the

John did provide text, and it is in
draft-narten-nvo3-overlay-problem-statement-03.txt, which was posted
last week. It says:

   For IP/MPLS networks, Ethernet Virtual Private Network (E-VPN)
   [I-D.ietf-l2vpn-evpn] provides an emulated Ethernet service in which
   each tenant has its own Ethernet network over a common IP or MPLS
   infrastructure and a BGP/MPLS control plane is used to distribute the
   tenant MAC addresses and the MPLS labels that identify the tenants
   and tenant MAC addresses.  Within the BGP/MPLS control plane a thirty
   two bit Ethernet Tag is used to identify the broadcast domains
   (VLANs) associated with a given L2 VLAN service instance and these
   Ethernet tags are mapped to VLAN IDs understood by the tenant at the
   service edges.  This means that the limit of 4096 VLANs is associated
   with an individual tenant service edge, enabling a much higher level
   of scalability.  Interconnectivity between tenants is also allowed in
   a controlled fashion.

   IP/MPLS networks also provide an IP VPN service (L3 VPN) [RFC4364] in
   which each tenant has its own IP network over a common IP or MPLS
   infrastructure and a BGP/MPLS control plane is used to distribute the
   tenant IP routes and the MPLS labels that identify the tenants and
   tenant IP routes.  As with E-VPNs, interconnectivity between tenants
   is also allowed in a controlled fashion.

   VM Mobility [I-D.raggarwa-data-center-mobility] introduces the
   concept of a combined L2/L3 VPN service in order to support the
   mobility of individual Virtual Machines (VMs) between Data Centers
   connected over a common IP or MPLS infrastructure.

   There are a number of VPN approaches that provide some if not all of
   the desired semantics of virtual networks.  A gap analysis will be
   needed to assess how well existing approaches satisfy the
   requirements.

Does that text work for folk?

Thomas

---

From: Kireeti Kompella <[email-masked]>
Date: Fri, 27 Jul 2012 13:26:26 -0700

Looks okay. 

Kireeti

---

From: David Freedman <[email-masked]>
Date: Fri, 27 Jul 2012 20:36:53 +0000

Yes, looks good.

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Fri, 27 Jul 2012 20:48:34 +0000

One ("organizational") comment is that since L3VPN is not only an RFC but it has been widely deployed for 14 years, it should precede the others mentioned in the proposed text. It seems to be mentioned as an "afterthought"..
The L3VPN description itself is not precise and not sufficient (and needs to include a reference to draft-marques-end-system).

Maria

> Sent: Friday, July 27, 2012 3:57 PM
> To: John E Drake
> Cc: [email-masked]

---

From: Lucy yong <[email-masked]>
Date: Fri, 27 Jul 2012 20:47:39 +0000

Hi Tom,

It is not clear what Ethernet service that E-VPN emulates for.
MEF has E-Line, E-LAN, E-Tree, and E-access service. IETF has VPLS. IMHO E-VPN does not emulate any of them. It emulates a bridging device connecting to multiple sites and being configured with multiple VLANs that may have different topologies among the sites. E-VPN is very like L3VPN [RFC4364] but implemented in L2. Another outstanding feature E-VPN providing is multi-homing with active-active mode. None of existing Ethernet Services support that. Thus, E-VPN does not emulate existing Ethernet service. It seems that SP will offer a new service by using E-VPN.

Thanks,
Lucy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Narten
Sent: Friday, July 27, 2012 2:57 PM
To: John E Drake
Cc: [email-masked]
Subject: Re: [nvo3] VRF text (take 3) in draft-narten-nvo3-overlay-problem-statement-02.txt

WG:

Thomas

---

From: John E Drake <[email-masked]>
Date: Sat, 28 Jul 2012 16:30:30 -0700

Also MAC address mobility and constant convergence time after failure regardless of the number advertised routes.

Sent from my iPhone

---

From: John E Drake <[email-masked]>
Date: Sun, 29 Jul 2012 09:33:09 -0700

Sent from my iPhone

> Sent: Friday, July 27, 2012 1:49 PM
> To: Thomas Narten; John E Drake
> Cc: [email-masked]
> One ("organizational") comment is that since L3VPN is not only an RFC
> but it has been widely deployed for 14 years, it should precede the
> others mentioned in the proposed text.

JD:  Totally irrelevant

> It seems to be mentioned as an
> "afterthought"..

JD:  No.  The section starts by describing issues with VLANs.  E-VPN is then described as it does not have these issues.  L3VPN is then introduced in the context of VM mobility
  
> The L3VPN description itself is not precise and not sufficient

JD:  This is a helpful comment

> (and needs to include a reference to draft-marques-end-system).

JD: Why, other than you like the draft?

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 30 Jul 2012 00:13:16 +0200

Hi John,

>> One ("organizational") comment is that since L3VPN is not only an
>> RFC but it has been widely deployed for 14 years, it should precede
>> the others mentioned in the proposed text.
>
> JD:  Totally irrelevant

I don't think this is irrelevant. If operational experience with pretty
mature technology which can address the problem at hand is not important
then what is ?

> JD: Why, other than you like the draft?

Because it proposes conversion of the L3VPN interface to the end host to 
be host friendly.

So let me ask the reverse .. why not ? Because you do not like the draft 
? Or because the draft does not help legacy vendors to be part of the pie ?

Best,
R.

---

From: Lucy yong <[email-masked]>
Date: Mon, 30 Jul 2012 20:08:06 +0000

Then the question is why we need to describe E-VPN in the problem draft? E-VPN is still in developing stage and operator yet deploy it. Why we describe it as a "traditional" solution?

Lucy

-----Original Message-----
From: John E Drake [mailto:[email-masked]] 
Sent: Saturday, July 28, 2012 6:31 PM
To: Lucy yong
Cc: Thomas Narten; [email-masked]
Subject: Re: [nvo3] VRF text (take 3) in draft-narten-nvo3-overlay-problem-statement-02.txt

Sent from my iPhone

---

From: <[email-masked]>
Date: Mon, 30 Jul 2012 17:02:07 -0400

Lucy,

Good question - that might fit better under the L2VPN related work section.

Thanks,
--David

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Lucy
> yong
> Sent: Monday, July 30, 2012 4:08 PM
> To: John E Drake; Lucy yong
> Then the question is why we need to describe E-VPN in the problem draft? E-VPN
> is still in developing stage and operator yet deploy it. Why we describe it as
> a "traditional" solution?
> 
> Lucy
> 
> Also MAC address mobility and constant convergence time after failure
> regardless of the number advertised routes.
> 
> Sent from my iPhone
> 
> > MEF has E-Line, E-LAN, E-Tree, and E-access service. IETF has VPLS. IMHO E-
> VPN does not emulate any of them. It emulates a bridging device connecting to
> multiple sites and being configured with multiple VLANs that may have
> different topologies among the sites. E-VPN is very like L3VPN [RFC4364] but
> implemented in L2. Another outstanding feature E-VPN providing is multi-homing
> with active-active mode. None of existing Ethernet Services support that.
> Thus, E-VPN does not emulate existing Ethernet service. It seems that SP will
> offer a new service by using E-VPN.
> >
> > Thanks,
> > Lucy
> >
> >
> >
> >