Subject: Re: [nvo3] Updated Charter

From: Christopher Liljenstolpe <[email-masked]>
Date: Fri, 30 Mar 2012 14:10:10 +0200

I support, and echo Dave's recommendation

Sent from a slightly less silly keyboard.

On Mar 30, 2012, at 1:13, Benson Schliesser <[email-masked]> wrote:

> Dear NVO3 Participants -
> 
> Based on feedback received during the BOF on Wednesday, we have made a few minor updates to the draft charter text. Specifically, we added a sentence to the 4th paragraph that explicitly says NVO3 will consider maintenance, troubleshooting, and security requirements. We also added dates to the milestones.
> 
> Please note that this is still a draft charter, pending IESG approval. My understanding is that the IESG will be discussing this at their meeting tomorrow and making a decision (on whether to form an NVO3 WG) at their 12-April meeting.
> 
> Cheers,
> -Benson
> 
> 
> NVO3: Network Virtualization Overlays
> 
> Support for multi-tenancy has become a core requirement of data centers (DCs), especially in the context of data centers supporting virtualized servers known as virtual machines (VMs).  A key multi-tenancy requirement is traffic isolation, so that a tenant's traffic (and internal address usage) is not visible to any other tenant and does not collide with addresses used within the data center itself. Another key requirement is to support the placement and live migration of VMs anywhere within a data center, without being limited by DC network constraints such as the IP subnet boundaries of the underlying DC network. (Note: Although this charter uses the term VM throughout, the scope of NVO3 is intended to include non-virtualized end systems.)
> 
> This WG will develop an approach to multi-tenancy that does not rely on any underlying L2 isolation mechanisms to support multi-tenancy. In particular, the WG will develop an approach where multi-tenancy is provided at the IP layer using an encapsulation header that resides above IP. This approach should provide an Ethernet service. It may provide an IP service; an important goal is to develop a "layer agnostic" framework and architecture meeting data center requirements.
> 
> NVO3 will document the problem statement, the applicability, and an architectural framework for overlay networks within a data center environment. Within this framework, functional blocks will be defined to allow the dynamic attachment / detachment of VMs to the overlay network, the mapping of "inner" (tenant VM) addresses into "outer" (underlying network) addresses in order to tunnel a packet to the destination VM, and the migration of VMs within the network in a dynamic fashion. 
> 
> Based on this framework, the WG will develop requirements for both control plane protocol(s) and data plane encapsulation format(s), and perform a gap analysis of existing candidate mechanisms. In addition to functional and architectural requirements, NVO3 will develop maintenance, troubleshooting, and security requirements. Driven by the requirements and consistent with the gap analysis, the WG will document solutions consisting of one or more data plane(s) and control plane(s), as applicable. 
> 
> The WG will investigate interconnection and interoperation issues between overlays and VPNs to determine if any specific work is needed.
> 
> The WG will liaise with the appropriate external organizations including the IEEE, as appropriate. 
> 
> Milestones:
> 
> 0) Problem Statement (Informational RFC) - August 2012
> 
> 1) Framework document (Informational RFC) - August 2012
> 
> 2) Control plane requirements document (Informational RFC) - November 2012
> 
> 3) Data plane requirements document (Informational RFC) - November 2012
> 
> 4) Gap Analysis (Informational RFC) - November 2012
> 
> 5) Recharter for Standards Track solution document(s) as appropriate - November 2012
> 
> 
> 
> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3

---

From: David Meyer <[email-masked]>
Date: Mon, 2 Apr 2012 08:13:22 -0700

Excellent points Dave. Agreed and also agree the charter looks good.

Dave

On Fri, Mar 30, 2012 at 1:43 AM, Mcdysan, David E
<[email-masked]> wrote:
> Hi,
>
> This looks good, with one major comment.
>
> I suggest that the charter needs to better describe the scope because this
> will impact the virtual tenant namespace requrements. The text below could
> be interpreted as:
> 1) a single data center
> 2) a set of data centers under control of one administrative domain
> 3) multiple sets of data centers under control of multiple administrative
> domains
>
> 1 and 2 are probably easier, but if the scope could at least consider 3.
> Using definition 3 we may not have to introduce additional levels on
> indirection as alluded to in David Black's slides or the comment Kireeti
> made at the mike that this is reminiscent of inter-AS VPNs, which is
> complexity I suggest that we strive to avoid.
>
> Thanks,
>
> Dave
>
> On 3/29/12 7:13 PM, "Benson Schliesser" <[email-masked]> wrote:
>
>>Dear NVO3 Participants -
>>
>>Based on feedback received during the BOF on Wednesday, we have made a
>>few minor updates to the draft charter text. Specifically, we added a
>>sentence to the 4th paragraph that explicitly says NVO3 will consider
>>maintenance, troubleshooting, and security requirements. We also added
>>dates to the milestones.
>>
>>Please note that this is still a draft charter, pending IESG approval. My
>>understanding is that the IESG will be discussing this at their meeting
>>tomorrow and making a decision (on whether to form an NVO3 WG) at their
>>12-April meeting.
>>
>>Cheers,
>>-Benson
>>
>>
>>Support for multi-tenancy has become a core requirement of data centers
>>(DCs), especially in the context of data centers supporting virtualized
>>servers known as virtual machines (VMs).  A key multi-tenancy requirement
>>is traffic isolation, so that a tenant's traffic (and internal address
>>usage) is not visible to any other tenant and does not collide with
>>addresses used within the data center itself. Another key requirement is
>>to support the placement and live migration of VMs anywhere within a data
>>center, without being limited by DC network constraints such as the IP
>>subnet boundaries of the underlying DC network. (Note: Although this
>>charter uses the term VM throughout, the scope of NVO3 is intended to
>>include non-virtualized end systems.)
>>
>>This WG will develop an approach to multi-tenancy that does not rely on
>>any underlying L2 isolation mechanisms to support multi-tenancy. In
>>particular, the WG will develop an approach where multi-tenancy is
>>provided at the IP layer using an encapsulation header that resides above
>>IP. This approach should provide an Ethernet service. It may provide an
>>IP service; an important goal is to develop a "layer agnostic" framework
>>and architecture meeting data center requirements.
>>
>>NVO3 will document the problem statement, the applicability, and an
>>architectural framework for overlay networks within a data center
>>environment. Within this framework, functional blocks will be defined to
>>allow the dynamic attachment / detachment of VMs to the overlay network,
>>the mapping of "inner" (tenant VM) addresses into "outer" (underlying
>>network) addresses in order to tunnel a packet to the destination VM, and
>>the migration of VMs within the network in a dynamic fashion.
>>
>>Based on this framework, the WG will develop requirements for both
>>control plane protocol(s) and data plane encapsulation format(s), and
>>perform a gap analysis of existing candidate mechanisms. In addition to
>>functional and architectural requirements, NVO3 will develop maintenance,
>>troubleshooting, and security requirements. Driven by the requirements
>>and consistent with the gap analysis, the WG will document solutions
>>consisting of one or more data plane(s) and control plane(s), as
>>applicable.
>>
>>The WG will investigate interconnection and interoperation issues between
>>overlays and VPNs to determine if any specific work is needed.
>>

---

From: Miya Kohno <[email-masked]>
Date: Tue, 3 Apr 2012 21:41:41 +0900

Hi,

[!--snip]
>  A key multi-tenancy requirement is traffic isolation, so that a tenant's traffic (and internal address usage) is not visible to any other tenant and does not collide with addresses used *within the data center itself*. Another key requirement is to support the placement and live migration of VMs anywhere *within a data center* [snip--]

Sorry if I miss something, but does this mean to exclude
inter-datacenter from the scope? If so, what are reasons?
The underlying network could span multiple locations, couldn't it?

Thanks,
Miya

On Wed, Mar 28, 2012 at 2:11 AM, Benson Schliesser <[email-masked]> wrote:
> Dear NVO3 Participants -
>
> Please see the revised draft charter, below.  Based on mailing list feedback, the BOF chairs and Thomas have attempted to re-factor and simplify the text.
>
> Cheers,
> -Benson, Matthew, Thomas
>
>
> NVO3: Network Virtualization Overlays (Version 2012032705)
>
> Based on this framework, the WG will develop requirements for both control plane protocol(s) and data plane encapsulation format(s), and perform a gap analysis of existing candidate mechanisms. Driven by the requirements and consistent with the gap analysis, the WG will document solutions consisting of one or more data plane(s) and control plane(s), as applicable.
>
> 0) Problem Statement (Informational RFC)
>
> 1) Framework document (Informational RFC)
>
> 2) Control plane requirements document (Informational RFC)
>
> 3) Data plane requirements document (Informational RFC)
>
> 4) Gap Analysis (Informational RFC)
>
> 5) Recharter for Standards Track solution document(s) as appropriate
>
>
>

---

From: Randy Bush <[email-masked]>
Date: Tue, 03 Apr 2012 09:38:36 -0700

>>  A key multi-tenancy requirement is traffic isolation, so that a
>>  tenant's traffic (and internal address usage) is not visible to any
>>  other tenant and does not collide with addresses used *within the
>>  data center itself*. Another key requirement is to support the
>>  placement and live migration of VMs anywhere *within a data center*
>>  [snip--]
> 
> inter-datacenter from the scope?

as benson's requirement for true traffic isolation requires a whole
lotta real wires, you're gonna be running a lot of wires between the
various locations.

i basically think his requirements are in conflict with eachother, and
hence i hit delete.

randy

---

From: Randy Bush <[email-masked]>
Date: Tue, 03 Apr 2012 11:24:44 -0700

>>>>  other tenant
> It is not expressing a requirement for physical separation.

well then you are postulating a way to get 2g out of 1g pipe if they
do not want their traffic (bandwidth use in this case) to be perceived
by the other.  i must have missed the april fool mpls rfc on this.

randy

---

From: Chris Wright <[email-masked]>
Date: Tue, 3 Apr 2012 11:45:01 -0700

> by the other.  i must have missed the april fool mpls rfc on this.

Operators tend to understand the tradeoffs between overprovisioning and
non-work conserving scheduling and have SLAs which reflect that.

Is there specific wording in the current draft charter you think is
insufficient?

---

From: Chris Wright <[email-masked]>
Date: Tue, 3 Apr 2012 10:43:50 -0700

---

From: "PAULING, JOEL (JOEL)" <[email-masked]>
Date: Wed, 4 Apr 2012 02:36:00 +0200

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Chris Wright
Sent: Wednesday, 4 April 2012 5:44 a.m.
To: Randy Bush
Cc: [email-masked]; Miya Kohno; Benson Schliesser
Subject: Re: [nvo3] Updated Charter

Playing devils advocate here
I read this as - "requires encryption" if sharing the same wire. 

-JoelW

---

From: Larry Kreeger <[email-masked]>
Date: Tue, 03 Apr 2012 20:40:03 -0700

> I read this as - "requires encryption" if sharing the same wire.

I don't think Benson was trying leave his meaning to the reader.  I think it
means exactly what it says after the comma in the sentence ", so that a
tenant's traffic (and internal address usage) is not visible to any other
tenant and does not collide with addresses used within the data center
itself".

So, if my VMs within my own Virtual Data Center send a packet to 10.1.2.3
they do not accidentally show up on one of your VMs in your Virtual Data
Center which is also using 10.1.2.3.  If the data center network itself is
secure and keeps the traffic isolated from the VMs there is no need for
either separate wires or encryption to perform this isolation.  It can be
performed by using separate overlay networks running over a common shared
underlying network.

 - Larry

---

From: Chris Wright <[email-masked]>
Date: Tue, 3 Apr 2012 20:42:34 -0700

* PAULING, JOEL (JOEL) ([email-masked]) wrote:
> * Chris Wright ([email-masked]) wrote:

It's meant to convey the same isolation properties provided by vlans.
The overlay encap/decap point has the same trust as a switch forwarding
between trunk and access ports.  Do you think it needs better wording?

thanks,
-chris

---

From: Stewart Bryant <[email-masked]>
Date: Wed, 04 Apr 2012 11:52:30 +0100

Dave McDyson asked about three classes of
connectivity:

1) a single data center
2) a set of data centers under control of one administrative domain
3) multiple sets of data centers under control of multiple 
administrative domains

Which of these do we *need* to address in NVO3?

Once we have decided which of the above we need to address
we can specify the requirements scope appropriately
and if we decide we need to address 2 and or 3 we can
add a deliverable.

My concern is that we should only add anything on inter-DC
that impacts the internal DC design. The more
we can cleanly separate internal and external communication
the greater the chance that we can construct a small set
of simple clear work programmes that will complete in
a timely manner.

I think that we need to make it clear that how
any external communications requirements
are addressed is outside the scope of the current charter
discussion, and that there should be no expectation
set as to where the work needed to address those
requirements will be undertaken.

Stewart

---

From: David Freedman <[email-masked]>
Date: Wed, 04 Apr 2012 12:13:01 +0100

> Which of these do we *need* to address in NVO3?

I have to agree with Stewart here, I think the scope should be
constrained to initially the intra-DC (and any inter-DC parts
which impact intra-DC) , we can then start achieving and choose
to expand the charter when we feel capable.

Dave.

---

From: Shane Amante <[email-masked]>
Date: Wed, 4 Apr 2012 09:17:30 -0600

Benson, All,

I have a question re: the charter.  Please see below.

> Support for multi-tenancy has become a core requirement of data centers (DCs), especially in the context of data centers supporting virtualized servers known as virtual machines (VMs).  A key multi…

When I read the above text, ("... without being limited by DC network constraints such as the IP subnet boundaries of the underlying DC network"), and think about the NVO3 problem space in general, I haven't seen any discussions (yet?) around accounting for something like IGP metrics (something related to "distance", "hops", etc.) within whatever NVO3 defines (or, re-uses) as its Control Plane, which ultimately will translate into forwarding decisions by various NVO3 end-points.  Specifically, from what I've read, so far, about NVO3 it seems to assume (?) a completely flat topology within/inside the "overlay" and there is no notion of feeding 'hints' from the underlying physical topology into the 'overlay' topology.  And, to take that a step further, how those hints are consumed by NVO3 encap/de-encap points to try to make "optimal" routing/forwarding decisions?

The reason why I raise the above concern is, primarily, /within/ a Layer-2 domain (i.e.: VLAN) there is no notion of "distance", from the perspective of hosts/end-points attached to that subnet.  Unfortunately, the further that Layer-2 domain is stretched (either across a DataCenter and, even moreso, across a WAN connecting DataCenters and/or to the Internet) you need to understand 'distance', because of latency & cost reasons, i.e.: you prefer traffic to stay *within* the DataCenter, if possible.  One result of not knowing/understanding 'distance' is the "classic" problem of traffic tromboning, particularly after, say, a VM (or, several VM's) is [live] migrated from DC #1 in SJO to DC #2 in NYC (or, further afield).  In that case, the VM most likely has it's "default GW", to exit the IP subnet, pointed at the same next-hop IP address that physically still resides in SJO. Ideally, you'd want the VM's idea of its "default GW" to move with it so its traffic can exit the IP subnet in NYC to reduce latency & cost.

So, IMO, it would be helpful to understand here on the list, and in the charter, if NVO3 'solutions' care about optimizing traffic flows that are transmitted by one VM and are required to traverse an IP subnet boundary, for a variety of reasons, e.g.:
- traffic is required to pass through a "policy enforcement point", such as a firewall/NAT/middle-box because:
  a) it's traversing between two 'tenants'; and/or,
  b) one VM is consuming a common service/application from the DC operator
- traffic is coming from a VM and destined toward the Internet
- traffic is coming from a VM and destined toward an (IP)VPN, which may have corporate offices spread all over the world that are using services provided by those VM's

Thanks,

-shane

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 04 Apr 2012 19:36:18 +0200

Hello Shane,

My view is that VM to VM/Internet/Appliance crossing subnet boundary 
should be made optimal and not necessarily optimal in regards to IGP 
metric.

Said this if we assume that we communicate over L3/IP I see no reason 
why already solutions provided by IETF in other working groups could not 
be directly applicable without any relation to NVO3.

Examples:

* BGP can compute optimal exit points for tunneled VM to XYZ paths

* VM can query DNS which works in concert with ALTO/NPS and return 
closest (based on what set of rules one would consider as closest) 
target IP address for a given service or destination site

* IGP could business as normal ECMP forward encapsulated VM to XYZ traffic.

So I am not clear where would one see within NVO3 charter to consider 
such set of solutions to optimize traffic flows. IMHO I think we should 
leave those out of scope as they seems to be already solved by other 
tools and by different layers.

Regards,
R.

> The reason why I raise the above concern is, primarily, /within/ a Layer-2 domain (i.e.: VLAN) there is no notion of "distance", from the perspective of hosts/end-points attached to that subnet.  Unfortunately, the further that Layer-2 domain is stretched (either across a DataCenter and, even moreso, across a WAN connecting DataCenters and/or to the Internet) you need to understand 'distance', because of latency&  cost reasons, i.e.: you prefer traffic to stay *within* the DataCenter, if possible.  One result of not knowing/understanding 'distance' is the "classic" problem of traffic tromboning, particularly after, say, a VM (or, several VM's) is [live] migrated from DC #1 in SJO to DC #2 in NYC (or, further afield).  In that case, the VM most likely has it's "default GW", to exit the IP subnet, pointed at the same next-hop IP address that physically still resides in SJO. Ideally, you'd want the VM's idea of its "default GW" to move with it so its traffic can exit the IP su
 bn
>   et in NYC to reduce latency&  cost.
>

---

From: Thomas Narten <[email-masked]>
Date: Wed, 04 Apr 2012 15:15:59 -0400

Hi Shane.

Shane Amante <[email-masked]> writes:

> When I read the above text, ("... without being limited by DC
>  network constraints such as the IP subnet boundaries of the
>  underlying DC network"), and think about the NVO3 problem space in
>  general, I haven't seen any discussions (yet?) around accounting
>  for something like IGP metrics (something related to "distance",
>  "hops", etc.) within whatever NVO3 defines (or, re-uses) as its
>  Control Plane, which ultimately will translate into forwarding
>  decisions by various NVO3 end-points.  Specifically, from what I've
>  read, so far, about NVO3 it seems to assume (?) a completely flat
>  topology within/inside the "overlay" and there is no notion of
>  feeding 'hints' from the underlying physical topology into the
>  'overlay' topology.  And, to take that a step further, how those
>  hints are consumed by NVO3 encap/de-encap points to try to make
>  "optimal" routing/forwarding decisions?

I think we are best off not including the above in NVO3. What you
refer to is the common problem of having "large" subnets spanning
large "distances", where one wants to factor in that some nodes
attached to the subnet are "closer" to each other than they are to
others. 

The problem with that is that we just don't know how to do it
effectively at the IP level and we have not to date come up with a
general purpose way of doing that.  At least as far as I know. Or do
you have some approaches in mind?

In other words, this isn't an NVO3 problem per se, this is an inherent
limitation in the IP subnet model.

Including "distance" in the scope of NVO3 would seem to me to add
significant complexity to our scope. I would not feel comfortable
adding that (at least not yet) without having a much better idea of
the implications, or how to scope the problem.

Besides, migrating VMs between geographically separated DCs poses a
number of challenges already. The one you mention is but one of
several.

> The reason why I raise the above concern is, primarily, /within/ a
>  Layer-2 domain (i.e.: VLAN) there is no notion of "distance", from
>  the perspective of hosts/end-points attached to that subnet.
>  Unfortunately, the further that Layer-2 domain is stretched (either
>  across a DataCenter and, even moreso, across a WAN connecting
>  DataCenters and/or to the Internet) you need to understand
>  'distance', because of latency & cost reasons, i.e.: you prefer
>  traffic to stay *within* the DataCenter, if possible.  One result
>  of not knowing/understanding 'distance' is the "classic" problem of
>  traffic tromboning, particularly after, say, a VM (or, several
>  VM's) is [live] migrated from DC #1 in SJO to DC #2 in NYC (or,
>  further afield).  In that case, the VM most likely has it's
>  "default GW", to exit the IP subnet, pointed at the same next-hop
>  IP address that physically still resides in SJO. Ideally, you'd
>  want the VM's idea of its "default GW" to move with it so its
>  traffic can exit the IP subn et in NYC to reduce latency & cost.

Well, but that is only for the outgoing path... How do you ensure that
return traffic doesn't enter back into the overlay at the other DC?
There are a number of related problems lurking here that go beyond
just the NVO3 part, I think.

> So, IMO, it would be helpful to understand here on the list, and in
> the charter, if NVO3 'solutions' care about optimizing traffic flows
> that are transmitted by one VM and are required to traverse an IP
> subnet boundary, for a variety of reasons, e.g.:

> - traffic is required to pass through a "policy enforcement point",
>   such as a firewall/NAT/middle-box because:

This will be something that has to be done, but it may be done
completely outside of NVO3. E.g., you can have a single default router
for the overlay instance, and all traffic would go through it (and the
firewall beyond). It is unclear to me NVO3 has to know this is going
on.
    
>   a) it's traversing between two 'tenants'; and/or,

By definition, we have been saying that if you go from one tenant to
another, that requires that you  (conceptually) exit the overlay and
re-enter it. So, again, this could be outside of NVO3's scope.

>   b) one VM is consuming a common service/application from the DC

Can you expand on what this means? A VM residing within an overlay
will certainly (at times) interact with servers outside the
overlay. But they can go in/out of a default outer for that (in the
simple case). Are you thinking of something more than that?
    
> - traffic is coming from a VM and destined toward the Internet

To start, I would keep things simple and have one exit router for the
overlay. Everything would then funnel through that point, which can do
whatever FW/ACL/etc. type stuff is appropriate.

> - traffic is coming from a VM and destined toward an (IP)VPN, which
>   may have corporate offices spread all over the world that are
>   using services provided by those VM's

Not sure of the details of this case. Where is the overlay boundary?
Are the VMs both inside an overlay instance, or is one outside? (And
if the latter, isn't this  similar to previous cases?)

Thomas

---

From: Thomas Narten <[email-masked]>
Date: Wed, 04 Apr 2012 16:44:38 -0400

Hi Stewart.

> 1) a single data center

Absolutely. If we don't do this, what are we here for? :-)

> 2) a set of data centers under control of one administrative domain

I think we need to do this as well. Operators will want this.

But I will be honest and say this could be the most challenging piece
of work for this WG.

If we take a (simple) approach that works for a single DC (say in one
building), and then try to make it work across geographically
separated DCs, we will surely have problems making it work. Latencies
will be too high, performance may not be adequate, etc.. Or, if the
connection between DCs goes down, we may find that local stuff stops
working (i.e., it impacts VM-VM communication between VMs that are in
the same DC). These sorts of things could easily become show stoppers
for multi-DC deployments.

I expect we'll have to do a pretty careful analysis of requirements
for the single administrative domain/multiple DCs before we know the
scope of what is  required. (That is IMO one of the work items in the
proposed charter.)

And I believe IETF role would be that in the multi-DC case you want to
have separate instances of an "oracle" running at each DC that has the
knowledge about the address mappings. The local tunnel endpoints would
query (or interact with) the local "oracle" at its DC to get the
necessary mappings it needs.

Some sort of on-the-wire protocol is presumably need for synchronizing
the differning instances of oracle DBs. This could be something based
on BGP, something new, or who knows. I don't think we really have a
good idea of the requirements and hence what protocol might be
appropriate.

> 3) multiple sets of data centers under control of multiple 

I fear that folk will want this (think mergers/aquistions and other
non technical motivations). But I'm very hesitant to try and tackle
this up front. At least not without limiting the scope to something
manageable. I'd say wait until later when the WG is off and running
and has developed a good track record and we better understand what
some of the issues would be.

> My concern is that we should only add anything on inter-DC that
> impacts the internal DC design. The more we can cleanly separate
> internal and external communication the greater the chance that we
> can construct a small set of simple clear work programmes that will
> complete in a timely manner.

I agree.

> I think that we need to make it clear that how any external
> communications requirements are addressed is outside the scope of
> the current charter discussion, and that there should be no
> expectation set as to where the work needed to address those

I think how any of the requirements are addressed is effectively out
of the current charter. The current charter has no protocol or
solution deliverables. Just requirements/framework/gap analysis.

Thomas

---

From: "Balus, Florin Stelian (Florin)" <[email-masked]>
Date: Wed, 4 Apr 2012 16:14:25 -0500

We can simplify and deal only with 1) if we assume that VPN technology will be used outside the data center. There is a lot of VPN Inter-domain stuff (Inter-AS/Carrier of Carrier etc) that was developed for 2) and 3).

I suggest we add some text to the charter to clarify the scope and maybe a specific deliverable for VPN Interoperability requirements. Some example below:

"The focus of this WG will be Intra-DC use cases. Initially VPN Interoperability with NVO3 technology will be used to interconnect a set of data centers under the control of one administrative domain or multiple sets of data centers under control of multiple administrative domains. 

[..]

Milestones:

Insert this one here> 4) VPN interoperability requirements (Informational RFC)

5) Gap Analysis (Informational RFC)

6) Recharter for Standards Track solution document(s) as appropriate

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Stewart Bryant
> Sent: Wednesday, April 04, 2012 3:53 AM
> To: [email-masked]
> Once we have decided which of the above we need to address we can
> specify the requirements scope appropriately and if we decide we need
> to address 2 and or 3 we can add a deliverable.
> 
> My concern is that we should only add anything on inter-DC that impacts
> the internal DC design. The more we can cleanly separate internal and
> external communication the greater the chance that we can construct a
> small set of simple clear work programmes that will complete in a
> timely manner.
> 
> communications requirements are addressed is outside the scope of the
> current charter discussion, and that there should be no expectation set
> as to where the work needed to address those requirements will be
> undertaken.
> 
> Stewart
> 
> 
> 
> 
>

---

From: Thomas Narten <[email-masked]>
Date: Wed, 04 Apr 2012 17:28:36 -0400

"Balus, Florin Stelian (Florin)" <[email-masked]> writes:

> We can simplify and deal only with 1) if we assume that VPN
> technology will be used outside the data center. There is a lot of
> VPN Inter-domain stuff (Inter-AS/Carrier of Carrier etc) that was
> developed for 2) and 3).

> I suggest we add some text to the charter to clarify the scope and
> maybe a specific deliverable for VPN Interoperability
> requirements. Some example below:

> "The focus of this WG will be Intra-DC use cases. Initially VPN
>  Interoperability with NVO3 technology will be used to interconnect
>  a set of data centers under the control of one administrative
>  domain or multiple sets of data centers under control of multiple
>  administrative domains.

This would seem to rule out the rather straightforward scenario where
two DCs are under the same administrative domain, but are
interconnected by some (existing) VPN technology, of which we don't
care what the details are. The overlay just tunnels end-to-end and
couldn't care less about the existance of a VPN interconnecting parts
of the network.

I think where the NVO3/VPN interaction gets potentially interesting is
when you can't just run NVO3 end-to-end and have to do some sort of
translation/mapping/interoperation at the boundaries between two
administrative domains.  David Black talked to these cases at the BOF.

This would correspond to case 3 where there are different
administrative domains. I would prefer not tackling that problem in
NVO3 now. Perhaps later, when we better understand what might be
needed and have gotten a good start on the other core work that needs
to get done.

Thomas

---

From: Shane Amante <[email-masked]>
Date: Wed, 4 Apr 2012 20:54:53 -0600

Hi Thomas,

FWIW, the reason I bring up this issue is that I have encountered this exact situation when I contributed to a redesign of DataCenters recently.  See below.

On Apr 4, 2012, at 1:15 PM, Thomas Narten wrote:
> general purpose way of doing that.  At least as far as I know.

I find the above somewhat ironic given that this is the IETF, where we've defined both ARP and IPv6 Router + Neighbor Discovery ...

> Or do

As I look at:
http://tools.ietf.org/html/draft-kreeger-nvo3-overlay-cp-00
It seems fairly clear that NVO3 is, eventually, headed in a map & encaps direction, (cf: Section 3.1, "Inner to Outer Address Mapping").  If this is true, then it would seem wise to think about requirements for additional attributes that can/should be included in the requisite NVO3 mapping protocol(s).  For example, it's conceivable that a Hypervisor, implementing NVO3, could dynamically learn or be statically programmed with an idea as to its physical and/or logical location, (e.g.: NYC1 DC).  In addition, the "exit points" from the NVO3 overlay (would these exist on routers or switches or would it be another VM/server acting as a "router"?) could also dynamically learn and/or be programmed with similar information, since they need to (at a bare minimum) implement a encaps/de-encaps function to get traffic in/out of the overlay, right?  Then, when "mapping" is performed in/by the NVO3 control plane for Inner <-> Outer Address Mapping the answer handed back as a result of that function /could/ be influenced by additional criteria so that you get back something like: "here is the topologically closest exit point from the NVO3 overlay".  Ultimately, the NVO3 "FIB" gets programmed with that information and, perhaps, propagated further into the VM's ARP and/or ND tables ...

> limitation in the IP subnet model.

True.  However, NVO3 appears headed down a road of needing and, eventually, defining (or, re-using?) a "mapping" function.  If that is the case, then there's room for 'enhancement' above and beyond standard ARP & ICMPv6 RD/ND ...

> the implications, or how to scope the problem.

Take a look at the above and see if that satisfies your concerns.

> number of challenges already. The one you mention is but one of

Yes, but the problem I point out is quite well-known, thus would seem wise to at least consider.

> just the NVO3 part, I think.

I can use a variety of techniques in IP routing to influence the return traffic to go "the right way" back into the DC, e.g.: announcing longer prefixes into IGP/BGP to ensure that 'return traffic' is optimally routed back.  The key problem is, without running a dynamic routing protocol or using static routes on the server/VM, I have no way of influencing traffic that is transmitted *from* the server/VM to a optimal exit point.

> firewall beyond). It is unclear to me NVO3 has to know this is going

Yikes.  I hope we're NOT designing for a single point-of-failure, out of the NVO3 overlay, from Day 1.  It seems we should be considering a minimum of 2 x exit points from the overlay from Day 1 or, at the very least, the ability for some type of *extremely* fast fail-over from one 'physical' piece of hardware acting as the primary exit point to a second piece of hardware acting as a secondary/standby exit point from the overlay.

> Can you expand on what this means?

Let's say I'm an Amazon customer and have my own IP subnet.  At times, I may want to use their S3 or other services they provide.  The key point is I need my servers to talk to their S3 service, but I do not want their other customers to be able to see and/or talk to my servers, for security reasons.

> A VM residing within an overlay

> whatever FW/ACL/etc. type stuff is appropriate.

Again, we need to be careful to include some form(s) of redundancy in the plans for NVO3, specifically as it relates to traffic that will enter/exit the overlay.  The way I see it there are three or four things that should be considered:
a)  No redundancy whatsoever for traffic entering/leaving the NVO3 overlay -- good luck trying to sell this to Mega/Tera/Yotta DataCenter customers ...
b)  Active/Backup.  One exit is active at any given time; only when it dies does the "backup" take over (instantaneously).
c)  Active/Active.  Two, or more, exit points are active at any given time.  This is the scenario I think we should aim for, because it can be used to provide two physically diverse exit points from the same, physical DataCenter as well as from geographically diverse DataCenters.  Ultimately, this allows the possibility for traffic to enter/exit through the most optimal "active" entry/exit point.

> if the latter, isn't this  similar to previous cases?)

VM's are inside the overlay, acting as application servers, but they are talking to clients (think: desktops/laptops/tablets/etc.) that are sitting on a Corp WAN/LAN.  The latter, clients, are not aware of NVO3 and are speaking "regular" IP toward the DataCenter(s) to access various applications/resources.  It's the former, VM's inside the NVO3 overlay, that I care about in the context of this conversation, reqmt's and design space.

Thanks,

-shane

---

From: Xuxiaohu <[email-masked]>
Date: Thu, 5 Apr 2012 09:40:59 +0000

> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代表 Shane
> Amante
> 发送时间: 2012年4月5日 10:55
> 收件人: Thomas Narten
> 抄送: Matthew (Matthew) Bocci; [email-masked]; Benson Schliesser
> 主题: Re: [nvo3] Updated Charter
> 
> Hi Thomas,
> 
> FWIW, the reason I bring up this issue is that I have encountered this exact
> situation when I contributed to a redesign of DataCenters recently.  See
> below.
> 
> I find the above somewhat ironic given that this is the IETF, where we've
> defined both ARP and IPv6 Router + Neighbor Discovery ...
> 
> 
> > Or do
> It seems fairly clear that NVO3 is, eventually, headed in a map & encaps
> direction, (cf: Section 3.1, "Inner to Outer Address Mapping").  If this is true,
> then it would seem wise to think about requirements for additional attributes
> that can/should be included in the requisite NVO3 mapping protocol(s).  For
> example, it's conceivable that a Hypervisor, implementing NVO3, could
> dynamically learn or be statically programmed with an idea as to its physical
> and/or logical location, (e.g.: NYC1 DC).  In addition, the "exit points" from
> the NVO3 overlay (would these exist on routers or switches or would it be
> another VM/server acting as a "router"?) could also dynamically learn and/or
> be programmed with similar information, since they need to (at a bare
> minimum) implement a encaps/de-encaps function to get traffic in/out of the
> overlay, right?  Then, when "mapping" is performed in/by the NVO3 control
> plane for Inner <-> Outer Address Mapping the answer handed back as a result
> of tha
>  t function /could/ be influenced by additional criteria so that you get back
> something like: "here is the topologically closest exit point from the NVO3
> overlay".  Ultimately, the NVO3 "FIB" gets programmed with that information
> and, perhaps, propagated further into the VM's ARP and/or ND tables ...
> 
> 
> True.  However, NVO3 appears headed down a road of needing and,
> eventually, defining (or, re-using?) a "mapping" function.  If that is the case,
> then there's room for 'enhancement' above and beyond standard ARP &
> ICMPv6 RD/ND ...
> 
> 
> I can use a variety of techniques in IP routing to influence the return traffic to
> go "the right way" back into the DC, e.g.: announcing longer prefixes into
> IGP/BGP to ensure that 'return traffic' is optimally routed back.  The key
> problem is, without running a dynamic routing protocol or using static routes
> on the server/VM, I have no way of influencing traffic that is transmitted
> *from* the server/VM to a optimal exit point.

Hi Shane,

Maybe the host route based overlay (e.g. IP over IP) is the best candidate solution to addressing the route optimization issue caused by subnet extension. 

Alternatively, those L2 overlays (i.e., MAC over IP) with control-plane MAC learning capability could also be considered. With these overlays, the outbound traffic would be forwarded to the nearest default gateway router. Of course, to ensure the return traffic to travel through the right default gateway router, you still need to make the default gateway routers to advertise host routes to the outside world. 

Best regards,
Xiaohu

> Yikes.  I hope we're NOT designing for a single point-of-failure, out of the
> NVO3 overlay, from Day 1.  It seems we should be considering a minimum of 2
> x exit points from the overlay from Day 1 or, at the very least, the ability for
> some type of *extremely* fast fail-over from one 'physical' piece of hardware
> acting as the primary exit point to a second piece of hardware acting as a
> secondary/standby exit point from the overlay.
> 
> 
> Let's say I'm an Amazon customer and have my own IP subnet.  At times, I
> may want to use their S3 or other services they provide.  The key point is I
> need my servers to talk to their S3 service, but I do not want their other
> customers to be able to see and/or talk to my servers, for security reasons.
> 
> 
> Again, we need to be careful to include some form(s) of redundancy in the plans
> for NVO3, specifically as it relates to traffic that will enter/exit the overlay.
> The way I see it there are three or four things that should be considered:
> a)  No redundancy whatsoever for traffic entering/leaving the NVO3 overlay --
> good luck trying to sell this to Mega/Tera/Yotta DataCenter customers ...
> b)  Active/Backup.  One exit is active at any given time; only when it dies
> does the "backup" take over (instantaneously).
> c)  Active/Active.  Two, or more, exit points are active at any given time.
> This is the scenario I think we should aim for, because it can be used to provide
> two physically diverse exit points from the same, physical DataCenter as well
> as from geographically diverse DataCenters.  Ultimately, this allows the
> possibility for traffic to enter/exit through the most optimal "active" entry/exit
> point.
> 
> 
> VM's are inside the overlay, acting as application servers, but they are talking
> to clients (think: desktops/laptops/tablets/etc.) that are sitting on a Corp
> WAN/LAN.  The latter, clients, are not aware of NVO3 and are speaking
> "regular" IP toward the DataCenter(s) to access various applications/resources.
> It's the former, VM's inside the NVO3 overlay, that I care about in the context
> of this conversation, reqmt's and design space.
> 
> Thanks,
> 
> -shane

---

From: Thomas Narten <[email-masked]>
Date: Thu, 05 Apr 2012 12:11:40 -0400

Shane,

> FWIW, the reason I bring up this issue is that I have encountered
> this exact situation when I contributed to a redesign of DataCenters
> recently.  See below.

I'm not saying it can't or won't happen. I'm saying this is a
relatively hard problem to solve, and just wishing we had a solution
shouldn't be enough to make it a charter item for NVO3.

Furthermore, this is arguably a general IP issue and not specific to
NVO3. That raises fundamental questions about whether NVO3 is the
right place to attempt a solution. Presumably we'd want general
solutions that apply also in non-NVO3 scenarios.

> I find the above somewhat ironic given that this is the IETF, where
>  we've defined both ARP and IPv6 Router + Neighbor Discovery ...

There are plenty of examples of problems where we wish we had a
solution, but there isn't necessarily one... No matter how much we'd
like one...

> > Or do you have some approaches in mind?

> It seems fairly clear that NVO3 is, eventually, headed in a map &
> encaps direction, (cf: Section 3.1, "Inner to Outer Address
> Mapping").

Absolutely. This is a given. That is what the data plane does.

> If this is true, then it would seem wise to think about
> requirements for additional attributes that can/should be included
> in the requisite NVO3 mapping protocol(s).  For example, it's
> conceivable that a Hypervisor, implementing NVO3, could dynamically
> learn or be statically programmed with an idea as to its physical
> and/or logical location, (e.g.: NYC1 DC).  In addition, the "exit
> points" from the NVO3 overlay (would these exist on routers or
> switches or would it be another VM/server acting as a "router"?)
> could also dynamically learn and/or be programmed with similar
> information, since they need to (at a bare minimum) implement a
> encaps/de-encaps function to get traffic in/out of the overlay,
> right?  Then, when "mapping" is performed in/by the NVO3 control
> plane for Inner <-> Outer Address Mapping the answer handed back as
> a result of that function /could/ be influenced by additional
> criteria so that you get back something like: "here is the
> topologically closest exit point from the NVO3 overlay".
> Ultimately, the NVO3 "FIB" gets programmed with that information

For the special case of a router, I can imagine optimizations. But
what about just using VRRP? There can be multiple instances of VRRP,
and it is (arguably) VRRP's job to figure out how to the optimal path
selection.  I'm not familiar with VRRP's details with regards to load
balancing, but it does allow for load balancing. That might be a
better place to handle the router exit selection problem...

> eventually, defining (or, re-using?) a "mapping" function.  If that
> is the case, then there's room for 'enhancement' above and beyond
> standard ARP & ICMPv6 RD/ND ...

The problem I see with this approach is that the mapping is based on
IP addresses. Given a target IP address, the mapping system presumably
has no information about the kind of service residing at that address
and whether or not there are multiple instances of such a service.

I gather you are suggesting that the mapping system should be
augmented with such information, so that the mapping function can be
smarter about what it returns.

This would certainly add to the complexities the mapping system has to
deal with.

Are you willing to scope the problem to the one case of wanting to
optimize the selection of exit routers? Or do you see additional
places where such optimizations would be desireable?

> I can use a variety of techniques in IP routing to influence the
> return traffic to go "the right way" back into the DC, e.g.:
> announcing longer prefixes into IGP/BGP to ensure that 'return
> traffic' is optimally routed back.  The key problem is, without
> running a dynamic routing protocol or using static routes on the
> server/VM, I have no way of influencing traffic that is transmitted

Understood.

> Yikes.  I hope we're NOT designing for a single point-of-failure,
> out of the NVO3 overlay, from Day 1.  It seems we should be
> considering a minimum of 2 x exit points from the overlay from Day 1
> or, at the very least, the ability for some type of *extremely* fast
> fail-over from one 'physical' piece of hardware acting as the
> primary exit point to a second piece of hardware acting as a

Don't we have existing techniques already for dealing with router
failover?  (e.g., VRRP).

And if you have multiple exit points, you need to deal with the
multiple (possibly stateful) FW issues (in some cases). Not saying
that can't be done, just that it may not be as simple as having
multiple exit routers.

> Let's say I'm an Amazon customer and have my own IP subnet.  At
> times, I may want to use their S3 or other services they provide.
> The key point is I need my servers to talk to their S3 service, but
> I do not want their other customers to be able to see and/or talk to
> my servers, for security reasons.

Not sure this is a new problem to NVO3. Don't you have the same
scenario today using (say) VLANs? How does NVO3 change this scenario?

> Again, we need to be careful to include some form(s) of redundancy
>  in the plans for NVO3, specifically as it relates to traffic that
>  will enter/exit the overlay.  The way I see it there are three or
>  four things that should be considered:
  
> a)  No redundancy whatsoever for traffic entering/leaving the NVO3
> overlay -- good luck trying to sell this to Mega/Tera/Yotta
> DataCenter customers ...

OK, but at a first cut, I see nor reason why one cannot have multiple
routers on a overlay given instance.

> b)  Active/Backup.  One exit is active at any given time; only when
> it dies does the "backup" take over (instantaneously).

OK.

> c)  Active/Active.  Two, or more, exit points are active at any
> given time.  This is the scenario I think we should aim for, because
> it can be used to provide two physically diverse exit points from
> the same, physical DataCenter as well as from geographically diverse
> DataCenters.  Ultimately, this allows the possibility for traffic to
> enter/exit through the most optimal "active" entry/exit point. 

OK.

I'm interested what others think about the need to add these to the
NVO3 requirements.

> VM's are inside the overlay, acting as application servers, but they
> are talking to clients (think: desktops/laptops/tablets/etc.) that
> are sitting on a Corp WAN/LAN.  The latter, clients, are not aware
> of NVO3 and are speaking "regular" IP toward the DataCenter(s) to
> access various applications/resources.  It's the former, VM's inside
> the NVO3 overlay, that I care about in the context of this
> conversation, reqmt's and design space.

Understood.

To summarize, what you are saying about all these cases is you want to
have multiple exit/entry points into a specific overlay, and you want
NVO3 to help make sure the "closest" exit is used when possible.
Right?

Thomas

---

From: "Stiliadis, Dimitrios (Dimitri)" <[email-masked]>
Date: Thu, 5 Apr 2012 11:14:36 -0500

Thomas:

The solution space if we do as below can be completely different.

> for multi-DC deployments.

[Dimitri] If we are to produce a solution for (2), then I would argue
that we should just do that, since then (1) is just a subset of (2).
Indeed, I can see cases where the "wrong" solution for (1) can make
(2) almost impossible to deal with. On the opposite, solving (2)
would also solve (1) with DC=1.

For example the state space for (2) can explode to much higher numbers
than (1). There have been scalability numbers quoted and methods 
for pre-populating routes in hypervisors that can scale to large number
of VMs (I believe that Igor mentioned a number like 400K/DC). 
Pre-populating routes for multiple DCs would raise this number 
into several millions and obviously such a pre-populating approach
won't make much sense.

Solving the scalability problems of multiple DCs, would immediately bring 
out the need for route aggregation  methods, caching, etc., that would lead to 
a completely different solution space than just doing (1).

I believe it is extremely important to make this decision early in the
process.

---

From: Shane Amante <[email-masked]>
Date: Thu, 5 Apr 2012 10:23:08 -0600

Hi Thomas,

Cutting to the chase, hopefully without losing too much context.

On Apr 5, 2012, at 10:11 AM, Thomas Narten wrote:
[--snip--]
> smarter about what it returns.

Correct.

> optimize the selection of exit routers?

Yes.

> Or do you see additional

I can't think of any at the moment.

[--snip--]

> NVO3 to help make sure the "closest" exit is used when possible.

Yes.

Thanks!

-shane

---

From: "Balus, Florin Stelian (Florin)" <[email-masked]>
Date: Thu, 5 Apr 2012 12:05:26 -0500

Thomas,

You asked below what others think about Shane use cases/requirements. 

I think what he described in his email are DC networking issues we need at least to discuss in NVO3 data plane/control plane requirements and gap analysis. In particular the requirements to address traffic tromboning to/from NVO3 and resiliency at NVO3 gateways came up a number of times in the past in various IETF WG sessions. We can see after that where/how exactly is the best place/way to address them. But we should not just ignore them.

My 2 cents...

Florin

> Thomas Narten
> Sent: Thursday, April 05, 2012 9:12 AM
> To: Shane Amante
> Cc: Bocci, Matthew (Matthew); [email-masked]; Benson Schliesser
> this
> > exact situation when I contributed to a redesign of DataCenters
> > recently.  See below.
> 
> NVO3. That raises fundamental questions about whether NVO3 is the right
> place to attempt a solution. Presumably we'd want general solutions
> that apply also in non-NVO3 scenarios.
> 
> > If this is true, then it would seem wise to think about requirements
> > for additional attributes that can/should be included in the
> requisite
> > NVO3 mapping protocol(s).  For example, it's conceivable that a
> > Hypervisor, implementing NVO3, could dynamically learn or be
> > statically programmed with an idea as to its physical and/or logical
> > location, (e.g.: NYC1 DC).  In addition, the "exit points" from the
> > NVO3 overlay (would these exist on routers or switches or would it be
> > another VM/server acting as a "router"?) could also dynamically learn
> > and/or be programmed with similar information, since they need to (at
> > a bare minimum) implement a encaps/de-encaps function to get traffic
> > in/out of the overlay, right?  Then, when "mapping" is performed
> in/by
> > the NVO3 control plane for Inner <-> Outer Address Mapping the answer
> > handed back as a result of that function /could/ be influenced by
> > additional criteria so that you get back something like: "here is the
> > Ultimately, the NVO3 "FIB" gets programmed with that information and,
> > perhaps, propagated further into the VM's ARP and/or ND tables ...
> 
> For the special case of a router, I can imagine optimizations. But what
> about just using VRRP? There can be multiple instances of VRRP, and it
> is (arguably) VRRP's job to figure out how to the optimal path
> balancing, but it does allow for load balancing. That might be a better
> place to handle the router exit selection problem...
> 
> > > In other words, this isn't an NVO3 problem per se, this is an
> > > inherent limitation in the IP subnet model.
> 
> is
> > the case, then there's room for 'enhancement' above and beyond
> The problem I see with this approach is that the mapping is based on IP
> addresses. Given a target IP address, the mapping system presumably has
> no information about the kind of service residing at that address and
> whether or not there are multiple instances of such a service.
> 
> I gather you are suggesting that the mapping system should be augmented
> with such information, so that the mapping function can be smarter
> about what it returns.
> 
> optimize the selection of exit routers? Or do you see additional places
> where such optimizations would be desireable?
> 
> > >> the charter, if NVO3 'solutions' care about optimizing traffic
> > >> flows that are transmitted by one VM and are required to traverse
> > >> an IP subnet boundary, for a variety of reasons, e.g.:
> > >
> > > completely outside of NVO3. E.g., you can have a single default
> > > router for the overlay instance, and all traffic would go through
> it
> > > (and the firewall beyond). It is unclear to me NVO3 has to know
> this
> > > is going on.
> 
> > Yikes.  I hope we're NOT designing for a single point-of-failure, out
> > of the NVO3 overlay, from Day 1.  It seems we should be considering a
> > minimum of 2 x exit points from the overlay from Day 1 or, at the
> very
> > least, the ability for some type of *extremely* fast fail-over from
> > one 'physical' piece of hardware acting as the primary exit point to
> a
> > second piece of hardware acting as a secondary/standby exit point
> from
> > the overlay.
> 
> multiple (possibly stateful) FW issues (in some cases). Not saying that
> can't be done, just that it may not be as simple as having multiple
> exit routers.
> 
> times,
> > I may want to use their S3 or other services they provide.
> > The key point is I need my servers to talk to their S3 service, but I
> > do not want their other customers to be able to see and/or talk to my
> > servers, for security reasons.
> 
> in
> > the plans for NVO3, specifically as it relates to traffic that  will
> > enter/exit the overlay.  The way I see it there are three or  four
> > things that should be considered:
> 
> DataCenter
> > customers ...
> it
> > dies does the "backup" take over (instantaneously).
> 
> OK.
> 
> > c)  Active/Active.  Two, or more, exit points are active at any given
> > time.  This is the scenario I think we should aim for, because it can
> > be used to provide two physically diverse exit points from the same,
> > physical DataCenter as well as from geographically diverse
> are
> > sitting on a Corp WAN/LAN.  The latter, clients, are not aware of
> NVO3
> > and are speaking "regular" IP toward the DataCenter(s) to access
> > various applications/resources.  It's the former, VM's inside the
> NVO3
> > overlay, that I care about in the context of this conversation,
> > reqmt's and design space.
> 
> Understood.
>

---

From: AshwoodsmithPeter <[email-masked]>
Date: Thu, 5 Apr 2012 18:26:13 +0000

> > c)  Active/Active.  
.....
> I'm interested what others think about the need to add these to the

Something is required in the routers that terminate the tunnels to support active/active. At the very least the charter should recommend something that exists like VRRP and show how it maps. I suppose its also possible for the router to have some NVO3 tunnels terminating into a subnet handled by some VRF but also some other non NVO3 mechanisms terminating into the same subnet/VRF, in which case any VRRP like mechanism would have to operate between the two different technologies.

Tromboning caused by picking the wrong gateway from outside the DC is LISPs realm and I suppose it begs the question of why not adopt a LISP style encap inside the DC too. Kill 2 birds with one stone.

So active/active I think should be in the charter.
External tromboning if included would be a pull towards LISP and would be nice to have just one encap for both purposes.

Peter Ashwood-Smith

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Balus, Florin Stelian (Florin)
Sent: Thursday, April 05, 2012 1:05 PM
To: Thomas Narten; Shane Amante
Cc: Bocci, Matthew (Matthew); [email-masked]; Benson Schliesser
Subject: Re: [nvo3] Updated Charter

Thomas,

My 2 cents...

Florin

---

From: Shane Amante <[email-masked]>
Date: Thu, 5 Apr 2012 12:46:28 -0600

Florin,

> I think what he described in his email are DC networking issues we need at least to discuss in NVO3 data plane/control plane requirements and gap analysis. In particular the requirements to address…

I think the above is an excellent way to summarize the issue and I agree with what you recommend.  Specifically, we should look at the use case I outlined, in NVO3, during the requirements and gap analysis phase(s).  From there, we should determine, as Thomas has said whether it makes more sense for NVO3 to solve for it -or- this has broader applicability, in which case, the "problem/use-case" may be brought to a different (existing?) WG at the IETF to look at and see if they are interested in solving for it.

Thanks,

-shane

---

From: Dino Farinacci <[email-masked]>
Date: Thu, 5 Apr 2012 11:53:04 -0700

> External tromboning if included would be a pull towards LISP and would be nice to have just one encap for both purposes.

Just an FYI that LISP and VxLAN use the same encapsulation headers. An 8 byte UDP header followed by a 8 byte LISP header. What is the inner versus the outer outer headers depends on the address type for your EID or RLOC respectively.

Dino

---

From: "Puneet Agarwal" <[email-masked]>
Date: Thu, 5 Apr 2012 19:00:30 +0000

I agree with Florin - Resiliency of NVO3 gateways should be part of the charter.

Thanks.
 
-Puneet

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Balus, Florin Stelian (Florin)
Sent: Thursday, April 05, 2012 10:05 AM
To: Thomas Narten; Shane Amante
Cc: Bocci, Matthew (Matthew); [email-masked]; Benson Schliesser
Subject: Re: [nvo3] Updated Charter

Thomas,

My 2 cents...

Florin

---

From: AshwoodsmithPeter <[email-masked]>
Date: Thu, 5 Apr 2012 19:09:08 +0000

Thanks Dino,

I did not realize that. The discussion at the BOF was heading towards reusing formats that speed up the servers but no discussion on the impacts to the gateway routers/firewalls etc. which are typically stressed out at 40/100G.

Using a common format seems very sensible and forward looking.

Peter 

-----Original Message-----
From: Dino Farinacci [mailto:[email-masked]] 
Sent: Thursday, April 05, 2012 2:53 PM
To: AshwoodsmithPeter
Cc: Balus, Florin Stelian (Florin); Thomas Narten; Shane Amante; Bocci, Matthew (Matthew); [email-masked]; Benson Schliesser
Subject: Re: [nvo3] Updated Charter

Dino

---

From: Vishwas Manral <[email-masked]>
Date: Thu, 5 Apr 2012 12:19:57 -0700

Hi folks,

I am sorry I joined the list recently and realized all the juicy
conversations happening here (so may be repeating what has already been
said).

I see the main requirements as below, which may be conflicting in some
contexts:

1. Appliances which can work better as they look at forward lesser number
of packets. So a mechanism that can easily help enhance Appliance
throughput. (Things like STT help that easily)
2. Routers have concerns with performance when stateful reassembly is
required. (As Dino says VxLAN and LISP)
3. We need the same (or stateless translatable) encapsulation, for both
Inter and Intra DC traffic.

Thanks,
Vishwas

On Thu, Apr 5, 2012 at 12:09 PM, AshwoodsmithPeter <
[email-masked]> wrote:

>
> Thanks Dino,
>
> I did not realize that. The discussion at the BOF was heading towards
> reusing formats that speed up the servers but no discussion on the impacts
> to the gateway routers/firewalls etc. which are typically stressed out at
> 40/100G.
>
> Cc: Balus, Florin Stelian (Florin); Thomas Narten; Shane Amante; Bocci,
> Matthew (Matthew); [email-masked]; Benson Schliesser
> > External tromboning if included would be a pull towards LISP and would
> be nice to have just one encap for both purposes.
>
> Just an FYI that LISP and VxLAN use the same encapsulation headers. An 8
> byte UDP header followed by a 8 byte LISP header. What is the inner versus
> the outer outer headers depends on the address type for your EID or RLOC
> respectively.
>
> Dino
>

---

From: Yakov Rekhter <[email-masked]>
Date: Thu, 5 Apr 2012 12:30:35 -0700

Peter,

> Something is required in the routers that terminate the tunnels to
> support active/active. At the very least the charter should recommend
> something that exis ts like VRRP and show how it maps. I suppose
> its also possible for the router to have some NVO3 tunnels terminating
> into a subnet handled by some VRF but also some other non NVO3
> mechanisms terminating into the same subnet/VRF, in which case any
> VRRP like mechanism would have to operate between the two different
> te chnologies.
> 
> Tromboning caused by picking the wrong gateway from outside the DC
> is LISPs realm and I suppose it begs the question of why not adopt
> a LISP style encap inside the DC too. Kill 2 birds with one stone.

You may look at draft-raggarwa-data-center-mobility-02.txt
for a description of mechanism that allow to eliminate
"tromboning". 

> So active/active I think should be in the charter.  External
> tromboning if included would be a pull towards LISP and would be
> nice to have just one encap for both purposes.

Yakov.

> I think what he described in his email are DC networking issues we need at le
ast to discuss in NVO3 data plane/control plane requirements and gap analysis. 
In particular the requirements to address traffic tromboning to/from NVO3 and r
esiliency at NVO3 gateways came up a number of times in the past in various IET
F WG sessions. We can see after that where/how exactly is the best place/way to
 address them. But we should not just ignore them.

---

From: Yakov Rekhter <[email-masked]>
Date: Thu, 5 Apr 2012 12:47:49 -0700

Dino,

> > External tromboning if included would be a pull towards LISP and would be 
> Just an FYI that LISP and VxLAN use the same encapsulation headers. An 8 byte
> UDP header followed by a 8 byte LISP header. What is the inner versus the 
> outer outer headers depends on the address type for your EID or RLOC 
> respectively.

Here is the LISP header:

       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   L   |N|L|E|V|I|flags|            Nonce/Map-Version                  |
   I \ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   S / |                 Instance ID/Locator Status Bits               |
   P   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Here is the VxLAN header:

        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
        |R|R|R|R|I|R|R|R|            Reserved                           | 
        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
        |                VXLAN Network Identifier (VNI) |   Reserved    | 
        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 

So, if one measure similarity by the size of the header, then indeed
both LISP and VxLAN use "the same encapsulation header" - both are
8 bytes long.

However, if we look inside these 8 bytes, we find that the two
encapsulation headers are quite different.

So, your statement that "LISP and VxLAN use the same encapsulation
headers" is incorrect.

Yakov.

---

From: Dino Farinacci <[email-masked]>
Date: Thu, 5 Apr 2012 12:53:43 -0700

>> outer outer headers depends on the address type for your EID or RLOC 

When the I-bit is set in the LISP header, then the format is:

     x x x x 1 x x x
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |N|L|E|V|I|flags|            Nonce/Map-Version                  |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                 Instance ID                   |     LSBs      |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

VxLAN's header is a subset functionality of the LISP header and always sets the I-bit.

> So, your statement that "LISP and VxLAN use the same encapsulation

Not true Yakov.

Dino

---

From: <[email-masked]>
Date: Fri, 6 Apr 2012 09:50:20 +0200

Shane, Florin, folks,

I fully agree that what will be done in NVO3 has to take into account 
these issues of (avoiding) traffic trombonning and gateways resiliency, 
and how Florin you suggest we address them.

-Thomas(M)

Shane Amante :
>>>> It seems fairly clear that NVO3 is, eventually, headed in a map&
>>>> the NVO3 control plane for Inner<->  Outer Address Mapping the answer
>>>> standard ARP&  ICMPv6 RD/ND ...

---

From: <[email-masked]>
Date: Fri, 6 Apr 2012 09:58:36 +0200

Hi Stewart,

Dimitri hinted that we should address (2), and (1) as a subset of (2). 
To me, this makes a lot of sense: assuming the overlay simply relies on 
IP connectivity and that there is no administrative border to cross, 
then (1) and (2) can resolve to the same problem, except that (2) may 
bring higher scales in the pictures, which we definitively have to 
anticipate.

Just to make the link with another thread in progress, the one initiated 
by Shane's email... There is a fourth class of connectivity (4), that is 
at least as important as (3) below: connectivity between an NVO3 
Ethernet overlay and external subnets.

-Thomas

---

From: Xuxiaohu <[email-masked]>
Date: Fri, 6 Apr 2012 08:29:42 +0000

Hi,

I also agree that if data center interconnection is included in the NVO3 charter, the path optimization issue associated with subnet extension would be much worthwhile to consider. Furthermore, I think it would be better if the requirements for DCN and DCI could be either described in two drafts respectively or explicitly differentiated in one draft, although there are a lot of overlaps between them.

Best regards,
Xiaohu

> 发送时间: 2012年4月6日 2:46
> 收件人: Balus, Florin Stelian (Florin)
> 抄送: Thomas Narten; Bocci, Matthew (Matthew); [email-masked]; Benson
> Schliesser
> > I think what he described in his email are DC networking issues we need at
> least to discuss in NVO3 data plane/control plane requirements and gap
> analysis. In particular the requirements to address traffic tromboning to/from
> NVO3 and resiliency at NVO3 gateways came up a number of times in the past
> in various IETF WG sessions. We can see after that where/how exactly is the
> best place/way to address them. But we should not just ignore them.
> >
> > My 2 cents...
> 
> I think the above is an excellent way to summarize the issue and I agree with
> what you recommend.  Specifically, we should look at the use case I outlined,
> in NVO3, during the requirements and gap analysis phase(s).  From there, we
> should determine, as Thomas has said whether it makes more sense for NVO3
> to solve for it -or- this has broader applicability, in which case, the
> "problem/use-case" may be brought to a different (existing?) WG at the IETF to
> look at and see if they are interested in solving for it.
> 
> Thanks,
> 
> -shane
> 
> 
> > Florin
> >

---

From: "Mcdysan, David E" <[email-masked]>
Date: Fri, 6 Apr 2012 06:57:01 -0400

Stewart,

What I asked about was the following (not connectivity):

"I suggest that the charter needs to better describe the scope because this
will impact the virtual tenant namespace requrements. The text below could
be interpreted as:
1) a single data center
2) a set of data centers under control of one administrative domain
3) multiple sets of data centers under control of multiple administrative
domains"

Quoting the draft charter "...the mapping of "inner" (tenant VM) addresses
into "outer" (underlying
network) addresses in order to tunnel a packet to the destination VM ..."

my comment was regarding the scope of the tenant VM address namespace, not
the underlying network addresses (which could be viewed as providing
connectivity). I am not sure if that is what you meant by "connectivity."
Could you please clarify.

As I understand the draft charter, the nvo3 wg would define requirements
for a data and control plane for an addressing scheme for tenant VM
addresses and how this would be mapped into underlying network addresses.
I believe an important requirement is for the namespace to be large enough
to handle the scope of overlay networked VM addresses that a multi-tenant
data center environment will need.

I agree with the sentiment expressed by you and Thomas of not going into
great detail for the
underlying network (which may be what you meant by "connectivity" and
"external communication")

Some questions in line below trying to better understand the terminology
used in your comments
since you did not use the terminology of the draft charter.

Thanks,

Dave

>and if we decide we need to address 2 and or 3 we can

You may be making this more complex than necessary. Either we decide on a
namespace with some scope of appropriate size and then the
framework/requirements/gap analysis needs to identify some way(s) to
handle cases with larger scope.

>that impacts the internal DC design.

I don't see "design" in the draft charter. Please clarify the charter
reference.

>The more

I don't see these terms in the draft charter. Please clarify the charter
reference.

---

From: "Mcdysan, David E" <[email-masked]>
Date: Fri, 6 Apr 2012 07:01:33 -0400

Stewart,

Quoting the draft charter "...the mapping of "inner" (tenant VM) addresses
into "outer" (underlying network) addresses in order to tunnel a packet to
the destination VM ..."

My comment was regarding the scope of the tenant VM address namespace, not
the underlying network addresses (which could be viewed as providing
connectivity). Is this what you meant by "connectivity?"

I agree with the sentiment expressed by you and Thomas of not going into
great detail for the
underlying network (which may be what you meant by "connectivity" and
"external communication"), since as the charter states "The WG will
investigate interconnection and interoperation issues between
overlays and VPNs to determine if any specific work is needed."

I think that the "Internet" needs to be added to the above sentence in the
charter as well.

Some questions in line below trying to better understand the
terminologyused in your comments
since you did not use the terminology of the draft charter.

Thanks,

Dave

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Fri, 6 Apr 2012 13:08:47 +0200

These are certainly issues that we have to discuss and analyze.

On one hand, one must rely on the network to be self-healing but at the same time one can rely on optimal placement of workloads based upon underlay topological information that gets exchanged with "applications" (e.g. orchestration tools).

Specific "cost" metrics (e.g. length of network paths, path usage, path latency) can be used for optimized traffic placement.

Hence, we need to study all the possible avenues in order to achieve traffic optimality and, as Florin mentioned, decide in which WG(s) to tackle this problem.

Marc

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Friday, April 06, 2012 9:50 AM
To: [email-masked]
Subject: Re: [nvo3] Updated Charter

Shane, Florin, folks,

-Thomas(M)

---

From: Thomas Narten <[email-masked]>
Date: Fri, 06 Apr 2012 10:23:18 -0400

Hi Dave.

> Quoting the draft charter "...the mapping of "inner" (tenant VM)
> addresses into "outer" (underlying network) addresses in order to
> tunnel a packet to the destination VM ..."

> Could you please clarify.

To be clear, when you say "tenant VM address namespace", do you mean
what draft-kreeger-nvo3-overlay-cp-00.txt refers to as:

   VN-ID:  Virtual Network Identifier.  This is the ID value that is
      carried in each data packet in the overlay encapsulation that
      identifies the Virtual Network the packet belongs to.  It should
      be a large enough ID space to not be a limiting factor within an
      administrative domain managing the ID space.  There are several
      technologies which encapsulate using a 24 bit ID value, e.g.  PBB,
      SPBM, LISP, OTV, TRILL Fine-grained labels, VXLAN, NVGRE.

Assuming the answer is yes, I would say the following.

The VN-ID namespace needs to be large enough to support a single
administrative domain. I view the VN-ID as having local significance
within a particular administrative domain.

Going further, it may be also be tied to a specific underlying
technology as the text above states.

That said, I think there is a second namespace that may be useful to
define. In the case where a "virtual network" spans multiple
administrative domains, e.g., including perhaps a VPN and multiple
data centers under different administrative domains, it would seem
useful to have a global name as well, one that each administrative
domain can use, but would map into its own local VN-ID.

For example, if you have 3 different administrative domains
cooperating to build one virtual network, it seems useful to have a
global name that all 3 can use to name the virtual network. This would
be a high-level name, one that could presumably go into email,
configuration files, etc., so that the different administrative
domains have a way of refering to the same virtual network.

Each administrative domain would need to such a string into a local
VN-ID for use, but the details of how that is done could be a local
matter.

Wherever there is a boundary between adminstrative domains (and the
two domains use different namespaces) having a common name that both
can refer to and understand in a consistent manner seems necessary.

The charter doesn't mention the above, but we've had some discussions
about it.

To summarize, I think  two different name spaces are probably
needed. One is the low-level "on-the-wire"  namespace  called the
VN-ID. But it is *not* global. This is in scope for NVO3

For a cross-domain namespace, I would think a string (rather than a
fixed sized number) would be more appropriate because of how it would
be used. The string could still have some syntactic structure to
ensure its globally unique (e.g., embedding a domain name that
identifies the actual tenant of that VN). This isn't listed in the
charter, but MAY be something we should pursue.

I'd be interested if folk think having a 2nd, higher-level namespace
as outlined above is useful, unnecessary or something else.

Thomas

---

From: "Mcdysan, David E" <[email-masked]>
Date: Fri, 6 Apr 2012 10:38:12 -0400

Hi Thomas,

Your post points out that we need to define and use consistent terminology.

A few responses in line below.

Thanks,

Dave

On 4/6/12 10:23 AM, "Thomas Narten" <[email-masked]> wrote:

>>administrative
>Assuming the answer is yes,

Need to decide on whether we should use "VN-ID" instead of "VM address"
from the draft charter. Suggest that the charter be updated to use
consistent terminology.

>I would say the following.
>

Which would correspond to 1 and 2 in my taxonomy above.
>technology as the text above states.

I would hope that a VN-ID space would not be tied to the underlying
network -- this is a foundational  idea of an overlay where the overlay
namespace need not be tied to the underlying network. A mapping, of
course, is needed.

>useful to have a global name as well,

This is what I had in mind with case 3 in the naming taxonomy.

>one that each administrative

This adds a level of indirection and hence complexity. If this must be
done, then so be it, but I suggest this would be a good item to include in
the framework - pick a global namespace, or a hierarchical one to handle
case 3. The hierarchical one could address the historical artifact of the
VN-IDs cited in the problem statement that have some association with a
specific underlying network technology.

>Each administrative domain would need to

??map??

>such a string into a local

I agree. This is what I had in mind with case 3 in the naming taxonomy.
>as outlined above is useful, unnecessary or something else.

I think it is very useful, and gives an opportunity to do naming in a
scalable, extensible manner.

---

From: Thomas Narten <[email-masked]>
Date: Fri, 06 Apr 2012 11:02:15 -0400

> from the draft charter. Suggest that the charter be updated to use

Actually, I think VN-ID and "VM address" are two very different
things. The VN-ID refers to a virtual network itself, whereas "VM
address" (I assume) would refer to the address of a machine attached
to specific virtual network.  Also, the charter doesn't even use the
term VN-ID

OK.

> namespace need not be tied to the underlying network. A mapping, of

But we need a name for the namespace of an underlying network.

> This is what I had in mind with case 3 in the naming taxonomy.

So, one thought about this is that within an administrative domain, it
would be OK to use a fixed-field number (whether 24 bits, 32 bits,
etc.) It would be the same number used by the underlying
technology. This field/number would appear "on the wire" in whatever
control protocols end up being used as well as in data plane
encapsulations.

On the other hand, we could also aim to use a higher-level name space,
pass those names (which maybe strings or have some other format)
around "on the wire" in control protocols, and then have various
components that actually need the underlying VN-ID (e.g., for
encapsulation) map the name to the VN-ID.

That is an architectural decision to make. I can see benefits to
either approach.

For example, for provisioning VMs, would it be better to use the VN-ID
in the meta-data associated with a VM image, or would it be better to
use a higher-level name, which presumably would be more portable? 

> specific underlying network technology.

I guess I am skeptical that we can have a single global name space for
VN-IDs that is compact enough to be used in packet headers on an
underlying network. But a heirarchical one that gets mapped into a
VN-ID that can be used locally seems doable.

Isn't this effectively already done today, to some extent, with VPNs?

> ??map??

Yes. :-)

OK.

Thomas

---

From: John E Drake <[email-masked]>
Date: Fri, 6 Apr 2012 08:09:51 -0700

Thomas,

Are you familiar with how L3/L2 VPNs are constructed using Route Targets, which, as was pointed out by Igor in the NV03 BoF, have extremely rich and useful semantics?

Thanks,

John  

Sent from my iPhone

> Sent: Friday, April 06, 2012 7:23 AM
> To: Mcdysan, David E
> Cc: [email-masked]; [email-masked]
> > "I suggest that the charter needs to better describe the scope
> because
> > this will impact the virtual tenant namespace requrements. The text
> > below could be interpreted as:
> > 1) a single data center
> > administrative domains"
> > my comment was regarding the scope of the tenant VM address
> namespace,
> > not the underlying network addresses (which could be viewed as
> > providing connectivity). I am not sure if that is what you meant by
> "connectivity."
> administrative domains, e.g., including perhaps a VPN and multiple data
> centers under different administrative domains, it would seem useful to
> have a global name as well, one that each administrative domain can
> use, but would map into its own local VN-ID.
> 
> For example, if you have 3 different administrative domains cooperating
> to build one virtual network, it seems useful to have a global name
> that all 3 can use to name the virtual network. This would be a high-
> level name, one that could presumably go into email, configuration
> files, etc., so that the different administrative domains have a way of
> refering to the same virtual network.
> 
> Each administrative domain would need to such a string into a local VN-
> ID for use, but the details of how that is done could be a local
> matter.
> 
> Wherever there is a boundary between adminstrative domains (and the two
> domains use different namespaces) having a common name that both can
> refer to and understand in a consistent manner seems necessary.
> 
> To summarize, I think  two different name spaces are probably needed.
> One is the low-level "on-the-wire"  namespace  called the VN-ID. But it
> is *not* global. This is in scope for NVO3
> 
> be used. The string could still have some syntactic structure to ensure
> its globally unique (e.g., embedding a domain name that identifies the
> actual tenant of that VN). This isn't listed in the charter, but MAY be
> something we should pursue.
> 
> I'd be interested if folk think having a 2nd, higher-level namespace as
> outlined above is useful, unnecessary or something else.
> 
> Thomas
>

---

From: "Balus, Florin Stelian (Florin)" <[email-masked]>
Date: Fri, 6 Apr 2012 10:45:11 -0500

Dave I have a question on your following paragraph:

> done, then so be it, but I suggest this would be a good item to include
> in the framework - pick a global namespace, or a hierarchical one to
> handle case 3. The hierarchical one could address the historical
> artifact of the VN-IDs cited in the problem statement that have some
> association with a specific underlying network technology.
> 
[FB>] We hinted in the current framework (see http://www.ietf.org/id/draft-lasserre-nvo3-framework-01.txt) section 3.1.3 at the possibility to have a global identifier and we gave as an example the Route Target usage in L2/L3 VPNs. 

So one option is to re-use the RTs. One way to do that is for NVO3 to terminate into VPNs when it wants to go outside an administrative domain: i.e. the VN-ID appears as one local interface in the IP VPN or VPLS which then use RTs to get out into other admin domains. This is very similar to the current VLAN case where the VLAN can then stay local to the admin domain.

Or do you imply something beyond that: e.g. using a generic identifier for the tenant service, something of the format [email-masked] which is the same for customer W VPN Y across different admin domains? 

> Mcdysan, David E
> Sent: Friday, April 06, 2012 7:38 AM
> To: Thomas Narten
> >>because this  will impact the virtual tenant namespace requrements.
> >>The text below could  be interpreted as:
> >> 1) a single data center
> >>namespace, not  the underlying network addresses (which could be
> >>viewed as providing  connectivity). I am not sure if that is what you
> >>meant by "connectivity."
> data
> >have a global name as well,
> 
> done, then so be it, but I suggest this would be a good item to include
> in the framework - pick a global namespace, or a hierarchical one to
> handle case 3. The hierarchical one could address the historical
> artifact of the VN-IDs cited in the problem statement that have some
> association with a specific underlying network technology.
> 
> >
> cooperating
> >that all 3 can use to name the virtual network. This would be a
> >high-level name, one that could presumably go into email,
> configuration
> two
> ensure
> as

---

From: "Mcdysan, David E" <[email-masked]>
Date: Fri, 6 Apr 2012 14:48:02 -0400

Yes, route targets are useful for intranets and extranets. However; I
believe that operationally, the complexity required to use route targets
limits their usage to relatively simple relationships.

I am not aware of a use case where these are employed to implement
forwarding between a graph of hosts, such as real (or virtual) middleboxes
which are relatively  common in multi-tenant data centers. As I stated at
the mike, I believe this is an important framework, and data/control
requirement.

Thanks,

Dave

On 4/6/12 11:09 AM, "John E Drake" <[email-masked]> wrote:

>Thomas,
>
>Are you familiar with how L3/L2 VPNs are constructed using Route Targets,
>which, as was pointed out by Igor in the NV03 BoF, have extremely rich
>and useful semantics?
>
>Thanks,
>
>John  
>
>Sent from my iPhone
>
>

---

From: Anoop Ghanwani <[email-masked]>
Date: Fri, 6 Apr 2012 11:51:25 -0700

> 3) multiple sets of data centers under control of multiple administrative

I think all of these need to be considered...especially since people are
already thinking on those lines and anything that doesn't account for all 3
is falling short. There is the transport issue which can be handled by any
of the L2/L3 VPN solutions, but there are issues around the NVO3 control
plane specifically which may require work.

On Wed, Apr 4, 2012 at 7:54 PM, Shane Amante <[email-masked]> wrote:

>
> Again, we need to be careful to include some form(s) of redundancy in the
> plans for NVO3, specifically as it relates to traffic that will enter/exit
> the overlay.  The way I see it there are three or four things that should
> be considered:
> a)  No redundancy whatsoever for traffic entering/leaving the NVO3 overlay
> -- good luck trying to sell this to Mega/Tera/Yotta DataCenter customers ...
> b)  Active/Backup.  One exit is active at any given time; only when it
> time.  This is the scenario I think we should aim for, because it can be
> used to provide two physically diverse exit points from the same, physical
> DataCenter as well as from geographically diverse DataCenters.  Ultimately,
> this allows the possibility for traffic to enter/exit through the most
> optimal "active" entry/exit point.

I think a solution without any redundancy is a non-starter.  I strongly
believe we should strive for active-active.

> administrative domains.  David Black talked to these cases at the BOF.

Given that we're specifying a single encapsulation is a non-goal, I think
becomes critical to address the issue of how to get VMs that are sitting on
hypervisors with differing encapsulations to talk to each other.

Anoop

---

From: "Mcdysan, David E" <[email-masked]>
Date: Fri, 6 Apr 2012 15:01:07 -0400

Hi Florin,

Responses in line below.

Thanks,

Dave

On 4/6/12 11:45 AM, "Balus, Florin Stelian (Florin)"
<[email-masked]> wrote:

>[FB>] We hinted in the current framework (see
>http://www.ietf.org/id/draft-lasserre-nvo3-framework-01.txt) section
>3.1.3 at the possibility to have a global identifier and we gave as an
>example the Route Target usage in L2/L3 VPNs.

I would like to see the final framework deliverable do more than hint. ;-)
> 
>
>So one option is to re-use the RTs. One way to do that is for NVO3 to
>terminate into VPNs when it wants to go outside an administrative domain:
>i.e. the VN-ID appears as one local interface in the IP VPN or VPLS which
>then use RTs to get out into other admin domains. This is very similar to
>the current VLAN case where the VLAN can then stay local to the admin
>domain.

I believe that we first need to agree on the requirements. As I responded
to John Drake, RTs are good for (relatively simple) intranets and
extranets. I am not sure if they are a good solution for forwarding
packets between graphs (e.g., a chain) of hosts in an overlay as is
becoming increasing important in multi-tenant data-center deployments.

>
>Or do you imply something beyond that: e.g. using a generic identifier
>for the tenant service, something of the format [email-masked]
>which is the same for customer W VPN Y across different admin domains?

That is what I understood Thomas Naarten to be proposing. And he said
"string" not something based upon a DNS name that your example above
implies.

---

From: "Mcdysan, David E" <[email-masked]>
Date: Fri, 6 Apr 2012 15:02:30 -0400

Hi Thomas,

Some responses in line below, primarily attempting to clarify charter
terminology and items to be covered in framework and requirements.

Dave

On 4/6/12 11:02 AM, "Thomas Narten" <[email-masked]> wrote:

>to specific virtual network.

Here is a quote from the updated charter:

"NVO3 will document the problem statement, the applicability, and an
architectural framework for overlay networks within a data center
environment. Within this framework, functional blocks will be defined to
allow the dynamic attachment / detachment of VMs to the overlay network,
the mapping of "inner" (tenant VM) addresses into "outer" (underlying
network) addresses in order to tunnel a packet to the destination VM, and
the migration of VMs within the network in a dynamic fashion."

I assumed that the tenant VM addresses were a new item to be defined by
this WG. The guest OS stack IP and MAC address is certainly present and
"tunneled," but there may be duplicate IP and MAC addresses on guest OSes
served by the same hypervisor and there is a need for some other address.
The above paragraph only lists two "addresses" and one appears to be
missing. 

> Also, the charter doesn't even use the
>term VN-ID

Possibly it should? SOME SUGGESTED ADDITIONAL WORDS IN THE CHARTER
PARAGRAPH.

"NVO3 will document the problem statement, the applicability, and an
architectural framework for overlay networks within a data center
environment. Within this framework, functional blocks will be defined to
allow the dynamic attachment / detachment of VMs to the overlay network,
the mapping of "inner" (tenant VM) addresses into A VIRTUAL NETWORK ID
(VN-ID) AND INTO "outer" (underlying
network) addresses in order to tunnel a packet to the destination VM, and
the migration of VMs within the network in a dynamic fashion."

>But we need a name for the namespace of an underlying network.

I had assumed that nvo3 would use the namespace of existing underlying
networks (MAC, VLAN, PBB, Trill, IP, GRE, BGP MPLS VPN, VPLS, etc.).
>use a higher-level name, which presumably would be more portable?

These seem like excellent points to document in the framework deliverable
in the proposed charter.

>VN-ID that can be used locally seems doable.

And there will be a need for gateways to do all of these mappings. Another
framework decision is whether many of these mappings can be made to some
canonical more common form, instead of the O(N^2) mappings needed between
N different underlying network technologies.

---

From: John E Drake <[email-masked]>
Date: Sat, 7 Apr 2012 06:48:29 -0700

Dave,

I think you switched topics.

Thanks,

John

Sent from my iPhone

> From: Mcdysan, David E [mailto:[email-masked]]
> Sent: Friday, April 06, 2012 11:48 AM
> To: John E Drake; Thomas Narten
> believe that operationally, the complexity required to use route
> targets limits their usage to relatively simple relationships.
> 
> forwarding between a graph of hosts, such as real (or virtual)
> middleboxes which are relatively  common in multi-tenant data centers.
> As I stated at the mike, I believe this is an important framework, and
> data/control requirement.
> 
> Thanks,
> 
> Dave
> 
> >Are you familiar with how L3/L2 VPNs are constructed using Route
> >Targets, which, as was pointed out by Igor in the NV03 BoF, have
> >extremely rich and useful semantics?
> >
> >Thanks,
> >
> >John
> >
> >Sent from my iPhone
> >
> >
> >> From: [email-masked] [mailto:[email-masked]] On Behalf
> >> Of Thomas Narten
> >> global name that all 3 can use to name the virtual network. This
> >> would be a high- level name, one that could presumably go into
> email,
> >> VN- ID for use, but the details of how that is done could be a local
> >> matter.
> >>
> needed.
> >> One is the low-level "on-the-wire"  namespace  called the VN-ID. But
> >> it is *not* global. This is in scope for NVO3
> >>

---

From: Kireeti Kompella <[email-masked]>
Date: Sat, 7 Apr 2012 10:10:21 -0700

Hi Thomas, Dave:

On Apr 6, 2012, at 8:02, Thomas Narten <[email-masked]> wrote:

> The VN-ID refers to a virtual network itself, whereas "VM

In general, we need three terms for the IDs that distinguish VNs at each of the management/orchestration, control and and data planes, to be used respectively for provisioning, signaling and overlay traffic.  Each of these planes would map back and forth to the next plane. Right now, we're conflating these terms. (Certainly, in some realizations of VNs, the maps could be identity maps (no pun intended).)

The management plane ID for a VN could very well be string-based and hierarchical. The data plane ID should be something easily and quickly processed at the packet level. Given that there would be a mapping between management plane IDs and control plane IDs, perhaps a relatively simple control plane ID structure suffices.

To use John's example of Route Targets (control plane IDs), one structure is simply AS number + 32-bit ID.  Each DC belonging to a separate autonomous entity could map the global, hierarchical management name for a VN to their local control plane ID. 

Kireeti

---

From: <[email-masked]>
Date: Mon, 9 Apr 2012 15:32:54 -0400

Hi Kireeti,

> In general, we need three terms for the IDs that distinguish VNs at each of
> the management/orchestration, control and data planes, to be used
> respectively for provisioning, signaling and overlay traffic.  Each of these
> planes would map back and forth to the next plane. Right now, we're conflating
> these terms. (Certainly, in some realizations of VNs, the maps could be
> identity maps (no pun intended).)

I think that's right - in particular at least I have been assuming that the
control-data ID mapping would be an identity map in preparing for the BOF,
which in turn implies a constant single-valued data plane ID. VLANs happen to
work that way, and this is a fine design approach, but I agree that we ought
to have separate terms for all the IDs involved.

> The management plane ID for a VN could very well be string-based and
> hierarchical. The data plane ID should be something easily and quickly
> processed at the packet level. Given that there would be a mapping between
> management plane IDs and control plane IDs, perhaps a relatively simple
> control plane ID structure suffices.

The management plane ID is more "interesting" as that ID is likely to have
multi-admin-domain requirements - see slides 5 and 6 from my nvo3 presentation
in Paris, where a single management plane ID that represents the overall L2
connectivity across the admin domains is desirable.  A hierarchical
string-based ID sounds like a good choice for that.

Thanks,
--David

> Kireeti Kompella
> Sent: Saturday, April 07, 2012 1:10 PM
> To: Thomas Narten
> Cc: Mcdysan, David E; [email-masked]; [email-masked]
> In general, we need three terms for the IDs that distinguish VNs at each of
> the management/orchestration, control and and data planes, to be used
> respectively for provisioning, signaling and overlay traffic.  Each of these
> planes would map back and forth to the next plane. Right now, we're conflating
> these terms. (Certainly, in some realizations of VNs, the maps could be
> identity maps (no pun intended).)
> 
> The management plane ID for a VN could very well be string-based and
> hierarchical. The data plane ID should be something easily and quickly
> processed at the packet level. Given that there would be a mapping between
> management plane IDs and control plane IDs, perhaps a relatively simple
> control plane ID structure suffices.
> 
> To use John's example of Route Targets (control plane IDs), one structure is
> simply AS number + 32-bit ID.  Each DC belonging to a separate autonomous
> entity could map the global, hierarchical management name for a VN to their
> local control plane ID.
> 
> Kireeti
>

---

From: AshwoodsmithPeter <[email-masked]>
Date: Mon, 9 Apr 2012 19:37:08 +0000

> Each DC belonging to a separate autonomous entity could map the global, hierarchical management name for a VN to their local control plane ID. 

Speaking of local data plane IDs (plural).

Is it a requirement to support different tunnel encapsulation types in the same DC?

It would seem that a very large DC could well end up with several different kinds of tunnel encasulations that would need to somehow be bridged if they terminate VMs in the same subnet.

I.E
                                          ?
VM A 1.1.1.1/24  |---------Encap A--------|      V
                                          |--- o 
VM B 1.1.1.2/24  |---------Encap B--------|      R
                                                 
VM C 2.1.1.1/24  |---------Encap A-------------o F

Regards,

Peter Ashwood-Smith

---

From: Yakov Rekhter <[email-masked]>
Date: Wed, 18 Apr 2012 08:15:24 -0700

Dino,

> When the I-bit is set in the LISP header, then the format is:

Functionality of a used car is a subset functionality of a new car.
Yet according to your logic the two are the same...

> > So, your statement that "LISP and VxLAN use the same encapsulation

I guess your definition of "the same" is different from mine.

Yakov.