Subject: [nvo3] Requirements + some non-requirement suggestions

From: <[email-masked]>
Date: Mon, 9 Apr 2012 16:51:30 -0400

> Dave McDyson asked about three classes of connectivity:
> 
> 1) a single data center
> 2) a set of data centers under control of one administrative domain
> 3) multiple sets of data centers under control of multiple 
>	administrative domains
>
> Which of these do we *need* to address in NVO3?

I agree with a number of other people that we have to start with 1), and then I'd
suggest addressing as much of 2) as "makes sense" without significantly affecting
the design and applicability of the result to data centers.  For example, a single
instance of an overlay that spans data centers "makes sense", at least to me, or
as Thomas Narten described it:

> two DCs are under the same administrative domain, but are
> interconnected by some (existing) VPN technology, of which we don't
> care what the details are. The overlay just tunnels end-to-end and
> couldn't care less about the existence of a VPN interconnecting parts
> of the network.

Looking at draft-meyer-loc-id-implications-01:

	http://tools.ietf.org/html/draft-meyer-loc-id-implications-01

I would suggest that for initial nvo3 work, reachability between all NVEs in a
single overlay instance should be assumed, as there will be an IGP routing protocol
(e.g., OSPF with ECMP) running on the underlying data center network which will
handle link failures.  Specifically, for initial nvo3 work I'd suggest assuming
that the underlying network handles reachability of the NVE at the other side of
the overlay (other end of the tunnel) that does the decapsulation.  In terms
of that draft, within a single data center (and hence for the scope of initial
nvo3 work), I'd suggest that the underlying network be responsible for handling
the Locator Path Liveness problem.

This suggestion also applies to the Multi-Exit problem, although on a related
note, I think it's a good idea to make sure that nvo3 doesn't turn any crucial
NVE into a single point of failure. Techniques like VRRP address this in the
absence of nvo3, so this could be mostly a matter of paying attention to ensure
that they're applicable to NVE failure.  Regardless of whether things work out
that way, I'd suggest that availability concerns be in scope.

Turning to the topic of IGP metrics and "tromboning", I'd suggest that having
nvo3 add a full IGP routing protocol (or even an IGP metrics infrastructure for
one that has to be administered) beyond what's already running in the underlying
network is not a good idea.  It seems like a large portion of the "tromboning"
concerns could be resolved by techniques that distribute the default gateway in
a virtual network so that moving a VM (virtual machine) automatically sends
traffic to the locally-applicable instance of the default gateway for the VM's
new location, based on the same L2 address. There are multiple examples of this
sort of approach - OTV and draft-raggarwa-data-center-mobility-02 are among them.

One more - Peter Ashwood-Smith writes:

> Is it a requirement to support different tunnel encapsulation types in the same DC?
> 
> It would seem that a very large DC could well end up with several different kinds of
> tunnel encapsulations that would need to somehow be bridged if they terminate VMs in
> the same subnet.

I'd suggest that the latter scenario be out of scope and that crossing virtual
networks initially involve routing in preference to bridging, so that an NVE
receiving an unencapsulated packet can determine the overlay and encapsulation by
knowing which virtual network the packet belongs to.  An implication is that I'd
suggest figuring out how to optimize the following structure into a single
network node later (or at least as a cleanly separable work effort:

... (Overlay1)---- NVE1 ---- (VLANs) ---- NVE2 ---- (Overlay2) ...

In the above, NVE1 and NVE2 are separate nodes, and the parenthesized terms are
the means of virtual network separation.

That suggests that a starting point for whether different tunnel encapsulation types
should be supported in a single data center could be "if they don't have an NVE
node in common, they can be made to work" and optimizations can be considered later.

Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

---

From: Anoop Ghanwani <[email-masked]>
Date: Mon, 9 Apr 2012 20:13:30 -0700

> node in common, they can be made to work" and optimizations can be

Perhaps there needs to be yet another overlay between NVE1 and
NVE2.  Otherwise, how do we squeeze 16 million virtual networks
at the edges into 4K VLANs between the NVEs?  Or maybe I am
missing something very fundamental.

Anoop

---

From: David Meyer <[email-masked]>
Date: Tue, 10 Apr 2012 04:54:30 -0700

On Mon, Apr 9, 2012 at 1:51 PM,  <[email-masked]> wrote:
> handle link failures.

That may be a reasonable assumption, but the fact that an IGP is
running doesn't ease the "locator liveness" problem unless the routing
system is carrying /32s (or /128s) and the corresponding /32 (/128)
isn't injected into the IGP unless the decapsulator is up (that in and
of itself might not be sufficient as we've learned from our
experiences with anycast DNS overlays). In any event what we would be
doing in this case is using the routing system to signal a live path
to the decapsulator. Of course, carrying such long prefixes has its
own set of problems.

> Specifically, for initial nvo3 work I'd suggest assuming

Not sure what you mean here by underlying network. In the LISP case,
does the underlying network handle this problem? In any event can you
be a bit more explicit in what you mean here?
> node in common, they can be made to work" and optimizations can be considered later.

Agree with this latter statement.

Dave

---

From: <[email-masked]>
Date: Tue, 10 Apr 2012 09:24:18 -0400

Anoop,

> missing something very fundamental.

That's a good question.  I'm not sure about "very fundamental", but network
locality is your friend ;-).

Seriously, overlay deployments will likely have initial scope large
enough to contain most of the virtual networks (e.g., drop an overlay
onto the collection of networking gear that's hitting VLAN count limits),
and the initial numbers of virtual networks are going to be orders of
magnitude short of 16 million (if we hit 16 million immediately, then
24 bits clearly aren't enough ...).

Longer term, I agree that 4K VLANs will be a limiting constraint on
overlay-to-overlay interfaces, as well as on overlay-to-VPN interfaces,
but it doesn't seem crucial to address that limit as part of the initial
overlay work focused on the data center use case.  I could see addressing
widening of both of these interfaces by taking out the VLANs in parallel
to the data center use case, but I'd prefer to be cautious about how
strongly to link these two areas of work.

Thanks,
--David

From: [email-masked] [mailto:[email-masked]] On Behalf Of Anoop Ghanwani
Sent: Monday, April 09, 2012 11:14 PM
To: Black, David
Cc: [email-masked]
Subject: Re: [nvo3] Requirements + some non-requirement suggestions

On Mon, Apr 9, 2012 at 1:51 PM, <[email-masked]<mailto:[email-masked]>> wrote:

Anoop

---

From: John E Drake <[email-masked]>
Date: Tue, 10 Apr 2012 06:34:11 -0700

In E-VPN, the internal identifier of a broadcast domain, an Ethernet Tag, is a 32 bit number, which is mapped into whatever broadcast domain identifier, e.g., VLAN ID, is understood by the attaching CE device.  So, the 4K limit is just between the E-VPN network and *each* attaching  CE device.

Sent from my iPhone

From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Tuesday, April 10, 2012 6:24 AM
To: [email-masked]
Cc: [email-masked]
Subject: [nvo3] VLANs between overlays

Anoop,

Thanks,
--David

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Anoop Ghanwani
Sent: Monday, April 09, 2012 11:14 PM
To: Black, David
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] Requirements + some non-requirement suggestions

Anoop

---

From: <[email-masked]>
Date: Tue, 10 Apr 2012 09:41:33 -0400

Dave,

In reverse order ...

> be a bit more explicit in what you mean here?

The underlying network is what connects the NVEs (encap/decap locations) in
the data center, and it runs an IGP (e.g., OSPF with ECMP) for its routing. 
That IGP is going to be configured with relatively conventional IP address
blocks (e.g., /24s for IPv4 if that's what the data center is already using).

Backing up ...

> to the decapsulator. Of course, carrying such long prefixes has its

I strongly disagree with that statement.

First, what may be an important difference is that when the NVE is not
in the end system (e.g., NVE in hypervisor softswitch or top-of-rack switch),
the locator (outer IP destination address) points at the NVE (tunnel endpoint,
decapsulation location), not the end system.  The end system is beyond the NVE,
so the NVE decaps the L2 frame and forwards based on that frame's L2 destination
(MAC) address (so the NVE serves multiple end systems.

To take a specific IPv4 example, it should be fine to have a bunch of NVEs
in 10.1.1.0/24 and expect OSPF or the like to deal with that /24 and its kin
as opposed to a /32 per NVE.  This example also applies to NVEs in end systems
- the IGP can still work with /24s and the like and does not need /32s (in this
case, each /24 points to an L2 domain with potentially many end system NVEs).

> Agree with this latter statement.

Thank you.

Thanks,
--David

> -----Original Message-----
> From: David Meyer [mailto:[email-masked]]
> Sent: Tuesday, April 10, 2012 7:55 AM
> To: Black, David
> Cc: [email-masked]

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 10 Apr 2012 10:15:04 -0400

In concluding that there is no locator liveness problem, is you 
assumption that there is always only one locator that can be used?

The reason I ask is that there are two basic, related, causes that drive 
the locator liveness issue.
First, if an end-point can be reached with two locators, it is important 
to know that one of them is no longer working, so that you switch to 
using the other.
\Second, if an end-point can change locators, one of the usual 
techniques for handling things is to have the original locator send new 
information.  If one relies on this, then if the original locator is 
dead one needs that information as a proxy to prompt refetching the 
information.

If neither of those applies, then maybe there is no locator liveness 
problem for VMs.

With regard to site exit, the issues seem more complex than your initial 
description painted.  It may be possible to solve.  But it is not free 
if one is using the locator of the site exit, instead of underlying 
routing to the ultimate destination.

Yours,
Joel

On 4/10/2012 9:41 AM, [email-masked] wrote:
>> On Mon, Apr 9, 2012 at 1:51 PM,<[email-masked]>  wrote:
> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3
>

---

From: <[email-masked]>
Date: Tue, 10 Apr 2012 10:51:21 -0400

Hi Joel,

> assumption that there is always only one locator that can be used?

No, at least not yet ... although it is an attractive simplification *if*
the related problems can be solved with a single locator ...

> to know that one of them is no longer working, so that you switch to

... starting with exactly that availability problem ;-).

I would not assume a single locator until the options for availability are
explored, although there are commonly-used data center approaches that
effectively fail-over a single locator (the MAC address moves at L2) -
I hope VRRP is a familiar example.

> dead one needs that information as a proxy to prompt refetching the

That's typically not done for virtual machines (VMs); the typical currently-
used technique for a VM location change is to send a gratuitous ARP or RARP
from the new location to trigger data plane learning, and not worry about
dropping a few packets that go to the old location in the interim.  The
draft charter envisions work on an explicit attach/detach protocol that
offers opportunities for improvement, but not having to retain state at
the old location is a "feature" for VMs.

> routing to the ultimate destination.

As an initial swipe at this, I'd expect "routing to the ultimate destination"
to select the site exit based on the inner IP header, with the encapsulation
effectively functioning as a glorified L2 link to get to it via the site exit's
locator.  If the initial nvo3 overlay deployments focus on the access layer,
then the core L3 routing on the "other side" of the overlay should get this
right, but I recognize that this paragraph is still somewhat of a high-level
handwave.  If the overlay reaches all the way to the site exits, then there's
no L3 forwarding of the end system's encapsulated L2 traffic between the end
system and the site exit (i.e., all of the L3 forwarding acts on the outer IP
header, not the inner one in the encapsulated L2 frame).  In this case, the
end system winds up selecting the site exit; there are environments in which
that may not be a good design ... or, in other words ..."Doctor, it hurts when
I do <this>." ... "Don't do <that>!" :-) :-).

Thanks,
--David

> From: Joel M. Halpern [mailto:[email-masked]]
> Sent: Tuesday, April 10, 2012 10:15 AM
> To: Black, David
> Cc: [email-masked]; [email-masked]
> Subject: Re: [nvo3] Reachability and Locator Path Liveness
> 
> >>> the design and applicability of the result to data centers.  For example,
> a
> >> single
> >>> I would suggest that for initial nvo3 work, reachability between all NVEs
> in
> >> a
> >>> nvo3 add a full IGP routing protocol (or even an IGP metrics
> infrastructure
> >> for
> >>> concerns could be resolved by techniques that distribute the default
> gateway
> >> in
> >>> sort of approach - OTV and draft-raggarwa-data-center-mobility-02 are
> among
> >> them.
> >>>
> >>>> Is it a requirement to support different tunnel encapsulation types in
> the
> >> same DC?
> >>>>
> >>>> It would seem that a very large DC could well end up with several
> different
> >> kinds of
> >>> knowing which virtual network the packet belongs to.  An implication is
> that
> >> I'd
> >>> In the above, NVE1 and NVE2 are separate nodes, and the parenthesized
> terms
> >> are
> >>> should be supported in a single data center could be "if they don't have
> an
> >> NVE

---

From: David Meyer <[email-masked]>
Date: Tue, 10 Apr 2012 09:19:56 -0700

David,

> (MAC) address (so the NVE serves multiple end systems.

Well, the endpoint I was talking about is the decapsulator (e.g., a
LISP ETR). So maybe
you can get away with just injecting /32s for those (if one decided to
do it this way).
> case, each /24 points to an L2 domain with potentially many end system NVEs).

Just because there is a covering prefix doesn't mean that any of the
longer prefixes that are
covered are reachable. It is just this loss of information (in this
case due to aggregation) that causes
the problem.

Dave

---

From: David Meyer <[email-masked]>
Date: Tue, 10 Apr 2012 09:25:03 -0700

On Tue, Apr 10, 2012 at 7:15 AM, Joel M. Halpern <[email-masked]> wrote:
> In concluding that there is no locator liveness problem, is you assumption
> that there is always only one locator that can be used?

No, in fact that is why the problem is essentially O(m*n)...m source
addresses, n destination addresses.

>
> The reason I ask is that there are two basic, related, causes that drive the
> locator liveness issue.
> First, if an end-point can be reached with two locators, it is important to
> know that one of them is no longer working, so that you switch to using the
> other.

I think the answer is yes, but generally we use the term "locator" to
refer to the address of a decapsulator, not the endpoint. Perhaps
clearing this up would help.

> \Second, if an end-point can change locators, one of the usual techniques
> for handling things is to have the original locator send new information.
>  If one relies on this, then if the original locator is dead one needs that
> information as a proxy to prompt refetching the information.

Again, I think the answer would be yes but I'm not sure about the
definition of terms here.

Dave

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 10 Apr 2012 18:44:23 +0200

Hi David,

> clearing this up would help.

Decapsulator will be the hypervisor as it will need to decapsulate 
between N tenants VMs which are running on top of it.

If you push the decapsulator to be outside of the host then effectively 
you are running into two main problems:

- short lived overlays (and we see more and more very dynamic overlay 
request) need to now span multiple devices and multiple operational and 
control planes

- the demux has to be embedded in the data plane between the ETR as you 
call it and hypervisor so final decapsulation will still happen on the 
host.

> So maybe

It is not about another set of /32s. It is about another layer of 
encap/decap if you move your demux point anywhere from hypervisor.

Rgs,
R.

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 10 Apr 2012 10:31:11 -0700

Here are two presentations on the subject I gave at the Stockholm LISP WG meeting in July 2009.

Read the lisp-ietf-arn-lra.ppt attachment first, it leads you into the second one. It turns out we put RLOC-probing and Echo-Noncing in the LISP spec as possible solutions with different scaling tradeoffs.

Dino

On Apr 10, 2012, at 7:15 AM, Joel M. Halpern wrote:

> In concluding that there is no locator liveness problem, is you assumption that there is always only one locator that can be used?
> 
> The reason I ask is that there are two basic, related, causes that drive the locator liveness issue.
> First, if an end-point can be reached with two locators, it is important to know that one of them is no longer working, so that you switch to using the other.
> \Second, if an end-point can change locators, one of the usual techniques for handling things is to have the original locator send new information.  If one relies on this, then if the original locator is dead one needs that information as a proxy to prompt refetching the information.
> 
> If neither of those applies, then maybe there is no locator liveness problem for VMs.
> 
> With regard to site exit, the issues seem more complex than your initial description painted.  It may be possible to solve.  But it is not free if one is using the locator of the site exit, instead of underlying routing to the ultimate destination.
> 
> Yours,
> Joel
>

---

From: "Pat Thaler" <[email-masked]>
Date: Tue, 10 Apr 2012 18:37:58 +0000

The situation is similar if the external (i.e. data center to data center) network is an IEEE 802.1Q Provider Backbone Bridging network - there is a 24-bit I-CID to identify the customer/tenant on the provider network and it can be mapped into a link local VLAN identifier on the link between the Provider Edge Bridge and the NVE device.

From: [email-masked] [mailto:[email-masked]] On Behalf Of John E Drake
Sent: Tuesday, April 10, 2012 6:34 AM
To: [email-masked]; [email-masked]
Cc: [email-masked]
Subject: Re: [nvo3] VLANs between overlays

Sent from my iPhone

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]<mailto:[mailto:[email-masked]]> On Behalf Of [email-masked]<mailto:[email-masked]>
Sent: Tuesday, April 10, 2012 6:24 AM
To: [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>
Subject: [nvo3] VLANs between overlays

Anoop,

Thanks,
--David

Anoop

---

From: AshwoodsmithPeter <[email-masked]>
Date: Tue, 10 Apr 2012 21:21:06 +0000

What prevents a TOR, or core switch from de-encapsulating, putting on a private VLAN and sending to a specific server port. You'd have to do that anyway where the end device is not a hypervisor or device that supports the given encapsulation.

Take for example the 'other end' of these tunnels when we terminate into a router. Either the router has to de-encap and then run its L2/L3, or the core switch facing the router can de-encap, stick a private VLAN on and up it goes to the router which now requires no changes.

I'm not sure what the overhead is relative to the line rates at 100G on the core routers but splitting the behavior allows one to decide on which side of the interface to spend the money which seems like a useful architectural bit of flexibility to maintain.

Peter Ashwood-Smith

---

From: <[email-masked]>
Date: Tue, 10 Apr 2012 18:29:10 -0400

> Just because there is a covering prefix doesn't mean that any of the longer prefixes that are
> covered are reachable. It is just this loss of information (in this case due to aggregation) that causes
> the problem.

That is an assumption that can be made in a data center for the NVEs (which
don't move) based on the IGP in the underlying network (e.g., every NVE in
10.1.1.0/24 is reachable via 10.1.1.0/24).  The initial endpoint identifiers
for nvo3 are L2 MACs, which are inherently non-aggregatable, and I agree that
aggregation assumptions for IP address endpoint identifiers are potentially
problematic.

Thanks,
--David

> From: [email-masked] [mailto:[email-masked]] On Behalf Of David
> Meyer
> Sent: Tuesday, April 10, 2012 12:20 PM
> To: Black, David
> Cc: [email-masked]
> in a
> an NVE
> >> > single overlay instance should be assumed, as there will be an IGP
> routing
> >> protocol
> >> > (e.g., OSPF with ECMP) running on the underlying data center network
> which
> >> will
> >> > handle link failures.
> >>
> >> > that they're applicable to NVE failure.  Regardless of whether things
> work
> >> out
> >> > new location, based on the same L2 address. There are multiple examples
> of
> >> this

---

From: David Allan I <[email-masked]>
Date: Tue, 10 Apr 2012 18:47:09 -0400

I don't think you can actually achieve geographic independence, aggregatable addresses, mobilty and locality in the same breath let alone network.

Currently L3 over L2 in some form of overlay is the level of indirection that decouples what I would call macro locality (where in the network) from micro locality (where in the cloud). Aggregatable addresses directly over aggregatable addresses without the level of indirection looks problematic to me and requires, at first blush, excessive gymnastics...and actually eliminates any potential value as it becomes more subsetting...you'd might as well concatenate the prefixes together

In that regard, I think we are saying the same thing...

Cheers
D

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Tuesday, April 10, 2012 3:29 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [nvo3] Reachability and Locator Path Liveness

> > To take a specific IPv4 example, it should be fine to have a bunch 
> > of NVEs in 10.1.1.0/24 and expect OSPF or the like to deal with that 
> > /24 and its kin as opposed to a /32 per NVE.  This example also 
> > applies to NVEs in end systems
> > - the IGP can still work with /24s and the like and does not need 
> > /32s (in this case, each /24 points to an L2 domain with potentially many end system NVEs).
> 
> longer prefixes that are covered are reachable. It is just this loss 
> of information (in this case due to aggregation) that causes the problem.

That is an assumption that can be made in a data center for the NVEs (which don't move) based on the IGP in the underlying network (e.g., every NVE in
10.1.1.0/24 is reachable via 10.1.1.0/24).  The initial endpoint identifiers for nvo3 are L2 MACs, which are inherently non-aggregatable, and I agree that aggregation assumptions for IP address endpoint identifiers are potentially problematic.

Thanks,
--David

> From: [email-masked] [mailto:[email-masked]] On Behalf 
> Of David Meyer
> routing protocol
> which will
> >> running doesn't ease the "locator liveness" problem unless the 
> >> routing system is carrying /32s (or /128s) and the corresponding 
> >> /32 (/128) isn't injected into the IGP unless the decapsulator is 
> >> up (that in and of itself might not be sufficient as we've learned 
> >> from our experiences with anycast DNS overlays). In any event what 
> >> we would be doing in this case is using the routing system to 
> >> signal a live path to the decapsulator. Of course, carrying such 
> >> long prefixes has its own set of problems.
> >
> > First, what may be an important difference is that when the NVE is 
> > not in the end system (e.g., NVE in hypervisor softswitch or 
> > top-of-rack
> switch),
> > decapsulation location), not the end system.  The end system is 
> > beyond the
> NVE,
> LISP ETR). So maybe you can get away with just injecting /32s for 
> those (if one decided to do it this way).
> >
> > To take a specific IPv4 example, it should be fine to have a bunch 
> > of NVEs in 10.1.1.0/24 and expect OSPF or the like to deal with that 
> > /24 and its kin as opposed to a /32 per NVE.  This example also 
> > applies to NVEs in end
> systems
> > - the IGP can still work with /24s and the like and does not need 
> > /32s (in
> this
> > case, each /24 points to an L2 domain with potentially many end 
> > system
> NVEs).
> 
> longer prefixes that are covered are reachable. It is just this loss 
> of information (in this case due to aggregation) that causes the 
> problem.
> 
> Dave
> 
> >
> >> > node in common, they can be made to work" and optimizations can 
> >> > be
> considered later.
> >>
> >> > I agree with a number of other people that we have to start with 
> >> > 1), and
> >> then I'd
> >> > suggest addressing as much of 2) as "makes sense" without 
> >> > significantly
> >> affecting
> >> > instance of an overlay that spans data centers "makes sense", at 
> >> > least to
> >> me, or
> >> >> interconnected by some (existing) VPN technology, of which we 
> >> >> don't care what the details are. The overlay just tunnels 
> >> >> end-to-end and couldn't care less about the existence of a VPN 
> >> >> interconnecting parts of the network.
> >> >
> >> running doesn't ease the "locator liveness" problem unless the 
> >> routing system is carrying /32s (or /128s) and the corresponding 
> >> /32 (/128) isn't injected into the IGP unless the decapsulator is 
> >> up (that in and of itself might not be sufficient as we've learned 
> >> from our experiences with anycast DNS overlays). In any event what 
> >> we would be doing in this case is using the routing system to 
> >> signal a live path to the decapsulator. Of course, carrying such 
> >> long prefixes has its own set of problems.
> >>
> >> > Specifically, for initial nvo3 work I'd suggest assuming that the 
> >> > underlying network handles reachability of the NVE at the other
> >> side of
> >> > the overlay (other end of the tunnel) that does the 
> >> > decapsulation.  In
> terms
> >> > of that draft, within a single data center (and hence for the 
> >> > scope of
> >> initial
> >> > nvo3 work), I'd suggest that the underlying network be 
> >> > responsible for
> >> handling
> >> Not sure what you mean here by underlying network. In the LISP 
> >> case, does the underlying network handle this problem? In any event 
> >> can you be a bit more explicit in what you mean here?
> >> >
> >> > This suggestion also applies to the Multi-Exit problem, although 
> >> > on a
> >> related
> >> > note, I think it's a good idea to make sure that nvo3 doesn't 
> >> > turn any
> >> crucial
> >> > NVE into a single point of failure. Techniques like VRRP address 
> >> > this in
> the
> >> > absence of nvo3, so this could be mostly a matter of paying 
> >> > attention to
> >> ensure
> >> > Turning to the topic of IGP metrics and "tromboning", I'd suggest 
> >> > that
> >> having
> >> > one that has to be administered) beyond what's already running in 
> >> > the
> >> underlying
> >> > a virtual network so that moving a VM (virtual machine) 
> >> > automatically
> sends
> >> > traffic to the locally-applicable instance of the default gateway 
> >> > for the
> >> VM's
> >> >> tunnel encapsulations that would need to somehow be bridged if 
> >> >> they
> >> terminate VMs in
> >> >> the same subnet.
> >> >
> >> > I'd suggest that the latter scenario be out of scope and that 
> >> > crossing
> >> virtual
> >> > networks initially involve routing in preference to bridging, so 
> >> > that an
> NVE
> >> > suggest figuring out how to optimize the following structure into 
> >> > a
> single
> >> > node in common, they can be made to work" and optimizations can 
> >> > be
> >> considered later.
> >>

---

From: Santosh Rajagopalan <[email-masked]>
Date: Tue, 10 Apr 2012 15:52:17 -0700

Hi, Robert
I prefer the "locator" term because it allows the decapsulator to be 
either within the hypervisor, or external, as will be the case when you 
use gateways or appliances. 

For example, if your data center started out as a single tenant network, 
and then extended into a multi-tenant "public" cloud, it is quite possible 
that the your traffic travels nvo3-enapsulated in the "public" sphere, and 
gets decap-ed at the entry into your private network by a gateway.

Also, at some point, you will begin to load balance your traffic among 
multiple appliances (firewalls for example) which might need to strip the 
outer header, examine the payload and then put in a new outer-header, and 
this may or may not be a virtual appliance located in a server. So for the 
firewall case, the name resolution at the source server could give 2 
firewalls as ECMP "locators", and the name resolution at the firewall 
could give 2 ecmp router/gateways at the edge of the private network. 

The summary is that we need to be open to "locators" that are not 
server-bound and be prepared for name resolution which is tailored to 
who's making the request. (in the example earlier, the answer to "resolve 
a.b.c.d" depends on whether it's the hypervisor or the firewall that's 
asking.

--
Sunny Rajagopalan

From:   Robert Raszuk <[email-masked]>
To:     David Meyer <[email-masked]>
Cc:     [email-masked], "Joel M. Halpern" <[email-masked]>, 
[email-masked]
Date:   04/10/2012 09:51 AM
Subject:        Re: [nvo3] Reachability and Locator Path Liveness
Sent by:        [email-masked]

Hi David,

Rgs,
R.

---

From: Roger Jørgensen <[email-masked]>
Date: Wed, 11 Apr 2012 10:22:53 +0200

On Tue, Apr 10, 2012 at 3:24 PM,  <[email-masked]> wrote:
<snip>

> That’s a good question.  I’m not sure about “very fundamental”, but network
>
> onto the collection of networking gear that’s hitting VLAN count limits),
> 24 bits clearly aren’t enough ...).

... why 24bit? Are there any underlaying header 24bit fit better into
than 32bits?
If there ain't any surrounding header fitting to consider, why use 24
and not 32?

-- 

Roger Jorgensen           |
[email-masked]          | - IPv6 is The Key!
http://www.jorgensen.no   | [email-masked]

---

From: Thomas Narten <[email-masked]>
Date: Wed, 11 Apr 2012 06:05:42 -0400

Robert,

> request) need to now span multiple devices and multiple operational and 

Maybe I'm confused by what you are saying, but overlays will most
definitely span multiple devices. The whole point is that an overlay
is a Virtual Network to which arbitrary VMs can attach. Those VMs (by
definition) can be anywhere in the network and should not need to be
on the same device. 

> call it and hypervisor so final decapsulation will still happen on the 

Maybe maybe not. In NVO3, the architecture should allow decapsulation
to take place on the hypervisor or on the switch attached to the
device (e.g., to support "bare metal" servers).

Thomas

---

From: Thomas Narten <[email-masked]>
Date: Wed, 11 Apr 2012 06:09:26 -0400

Peter,

> What prevents a TOR, or core switch from de-encapsulating, putting
> on a private VLAN and sending to a specific server port. You'd have
> to do that anyway where the end device is not a hypervisor or device
> that supports the given encapsulation.

Exactly. This is needed, e.g., to allow non-overlay aware systems
(e.g., network appliances, bare metal machines, etc.) to be part of an
overlay instance.

> Take for example the 'other end' of these tunnels when we terminate
> into a router. Either the router has to de-encap and then run its
> L2/L3, or the core switch facing the router can de-encap, stick a
> private VLAN on and up it goes to the router which now requires no
> changes.

> I'm not sure what the overhead is relative to the line rates at 100G
> on the core routers but splitting the behavior allows one to decide
> on which side of the interface to spend the money which seems like a
> useful architectural bit of flexibility to maintain.

Fully Agree.

Thomas

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 11 Apr 2012 12:34:39 +0200

Hi Thomas,

 > definitely span multiple devices.

Very much agreed.

But reading your mail I actually observed that we do not have clear 
definition of what "entity" can become part of an overlay.

I assumed VMs behind hypervisors would be the most popular, but I agree 
VMs running over bear metal or just entire physical hosts or other 
devices could become part of them too.

I am not trying to say that we should exclude TORs or aggregation 
routers to be part of an overlay demux. I am just trying to point out 
that I think the most common case would be VMs behind some host stack 
(kernel/hypervisor/ovs) and that VMs or appliances running over bear 
metal could easily terminate the overlay's encapsulation too. So even in 
those cases placing a real hardware in front of them may be optional.

 > device (e.g., to support "bare metal" servers).

Yes. But I think we are not clear on what the topology(ies) of an 
overlay would be. Would it be cloud like overlay that tenant members 
join or would that be multi-star like that each member is directly 
connected with other members of a given overlay. I think both topologies 
may have their applicability.

Regards,
R.

---

From: Thomas Narten <[email-masked]>
Date: Wed, 11 Apr 2012 06:51:55 -0400

> definition of what "entity" can become part of an overlay.

Well, we tried to come up with such terminology in
draft-kreeger-nvo3-overlay-cp-00.txt with:

   OBP:  Overlay Boundary Point.  This is a network entity that is on
      the edge boundary of the overlay.  It performs encapsulation to
      send packets to other OBPs across an Underlying Network for
      decapsulation.  An OBP could be implemented as part of a virtual
      switch within a hypervisor, a physical switch or router, a Network
      Service Appliance or even be embedded within an End Station.

draft-lasserre-nvo3-framework-01.txt used different terminology:

   A Tenant End System attaches to a Network Virtualization Edge (NVE)
   node, either directly or via a switched network (typically
   Ethernet).

And now this thread has morphed into talking about "locators",
terminology which NVO3 doesn't define or use...

> those cases placing a real hardware in front of them may be

For bare metal machines and other "legacy" (read non overlay aware
systems) you need a transition path. Hence, there will be a need for
switches attached to such machines performing the functions of
encap/decap/etc. Performing the functions of an OBP.

> may have their applicability.

I don't understand this notion of "topology" of an overlay. An overlay
is a "flat" construct. It is a virtual network to which arbitrary VMs
(that are part of that overlay) connect. An overlay instance doesn't
have "topology" in the conventional sense. An overlay resides on an
underlying transport network (e.g., the DC network) which has
structure, but NVO3 doesn't concern itself with the details of that
structure. We just assume its an IP network and that any device (and
OBP in particular) on the underlying transport network is reachable
via any other device/OBP.

So, within an overlay, you can view each attached VM is being
"directly" connected to all other VMs. I.e., in the L2overL3 case, we
are emulating the connectivity model of a single L2 broadcast
domain. But I don't think it is helpful to talk about the "topology"
of an overlay.

Thomas

---

From: <[email-masked]>
Date: Wed, 11 Apr 2012 09:01:28 -0400

> If there ain't any surrounding header fitting to consider, why use 24

Lots of people seem to be settling on 24 bits as the next place to go beyond
the 12-bit VLAN tag.  Both VXLAN and NVGRE are currently using 24 bits:

NVGRE: http://datatracker.ietf.org/doc/draft-sridharan-virtualization-nvgre/
VXLAN: http://datatracker.ietf.org/doc/draft-mahalingam-dutt-dcops-vxlan/

In addition, in related technologies, SPBM ISIDs are 24 bits and TRILL
fine grain labels will result in 24 bits of addressing.

Thanks,
--David

> From: Roger Jørgensen [mailto:[email-masked]]
> Sent: Wednesday, April 11, 2012 4:23 AM
> To: Black, David
> Cc: [email-masked]; [email-masked]

---

From: <[email-masked]>
Date: Wed, 11 Apr 2012 09:06:50 -0400

> domain. But I don't think it is helpful to talk about the "topology"

IMHO, this is crucial to get started - in a data center, not only is the
overlay flat, but it's ok to assume complete reachability among all the
NVEs (encapsulators/decapsulators) that participate by appealing to the IGP
(e.g., OSPF with ECMP) that runs in the underlying network.

Also, the whole discussion of /32s (or /128s for IPv6) is tangential
wrt getting started with the data center use case, as those sorts of host
routes aren't typically used in a data center network (and hence won't be
used or needed in the underlying network within a data center).

Thanks,
--David

> From: Thomas Narten [mailto:[email-masked]]
> Sent: Wednesday, April 11, 2012 6:52 AM
> To: [email-masked]
> Cc: David Meyer; Black, David; Joel M. Halpern; [email-masked]

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Wed, 11 Apr 2012 15:49:17 +0200

While any-to-any connectivity is assumed in an overlay network, there is some topological information that can be useful to select a path among diverse paths specifically in the case of multi-homing. Some entries might get precedence based on path cost or latency for instance.

Marc

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Wednesday, April 11, 2012 3:07 PM
To: [email-masked]
Cc: [email-masked]
Subject: [nvo3] Topology

Thanks,
--David

---

From: AshwoodsmithPeter <[email-masked]>
Date: Wed, 11 Apr 2012 13:48:11 +0000

> Lots of people seem to be settling on 24 bits as the next place to go >beyond
>he 12-bit VLAN tag.  Both VXLAN and NVGRE are currently using 24 bits:

>NVGRE: http://datatracker.ietf.org/doc/draft-sridharan-virtualization->nvgre/

>--David

Yes, there is a lot of deployed PBB stuff with a 24 bit ISID which seems to be proving adequate although if any of the folks running PBB/SPBM think more is required now would be a very good time to speak up ;)

Peter

---

From: Thomas Narten <[email-masked]>
Date: Wed, 11 Apr 2012 10:01:17 -0400

Hi Marc.

"LASSERRE, MARC (MARC)" <[email-masked]> writes:

> While any-to-any connectivity is assumed in an overlay network,
>  there is some topological information that can be useful to select
>  a path among diverse paths specifically in the case of
>  multi-homing. Some entries might get precedence based on path cost
>  or latency for instance.

Let's be clear about what we are specifically talking about.

What does multi-homing actually mean?

Do you mean where there may be 2 tunnel endpoints for the same target
VM, in which case the mapping function needs to be smart about
choosing which of the multiple choices?  It remains to be seen whether
and to what degree NVO3 should be allowing such multi-homing. 

Or, do you mean multiple paths on the underlying network to the same
tunnel endpoint? If the latter, I would say completely orthogonal and
out of scope for NVO3. It would be a matter for the underlying network
for how to handle this case (i.e., use TE, ECMP, etc.). It would seem
to me to be a standard case where the underlying transport network is
being asked to deliver traffic to a specific destination, and it may
have multiple paths for reaching that destination. Existing tools
would be used for this.

Or are you thinking something different?

Thomas

---

From: Paul Unbehagen <[email-masked]>
Date: Wed, 11 Apr 2012 08:58:13 -0600

16 million VLANs has been more than enough in many enterprise data centers.  

Additionally, since it's running on the same switch/routers that are running OSPF the LANs are extended naturally under the IP/OSPF layer using the same link state calculation/topology to reach any server regardless of its location in the data center. 

This eliminates the L2 edge/L3 core problem of connecting the servers across the data center as the LAN for any subnet can be extended anywhere without affecting the IP subnet routing points.

For many enterprises the overlay solves a bigger issue of trying to connect the multiple data centers over a IP WAN. Hence reusing the 24 bits identifier is a very useful property of the approach. 
--
Paul Unbehagen

Sent from my iPad

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Wed, 11 Apr 2012 16:59:24 +0200

Hi Thomas,

See my comments inline.

Thanks,
Marc

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Narten
Sent: Wednesday, April 11, 2012 4:01 PM
To: LASSERRE, MARC (MARC)
Cc: [email-masked]; [email-masked]
Subject: Re: [nvo3] Topology

Hi Marc.

[ML] Yes, this is the use case that I had in mind.
[ML] I think that we have to consider multi-homing from day 1 in nvo3.
[ML] Why do you think that this may not be a requirement?

Thomas

---

From: "Balus, Florin Stelian (Florin)" <[email-masked]>
Date: Wed, 11 Apr 2012 10:12:46 -0500

In-line...

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Roger Jørgensen
> Sent: Wednesday, April 11, 2012 1:23 AM
> To: [email-masked]
> >> Perhaps there needs to be yet another overlay between NVE1 and NVE2.
> 
> >> Otherwise, how do we squeeze 16 million virtual networks at the
> edges
> >> into 4K VLANs between the NVEs?  Or maybe I am missing something
> very
> >> fundamental.
> >
> >
> >
> > onto the collection of networking gear that's hitting VLAN count
> > limits), and the initial numbers of virtual networks are going to be
> > orders of magnitude short of 16 million (if we hit 16 million
> > immediately, then

[FB>] The previous IEEE tags (VLAN and for PBB ISID) were 16 and respectively 32 but that includes also 4 qos bits (dot1p and discard eligibility) and 4 reserved. The same approach should be taken in my opinion for any NVO3 tag that supports QoS transparency and extensibility.

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 11 Apr 2012 08:26:58 -0700

> fine grain labels will result in 24 bits of addressing.

For LISP, we have 24-bits in the data-plane and 32-bits in the control-plane.

Dino

---

From: "Stiliadis, Dimitrios (Dimitri)" <[email-masked]>
Date: Wed, 11 Apr 2012 10:30:23 -0500

> those cases placing a real hardware in front of them may be optional.

There is a significant issue of trust and security here. If the bare metal 
machine belongs to a tenant, then the tenant cannot be trusted with all 
the packets that the machine is sending into the network. In theory the 
tenant can send any packet to the network, and thus join any overlay, 
and this  will be a serious security risk. 

The overlay has to be initiated by the first device that is under the 
*administrative* control of the cloud provider and not the tenant. This 
first device must also make sure that any protocol that communicates
VM instantiation information is secure and cannot be spoofed. In addition
this first device must make sure that tenants cannot join arbitrary
overlays.

In  the case of VMs with vswitches this first device might be the vswitch.

However in the case of virtualization with SRIOV, bare metal machines or 
other tenant specific appliances the first device that is under the 
administrative control of the cloud provider is most likely the first 
physical switch.

In an effort to simplify, let's not keep security as an after-thought.

Dimitri

---

From: "Stiliadis, Dimitrios (Dimitri)" <[email-masked]>
Date: Wed, 11 Apr 2012 10:32:43 -0500

Thomas:

> encap/decap/etc. Performing the functions of an OBP.

I wouldn't characterize SRIOV supporting hypervisors and Seamicro type 
of compute systems as "legacy" .. rather the future ;)

But, see my previous e-mail about administrative boundaries and security.

Clarifying that the overlay always starts on the cloud provider administrative
domain is crucial to the security and isolation properties in the system.

Dimitri

---

From: Roger Jørgensen <[email-masked]>
Date: Wed, 11 Apr 2012 20:11:27 +0200

<snip>

That has been said earlier, the history tell the rest  ;)

anyhow, thanks for good answers to my question about the size.

-- 

Roger Jorgensen           |

---

From: Vishwas Manral <[email-masked]>
Date: Wed, 11 Apr 2012 13:40:52 -0700

Hi Dimtri,

You bring a great paradox here.

We got isolation/ security with virtualization, but the performance sucked.
With Virt I/Oand  SRIOV we are again sharing virtual resources and working
directly with Bare metal.

Thanks,
Vishwas

---

From: Anoop Ghanwani <[email-masked]>
Date: Wed, 11 Apr 2012 14:01:50 -0700

> [ML] Why do you think that this may not be a requirement?

Where is the location of the OBP in this case?  I don't see
a need for multi-homing if the OBP is in a vSwitch.  If in a
physical switch, then it's not just a "VM" we're talking about;
it could be a physical machine as well.  And in that case, I
think this requirement was discussed in a previous thread:
http://www.ietf.org/mail-archive/web/nvo3/current/msg00220.html
See the discussion about entry/exit from the overlay.