Subject: [nvo3] FW: New Version Notification for draft-merged-nvo3-vm-mobility-scheme-00.txt

From: Linda Dunbar <[email-masked]>
Date: Fri, 3 Oct 2014 17:22:13 +0000

As NVO3 new charter encourage solutions proposals, we added more comprehensive solutions for the issues described in the draft-ietf-nvo3-vm-mobility-issues. We now call it "draft-merged-nvo3-vm-mobility-scheme-00"

Comments and suggestions are greatly appreciated. 

Linda 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Friday, October 03, 2014 12:18 PM
To: Rahul Aggarwal; Wim Henderickx; Ravi Shekhar; Luyuan Fang; Linda Dunbar; Rahul Aggarwal; Luyuan Fang; Wim Henderickx; Ravi Shekhar; Yakov Rekhter; Yakov Rekhter; Linda Dunbar; Ali Sajassi; Ali Sajassi
Subject: New Version Notification for draft-merged-nvo3-vm-mobility-scheme-00.txt

A new version of I-D, draft-merged-nvo3-vm-mobility-scheme-00.txt
has been successfully submitted by Linda Dunbar and posted to the IETF repository.

Name:		draft-merged-nvo3-vm-mobility-scheme
Revision:	00
Title:		NVO3 VM Mobility Scheme
Document date:	2014-10-03
Group:		Individual Submission
Pages:		24
URL:            http://www.ietf.org/internet-drafts/draft-merged-nvo3-vm-mobility-scheme-00.txt
Status:         https://datatracker.ietf.org/doc/draft-merged-nvo3-vm-mobility-scheme/
Htmlized:       http://tools.ietf.org/html/draft-merged-nvo3-vm-mobility-scheme-00

Abstract:
   This document describes the schemes to overcome the network-related
   issues to achieve seamless Virtual Machine mobility in the data
   center and between data centers.

Please note that it may take a couple of minutes from the time of submission until the htmlized version and diff are available at tools.ietf.org.

The IETF Secretariat

---

From: Tom Herbert <[email-masked]>
Date: Fri, 3 Oct 2014 11:28:11 -0700

> Comments and suggestions are greatly appreciated.
Hi Linda,

It looks like this draft mostly discusses mobility for L2 overlay
virtual networks. Do you envision that there will be another draft to
describe mobility for L3?

Thanks,
Tom

> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3

---

From: Linda <[email-masked]>
Date: Fri, 3 Oct 2014 17:16:04 -0500

Tom, 

Thank you for the comment. 
The 00 version of this draft only adds the solutions for the issues in the original draft. We will look into L3. If it is small portion for L3, we will add. If there are a lot, a separate draft will be written.
Linda
Sent from my iPhone

> On Oct 3, 2014, at 1:28 PM, Tom Herbert <[email-masked]> wrote:
>

---

From: Tom Herbert <[email-masked]>
Date: Fri, 3 Oct 2014 18:03:44 -0700

> As NVO3 new charter encourage solutions proposals, we added more comprehensive solutions for the issues described in the draft-ietf-nvo3-vm-mobility-issues. We now call it "draft-merged-nvo3-vm-mob…
I would suggest that "Address mobility scheme" might be a better
title. VM migration is one instance of when we need address mobility,
but we'll need this in with container migration or when we just want
to move an virtual address between servers. Also, w.r.t. VM or
container migration, addresses are not the only networking state we
need to consider, we need consider how to move connection state (e.g.
open TCP connections bond to the address being moved)-- this seems to
be out of scope for this draft.

Tom

---

From: "Black, David" <[email-masked]>
Date: Sat, 4 Oct 2014 01:14:37 +0000

> to move an virtual address between servers.

I agree, and I think Larry pointed this out earlier.

> Also, w.r.t. VM or

OTOH, I would caution about getting too involved in this as both ends
of the spectrum of connection state preservation are reasonable and used
in practice:

- VM live migration preserves TCP connections and the like.
- IP address takeover on hardware failure doesn't preserve
	anything whose state was solely on the hardware that's now a
	smoking pile of parts.

Thanks,
--David

> From: Tom Herbert [mailto:[email-masked]]
> Sent: Friday, October 03, 2014 9:04 PM
> To: Linda Dunbar
> Cc: [email-masked]; Larry Kreeger (kreeger); Black, David
> Subject: Re: [nvo3] FW: New Version Notification for draft-merged-nvo3-vm-
> mobility-scheme-00.txt
> 
> > As NVO3 new charter encourage solutions proposals, we added more
> comprehensive solutions for the issues described in the draft-ietf-nvo3-vm-
> mobility-issues. We now call it "draft-merged-nvo3-vm-mobility-scheme-00"
> >
> > To: Rahul Aggarwal; Wim Henderickx; Ravi Shekhar; Luyuan Fang; Linda Dunbar;
> Rahul Aggarwal; Luyuan Fang; Wim Henderickx; Ravi Shekhar; Yakov Rekhter;
> Yakov Rekhter; Linda Dunbar; Ali Sajassi; Ali Sajassi
> > Please note that it may take a couple of minutes from the time of submission
> until the htmlized version and diff are available at tools.ietf.org.
> >
> > The IETF Secretariat
> >

---

From: Sharon Barkai <[email-masked]>
Date: Sat, 4 Oct 2014 03:17:39 +0000

Would like to hear service provider opinion but would expect end-client IP to be able to "show up" on any NVE, as is the case for server IPs. This can be due to any topology change or load rebalancing.

The "address mobility" characteristic of NVO3 should apply here as well and without breaking connections.

There shouldn't be a material difference is supporting address mobility for both server-function to server-function and client to server-function.

               Client Traffic  Client Traffic
                              /     \/     /
                            /    /     \ /
                        NVE        NVE
                          |-VM       |-VM
                          |-VM      |-VM
--szb

---

From: Linda Dunbar <[email-masked]>
Date: Mon, 6 Oct 2014 15:54:19 +0000

I agree with Tom & David that this draft only describes the solutions to address the issues associated with hosts' (VMs') addresses floating across multiple NVEs or PODs, but not cover the state aspects (on middleware boxes) of the VM mobility. 

What do people think of "NVO3 Address Mobility Scheme"? 

Anyone has better suggestions? 

Thank, Linda

-----Original Message-----
From: nvo3 [mailto:[email-masked]] On Behalf Of Black, David
Sent: Friday, October 03, 2014 8:15 PM
To: Tom Herbert
Cc: [email-masked]
Subject: Re: [nvo3] FW: New Version Notification for draft-merged-nvo3-vm-mobility-scheme-00.txt

> be out of scope for this draft.

OTOH, I would caution about getting too involved in this as both ends of the spectrum of connection state preservation are reasonable and used in practice:

Thanks,
--David

> Subject: Re: [nvo3] FW: New Version Notification for 
> draft-merged-nvo3-vm- mobility-scheme-00.txt
> 
> comprehensive solutions for the issues described in the 
> draft-ietf-nvo3-vm- mobility-issues. We now call it "draft-merged-nvo3-vm-mobility-scheme-00"
> >
> Rahul Aggarwal; Luyuan Fang; Wim Henderickx; Ravi Shekhar; Yakov 
> Rekhter; Yakov Rekhter; Linda Dunbar; Ali Sajassi; Ali Sajassi
> > Subject: New Version Notification for 
> > draft-merged-nvo3-vm-mobility-scheme-
> 00.txt
> >
> >
> > has been successfully submitted by Linda Dunbar and posted to the 
> > IETF
> repository.
> >

---

From: Tom Herbert <[email-masked]>
Date: Mon, 6 Oct 2014 12:15:40 -0700

On Mon, Oct 6, 2014 at 8:54 AM, Linda Dunbar <[email-masked]>
wrote:
>
> I agree with Tom & David that this draft only describes the solutions to
address the issues associated with hosts' (VMs') addresses floating across
multiple NVEs or PODs, but not cover the state aspects (on middleware
boxes) of the VM mobility.
> What do people think of "NVO3 Address Mobility Scheme"?
While it's a better title, I think my comment was meant to be more general.
VMs (and hypervisors) are references to specific mechanisms of server
virtualization, but not the only mechanisms that can be deployed (e.g.
container virtualization is not normally a VM and does not have an explicit
hypervisor). VM is really just one use case of network virtualization and
not in itself a networking term, so I think that mechanisms or protocols
for network virtualization really should be described without reference to
VMs unless there really is something VM specific about that.

Tom

> Subject: Re: [nvo3] FW: New Version Notification for
draft-merged-nvo3-vm-mobility-scheme-00.txt
> OTOH, I would caution about getting too involved in this as both ends of
the spectrum of connection state preservation are reasonable and used in
practice:
>> draft-ietf-nvo3-vm- mobility-issues. We now call it
"draft-merged-nvo3-vm-mobility-scheme-00"
>> > URL:
http://www.ietf.org/internet-drafts/draft-merged-nvo3-vm-
>> > Htmlized:
http://tools.ietf.org/html/draft-merged-nvo3-vm-mobility-
>> scheme-00

---

From: Linda Dunbar <[email-masked]>
Date: Mon, 6 Oct 2014 19:41:05 +0000

Tom,
Are you saying that the term “VM” should only be described in motivation and the proposed scheme should work for address mobility enabled by any methods?

Linda

From: nvo3 [mailto:[email-masked]] On Behalf Of Tom Herbert
Sent: Monday, October 06, 2014 2:16 PM

While it's a better title, I think my comment was meant to be more general. VMs (and hypervisors) are references to specific mechanisms of server virtualization, but not the only mechanisms that can be deployed (e.g. container virtualization is not normally a VM and does not have an explicit hypervisor). VM is really just one use case of network virtualization and not in itself a networking term, so I think that mechanisms or protocols for network virtualization really should be described without reference to VMs unless there really is something VM specific about that.

Tom

> From: nvo3 [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Black, David
> Cc: [email-masked]<mailto:[email-masked]>
>> From: Tom Herbert [mailto:[email-masked]<mailto:[email-masked]>]
>> Cc: [email-masked]<mailto:[email-masked]>; Larry Kreeger (kreeger); Black, David
>> On Fri, Oct 3, 2014 at 10:22 AM, Linda Dunbar <[email-masked]<mailto:[email-masked]>> wrote:
>> > From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>]
>> until the htmlized version and diff are available at tools.ietf.org<http://tools.ietf.org>.
>> >
>> > The IETF Secretariat
>> >
>> > [email-masked]<mailto:[email-masked]>
> [email-masked]<mailto:[email-masked]>

---

From: David Allan I <[email-masked]>
Date: Mon, 6 Oct 2014 19:59:10 +0000

IMO “EVPN VM mobility framework” would be a significantly more accurate title…

Cheers
Dave

From: nvo3 [mailto:[email-masked]] On Behalf Of Linda Dunbar
Sent: Monday, October 06, 2014 12:41 PM
To: Tom Herbert
Cc: Black, David; [email-masked]
Subject: Re: [nvo3] Poll for a better name for draft-merged-nvo3-vm-mobility-scheme-00.txt

Tom,

Linda

Tom

---

From: Tom Herbert <[email-masked]>
Date: Mon, 6 Oct 2014 13:00:36 -0700

On Mon, Oct 6, 2014 at 12:41 PM, Linda Dunbar <[email-masked]> wrote:
> Tom,
>
> Are you saying that the term “VM” should only be described in motivation and
> the proposed scheme should work for address mobility enabled by any methods?
>
Yes, that is my hope, with the constraint that this solution is for
address mobility in realm DC network virtualization (i.e. this should
not be reinventing mobile IP for instance)

Tom

> not in itself a networking term, so I think that mechanisms or protocols for
> network virtualization really should be described without reference to VMs
> unless there really is something VM specific about that.
>
>
>
>
>
>
>
>
>
> Tom
>
>

---

From: Linda Dunbar <[email-masked]>
Date: Mon, 6 Oct 2014 20:22:09 +0000

Dave,

Thank you very much for the suggestion. I thought about a title along the line with what you suggested, but then realized it wasn’t accurate for the following reasons:

·       For those three issues: Default Gateway, Triangular routing, and Layer 2 extension,  both  NVA based solution and E-VPN based solution are feasible (and are described by the draft).

·       For VLAN-ID conflict issue in multiple L2 access domains, only NVA based solution is described.

·       Based on what Tom suggested, we are adding L3 address mobility, which will not be based on E-VPN.

Linda

From: David Allan I [mailto:[email-masked]]
Sent: Monday, October 06, 2014 2:59 PM
To: Linda Dunbar; Tom Herbert
Cc: Black, David; [email-masked]
Subject: RE: [nvo3] Poll for a better name for draft-merged-nvo3-vm-mobility-scheme-00.txt

Cheers
Dave

From: nvo3 [mailto:[email-masked]] On Behalf Of Linda Dunbar
Sent: Monday, October 06, 2014 12:41 PM
To: Tom Herbert
Cc: Black, David; [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] Poll for a better name for draft-merged-nvo3-vm-mobility-scheme-00.txt

Tom,

Linda

Tom

---

From: Behcet Sarikaya <[email-masked]>
Date: Mon, 6 Oct 2014 15:43:47 -0500

> not be reinventing mobile IP for instance)

Is this a comment on draft-sarikaya-nvo3-vmm-dmm-pmip-03.txt?

Regards,

Behcet

---

From: Tom Herbert <[email-masked]>
Date: Mon, 6 Oct 2014 14:16:21 -0700

> Is this a comment on draft-sarikaya-nvo3-vmm-dmm-pmip-03.txt?

i don't know...

Consider in Iast week's Interim meeting I described the "virtualizing
existing jobs" use case of networking virtualization. In this model,
we would like to be able to migrate existing jobs between servers. For
the networking part of this, we need to migrate addresses and and
connection state. These jobs don't run in VMs, they run in the host's
stack with some sort of containerization. While job is not migrating
there is no impact, performance and behavior are unchanged. At most we
need to perform encapsulation, but we may even be able to obviate that
with some create use of IPv6 addressing (Identifier/Locator).

Is this use case considered in your draft?

Side note on your draft: From the introduction "Currently these
networks are organized as one large Layer 2 network in a single
building". I don't believe this is universally true and I hope it's
not a fundamental design point, there are Layer 3 switched data center
networks.

Thanks,
Tom

---

From: "Andrew G. Malis" <[email-masked]>
Date: Mon, 6 Oct 2014 17:23:25 -0400

Dave,

It's not really what we think of as a "framework" document, as it contains
normative language, framework documents are usually informative in nature
and used to set the stage for solution drafts.

I also agree with Tom that it's more about address mobility than just VM
mobility, which is an instance of address mobility.

I prefer the very simple title "NVO3 Address Mobility".

Cheers,
Andy

On Mon, Oct 6, 2014 at 3:59 PM, David Allan I <[email-masked]>
wrote:

> *From:* nvo3 [mailto:[email-masked]] *On Behalf Of *Linda Dunbar
> *Sent:* Monday, October 06, 2014 12:41 PM
> *To:* Tom Herbert
> *Cc:* Black, David; [email-masked]
> *Subject:* Re: [nvo3] Poll for a better name for
> Are you saying that the term “VM” should only be described in motivation
> and the proposed scheme should work for address mobility enabled by any
> methods?
>
>
>
> Linda
>
>
>
> *From:* nvo3 [mailto:[email-masked] <[email-masked]>] *On
> Behalf Of *Tom Herbert
> *Sent:* Monday, October 06, 2014 2:16 PM
>
> While it's a better title, I think my comment was meant to be more
> general. VMs (and hypervisors) are references to specific mechanisms of
> server virtualization, but not the only mechanisms that can be deployed
> (e.g. container virtualization is not normally a VM and does not have an
> explicit hypervisor). VM is really just one use case of network
> virtualization and not in itself a networking term, so I think that
> mechanisms or protocols for network virtualization really should be
> described without reference to VMs unless there really is something VM
> specific about that.
>
>
>
>
>
>
>
>
>
>
>
> Tom
>
>
> >> > Status:
> https://datatracker.ietf.org/doc/draft-merged-nvo3-vm-
> >> mobility-scheme/
> >> > Htmlized:

---

From: Behcet Sarikaya <[email-masked]>
Date: Mon, 6 Oct 2014 17:11:07 -0500

> Is this use case considered in your draft?

Has  this problem been addressed in PS draft
(draft-ietf-nvo3-overlay-problem-statement)? VM mobility is addressed
there.
Do not worry about the acronyms pmip, dmip, etc. in the draft.

If I understand what you are talking about I can better provide an
opinion. To me address mobility is vm mobility, what is the
difference? Is there a draft?

> not a fundamental design point, there are Layer 3 switched data center

Yes, you are right.

Regards,

Behcet

---

From: Linda Dunbar <[email-masked]>
Date: Mon, 6 Oct 2014 22:48:49 +0000

David,

Thanks for the comments. Replies are inserted below in Maroon.

From: David Allan I [mailto:[email-masked]]
Sent: Monday, October 06, 2014 3:53 PM
To: Linda Dunbar; Tom Herbert
Cc: Black, David
Subject: RE: [nvo3] Poll for a better name for draft-merged-nvo3-vm-mobility-scheme-00.txt

Point taken, given multiple solutions are presented even for a single technology, I still think “framework” would be appropriate.

As for VLAN ID conflict, VLAN uniqueness in a given administrative domain, and VID translation between domains is so well understood I’m surprised it is an issue that needs more than a couple of sentences. It would be table stakes for any approach.

[Linda] VID translation is not an issue. The draft describes a way for NVA to notify NVEs how to map a given VN to a local VID, when to free a VID, etc. The local VID for a given VN can be changing. NVE needs to free up VIDs when there is no VMs underneath under the VIDs.. When the VN is re-connected to the NVE, a different available VID is assigned.
 To support tens of thousands of virtual networks, the local VID associated with client payload under each NVE has to be locally significant. Therefore, the same L2-based VN MAY have either the same or different VLAN-IDs under different NVEs.

I know it is also bleedingly obvious, but the document never says addresses cannot be in two places at once. At least not that I could find  ;-)

[Linda] Thank you very much for pointing out this. Are you saying that we should have a requirement/assumption that one address can only be attached to one NVE? Do you think adding one assumption at the beginning of the draft to state “we assume that one host can only be attached to one NVE” is good enough?
Or are you suggesting that the draft needs to address detecting duplicated addresses under different NVEs?

And one last thing… The sentence in 5.2 “When a VM of a
   given VN is moved into a NVE for the first time (i.e. the NVE didn't
   have any VMs belonging to this VN yet), the NVA will notify the NVE
   for it to be connected to VN.
IMO is a problem.

As phrased seems to explicitly couple things that do not have to be coupled as well as treating migration as some sort of special case.  “When an NVE needs to support connectivity to a VN not currently supported (as a result of VM turn up, or VM migration), the NVA will push the necessary VN information into the NVE.”  Seems to cover the waterfront…

[Linda] Thank you very much for the suggestion. Your suggested wording is definitely much more clear. I will take your suggested wording.

Cheers
Dave
From: Linda Dunbar [mailto:[email-masked]]
Sent: Monday, October 06, 2014 1:22 PM
To: David Allan I; Tom Herbert
Cc: Black, David; [email-masked]<mailto:[email-masked]>
Subject: RE: [nvo3] Poll for a better name for draft-merged-nvo3-vm-mobility-scheme-00.txt

Dave,

Linda

Cheers
Dave

Tom,

Linda

Tom

---

From: David Allan I <[email-masked]>
Date: Mon, 6 Oct 2014 23:52:17 +0000

HI Linda:
<snipped>
[Linda] VID translation is not an issue. The draft describes a way for NVA to notify NVEs how to map a given VN to a local VID, when to free a VID, etc. The local VID for a given VN can be changing. NVE needs to free up VIDs when there is no VMs underneath under the VIDs.. When the VN is re-connected to the NVE, a different available VID is assigned.
I read the words but cannot quite parse the contents.
 IMO there are customer administered VIDs (usually hard coded in applications when they exist). Provider administered VIDs of local significance, and provider administered VIDs of global significance.
In the scenario where there are provider administered VIDs of local significance (e.g. NVE in a TOR), the value is selected from the pool of unused VIDs when the first local client of a VN is being added, and returned to the unused pool of VIDs when the last client leaves. And as the architecture is defined, the NVA would have a hand in this. Is this what you are referring to?
I am curious as to why this figures prominently here, as it is simply normal operation to any VM lifecycle and not unique to VM migration.
 To support tens of thousands of virtual networks, the local VID associated with client payload under each NVE has to be locally significant. Therefore, the same L2-based VN MAY have either the same or different VLAN-IDs under different NVEs.

Of course…

[Linda] Thank you very much for pointing out this. Are you saying that we should have a requirement/assumption that one address can only be attached to one NVE? Do you think adding one assumption at the beginning of the draft to state “we assume that one host can only be attached to one NVE” is good enough?
Or are you suggesting that the draft needs to address detecting duplicated addresses under different NVEs?
Ethernet does like duplicate MAC addresses in a broadcast domain. So ideally during VM migration a given MAC address within a VN can only exist at one vNIC at a time (the network state may not immediately synchronize, but if the NVEs have a hard and ordered cut over, at least it does not flap). That strikes me as a requirement worth mentioning when discussing solutions for mobility.

<snipped to end>

Cheers
Dave

Dave,

Linda

Cheers
Dave

Tom,

Linda

Tom

---

From: "Sri Gundavelli (sgundave)" <[email-masked]>
Date: Tue, 7 Oct 2014 00:49:49 +0000

On 10/6/14 2:16 PM, "Tom Herbert" <[email-masked]> wrote:
>connection state. 

Clarifying question. Trying to understand the comment here; Not promoting
a specific approach/technology.
Does migration of "IP addresses and connection state" does not equate to
Layer-3 mobility ? 

>These jobs don't run in VMs, they run in the host's

If the container has its own IP address and is changing its point of
attachment across layer-3 boundaries, then to me those container migration
properties are matching the mobility of a typical mobile device. The use
of encapsulation is to hide the transport topology from the end point
addressing; FWIW, Identifier is a stable address, locator is your current
reachable point; In some other language this is the home address/care-of
address.

Sri

---

From: Tom Herbert <[email-masked]>
Date: Mon, 6 Oct 2014 18:27:32 -0700

> Does migration of "IP addresses and connection state" does not equate to

I believe they are mostly equivalent.

> Yes, although the identifier is probably not routable. The locator
(routable) is either the outer IP address in an encapsulation, or maybe
something like upper 64 bits in IPv6 address (where lower 64 are set to
identifier). LISP and ILNP are good references here.

Tom

---

From: "Sri Gundavelli (sgundave)" <[email-masked]>
Date: Tue, 7 Oct 2014 02:40:58 +0000

On Mon, Oct 6, 2014 at 5:49 PM, Sri Gundavelli (sgundave) <[email-masked]<mailto:[email-masked]>> wrote:

On 10/6/14 2:16 PM, "Tom Herbert" <[email-masked]<mailto:[email-masked]>> wrote:

> I believe they are mostly equivalent.

Yes. That was point. I almost thought there is a new mobility requirement that stands out here and was trying to understand that requirement.

If you take the case of a VM Migration (or any other light weigh virtualization models such as LXC), the mobility requirements related to migration across Layer-3 boundaries are the exactly same as for what typical layer-3 mobility protocols are designed for.  So, looking at the properties:

A VM has an IPv4 address and/or a set of IPv6 prefixes;
It can change its point of attachment moving across layer-3 network boundaries
VM migration comes with Application continuity requirement;
Handover latency should be very small (Ex: 50 msec) is desired
VM's can move as a cluster or as a individual node.
VM's can be multi-homed with application based path selection
VM's can negotiate QoS policies and provide Guaranteed-bit rates for set of flows
 .. And so on and so forth

=====> Now, if I replace "VM" with "mobile node", that's layer-3 mobility and we have few solution options there…

> Yes, although the identifier is probably not routable. The locator (routable) is either the outer IP address in an encapsulation, or maybe something like upper 64 bits in IPv6 address (where lower 64 are set to identifier). LISP and ILNP are good references here.

Right. That's from the LISP jargon; Sure, its one of the protocol options.

Regards
Sri

---

From: "Reith, Lothar" <[email-masked]>
Date: Tue, 7 Oct 2014 09:50:39 +0000

What about:

  „overlay network endpoint mobility framework“

Which leaves open if L2-endpoint ID or L3 endpoint ID or both IDs require mobility, and avoids the term VM which may be too narrow when considering using containerization instead of VMs for isolation on the server side of things.

Lothar

Von: nvo3 [mailto:[email-masked]] Im Auftrag von David Allan I
Gesendet: Montag, 6. Oktober 2014 21:59
An: Linda Dunbar; Tom Herbert
Cc: Black, David; [email-masked]
Betreff: Re: [nvo3] Poll for a better name for draft-merged-nvo3-vm-mobility-scheme-00.txt

Cheers
Dave

Tom,

Linda

Tom

---

From: Lucy yong <[email-masked]>
Date: Tue, 7 Oct 2014 13:48:17 +0000

Maybe “NVO tenant system mobility solution”?

Lucy

From: nvo3 [mailto:[email-masked]] On Behalf Of Reith, Lothar
Sent: Tuesday, October 07, 2014 4:51 AM
To: David Allan I; Linda Dunbar; Tom Herbert
Cc: Black, David; [email-masked]
Subject: Re: [nvo3] Poll for a better name for draft-merged-nvo3-vm-mobility-scheme-00.txt

What about:

Lothar

Cheers
Dave

Tom,

Linda

Tom

---

From: Tom Herbert <[email-masked]>
Date: Tue, 7 Oct 2014 07:39:34 -0700

On Mon, Oct 6, 2014 at 7:40 PM, Sri Gundavelli (sgundave) <
[email-masked]> wrote:

>     Yes. That was point. I almost thought there is a new mobility
> requirement that stands out here and was trying to understand that
> requirement.
>
>  If you take the case of a VM Migration (or any other light weigh
> virtualization models such as LXC), the mobility requirements related to
> migration across Layer-3 boundaries are the exactly same as for what
> typical layer-3 mobility protocols are designed for.  So, looking at the
> properties:
>
>  =====> Now, if I replace "VM" with "mobile node", that's layer-3
> mobility and we have few solution options there…
>
> I like that term! If we add that a mobile node may have
interfaces/addresses that reside in on or more virtual networks and apply
this to L2 also, then it is an a good abstraction that would cover the use
cases I have in mind.

---

From: "Black, David" <[email-masked]>
Date: Tue, 7 Oct 2014 15:41:05 +0000

I might change “mobility” to migration to avoid possible confusion with
mobile IP and drop the “framework” word as this draft is laying out
specific solutions.

Thanks,
--David

From: Reith, Lothar [mailto:[email-masked]]
Sent: Tuesday, October 07, 2014 5:51 AM
To: David Allan I; Linda Dunbar; Tom Herbert
Cc: Black, David; [email-masked]
Subject: AW: [nvo3] Poll for a better name for draft-merged-nvo3-vm-mobility-scheme-00.txt

What about:

Lothar

Cheers
Dave

Tom,

Linda

Tom

---

From: David Allan I <[email-masked]>
Date: Tue, 7 Oct 2014 15:59:50 +0000

Yes, Andy pointed out that the solutions are normative, not simply informative, so “my bad” for suggesting framework in the first place…

Cheers
Dave

From: nvo3 [mailto:[email-masked]] On Behalf Of Black, David
Sent: Tuesday, October 07, 2014 8:41 AM
To: Linda Dunbar
Cc: [email-masked]
Subject: Re: [nvo3] Poll for a better name for draft-merged-nvo3-vm-mobility-scheme-00.txt

Thanks,
--David

What about:

Lothar

Cheers
Dave

Tom,

Linda

Tom

---

From: Linda Dunbar <[email-masked]>
Date: Tue, 7 Oct 2014 16:04:55 +0000

David,

Replies are inserted below:

From: David Allan I [mailto:[email-masked]]
Sent: Monday, October 06, 2014 6:52 PM
To: Linda Dunbar; Tom Herbert
Cc: Black, David; [email-masked]
Subject: RE: Solutions for resolving access domain's VLAN-ID conflict ( was name poll for draft-merged-nvo3-vm-mobility-scheme -00.txt)

HI Linda:
<snipped>
[Linda] VID translation is not an issue. The draft describes a way for NVA to notify NVEs how to map a given VN to a local VID, when to free a VID, etc. The local VID for a given VN can be changing. NVE needs to free up VIDs when there is no VMs underneath under the VIDs.. When the VN is re-connected to the NVE, a different available VID is assigned.
I read the words but cannot quite parse the contents.
 IMO there are customer administered VIDs (usually hard coded in applications when they exist). Provider administered VIDs of local significance, and provider administered VIDs of global significance.
In the scenario where there are provider administered VIDs of local significance (e.g. NVE in a TOR), the value is selected from the pool of unused VIDs when the first local client of a VN is being added, and returned to the unused pool of VIDs when the last client leaves. And as the architecture is defined, the NVA would have a hand in this. Is this what you are referring to?
[Linda] Thanks for the suggested wording. I changed the text to the following. Is it more clear?
To describe the solution more clearly, here are the terminologies used:

-   Customer administered VLAN-IDs (usually hard coded in a TS’s Guest OS and can’t be changed when the TS move from one NVE to another). Some TSs may not have VLAN-ID attached.

-   Provider administered VLAN-IDs of local significance, and

-   Provider administered VN-IDs of global significance.
In the scenario where there are provider administered VLAN-IDs of local significance (e.g. NVE in a TOR), the value is selected by NVA from the pool of unused VIDs when the first local TS of a VN is being added, and returned by NVA to the unused pool of VLAN-IDs when the last TS leaves. For TSs with hard coded VLAN-ID, it is necessary for an entity, most likely the first switch (virtual or physical) to which the TS is attached, to change the locally administered VLAN-IDs to the TSs’ hard coded VLAN-IDs. For un-tagged TSs, the first switch has to remove the locally administered VLAN-IDs before sending packets to TSs.
The section is intended to describe:
·       NVA manages unused VLAN-IDs pool in each access L2 domain
·       NVE reports to NVA when first local TS of a VN is reachable, or none of TS in a VN is reachable by the NVE
·       NVA can push the global VN ID <-> locally administered VID mapping to NVE, or NVE can pull upon detecting a newly attached VN.
·       NVA manages the first switch to which TS is attached on mapping between TS’s own VLAN-ID and “locally administered VID”.

[Linda] When VMs don’t move, the “locally administered VID <-> TS’s VID” mapping is static. With VMs moving around, some VIDs being freed up and reassigned, the mapping becomes dynamic. The entity, most likely the first switch (virtual or physical) to which the TS is attached, need to interface with NVA to get the proper mapping when new VMs is attached.

[Linda] Thanks. I added a section on “Managing duplicated addresses” with the following words:
This document assumes that during VM migration a given MAC address within a VN can only exist at one TS at a time. As TSs move around NVEs, it is possible that the network state may not be immediately synchronized. It is important for NVEs to report directly attached TSs to NVA on periodically bases so that NVA can generate alarms and fix duplicated address issues.

Thank you very much for the suggestions.

Linda

Dave,

Linda

Cheers
Dave

Tom,

Linda

Tom

---

From: Linda Dunbar <[email-masked]>
Date: Tue, 7 Oct 2014 16:24:37 +0000

NVO3 framework introduced terminology TS (Tenant System) to describe Endpoint.
Combining all the input, we will have this name

“Overlay Network Tenant System Address Migration”

Is it good ? or too many words for the title?

Linda

What about:

Lothar

Cheers
Dave

Tom,

Linda

Tom

---

From: Thomas Narten <[email-masked]>
Date: Tue, 07 Oct 2014 12:54:28 -0400

> mobility and we have few solution options there…

No. Let's please not go here. As David pointed out in a different
thread, NVO3 is about VM *migration*.

Mobile IP is a very different beast built on a number of fundamentally
different assumptions. E.g., in mobile IP, the entire physical device
is mobile, not just a VM. Also, the physical device is moving, i.e.,
because its owner is carrying it around. In the DC, the VM is moving
because the DC operator wants to move it. Finally, in MIP, the mobile
device itself *knows* it is mobile and actively participates in that
mobility. In data centers, the VMs are oblivious to being moved and
are not themselves actively involved in any of the signaling or steps
of the move.

Thomas

---

From: Tom Herbert <[email-masked]>
Date: Tue, 7 Oct 2014 10:27:22 -0700

On Tue, Oct 7, 2014 at 9:54 AM, Thomas Narten <[email-masked]> wrote:

> Migration may be a better term than mobility, but the other part of the
phrasing, "node", was to encompass non-VM uses cases of network
virtualization migration in DC-- like job migration, container migration.
Is nvo3 explicitly about "VM" migration so that these other cases are out
of scope?

Thanks,
Tom

Thomas

---

From: Thomas Narten <[email-masked]>
Date: Tue, 07 Oct 2014 14:08:22 -0400

Hi Tom.

> Migration may be a better term than mobility, but the other part of
> the phrasing, "node", was to encompass non-VM uses cases of network
> virtualization migration in DC-- like job migration, container
> migration. Is nvo3 explicitly about "VM" migration so that these
> other cases are out of scope?

Other cases have always been in scope (e.g., Linux containers, AIX
LPARs, Solaris Zones, etc.). We just use the terms VMs and hypervisors
all the time because that is the common case, at least at the
moment. Looking at the framework and problem statement docs, this
isn't called out as clearly as it should be.

The next version of the architecture document will make this
clearer. I.e., we had  a discussion a while back on the list related
to Linux containers.

Thomas

---

From: Linda Dunbar <[email-masked]>
Date: Tue, 7 Oct 2014 18:39:28 +0000

Linux Container performs the same task as a hypervisor, i.e. enabling multiple isolated hosts (or VMs) running on a single physical machine. Why need a separate discussion for Linux Container?

Linda 

-----Original Message-----
From: Thomas Narten [mailto:[email-masked]] 
Sent: Tuesday, October 07, 2014 1:08 PM
To: Tom Herbert
Cc: Sri Gundavelli (sgundave); [email-masked]; [email-masked]; Black, David; Linda Dunbar
Subject: Re: [nvo3] Poll for a better name for draft-merged-nvo3-vm-mobility-scheme-00.txt

Hi Tom.

> migration. Is nvo3 explicitly about "VM" migration so that these other 
> cases are out of scope?

Other cases have always been in scope (e.g., Linux containers, AIX LPARs, Solaris Zones, etc.). We just use the terms VMs and hypervisors all the time because that is the common case, at least at the moment. Looking at the framework and problem statement docs, this isn't called out as clearly as it should be.

The next version of the architecture document will make this clearer. I.e., we had  a discussion a while back on the list related to Linux containers.

Thomas

---

From: Thomas Narten <[email-masked]>
Date: Tue, 07 Oct 2014 14:45:36 -0400

Hi Linda.

> Linux Container performs the same task as a hypervisor, i.e. enabling
> multiple isolated hosts (or VMs) running on a single physical
> machine. Why need a separate discussion for Linux Container?

Who is asking for a separate discussion?

I was just clarifying that Linux containers and other variants of
virtualization are in-scope for NVO3 and always have been and thus WG
documents need to cover them adequately. Just clarifying some apparent
confusion about that.

Thomas

---

From: Behcet Sarikaya <[email-masked]>
Date: Tue, 7 Oct 2014 13:53:39 -0500

> are not themselves actively involved in any of the signaling or steps

I agree with this.

Should we continue using VMs and use VM mobility?

Regards,

Behcet

---

From: "Sri Gundavelli (sgundave)" <[email-masked]>
Date: Wed, 8 Oct 2014 02:33:25 +0000

Hi Thomas,

> No. Let's please not go here.

No .. no. My comment is not intended to argue against inventing a new
approach; I'm not in the way. But, if we do not discuss and show a
reasonable technical argument as why an option needs to be ruled out, I
suspect we will end with the exact same answer, but with a different
technology title and functional entities.  I'm more curious to understand
how the VM mobility properties/requirements are any different from IP
mobility requirements of a classical mobile node. In today's deployments,
that classical mobile node is not always a cellular device /laptop, but
can also be a mobile router, IOT device with no user association.

IP Mobility protocols are providing layer-3 mobility to a device. There is
very little relation to the user. If the device moves by itself, or if the
user moves it physically, the protocol has no clue. If there exists any
relation to the user, its only about using the access authentication of
the user and binding that identity to the mobility session. But, even such
user relation is not present in mobile router/IOT deployments. The core
protocol only deals with signaling and managing the forwarding state.
There is no user identity semantic in the data-plane.
 
If a mobile node changes its point of attachment in the network, because
the owner of the device moves it, or when a VM moves across L3 boundaries
due to the administrator triggering a VM migration may not mean any thing
to the protocol underneath. If we take CDMA2000, the MIP stack is in the
chipset and there are many IOT devices with the cellular interface and
there is still IP Mobility for the device without a user operating it.

> the entire physical device is mobile,

This is a good point. But, however you look at it, the "Virtual Machine"
construct is presenting the view of a separate IP node. It has an
Operating System, set of applications, a logical interface card, IP
address configuration, forward stack, and a set of resources. Applications
are able to bind to an address, TCP/UDP ports and are able to send/receive
IP traffic. That Virtual Machine entity as a whole is moving across
networks. Now, why would it matter for layer-3 mobility protocols to be
aware of this subtle difference on a real mobile device, vs a VM instance
? Why is it relevant from the forwarding point of view ?

> Finally, in MIP, the mobile device itself *knows* it is mobile and
>actively participates in that mobility.

There exists two mobility models; Client-based and network-based. For the
later, there is no such assumption on the client awareness. The network is
responsible for providing the mobility.

Regards
Sri

On 10/7/14 9:54 AM, "Thomas Narten" <[email-masked]> wrote:

>> mobility and we have few solution options thereŠ
>

---

From: Tom Herbert <[email-masked]>
Date: Tue, 7 Oct 2014 20:32:54 -0700

> ? Why is it relevant from the forwarding point of view ?

You've reverted to posing the networking virtualization problem in
terms of virtual machines which leads to a use case specific
solution-- this is exactly the reason I suggested to not use the term.
The general problem is not (virtual) machine migration, it is a
problem of moving networking state between hosts and adapting the
network to be aware of this.

---

From: "Sri Gundavelli (sgundave)" <[email-masked]>
Date: Wed, 8 Oct 2014 03:43:23 +0000

>network to be aware of this.

That "networking" state when it is migrated from physical device-1 to
device-2, does not migrate alone; It takes along with it, all the
applications, connections and forwarding states. Those application expect
IP mobility and that moves the problem to the general category of
(virtual) machine migration.

Sri

---

From: "Black, David" <[email-masked]>
Date: Wed, 8 Oct 2014 04:07:54 +0000

Sri,

> (virtual) machine migration.

No, it does not take all that with it, sorry.  As I wrote earlier ...

------------
... both ends of the spectrum of connection state preservation are
reasonable and used in practice:

------------

The common element in both cases is the network address movement -
what other state also moves depends on why the network address(es)
are being moved.

If you disagree, please explain how to extract TCP connection
state from "a smoking pile of parts" ;-).

Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

---

From: "Sri Gundavelli (sgundave)" <[email-masked]>
Date: Wed, 8 Oct 2014 04:26:44 +0000

David,

Ok. Fair enough. Any transient, ephemeral and hardware-specific state does
not go with it. But, the question comes back, how much of that state is
relevant for the forwarding plane in the network. May be the applications
will rebuild that state on the target node. Why should a layer-3 mobility
protocol care about that state ?  So, if there be a definition of a new
protocol, is there an assumption that the new protocol will be aware of
such state ?

> The common element in both cases is the network address movement -

That's precisely the IP Mobility protocols that are out there are designed
for. IP address state is moved and we make the network aware of that.

Regards
Sri

---

From: "Black, David" <[email-masked]>
Date: Wed, 8 Oct 2014 04:58:33 +0000

> for. IP address state is moved and we make the network aware of that.

Fair enough, but they are not the only to move IP addresses.

I'm not advocating any new protocol for moving IP addresses - speaking for
myself, the techniques that I'm most interested in move IP addresses as
a side effect of moving layer 2 MACs (e.g., "gratuitous ARP").  There is
an enormous amount of deployed "running code" that does that without
using mobile IP; the techniques used by that "running code" fall squarely
into the "ain't broken" category, and I see no need to replace them with
IP mobility protocols.

The nvo3 work is of interest to me because in its most basic form, it can
extend those layer-2-based techniques to work across IP forwarding hops
in the underlay by providing effective layer-2 adjacency in the overlay,
and more sophisticated forms can optimize those techniques based on NVA
knowledge of what's going on.

Thanks,
--David

> From: Sri Gundavelli (sgundave) [mailto:[email-masked]]
> Sent: Wednesday, October 08, 2014 12:27 AM
> To: Black, David
> Cc: [email-masked]

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 8 Oct 2014 06:28:01 -0700

There is a real problem that this working group could work on. It has been a problem with VM-mobility for a while now (4 years now). The fact that when the IP address moves and doesn't signal to the network is a REAL problem.

Come up with a protocol that says "hi-I-have-arrived" as well as "bye-I-am-leaving". This protocol needs to be explicit.

In the VM-mobility use-cases I have been involved in, you cannot depend on ARP because not all hypervisors will ARP when they come and go. The only solution today, when there is no involvement from the IP address entity, is to listen to the MAC address it is sending on. This is kludgy and causes all sorts of hoops the implementation has to jump through.

So focus on a "host to NVE" protocol would bring something to the table.

Dino

---

From: "Sri Gundavelli (sgundave)" <[email-masked]>
Date: Wed, 8 Oct 2014 14:44:01 +0000

> Come up with a protocol that says "hi-I-have-arrived" as well as
>"bye-I-am-leaving". This protocol needs to be explicit.

Agree.

The MIP working groups have looked at this interface, also referred to as
MN-AR interface (Mobile Node to Access Router interface). We attempted to
publish a reference interface, so the SDO's can realize this interface
using access specific approaches;
https://tools.ietf.org/html/draft-ietf-netlmm-mn-ar-if-03. As you stated,
the goal was to allow the mobile node to provide ATTACH/DETACH events in
an explicit manner.

In deployments where there is no such access-specific interface (also for
dealing with unmodified clients) MIP architectures used subscriber's
access authentication and additionally DHCP/ND triggers. For WLAN access,
its the 802.1x that provided the semantics. Attach event including the
authenticated identity was always available to the gateway. However,
DETACH is always an issue and we used unicast ARP/NS polling approaches,
radio specific and other approaches for detecting MN loss on the link.

Sri

On 10/8/14 6:28 AM, "Dino Farinacci" <[email-masked]> wrote:

>There is a real problem that this working group could work on. It has
>been a problem with VM-mobility for a while now (4 years now). The fact
>that when the IP address moves and doesn't signal to the network is a
>REAL problem.
>
>Come up with a protocol that says "hi-I-have-arrived" as well as
>"bye-I-am-leaving". This protocol needs to be explicit.
>
>In the VM-mobility use-cases I have been involved in, you cannot depend
>on ARP because not all hypervisors will ARP when they come and go. The
>only solution today, when there is no involvement from the IP address
>entity, is to listen to the MAC address it is sending on. This is kludgy
>and causes all sorts of hoops the implementation has to jump through.
>
>>> Subject: Re: [nvo3] Poll for a better name for

---

From: Linda Dunbar <[email-masked]>
Date: Wed, 8 Oct 2014 14:51:20 +0000

Dino, 

IEEE802.1Qbg developed this host to Edge (NVE) protocol more than 3 years ago. The problem is that the servers/hypervisors deployed in data centers are from many different vendors and they all have their own behavior. That is why 802.1Qbg hasn't be widely deployed. 

Sad to say that networking is only a very small portion (<15%) of entire data center cost. 

It is not Lack of Protocol, but it is lack of interests from server vendors.

However, network can utilize Server Management (e.g. vCenter or SystemCenter via NVA) to notify all the NVEs on the attached IP addresses. 

Linda

-----Original Message-----
From: nvo3 [mailto:[email-masked]] On Behalf Of Dino Farinacci
Sent: Wednesday, October 08, 2014 8:28 AM
To: Black, David
Cc: [email-masked]; Sri Gundavelli (sgundave)
Subject: Re: [nvo3] Poll for a better name for draft-merged-nvo3-vm-mobility-scheme-00.txt

Dino

On Oct 7, 2014, at 9:58 PM, Black, David <[email-masked]> wrote:

>> That's precisely the IP Mobility protocols that are out there are 
>> designed for. IP address state is moved and we make the network aware of that.
> 
> I'm not advocating any new protocol for moving IP addresses - speaking 
> for myself, the techniques that I'm most interested in move IP 
> addresses as a side effect of moving layer 2 MACs (e.g., "gratuitous 
> ARP").  There is an enormous amount of deployed "running code" that 
> does that without using mobile IP; the techniques used by that 
> "running code" fall squarely into the "ain't broken" category, and I 
> see no need to replace them with IP mobility protocols.
> 
> The nvo3 work is of interest to me because in its most basic form, it 
> can extend those layer-2-based techniques to work across IP forwarding 
> hops in the underlay by providing effective layer-2 adjacency in the 
> overlay, and more sophisticated forms can optimize those techniques 
> based on NVA knowledge of what's going on.
> 
> Thanks,
> --David
> 
>> draft-merged-nvo3-vm-mobility- scheme-00.txt
>> 
>> David,
>> 
>> Ok. Fair enough. Any transient, ephemeral and hardware-specific state 
>> does not go with it. But, the question comes back, how much of that 
>> state is relevant for the forwarding plane in the network. May be the 
>> applications will rebuild that state on the target node. Why should a 
>> layer-3 mobility protocol care about that state ?  So, if there be a 
>> definition of a new protocol, is there an assumption that the new 
>> protocol will be aware of such state ?
>> 
>> 
>> That's precisely the IP Mobility protocols that are out there are 
>> designed for. IP address state is moved and we make the network aware of that.
>> 
>> 
>> 
>> Regards
>> Sri
>> 
>> 
>> 
>>>> That "networking" state when it is migrated from physical device-1 
>>>> to device-2, does not migrate alone; It takes along with it, all 
>>>> the applications, connections and forwarding states. Those 
>>>> application expect IP mobility and that moves the problem to the 
>>>> general category of
>>> If you disagree, please explain how to extract TCP connection state 
>>> from "a smoking pile of parts" ;-).
>>> 
>>> Thanks,
>>> --David
>>> David L. Black, Distinguished Engineer EMC Corporation, 176 South 
>>> St., Hopkinton, MA  01748

---

From: Lucy yong <[email-masked]>
Date: Wed, 8 Oct 2014 15:00:22 +0000

Agree what you said. If NVE resides on the End device (host in your term), internal states can serve the purpose of the protocol. In the split-NVE case, i.e. use external NVE, such protocol between end device and external NVE is necessary. The protocol requirements are described in draft-ietf-nvo3-hyrv2nve-cp-req. 

Lucy

-----Original Message-----
From: nvo3 [mailto:[email-masked]] On Behalf Of Dino Farinacci
Sent: Wednesday, October 08, 2014 8:28 AM
To: Black, David
Cc: [email-masked]; Sri Gundavelli (sgundave)
Subject: Re: [nvo3] Poll for a better name for draft-merged-nvo3-vm-mobility-scheme-00.txt

Dino

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 8 Oct 2014 08:09:53 -0700

This protocol is needed for ANY IP address that moves, no matter what type of device or layer-2 physical technology is used.

Think of the equivalent of what IGMP does for receivers (but we needed a "request-to-send" for multicast sources that was never developed it).

Dino

---

From: Sharon <[email-masked]>
Date: Wed, 8 Oct 2014 10:14:20 -0700

I think this is exactly the point.

There are IP address mobility protocols which requires concentrated anchors so the internet doesn't think they move

This approach is not usable in a Datacenter because you can't create anchor bottlenecks. What does work is not the "gratuitous arp" per say, but the two tier ID-Location network aka the NVE.

This enables the dynamic allocation of distributed mobility anchors, albeit this requires some mechanism to distribute context aka the NVA.

Some "working code" defacto NVA mechanisms out there assume very few addresses can move e.g. just the servers on VMs. Some defacto NVAs out there Don't assume that, and allow Any IP address can move, server Or Client.

The WG insists on treating these two different defacto code bases the same eg the BGP and LISP text.
But they are not the same. As we dive into the all these detailed issues it will become further evident.

--szb

---

From: Behcet Sarikaya <[email-masked]>
Date: Thu, 16 Oct 2014 14:14:49 -0500

> clearer. I.e., we had  a discussion a while back on the list related

The architecture document now became an RFC, it is RFC 7365.
Which next version will make this clearer?

Regards,

Behcet

---

From: "Black, David" <[email-masked]>
Date: Thu, 16 Oct 2014 20:14:15 +0000

> Which next version will make this clearer?

That would be the framework document (draft-ietf-nvo3-framework).

There should be a new version of the architecture document
(draft-ietf-nvo3-arch) before the Honolulu draft cutoff.

Thanks,
--David

> From: nvo3 [mailto:[email-masked]] On Behalf Of Behcet Sarikaya
> Sent: Thursday, October 16, 2014 3:15 PM
> To: Thomas Narten
> Cc: [email-masked]