Subject: [nvo3] Comments on draft-kompella-nvo3-server2nve

From: <[email-masked]>
Date: Fri, 13 Jul 2012 16:01:19 -0400

Authors (Kireeti, Yakov and Thomas),

This is a good draft - it looks like a good foundation to focus discussion
around what the server-to-NVE (attach/detach) protocol needs to do.  I
like a lot of the contents - I have a few high level comments and some
more detailed feedback.

(1) This draft starts out dealing with the attach/detach (server-to-NVE)
protocol and then includes some material on the control protocol for
distributing and managing mapping information on the NVEs.  I suggest
focusing the draft on the attach/detach protocol, removing control
protocol discussion (e.g., Section 3), and minimizing assumptions about
the control protocol (see detailed comments for where I think assumptions
could be minimized).  The result should be more general and more useful.

(2) Section 2.2.3 on detach is trying to cover at least a couple of use
cases,  VM live migration, and VM removal (e.g., power-off) that probably
want to be separated. The current text really doesn't get the live migration
case right, D.4 comes before D.3 for the power-off case, and I think things
get more complex when the live migration detach functionality is corrected.

(3) Section 2.2.4 appears to assume a specific order of events between
the two servers involved in VM migration.  As those servers are operating
concurrently, that's not a robust assumption, and the NVE functionality
should be specified to not depend on the order of events.

--- Detailed comments by section ---

A pre-disassociate operation is defined in section 2.2.1 but not used 
in the rest of the draft.  Is it actually needed?

-- Section 2.2.2

   A.1:  Validate the authentication (if present).  If not, inform the
         provisioning system, log the error, and stop processing the
         associate message.

This step should also include an optional authorization check, as network
policy may limit which NVEs are allowed to participate in which VNs.

   A.3:  If the VID in the associate message is non-zero, look up <VNID,
         P>.  If the result is zero, or equal to VID, all's well.
         Otherwise, respond to S with an error, and stop processing the
         associate message.

Why is a zero VID lookup result ok for a non-zero VID in the associate
message?  Should the NVE copy the VID from the associate message to the
<VNID,P> entry before responding?

   A.5:  Communicate with each rNVE device to advertise the VM's
         addresses, and also to get the addresses of other VMs in the
         DCVPN.  Populate the table with the VM's addresses and
         addresses learned from each rNVE.

This assumes that the control protocol does active propagation of all
address info, and assumes that no other addresses for the VN are present
in the NVE.  Neither of those are good general assumptions, IMHO, and
in particular, lazy evaluation is possible (e.g., load address mappings
on demand to reduce the amount of invalidation traffic caused by
each mapping change).  I'd suggest rephrasing to something like:

   A.5:  Use the overlay control protocol to inform the network of the
         VM's addresses and the VM's association with this NVE. 

-- Section 2.2.3

   D.1:  Validate the authentication (if present).  If not, inform the
         provisioning system, log the error, and stop processing the
         associate message.

Like A.1, this should include an optional authorization check, as some
<VNID,P> -> VID mappings may be statically configured and hence not
permit removal.

   D.2:  If the hold time is non-zero, point the VM's addresses in the
         VNID table to the new location of the VM, if known, or to
         "discard", and start a timer for the period of the hold time.
         If hold time is zero, immediately perform step D.4, then go to
         D.3.

This is where the power-off and migration cases start to interact - 
Hold time would be zero for power-off, non-zero for detach.  For migration,
this change potentially races with a change to the VM's addresses received
via the control protocol, so the VM's address may already point somewhere
else if the control protocol did its update before the dissociate (in
which case nothing should be done to those addresses).

   D.3:  Set the VID for <VNID, P> as unassigned.  Respond to S saying
         that the operation was successful.

If there are multiple VMs using the VNID on that port, this
"pulls the rug" out from under the others by disabling their forwarding.
This <VNID,P> -> VID mapping needs a reference count of some form, and
corresponding changes would be needed to A.2 and A.3.  Not using a
reference count may be ok under the assumption that the NVE does not
share ports among VMs (or VSIs/vNICs), but that may not be a good
assumption for an external NVE (e.g., in a ToR switch).

   D.4:  When the hold timer expires, delete the VM's addresses from the
         VNID table.  Delete any VM-specific network policies associated
         with any of the VM addresses.  If the VNID table is empty after
         deleting the VM's addresses, optionally delete the table and
         any network policies for the VNID.

Well, that's the right thing to do in the power-off case, but not
when the VM has moved and there are other VMs on this NVE (possibly even
the same port) that still need to communicate with the moved VM.  Also,
the power-off case needs to include (at least optionally) informing the
control protocol of the withdrawal of the VM's addresses.

As noted in (2) above, I think it would be clearer if there were separate
versions of 2.2.3 for the migration departure and power-down use cases.

-- Section 2.2.4

   M.3:  S then gets a request to terminate the VM on S.

   M.4:  Finally, S' gets a request to start up the VM on S'.

Not exactly ;-).

Terminating the VM on S (and destroying its state) before confirming
its startup on S' risks losing the VM entirely if something goes wrong
on S'.  This level of detail isn't necessary - from the point of view
of the network:
- Startup on S' generates an associate request to the NVE for S'.
- The dissociate request from S to its NVE may occur before or after
	that S' associate request
- The dissociate request from S to its NVE may occur before or after
	control protocol propagation of the results of the S' associate
	request to the NVE for S.
The server-to-NVE functionality should be specified to operate properly
independent of the order of these events.

   PA.5:  Communicate with each rNVE device to advertise the VM's
      addresses but as non-preferred destinations(*).  Also get the
      addresses of other VMs in the DCVPN.  Populate the table with the
      VM's addresses and addresses learned from each rNVE.

That assumes aggressive push of the new address information by the
control protocol directly to the rNVEs - while a control protocol
may choose to do that, it's not strictly necessary and the interaction
may not be directly between the lNVE and the rNVEs.  Generalizing in
a fashion similar to A.5, I'd suggest something like:

   PA.5:  The overlay control protocol may be used to inform the
      network of the forthcoming change to the VM's addresses
      that will occur when the VM is associated with this NVE.

If this is done, withdrawal of the tentative address changes
needs to be discussed, as VM migrations can abort for a variety
of reasons (e.g., S' may crash during the copy).  This PA.5
step can be skipped for a control protocol only does on-demand
provisioning of the address mapping information.

-- Section 3

This appears to be entirely about the control protocol and (IMHO)
doesn't fit well with the rest of the draft.

Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 13 Jul 2012 22:47:36 +0200

All,

Since we are starting to discuss this document I have one fundamental 
comment/question.

The part of it's abstract says:

    This document proposes a "single-touch" approach for provisioning the
    networking parameters related to Virtual Machine creation, migration
    and termination on servers.

I would like to observe that in the tenant data center VMs are only a 
small part of the tenant network.

Single-touch is of course great idea and that is why extension to one of 
the common and very popular data center orchestration environments 
(openstack) developed a module called Quantum.

This module is exactly targeting to not only provision networking 
parameters to VMs but also to other network appliances including virtual 
routers, firewalls, loadbalancers etc .. which are part of the tenant DC 
network.

Ref: http://wiki.openstack.org/Quantum

So I would like to ask the authors of the draft-kompella-nvo3-server2nve 
why there is the an attempt to reinvent the wheel and how far they are 
going to maintain the vision of "single-touch" considering that much 
broader aspect of this has been already solved ?

Best regards,
R.

PS. I am perfectly fine to hear that NVO3 does not want to adopt any 
work which is done outside of IETF. However I think that such message 
needs to be clearly weighted against current deployments as well as 
existing market adoption of such solutions.

---

From: Kireeti Kompella <[email-masked]>
Date: Sat, 14 Jul 2012 17:16:26 -0700

Hi David,

Thanks for your detailed comments!  More inline.

> like a lot of the contents - I have a few high level comments and some

Thanks!

> could be minimized).  The result should be more general and more useful.

About the control plane: it really concerns me that the control plane
discussion has not happened so far (not really).  ARP doesn't scale;
neither does flooding.  The goal here is to signal networking parameters:
from server (vswitch) to local NVE to remote NVEs to remote servers.  Fine,
call the local NVE to remote NVEs part "control plane" -- but that's a
critical part of the picture.

What I take from your suggestion is to move the lNVE to rNVE part to a
different draft; I buy that, especially if there are other mechanisms for
doing this that can plug-in to this server2nve signaling, so that one can
mix-and-match server2nve signaling and lNVE2rNVE signaling.  Does that seem
reasonable?

> get more complex when the live migration detach functionality is corrected.

More on this below.

> should be specified to not depend on the order of events.

Ordering assumptions weren't intended, so we'll tweak the wording to remove
any such implications.

> in the rest of the draft.  Is it actually needed?

Good catch!  I'd put that in early, worked out the rest of the details,
couldn't figure out a use for it, but forgot to remove it.  I'll remove it.

> policy may limit which NVEs are allowed to participate in which VNs.

Okay.  Authorization locally, or from the provisioning system?  (Or either?)

> message?

Just means no mapping yet.  With respect to the refcounting suggested
below, good place to set it to 1; otherwise increment.

> Should the NVE copy the VID from the associate message to the

Good point.  Will fix.

> each mapping change).

I'm leery of on-demand/cache-based address mappings and lazy evaluation
(love it in general, but not for address mappings).  However, you're right:
there may be cases where this is a valid approach.

>          VM's addresses and the VM's association with this NVE.

Something like that.  Will work on text.

> <VNID,P> -> VID mappings may be statically configured and hence not

Okay, will copy wording from there once we've agreed on it.

> which case nothing should be done to those addresses).

Definitely worth looking at again, especially with respect to your comments
about the order for migration.

With regard to the race condition, I'll send a separate email on that.

> assumption for an external NVE (e.g., in a ToR switch).

Good point!  I'll go with refcounting.

> control protocol of the withdrawal of the VM's addresses.

See separate email.

> versions of 2.2.3 for the migration departure and power-down use cases.

Perhaps.  Let's get the semantics right first, then see if there are common
elements or not.

> on S'.

Interesting point.  However, if the VM starts on S' without first being
stopped on S, then (for some time) both S and S' are running, and I'd think
that the results would be unpredictable, especially if the VM is just about
to engage in some I/O.  However, I'll bow to those who've implemented VM
migration and know what they're doing.  Perhaps the VM is paused on S,
started on S'; if that's successful, the VM is destroyed on S, otherwise
the migration is aborted and the VM is continued on S.  I'd like to know,
as this affects the "tentative address changes" you talk about below, and
dealing with migration abort.

This level of detail isn't necessary - from the point of view

> independent of the order of these events.

Agreed.  Separate email thread to work this out.

>       that will occur when the VM is associated with this NVE.

Okay, something like.

> provisioning of the address mapping information.

Interesting thought.  Will follow up once we get the migration "right" (for
some value of right).

> doesn't fit well with the rest of the draft.

Will discuss putting this in a separate draft with co-authors.

Thanks again for the detailed comments!
Kireeti.

> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3
>

-- 
Kireeti

---

From: <[email-masked]>
Date: Sun, 15 Jul 2012 12:57:53 -0400

Hi Kireeti,

Your response looks good - we appear to be thinking along similar lines.
I'll add a few comments and wait for the next version of the draft ...

> What I take from your suggestion is to move the lNVE to rNVE part to a different draft;
> I buy that, especially if there are other mechanisms for doing this that can plug-in to
> this server2nve signaling, so that one can mix-and-match server2nve signaling and
> lNVE2rNVE signaling.  Does that seem reasonable?

Yes, that's *exactly* what I had in mind.

> Okay.  Authorization locally, or from the provisioning system?  (Or either?)

Either, both and more :-).  An important case is that this may be from the
"network" (e.g., via the control protocol) if the network imposes scope
boundaries on virtual networks.  For example, suppose that a tenant has not
paid the service provider for data-center-wide scope (or multiple-data-center
scope) of her virtual networks - in this case the service provider will impose
restrictions on where her VNs are usable, independent of what she may be
able to convince the provisioning system to ask for.

> Perhaps the VM is paused on S, started on S'; if that's successful, the VM is
> destroyed on S, otherwise the migration is aborted and the VM is continued on S.

Something like that - that summary is good enough for this draft, and the
important points (on which I believe we agree) are:
- There is a "paused" or "suspended" state of a VM in addition to the
powered-on and powered-off states.
- The concurrency (lack of ordering) requirements on the network-visible
events that we agree on and that you're going to send a separate
message about.

I have a related consideration based on thinking about this further.  The network
SHOULD NOT rely on dissociate messages always being sent - a server crash at the
wrong point during a VM migration may cause a dissociate to be missed (e.g., the
VM made it to S', but S crashed before sending the dissociate).  More importantly,
not relying on the dissociate messages (in particular, not having the inter-NVE
control protocol rely on them) helps if one wants to mix hypervisors that support
the attach/detach protocol with (exiting) ones that don't.  For existing hypervisors,
under suitable restrictions and assuming some advance configuration, "associate"
can be inferred from a gratuitous ARP or RARP, but nothing is sent for dissociate.
The inference of "associate" won't be possible if things have not been set up
to enable the gratuitous ARP or RARP.

Thanks,
--David

From: [email-masked] [mailto:[email-masked]] On Behalf Of Kireeti Kompella
Sent: Saturday, July 14, 2012 8:16 PM
To: Black, David
Cc: [email-masked]
Subject: Re: [nvo3] Comments on draft-kompella-nvo3-server2nve

Hi David,

Thanks for your detailed comments!  More inline.
On Fri, Jul 13, 2012 at 1:01 PM, <[email-masked]<mailto:[email-masked]>> wrote:
Authors (Kireeti, Yakov and Thomas),

Thanks!

About the control plane: it really concerns me that the control plane discussion has not happened so far (not really).  ARP doesn't scale; neither does flooding.  The goal here is to signal networking parameters: from server (vswitch) to local NVE to remote NVEs to remote servers.  Fine, call the local NVE to remote NVEs part "control plane" -- but that's a critical part of the picture.

What I take from your suggestion is to move the lNVE to rNVE part to a different draft; I buy that, especially if there are other mechanisms for doing this that can plug-in to this server2nve signaling, so that one can mix-and-match server2nve signaling and lNVE2rNVE signaling.  Does that seem reasonable?

More on this below.

Ordering assumptions weren't intended, so we'll tweak the wording to remove any such implications.

--- Detailed comments by section ---

Good catch!  I'd put that in early, worked out the rest of the details, couldn't figure out a use for it, but forgot to remove it.  I'll remove it.

-- Section 2.2.2

Just means no mapping yet.  With respect to the refcounting suggested below, good place to set it to 1; otherwise increment.

Good point.  Will fix.

I'm leery of on-demand/cache-based address mappings and lazy evaluation (love it in general, but not for address mappings).  However, you're right: there may be cases where this is a valid approach.

-- Section 2.2.3

Definitely worth looking at again, especially with respect to your comments about the order for migration.

See separate email.

Perhaps.  Let's get the semantics right first, then see if there are common elements or not.

-- Section 2.2.4

Not exactly ;-).

Interesting point.  However, if the VM starts on S' without first being stopped on S, then (for some time) both S and S' are running, and I'd think that the results would be unpredictable, especially if the VM is just about to engage in some I/O.  However, I'll bow to those who've implemented VM migration and know what they're doing.  Perhaps the VM is paused on S, started on S'; if that's successful, the VM is destroyed on S, otherwise the migration is aborted and the VM is continued on S.  I'd like to know, as this affects the "tentative address changes" you talk about below, and dealing with migration abort.

Okay, something like.

Interesting thought.  Will follow up once we get the migration "right" (for some value of right).

-- Section 3

Thanks,
--David
[email-masked]<mailto:[email-masked]>        Mobile: +1 (978) 394-7754<tel:%2B1%20%28978%29%20394-7754>

_______________________________________________
nvo3 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nvo3

--
Kireeti

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Tue, 17 Jul 2012 16:07:26 +0000

I also have a similar fundamental issue with this document.  The document seems to be dictating provisioning process for a VM. What service/cloud computing providers need are the building blocks, i.e. standard APIs and protocols to create and manage virtual networks in cloud computing environment. Most of those APIs already exist (e.g., OpenStack REST API, IF-MAP API, etc.). How to put the solution blocks together will be individually decided by the providers. There already exist OpenStack Tenant/Project and Quantum Project that allow creation and management of Virtual Networks. There are many available Quantum plugins. Actually, such models are proven to scale and have been already used by many large cloud computing providers. As Robert said, let's not "reinvent the wheel".

Besides this fundamental issue (which, in fact, makes any other comments less relevant) I would like to point out the following:

Section 1.
   "This document focuses on the case where the network elements in Step
   4 are not co-resident with the server, and shows how the provisioning
   in Step 4 can be replaced by signaling between server and network,"

Allowing network elements to signal provisioning information directly to each other is not a secure approach, especially if one of those elements is carrying guest workloads. It is especially insecure if the signaling protocol used cannot be authenticated. It is much more secure to delegate such coordination to a management system. Hence, devoting entire draft to this single, narrow issue, which for majority, especially large Service or Cloud Providers would be irrelevant, seems to miss the mark.
It is much more relevant to address the case where NVE is in the end-system and to address the signaling used between the end-system and its controller.

Section 2.3
   "There are several options for protocols to use to signal the above
   messages.  One could invent a new protocol for this purpose.  One
   could reuse existing protocols, among them LLDP, XMPP, HTTP REST, and
   VDP [VDP], a new protocol standardized for the purposes of signaling
   a VM's network parameters from server to lNVE"

I don't think that these protocols should be listed as "equal". There are major differences between them from security, extensibility, and applicability point of view.

Section 3
    "DCVPNs may be realized using BGP-based VPNs, for example, BGP/MPLS
     "VPNs ([RFC4364]),"

This approach is addressed in draft-fang-vpn4dc-problem-statement-01.

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Robert Raszuk
> Sent: Friday, July 13, 2012 4:48 PM
> To: [email-masked]

---

From: Xuxiaohu <[email-masked]>
Date: Thu, 19 Jul 2012 02:47:41 +0000

Does that mean the ARP could also be considered as an option for signaling the VM attachment/detachment event? For example, a gratuitous ARP packet can be inferred as an attachment event by the NVE which receives such packet via the NVE-TES interface. Meanwhile, for those L2VPN  (e.g., VPLS)  or L3VPN overlay approaches which only allows one next-hop to be available  for a given MAC route or a host route in the forwarding table,  a gratuitous ARP packet received from a remote NVE could be inferred as a detachment event by the NVE to which the ARP sending VM was previously attached. Moreover, in case a gratuitous ARP packet triggers the NVE which received that packet via the NVE-TES interface to generate a MAC route or a host route for the ARP sending VM, and the NVE to which that VM was previously attached, upon receiving that route,  could also infer that route as a detachment event of that VM.

Best regards,
Xiaohu

<skipped>

Thanks,
--David

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]<mailto:[mailto:[email-masked]]> On Behalf Of Kireeti Kompella
Sent: Saturday, July 14, 2012 8:16 PM
To: Black, David
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] Comments on draft-kompella-nvo3-server2nve

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: Linda Dunbar <[email-masked]>
Date: Mon, 23 Jul 2012 15:59:28 +0000

Millions of applications being deployed already use ARP to signal their presence. The widely deployed vMotion makes VMs in new location to send ARP (RARP) to inform the network of their new location. It doesn't hurt to utilize the available messages from applications.

My two cents.

Linda Dunbar

From: [email-masked] [mailto:[email-masked]] On Behalf Of Xuxiaohu
Sent: Wednesday, July 18, 2012 9:48 PM
To: [email-masked]; [email-masked]
Cc: [email-masked]
Subject: Re: [nvo3] Comments on draft-kompella-nvo3-server2nve

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: "Ivan Pepelnjak" <[email-masked]>
Date: Mon, 23 Jul 2012 18:22:47 +0200

Just because millions of applications misuse a simplistic protocol in a way it was never designed to handle doesn’t make it a good idea. Not to mention the total lack of security.

@Xiaohu: how would you distinguish a gratuitous ARP send from the hypervisor to indicate a VM move from a gratuitous ARP sent by a VM with a misconfigured IP address or a malicious gratuitous ARP sent by an intruder (physical or virtual)? Unless you can totally control the VM attachment point (= hypervisor switch unless you’re using something like 802.1BR) you cannot trust ARP ... but then if you do control the hypervisor switch, you don’t need ARP.

Ivan

From: [email-masked] [mailto:[email-masked]] On Behalf Of Linda Dunbar
Sent: Monday, July 23, 2012 5:59 PM
To: Xuxiaohu; [email-masked]; [email-masked]
Cc: [email-masked]
Subject: Re: [nvo3] Comments on draft-kompella-nvo3-server2nve

Millions of applications being deployed already use ARP to signal their presence. The widely deployed vMotion makes VMs in new location to send ARP (RARP) to inform the network of their new location. It doesn’t hurt to utilize the available messages from applications. 

My two cents. 

Linda Dunbar

Best regards,

Xiaohu

<skipped>

VM made it to S’, but S crashed before sending the dissociate).  More importantly,

the attach/detach protocol with (exiting) ones that don’t.  For existing hypervisors,

under suitable restrictions and assuming some advance configuration, “associate”

The inference of “associate” won’t be possible if things have not been set up

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Kireeti.

Thanks,
--David
[email-masked]        Mobile: +1 (978) 394-7754 <tel:%2B1%20%28978%29%20394-7754> 

 

-- 
Kireeti

---

From: Truman Boyes <[email-masked]>
Date: Mon, 23 Jul 2012 13:02:51 -0400

On Mon, Jul 23, 2012 at 12:22 PM, Ivan Pepelnjak <[email-masked]>wrote:

> Just because millions of applications misuse a simplistic protocol in a
> way it was never designed to handle doesn’t make it a good idea. Not to
> mention the total lack of security.****
>
> ** **
>
> @Xiaohu: how would you distinguish a gratuitous ARP send from the
> hypervisor to indicate a VM move from a gratuitous ARP sent by a VM with a
> misconfigured IP address or a malicious gratuitous ARP sent by an intruder
> (physical or virtual)? Unless you can totally control the VM attachment
> point (= hypervisor switch unless you’re using something like 802.1BR) you
> cannot trust ARP ... but then if you do control the hypervisor switch, you
> don’t need ARP.****
>
> ** **
>
> Ivan
>

Completely agree. ARP is the wrong protocol for any form of trusted
signaling. Additionally, IPv6-only networks would present their own set of
challenges if we had a reliance on  IPv4 address resolution protocol.

Truman

> ****
>
> ** **
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf
> Of *Linda Dunbar
> *Sent:* Monday, July 23, 2012 5:59 PM
> *To:* Xuxiaohu; [email-masked]; [email-masked]
>
> *Cc:* [email-masked]
> *Subject:* Re: [nvo3] Comments on draft-kompella-nvo3-server2nve****
>
> ** **
>
> Millions of applications being deployed already use ARP to signal their
> presence. The widely deployed vMotion makes VMs in new location to send ARP
> (RARP) to inform the network of their new location. It doesn’t hurt to
> utilize the available messages from applications. ****
>
> ** **
>
> My two cents. ****
>
> ** **
>
> Linda Dunbar****
>
> ** **
>
> ** **
>
> ** **
>
> *From:* [email-masked] [mailto:[email-masked]<[email-masked]>]
> *On Behalf Of *Xuxiaohu
> *Sent:* Wednesday, July 18, 2012 9:48 PM
> *To:* [email-masked]; [email-masked]
> *Cc:* [email-masked]
> *Subject:* Re: [nvo3] Comments on draft-kompella-nvo3-server2nve****
>
> ** **
>
> Does that mean the ARP could also be considered as an option for signaling
> the VM attachment/detachment event? For example, a gratuitous ARP packet
> can be inferred as an attachment event by the NVE which receives such
> packet via the NVE-TES interface. Meanwhile, for those L2VPN  (e.g., VPLS)
>  or L3VPN overlay approaches which only allows one next-hop to be
> available  for a given MAC route or a host route in the forwarding table,
> a gratuitous ARP packet received from a remote NVE could be inferred as a
> detachment event by the NVE to which the ARP sending VM was previously
> attached. Moreover, in case a gratuitous ARP packet triggers the NVE which
> received that packet via the NVE-TES interface to generate a MAC route or a
> host route for the ARP sending VM, and the NVE to which that VM was
> previously attached, upon receiving that route,  could also infer that
> route as a detachment event of that VM.****
>
> ** **
>
> Best regards,****
>
> Xiaohu****
>
> ** **
>
> <skipped>****
>
> I have a related consideration based on thinking about this further.  The
> network****
>
> SHOULD NOT rely on dissociate messages always being sent - a server crash
> at the****
>
> wrong point during a VM migration may cause a dissociate to be missed
> (e.g., the****
>
> VM made it to S’, but S crashed before sending the dissociate).  More
> importantly,****
>
> not relying on the dissociate messages (in particular, not having the
> inter-NVE****
>
> control protocol rely on them) helps if one wants to mix hypervisors that
> support****
>
> the attach/detach protocol with (exiting) ones that don’t.  For existing
> hypervisors,****
>
> under suitable restrictions and assuming some advance configuration,
> “associate”****
>
> can be inferred from a gratuitous ARP or RARP, but nothing is sent for
> dissociate.****
>
> The inference of “associate” won’t be possible if things have not been set
> up****
>
> to enable the gratuitous ARP or RARP.****
>
> ** **
>
> Thanks,
> --David****
>
> ** **
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf
> Of *Kireeti Kompella
> *Sent:* Saturday, July 14, 2012 8:16 PM
> *To:* Black, David
> *Cc:* [email-masked]
> *Subject:* Re: [nvo3] Comments on draft-kompella-nvo3-server2nve****
>
> ** **
>
> Hi David,****
>
> ** **
>
> Thanks for your detailed comments!  More inline.****
>
> On Fri, Jul 13, 2012 at 1:01 PM, <[email-masked]> wrote:****
>
> more detailed feedback.****
>
> ** **
>
> Thanks!****
>
>  ****
>
> could be minimized).  The result should be more general and more useful.**
> **
>
> ** **
>
> critical part of the picture.****
>
> ** **
>
> reasonable?****
>  ****
> ****
> ** **
> More on this below.****
>  ****
> should be specified to not depend on the order of events.****
>
> ** **
>
> Ordering assumptions weren't intended, so we'll tweak the wording to
> remove any such implications.****
>
>  ****
>
> in the rest of the draft.  Is it actually needed?****
>
> ** **
>
> ****
>  ****
> policy may limit which NVEs are allowed to participate in which VNs.****
>
> ** **
>
> Okay.  Authorization locally, or from the provisioning system?  (Or
> either?)****
>
>  ****
>
> message?  ****
> ** **
> below, good place to set it to 1; otherwise increment.****
>
>  ****
>
> <VNID,P> entry before responding?****
>
> ** **
>
> Good point.  Will fix.****
>
>  ****
>
> each mapping change).  ****
>
> ** **
>
> there may be cases where this is a valid approach.****
>
> ** **
>
>          VM's addresses and the VM's association with this NVE.****
>
> ** **
>
> Something like that.  Will work on text.****
>
>  ****
>
> -- Section 2.2.3
>
> permit removal.****
> ** **
> Okay, will copy wording from there once we've agreed on it.****
>
>  ****
>
> which case nothing should be done to those addresses).****
>
> ** **
>
> Definitely worth looking at again, especially with respect to your
> comments about the order for migration.****
>
> ** **
>
> With regard to the race condition, I'll send a separate email on that.****
>
> ** **
>
> assumption for an external NVE (e.g., in a ToR switch).****
>
> ** **
>
> Good point!  I'll go with refcounting.****
>
>  ****
>
> control protocol of the withdrawal of the VM's addresses.****
>
> ** **
>
> See separate email.****
>
>  ****
>
> versions of 2.2.3 for the migration departure and power-down use cases.***
> *
>
> ** **
>
> Perhaps.  Let's get the semantics right first, then see if there are
> common elements or not.****
>
>  ****
>
> -- Section 2.2.4
>
> on S'.  ****
> ** **
> dealing with migration abort.****
>
> ** **
>
>         request to the NVE for S. ****
>
> independent of the order of these events.****
>
> ** **
>
> Agreed.  Separate email thread to work this out.****
>
> ** **
>
>       that will occur when the VM is associated with this NVE.****
>
> ** **
>
> Okay, something like.****
>
>  ****
>
> provisioning of the address mapping information.****
>
> ** **
>
> Interesting thought.  Will follow up once we get the migration "right"
> (for some value of right).****
>
>  ****
>
> -- Section 3
>
> doesn't fit well with the rest of the draft.****
>
> ** **
>
> Will discuss putting this in a separate draft with co-authors.****
>
> ** **
>
> Thanks again for the detailed comments!****
>
> Kireeti.****
>
> ** **
>
>  ****
>
> Thanks,
> --David
> https://www.ietf.org/mailman/listinfo/nvo3****
>
>
>
> ****
>
> ** **
>
> --
> Kireeti****
>

---

From: <[email-masked]>
Date: Mon, 23 Jul 2012 17:49:28 -0400

**Provided that** the pre-associate operation that selects the virtual network
for the VM and sets up access it (e.g., VLAN between End Device and NVE),
is done via other (suitably secured) means, a gratuitous ARP or RARP on that
pre-configured access can be used to signal the VM's arrival to use the network
that's been preconfigured for it (i.e., the associate event).  As Linda noted,
there's lots of running code that does this, but (as Truman comments) ARP/RARP
cannot be trusted to carry explicit parameters to select a virtual network
(that has to be done via some other means).

Thanks,
--David

From: Truman Boyes [mailto:[email-masked]]
Sent: Monday, July 23, 2012 1:03 PM
To: Ivan Pepelnjak
Cc: Linda Dunbar; Xuxiaohu; Black, David; [email-masked]; [email-masked]
Subject: Re: [nvo3] Comments on draft-kompella-nvo3-server2nve

On Mon, Jul 23, 2012 at 12:22 PM, Ivan Pepelnjak <[email-masked]<mailto:[email-masked]>> wrote:
Just because millions of applications misuse a simplistic protocol in a way it was never designed to handle doesn't make it a good idea. Not to mention the total lack of security.

@Xiaohu: how would you distinguish a gratuitous ARP send from the hypervisor to indicate a VM move from a gratuitous ARP sent by a VM with a misconfigured IP address or a malicious gratuitous ARP sent by an intruder (physical or virtual)? Unless you can totally control the VM attachment point (= hypervisor switch unless you're using something like 802.1BR) you cannot trust ARP ... but then if you do control the hypervisor switch, you don't need ARP.

Ivan

Completely agree. ARP is the wrong protocol for any form of trusted signaling. Additionally, IPv6-only networks would present their own set of challenges if we had a reliance on  IPv4 address resolution protocol.

Truman

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Linda Dunbar
Sent: Monday, July 23, 2012 5:59 PM
To: Xuxiaohu; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>

My two cents.

Linda Dunbar

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Xuxiaohu
Sent: Wednesday, July 18, 2012 9:48 PM
To: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] Comments on draft-kompella-nvo3-server2nve

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David
+1 (508) 293-7953<tel:%2B1%20%28508%29%20293-7953>             FAX: +1 (508) 293-7786<tel:%2B1%20%28508%29%20293-7786>

--
Kireeti

---

From: Xuxiaohu <[email-masked]>
Date: Tue, 24 Jul 2012 01:15:40 +0000

Hi Ivan,

As for a gratuitous ARP sent by a VM with a misconfigured IP address, it’s  a network management issue and it should be avoided at the network management level. In fact, even if VDP or other Hypervisor-ToR protocol is used, such problem (i.e., IP address mis-configuration) will also occur.

As for a malicious gratuitous ARP sent by an intruder (physical or virtual), it’s the business of the tenant and the tenant itself should deploy some kind of security measures to prevent this issue.  Anyway, the usage of ARP here is like a PE-CE protocol, rather than PE-PE protocol.

Best regards,
Xiaohu

发件人: [email-masked] [mailto:[email-masked]] 代表 Ivan Pepelnjak
发送时间: 2012年7月24日 0:23
收件人: Linda Dunbar; Xuxiaohu; [email-masked]; [email-masked]
抄送: [email-masked]
主题: Re: [nvo3] Comments on draft-kompella-nvo3-server2nve

Ivan

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: Xuxiaohu <[email-masked]>
Date: Tue, 24 Jul 2012 01:37:11 +0000

Fully agree. Since the existing ARP has no such capability of carrying explicit parameters to select a virtual network for a given VM and such capability may never be needed (e.g., the orchestration system will do this work),  it seems there is no need for concerning the trust issue of ARP anymore. Hence I suggest considering ARP as one of the candidate protocol for signaling the VM attachment/detachment event in the related NVo3 drafts.

Best regards,
Xiaohu

发件人: [email-masked] [mailto:[email-masked]] 代表 [email-masked]
发送时间: 2012年7月24日 5:49
收件人: [email-masked]
抄送: [email-masked]
主题: [nvo3] Role of ARP/RARP (was: Comments on draft-kompella-nvo3-server2nve)

**Provided that** the pre-associate operation that selects the virtual network
for the VM and sets up access it (e.g., VLAN between End Device and NVE),
is done via other (suitably secured) means, a gratuitous ARP or RARP on that
pre-configured access can be used to signal the VM’s arrival to use the network
that’s been preconfigured for it (i.e., the associate event).  As Linda noted,
there’s lots of running code that does this, but (as Truman comments) ARP/RARP
cannot be trusted to carry explicit parameters to select a virtual network
(that has to be done via some other means).

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: Xuxiaohu <[email-masked]>
Date: Tue, 24 Jul 2012 02:02:32 +0000

Yes, it’s an already deployed usage, especially in Ethernet bridge networks. And as I have said in a previous email, in the L2VPN or L3VPN approaches where only one forwarding table entry is available for a given destination MAC or IP, it’s safe for the NVE to infer an ARP received on the TES-NVE interface as an attachment event, and it’s also safe for the NVE to which a moved VM was previously attached, to infer an ARP originated from that VM (or a new MAC or IP host route for that VM) which is received from a remote NVE as a detachment event. However, in the case where more than one forwarding table entry is allowed for a given destination MAC or IP address, the NVE should take some addition measures  to distinguish the VM movement event from the VM multi-homing event upon receiving the above ARP or a new MAC or IP host route from a remote NVE, e.g., check the current connectivity status of that VM upon receiving the above ARP or new route.

Best regards,
Xiaohu

发件人: Linda Dunbar
发送时间: 2012年7月23日 23:59
收件人: Xuxiaohu; [email-masked]; [email-masked]
抄送: [email-masked]
主题: RE: [nvo3] Comments on draft-kompella-nvo3-server2nve

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: Truman Boyes <[email-masked]>
Date: Tue, 24 Jul 2012 00:41:51 -0400

Xiaohu, if we plan to build ipv4 networks this could be considered, but the train has left the station and we need to support IPv6 in cloud and datacenters for applications and infrastructure. 

Now one might argue that we could use ARP and Neighbor discovery, but this is highly irregular and the mechanics are quite different. 

In your statement, "it's safe for the NVE to infer", I would say this is far from orchestration and much closer to standard Ethernet behavior, which in my experience leaves much room for manipulation. If ARP packets received on VAPs (TES-NVE) are capable of eventing to orchestration systems I would be highly worried. There would be many things to consider in trying to fix the security and authentication of such messages, or methods to enforce throttling or white listing of MAC addresses that the entire approach would be a retrofit to a protocol that was designed for a completely different purpose. 

Kind regards,
Truman Boyes

---

From: "Ivan Pepelnjak" <[email-masked]>
Date: Tue, 24 Jul 2012 09:40:49 +0200

*** Provided that *** we have a management channel that can send pre-associate message from the orchestration tool to the NVE, that same management channel can also send the associate message (including the MAC address of the VM). Don’t use two protocols for two steps of the same process, and don’t rely on protocols that (as Truman noted) work for IPv4 only (see also http://tools.ietf.org/html/draft-george-ipv6-support-01). 

Obviously we’ll have to re-architect the existing “squashed complexity sausage” ways of doing things, in which the networking industry tries to implement all sorts of kludges to accommodate simplistic VLAN/L2-only hypervisor switches. Evidently this approach led us nowhere (or we wouldn’t have this working group) and yesterday’s news told us VMware clearly recognized simplistic hypervisor switches are not the way to go. While re-architecting the existing mess, we just might admit that current bad practices are just that – bad.

Best,

Ivan

From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Monday, July 23, 2012 11:49 PM
To: [email-masked]
Cc: [email-masked]
Subject: [nvo3] Role of ARP/RARP (was: Comments on draft-kompella-nvo3-server2nve)

Thanks,
--David

On Mon, Jul 23, 2012 at 12:22 PM, Ivan Pepelnjak <[email-masked]> wrote:

Ivan

Truman

Cc: [email-masked]

My two cents. 

Linda Dunbar

Best regards,

Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Kireeti.

Thanks,
--David
+1 (508) 293-7953 <tel:%2B1%20%28508%29%20293-7953>              FAX: +1 (508) 293-7786 <tel:%2B1%20%28508%29%20293-7786> 

 

-- 
Kireeti

---

From: Xuxiaohu <[email-masked]>
Date: Tue, 24 Jul 2012 08:13:58 +0000

Yes, the orchestration system could inform the NVE of the VM attachment/detachment event directly and it works very well in the case where there is a control plane used for exchanging MAC or IP reachability information for VMs among NVEs. However, in some L2VPN approaches (e.g., VPLS) where only data-plane MAC learning is available, upon receiving the VM attachment notification from the orchestration system, the PE router would have to generate a customer multicast/broadcast frame on behalf of the moved VM and flood it to remote PE routers so as to update the corresponding VSI forwarding tables on those remote PE routers, which would otherwise be naturally accomplished by a gratuitous ARP packet originated from the moved VM itself.

Best regards,
Xiaohu

发件人: [email-masked] [mailto:[email-masked]] 代表 Ivan Pepelnjak
发送时间: 2012年7月24日 15:41
收件人: [email-masked]; [email-masked]
抄送: [email-masked]
主题: Re: [nvo3] Role of ARP/RARP (was: Comments on draft-kompella-nvo3-server2nve)

Best,
Ivan

Thanks,
--David

From: Truman Boyes [mailto:[email-masked]]
Sent: Monday, July 23, 2012 1:03 PM
To: Ivan Pepelnjak
Cc: Linda Dunbar; Xuxiaohu; Black, David; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] Comments on draft-kompella-nvo3-server2nve

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: Xuxiaohu <[email-masked]>
Date: Tue, 24 Jul 2012 09:40:08 +0000

By the way, there are many scenarios which need to be considered when using the orchestration system to notify the VM attachment event. For example, in the scenario where a TES is multi-homed to more than one L2 NVE (e.g., VPLS PE) via an Ethernet bridge or network which runs STP to avoid the forwarding loop, How could each NVE know whether or not it is responsible for generating such customer broadcast/multicast frame on behalf of the VM is a hard issue to be solved.

Anyway, different mechanisms of notifying the VM attachment/detachment events have their different target scenarios.

Best regards,
Xiaohu

发件人: [email-masked] [mailto:[email-masked]] 代表 Xuxiaohu
发送时间: 2012年7月24日 16:14
收件人: Ivan Pepelnjak; [email-masked]; [email-masked]
抄送: [email-masked]
主题: Re: [nvo3] Role of ARP/RARP (was: Comments on draft-kompella-nvo3-server2nve)

Best regards,
Xiaohu

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: <[email-masked]>
Date: Tue, 24 Jul 2012 09:03:05 -0400

Ivan,

> Obviously we’ll have to re-architect the existing “squashed complexity sausage”
> ways of doing things, in which the networking industry tries to implement all
> sorts of kludges to accommodate simplistic VLAN/L2-only hypervisor switches.
> Evidently this approach led us nowhere (or we wouldn’t have this working group)
> and yesterday’s news told us VMware clearly recognized simplistic hypervisor
> switches are not the way to go. While re-architecting the existing mess, we
> just might admit that current bad practices are just that – bad.

That’s a fine “greenfield” or “blank sheet of paper” design approach, but I think
you’re ignoring the “running code” that uses ARP/RARP for data plane learning.
It would be good to have a means of coping with that “running code” while we
transition to this brave new world ...

For clarity, I don’t think this is an either/or situation, but rather one in which
both approaches have merits and both should be pursued.  Specifically, I’d expect
better results from an explicit associate message (e.g., it provides an explicit
check that the interface [VSI or vNIC] that is associating is the one that is
expected based on the pre-associate), and would agree that this should be the
primary design approach.  OTOH, the ability to accommodate existing ARP/RARP usage
as an implicit associate is likely to make what we design easier to incrementally
deploy into existing data centers, especially enterprise data centers, whose admins
tend to show the door to anyone who talks about complete (“forklift”) upgrades ;-).

For IPv6, ND can be used, but I’m not sure how much of that is deployed for this
data-plane learning purpose, and I wouldn’t advocate this unless we have a
significant deployed base of “running code” to deal with.

Also, at the risk of really attracting responses ;-), this simplification of
incremental deployment is a reason to consider ARP/RARP as an additional implicit
associate primitive for IP-only solutions (and see previous discussion about this
L2 approach working well on a single L2 link but less well as the L2 and L3
topology between End Device and NVE gets more complex).

Thanks,
--David

From: [email-masked] [mailto:[email-masked]] On Behalf Of Ivan Pepelnjak
Sent: Tuesday, July 24, 2012 3:41 AM
To: Black, David; [email-masked]
Cc: [email-masked]
Subject: Re: [nvo3] Role of ARP/RARP (was: Comments on draft-kompella-nvo3-server2nve)

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: "Ivan Pepelnjak" <[email-masked]>
Date: Tue, 24 Jul 2012 17:38:57 +0200

I totally agree with you (in principle), but the NVO3 charter says ...

“NVO3 will consider approaches to multi-tenancy that reside at the 
network layer rather than using traditional isolation mechanisms 
that rely on the underlying layer 2 technology (e.g., VLANs). “

Which, according to my understanding, means using something-in-IP or something-in-MPLS encapsulation, done either in hypervisor or ToR switches. I don’t think there are too many data centers out there (particularly in enterprise environment) where such an idea wouldn’t require a forklift ToR switch upgrade or a replacement of existing hypervisor vSwitch (Open vSwitch is a bit of an exception).

Now, if our focus should be viability of deployment in existing networks (and I totally agree with that idea), it might be best to solve the problem within the hypervisor (with a decent vSwitch) and use the physical hardware just to provide IP transport.

Did I take a wrong turn somewhere to reach this conclusion?

Thank you!
Ivan

From: [email-masked] [mailto:[email-masked]] 
Sent: Tuesday, July 24, 2012 3:03 PM
To: [email-masked]
Cc: [email-masked]
Subject: RE: [nvo3] Role of ARP/RARP

Ivan,

Thanks,
--David

Best,

Ivan

Thanks,
--David

Ivan

Truman

Cc: [email-masked]

My two cents. 

Linda Dunbar

Best regards,

Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Kireeti.

Thanks,
--David

 

-- 
Kireeti

---

From: Linda Dunbar <[email-masked]>
Date: Tue, 24 Jul 2012 15:40:46 +0000

Changing running code is never easy. Considering "network" is not even the main business of any data centers, requiring all those deployed applications to change is a losing battle.

Besides, NVE shouldn't trust messages from VMs anyway, be it ARP or any newly invented message protocols. If you run a data center and lease out VMs to 3rd party, those VMs can be loaded with applications which you can't (and shouldn't)  trust.
Whatever messages come to NVE from VMs, NVE should check with its management system (or controller) to authenticate the messages.
The messages from VMs only serves the purpose of triggering NVE to start the process of  authenticating.
It is fine if some applications vendors want to use a new protocol to trigger NVE to start the authenticating process. But we should definitely include the option of letting NVE's authentication process being triggered by ARP messages.

Linda Dunbar

From: Truman Boyes [mailto:[email-masked]]
Sent: Monday, July 23, 2012 12:03 PM
To: Ivan Pepelnjak
Cc: Linda Dunbar; Xuxiaohu; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [nvo3] Comments on draft-kompella-nvo3-server2nve

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: Lucy yong <[email-masked]>
Date: Tue, 24 Jul 2012 15:51:08 +0000

Ivan,

IMO: the charter text means that the virtual network isolation should not rely on the VLAN, but it does not mean that the VLANs can not be used for the virtual access of a virtual network. In fact, this is common mechanism for the virtual access.
Lucy

From: [email-masked] [mailto:[email-masked]] On Behalf Of Ivan Pepelnjak
Sent: Tuesday, July 24, 2012 10:39 AM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [nvo3] Role of ARP/RARP

Thank you!
Ivan

Ivan,

Thanks,
--David

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Ivan Pepelnjak
Sent: Tuesday, July 24, 2012 3:41 AM
To: Black, David; [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] Role of ARP/RARP (was: Comments on draft-kompella-nvo3-server2nve)

Best,
Ivan

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of [email-masked]<mailto:[email-masked]>
Sent: Monday, July 23, 2012 11:49 PM
To: [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>
Subject: [nvo3] Role of ARP/RARP (was: Comments on draft-kompella-nvo3-server2nve)

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: "Ivan Pepelnjak" <[email-masked]>
Date: Tue, 24 Jul 2012 18:09:29 +0200

Lucy,

I don’t care where this WG eventually decides the MAC-to-something encapsulation (or translation) is done, but it will have to be done somewhere.

I’m just saying that there are not that many ToR switches deployed in today’s data centers that could do the encapsulation into L3-something with existing hardware (Q-in-Q or MAC-in-MAC is more common, but out of NVO3 scope), so it might make more sense to solve the problem in hypervisor software.

Ivan

From: Lucy yong [mailto:[email-masked]] 
Sent: Tuesday, July 24, 2012 5:51 PM
To: Ivan Pepelnjak; [email-masked]
Cc: [email-masked]
Subject: RE: [nvo3] Role of ARP/RARP

Ivan,

Lucy

Thank you!
Ivan

Ivan,

Thanks,
--David

Best,

Ivan

Thanks,
--David

Ivan

Truman

Cc: [email-masked]

My two cents. 

Linda Dunbar

Best regards,

Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Kireeti.

Thanks,
--David

 

-- 
Kireeti

---

From: "Ivan Pepelnjak" <[email-masked]>
Date: Tue, 24 Jul 2012 18:12:14 +0200

#1 – Changing applications is mission impossible, and I would never (again) be stupid enough to suggest that. Hypervisor upgrades are done quite often when the new features are worth the upgrade (people were pretty quick to upgrade to vSphere 5, for example). Claiming it’s impossible to change the behavior of the hypervisor vSwitch is not realistic – once VMware will roll out MAC-over-IP mechanism as part of its regular vSwitch code, people will start using it (like they’re using VCDNI today).

#2 – Totally agree that NVE (or hypervisor) shouldn’t trust messages from VM. However, the gratuitous ARP that we’re talking about isn’t sent by the VM, it’s usually generated by the hypervisor vSwitch that makes it look like it would be coming from the VM (you can even disable that functionality in vSphere, for example). We’re thus not talking about changes to the VM OS or app code, because they don’t even know they have moved (you can’t bounce a VM interface to generate a gratuitous ARP or you might lose all sessions).

Finally, as should be clear by now, ARP messages are generated by the VM TCP stack (VM-generated IP-to-MAC address resolution requests and replies), as well as “faked” by the hypervisor vSwitch (hypervisor-generated MAC move indication). Two totally different mechanisms and were thus overloaded on the same protocol because a decade ago someone wanted a quick fix to his problem-of-the-day. Is there any chance we could agree that this is bad, and focus on an out-of-band (management channel) MAC-move-indicating mechanism between hypervisors (or orchestration tools) and the rest of the network that makes sense and cannot be faked by a VM?

Ivan

From: Linda Dunbar [mailto:[email-masked]] 
Sent: Tuesday, July 24, 2012 5:41 PM
To: Truman Boyes; Ivan Pepelnjak
Cc: Xuxiaohu; [email-masked]; [email-masked]; [email-masked]
Subject: RE: [nvo3] Comments on draft-kompella-nvo3-server2nve

Changing running code is never easy. Considering “network” is not even the main business of any data centers, requiring all those deployed applications to change is a losing battle. 

Besides, NVE shouldn’t trust messages from VMs anyway, be it ARP or any newly invented message protocols. If you run a data center and lease out VMs to 3rd party, those VMs can be loaded with applications which you can’t (and shouldn’t)  trust.

It is fine if some applications vendors want to use a new protocol to trigger NVE to start the authenticating process. But we should definitely include the option of letting NVE’s authentication process being triggered by ARP messages.

Linda Dunbar

Ivan

Truman

Cc: [email-masked]

My two cents. 

Linda Dunbar

Best regards,

Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Kireeti.

Thanks,
--David

 

-- 
Kireeti

---

From: Lucy yong <[email-masked]>
Date: Tue, 24 Jul 2012 16:20:12 +0000

I agree your aspect in general. But the nvo3 solution does not limit to NVE on hypervisor only. The solution needs to work over the deployed devices too.
Lucy

From: [email-masked] [mailto:[email-masked]] On Behalf Of Ivan Pepelnjak
Sent: Tuesday, July 24, 2012 11:09 AM
To: Lucy yong; [email-masked]
Cc: [email-masked]
Subject: Re: [nvo3] Role of ARP/RARP

Lucy,

Ivan

Ivan,

To: [email-masked]<mailto:[email-masked]>

Thank you!
Ivan

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]
Sent: Tuesday, July 24, 2012 3:03 PM
To: [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>
Subject: RE: [nvo3] Role of ARP/RARP

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Tue, 24 Jul 2012 16:26:54 +0000

Lucy,

This is not about whether VLANs are used (to separate tenants on a shared physical port) but whether bridging is involved in the access.

Maria

From: [email-masked] [mailto:[email-masked]] On Behalf Of Lucy yong
Sent: Tuesday, July 24, 2012 11:51 AM
To: Ivan Pepelnjak; [email-masked]
Cc: [email-masked]
Subject: Re: [nvo3] Role of ARP/RARP

Ivan,

Thank you!
Ivan

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: <[email-masked]>
Date: Tue, 24 Jul 2012 12:31:23 -0400

IMHO ...

... if there’s agreement that this is ok and needs to be supported:

> whether VLANs are used (to separate tenants on a shared physical port)

... then this seems like fine fodder for the gap analysis plus future
selection of and work on associate/dissociate protocols:

> whether bridging is involved in the access.

Thanks,
--David

From: NAPIERALA, MARIA H [mailto:[email-masked]]
Sent: Tuesday, July 24, 2012 12:27 PM
To: Lucy yong; Ivan Pepelnjak; Black, David
Cc: [email-masked]
Subject: RE: [nvo3] Role of ARP/RARP

Lucy,

Maria

Ivan,

Thank you!
Ivan

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Tue, 24 Jul 2012 16:36:44 +0000

David> Also, at the risk of really attracting responses ;-), this simplification of

:-)

David> incremental deployment is a reason to consider ARP/RARP as an additional implicit
David> associate primitive for IP-only solutions (and see previous discussion about this
David> L2 approach working well on a single L2 link but less well as the L2 and L3
David> topology between End Device and NVE gets more complex).

If this “access” stretches out beyond ToR (not even mentioning hypervisor, which, IMO, it is the most optimal place to initiate the virtual network including the encapsulation), then I am not sure what value such solution will bring in terms of addressing one of the main requirements for virtualized data centers which is the independence of the data-center switching infrastructure.

When the technology moves forwards, the upgrades of hypervisor and switching equipment will be required. Trying to be forever backward compatible with an old technology will not be productive and will have to be abandoned.  This happened many times in, e.g., large Service Providers networks.

Maria

From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Tuesday, July 24, 2012 9:03 AM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [nvo3] Role of ARP/RARP

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: <[email-masked]>
Date: Tue, 24 Jul 2012 12:45:33 -0400

Maria,

We may be close to agreement.  I don’t see ARP/RARP as viable on all topologies between End Device
and NVE, but it clearly works fine over a single L2 link.  The moment the access has to cross an
L3 forwarding hop, ARP/RARP gets much more involved, and an IP-based solution is probably a better
idea.

I’m also not advocating “trying to be forever backwards compatible”, but ARP/RARP sure looks like
a useful mechanism to preserve for some transition scenarios.  The alternative of complete
(“forklift”) upgrades of everything at once gets ugly fast in large facilities.

Thanks,
--David

From: NAPIERALA, MARIA H [mailto:[email-masked]]
Sent: Tuesday, July 24, 2012 12:37 PM
To: Black, David; [email-masked]
Cc: [email-masked]
Subject: RE: [nvo3] Role of ARP/RARP

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: "Joe Pelissier (jopeliss)" <[email-masked]>
Date: Tue, 24 Jul 2012 17:22:07 +0000

I think there is a subtle difference between an associate and what the ARP/RARP (or ND) does.  There are potentially two steps:

1)      Associate: this sets up all of the network parameters in the switch/router/NVE but does not update the forwarding.  This can be a relatively slow process since it sets up things like ACLs, etc., and may require fetching data from other databases.  This process can take place while the VM is still operating in the old location.

2)       ARP/RARP/ND: This indicates that the VM is now becoming active in the new location (and that it is shut down in the old location) and forwarding tables need to be updated.  The nice thing about an ARP/RARP/ND approach is that it updates all of the tables in the L2 LAN, and is therefore a fundamental part of a VM move.  The ARP/RARP/ND essentially needs to be done anyway to quickly converge forwarding in the L2, so there may not be a lot of point in duplicating this.  The ARP/RARP/NDs could be snooped by an NVE to indicate that the VM is now active.

IAW: the ARP/RARP/ND is not an implicit associate; the associate and ARP/RARP/ND perform different functions.  The critical time is the period during which the VM is suspended during the move, so it is important to have all of the associate work ready to go prior to the move.  The ARP/RARP/ND indicates that the move is now complete and to activate the new forwarding, which can be done very quickly.

(One other observation: it really does not need to be an ARP/RARP/ND.  Essentially, any frame received with the source MAC or IP address of the VM (which is known as part of the associate process), is indicative of the VM becoming active.  ARP/RARP/ND approach is widely deployed since it has the effect of updating the L2 forwarding everywhere in the L2 network.)

My $0.02 worth…
Joe Pelissier

From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Tuesday, July 24, 2012 9:46 AM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [nvo3] Role of ARP/RARP

Maria,

Thanks,
--David

To: Black, David; [email-masked]<mailto:[email-masked]>

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Tue, 24 Jul 2012 17:22:22 +0000

David,

> We may be close to agreement.  I don’t see ARP/RARP as viable on all topologies between End Device

Good!

> and NVE, but it clearly works fine over a single L2 link.

In a layer 3 VPN solution we don’t want/don’t have to rely on gratuitous  ARP (for VM moves) even on a single link. We want to keep the end-system-NVE communication at layer 3 using an extensible protocol that can be secured.

> The moment the access has to cross an

Agree.

Maria

From: [email-masked] [mailto:[email-masked]]
Sent: Tuesday, July 24, 2012 12:46 PM
To: NAPIERALA, MARIA H
Cc: [email-masked]
Subject: RE: [nvo3] Role of ARP/RARP

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: <[email-masked]>
Date: Tue, 24 Jul 2012 13:26:13 -0400

Joe,

My assumption behind this has been that *if* all the network parameters can be set up in advance using the pre-associate, then ARP/RARP/ND can serve as the indication to put them into service and update the forwarding.

We’re in clear agreement that both operations (set up parameters, activate forwarding) are needed, and I had been using “pre-associate” as the term used in the VDP spec for the in-advance operation.

Thanks,
--David

From: [email-masked] [mailto:[email-masked]] On Behalf Of Joe Pelissier (jopeliss)
Sent: Tuesday, July 24, 2012 1:22 PM
To: [email-masked]
Subject: Re: [nvo3] Role of ARP/RARP

My $0.02 worth…
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: "Joe Pelissier (jopeliss)" <[email-masked]>
Date: Tue, 24 Jul 2012 17:58:22 +0000

Hi David,
Just to clarify, in the VDP specification, there are two types of pre-associate: with and without resource reservation.  What you are referring to is the pre-associate with resource reservation.  However, it does not operate quite the way you describe.

A pre-associate with resource reservation essentially ensures that the switch can perform a future associate with the same parameters (or indicates a failure).  It reserves the resources needed so that a future associate with the same parameters will be able to complete, but it does not necessarily configure those parameters into the switch; i.e., a pre-associate has no impact on the currently running configuration.  An associate actually configures the parameters so that when the new VM becomes active, everything is ready to go.  This is the part that can be a bit slow, so we want it to be completed before the VM comes alive in the new location.  However, we can’t update forwarding at this point since the VM is still active in the old location.  Forwarding gets updated through the normal learning process which ensures that forwarding is not impacted until the VM is alive in the new location.

If associate updated the forwarding in the local switch, then we would be forced to do the associate during the period that the VM is suspended which increases the time that the VM must be suspended.  This is undesirable.  By separating the associate from the learning, the associate is removed from the critical path.

Even if we used the associate to update the forwarding, it is insufficient.  It would only update the forwarding in the local switch, not the entire L2 LAN.  Therefore, the ARP/RARP/ND is still highly desirable.  Since we need the ARP/RARP/ND anyway for the rest of the L2 LAN, it might as well be used to update forwarding in the local switch also, with the benefit of moving associate out of the critical path.

Joe Pelissier

From: [email-masked] [mailto:[email-masked]]
Sent: Tuesday, July 24, 2012 10:26 AM
To: Joe Pelissier (jopeliss); [email-masked]
Subject: RE: [nvo3] Role of ARP/RARP

Joe,

Thanks,
--David

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Joe Pelissier (jopeliss)
Sent: Tuesday, July 24, 2012 1:22 PM
To: [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] Role of ARP/RARP

My $0.02 worth…
Joe Pelissier

To: [email-masked]<mailto:[email-masked]>

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: Lucy yong <[email-masked]>
Date: Tue, 24 Jul 2012 18:21:28 +0000

Maria,

Do you mean if the traffic between VMs at access should be bridged directly or not?

The discussion on this thread is about if ARP protocol is sufficient in VM moves.

Thanks,
Lucy

From: NAPIERALA, MARIA H [mailto:[email-masked]]
Sent: Tuesday, July 24, 2012 11:27 AM
To: Lucy yong; Ivan Pepelnjak; [email-masked]
Cc: [email-masked]
Subject: RE: [nvo3] Role of ARP/RARP

Lucy,

Maria

Ivan,

Thank you!
Ivan

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Tue, 24 Jul 2012 18:31:50 +0000

It is whether the access to the external first-hop switch/NVE is based on bridging or routing.

Maria

From: Lucy yong [mailto:[email-masked]]
Sent: Tuesday, July 24, 2012 2:21 PM
To: NAPIERALA, MARIA H; Ivan Pepelnjak; [email-masked]
Cc: [email-masked]
Subject: RE: [nvo3] Role of ARP/RARP

Maria,

Thanks,
Lucy

To: Lucy yong; Ivan Pepelnjak; [email-masked]<mailto:[email-masked]>

Lucy,

Maria

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Lucy yong
Sent: Tuesday, July 24, 2012 11:51 AM
To: Ivan Pepelnjak; [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] Role of ARP/RARP

Ivan,

Thank you!
Ivan

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: "Ivan Pepelnjak" <[email-masked]>
Date: Tue, 24 Jul 2012 20:43:28 +0200

Joe,

Thanks for a very concise explanation – I’m probably too focused on the hypervisor-based solutions and overlooked the associate action might take a while reality.

However, the use of ARP (or any other L2-broadcast-based mechanism) to propagate the MAC forwarding across the L2 virtual network has a severe scalability limitation – it relies on flooding. 

Flooding can be implemented with head-end replication, in which case we have to assume the NVE knows L3 addresses of all NVEs participating in the same virtual network. If that information is already available in NVE, we don’t need ARP but could use any out-of-band signaling mechanism. Alternatively, a central system (orchestration tool) could propagate that information using a routing protocol or a FIB update protocol (ex: OpenFlow).

Core-network-based flooding like today’s VLAN-based virtual networks is either too coarse (resulting in wasted bandwidth and NVE resources), or requires per-tenant state in the core (example: IP multicast-based flooding in VXLAN). Both approaches have scalability problems once the data center becomes big enough.

Am I missing something?

Ivan

From: [email-masked] [mailto:[email-masked]] On Behalf Of Joe Pelissier (jopeliss)
Sent: Tuesday, July 24, 2012 7:58 PM
To: [email-masked]
Subject: Re: [nvo3] Role of ARP/RARP

Hi David,

Joe Pelissier

Joe,

Thanks,
--David

My $0.02 worth…

Joe Pelissier

Maria,

idea.

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,

Ivan

Thanks,
--David

Ivan

Truman

Cc: [email-masked]

My two cents. 

Linda Dunbar

Best regards,

Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Kireeti.

Thanks,
--David

 

-- 
Kireeti

---

From: <[email-masked]>
Date: Tue, 24 Jul 2012 14:46:54 -0400

Ivan,

You’ve missed an important twist:

> However, the use of ARP (or any other L2-broadcast-based mechanism) to propagate the MAC
> forwarding across the L2 virtual network has a severe scalability limitation – it relies on flooding.

The idea is to have the NVE intercept the ARP and something more scalable than flooding to get the job done.  There are multiple -nvo3- drafts that envision this sort of approach to replace flooding at scale.

Thanks,
--David

From: [email-masked] [mailto:[email-masked]] On Behalf Of Ivan Pepelnjak
Sent: Tuesday, July 24, 2012 2:43 PM
To: 'Joe Pelissier (jopeliss)'; [email-masked]
Subject: Re: [nvo3] Role of ARP/RARP

Joe,

Am I missing something?
Ivan

Hi David,

Joe Pelissier

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]
Sent: Tuesday, July 24, 2012 10:26 AM
To: Joe Pelissier (jopeliss); [email-masked]<mailto:[email-masked]>
Subject: RE: [nvo3] Role of ARP/RARP

Joe,

Thanks,
--David

My $0.02 worth…
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: <[email-masked]>
Date: Tue, 24 Jul 2012 14:55:16 -0400

Missing word (oops) ...

The idea is to have the NVE intercept the ARP and *use* something more scalable than flooding to get the job done.

Thanks,
--David

From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Tuesday, July 24, 2012 2:47 PM
To: [email-masked]; [email-masked]
Subject: Re: [nvo3] Role of ARP/RARP

Ivan,

Thanks,
--David

Joe,

Am I missing something?
Ivan

Hi David,

Joe Pelissier

Joe,

Thanks,
--David

My $0.02 worth…
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: "Ivan Pepelnjak" <[email-masked]>
Date: Tue, 24 Jul 2012 21:01:56 +0200

So glad to hear we’re past end-to-end L2 flooding.

I’m probably missing something else at this moment (physical device integration comes to mind), but the only reason I see why we’re discussing ARP, VLANs (and potentially VDP) today is the idea to retain access network (including hypervisor switches) intact in the dumb L2-only stage and implement some complex mechanism in ToR switch (or whatever other device acts as NVO3 PE-router) to cope with the existing signaling mechanisms (gratuitous ARP) used by hypervisor switches.

If that’s the case, we’re missing the boat. All the hypervisor vendors got the “bridging doesn’t scale, IP does” message loud and clear a year ago. VMware has VXLAN (currently with Nexus 1000V, but very probably they won’t stay there, and they bought Nicira for a reason), Hyper-V will have NVGRE with Windows Server 2012 and Xen/KVM have GRE (or CAPWAP and eventually VXLAN)  support through OVS. Let’s acknowledge that reality and move on.

Best,

Ivan

From: [email-masked] [mailto:[email-masked]] 
Sent: Tuesday, July 24, 2012 8:47 PM
To: [email-masked]; [email-masked]
Subject: RE: [nvo3] Role of ARP/RARP

Ivan,

Thanks,
--David

Joe,

Am I missing something?

Ivan

Hi David,

Joe Pelissier

Joe,

Thanks,
--David

My $0.02 worth…

Joe Pelissier

Maria,

idea.

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,

Ivan

Thanks,
--David

Ivan

Truman

Cc: [email-masked]

My two cents. 

Linda Dunbar

Best regards,

Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Kireeti.

Thanks,
--David

 

-- 
Kireeti

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Tue, 24 Jul 2012 19:27:21 +0000

> bridging doesn’t scale, IP does
You nailed it.
Luyuan

From: [email-masked] [mailto:[email-masked]] On Behalf Of Ivan Pepelnjak
Sent: Tuesday, July 24, 2012 3:02 PM
To: [email-masked]; [email-masked]
Subject: Re: [nvo3] Role of ARP/RARP

Best,
Ivan

Ivan,

Thanks,
--David

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Ivan Pepelnjak
Sent: Tuesday, July 24, 2012 2:43 PM
To: 'Joe Pelissier (jopeliss)'; [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] Role of ARP/RARP

Joe,

Am I missing something?
Ivan

Hi David,

Joe Pelissier

Joe,

Thanks,
--David

My $0.02 worth…
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: AshwoodsmithPeter <[email-masked]>
Date: Tue, 24 Jul 2012 19:53:49 +0000

What does not scale well is flat addressing no matter what you call the bit field.

Peter

________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Luyuan Fang (lufang)
Sent: Tuesday, July 24, 2012 3:27 PM
To: Ivan Pepelnjak; [email-masked]; [email-masked]
Subject: Re: [nvo3] Role of ARP/RARP

> bridging doesn't scale, IP does
You nailed it.
Luyuan

So glad to hear we're past end-to-end L2 flooding.

I'm probably missing something else at this moment (physical device integration comes to mind), but the only reason I see why we're discussing ARP, VLANs (and potentially VDP) today is the idea to retain access network (including hypervisor switches) intact in the dumb L2-only stage and implement some complex mechanism in ToR switch (or whatever other device acts as NVO3 PE-router) to cope with the existing signaling mechanisms (gratuitous ARP) used by hypervisor switches.

If that's the case, we're missing the boat. All the hypervisor vendors got the "bridging doesn't scale, IP does" message loud and clear a year ago. VMware has VXLAN (currently with Nexus 1000V, but very probably they won't stay there, and they bought Nicira for a reason), Hyper-V will have NVGRE with Windows Server 2012 and Xen/KVM have GRE (or CAPWAP and eventually VXLAN)  support through OVS. Let's acknowledge that reality and move on.

Best,
Ivan

Ivan,

You've missed an important twist:

> forwarding across the L2 virtual network has a severe scalability limitation - it relies on flooding.

Thanks,
--David

Joe,

Thanks for a very concise explanation - I'm probably too focused on the hypervisor-based solutions and overlooked the associate action might take a while reality.

However, the use of ARP (or any other L2-broadcast-based mechanism) to propagate the MAC forwarding across the L2 virtual network has a severe scalability limitation - it relies on flooding.

Flooding can be implemented with head-end replication, in which case we have to assume the NVE knows L3 addresses of all NVEs participating in the same virtual network. If that information is already available in NVE, we don't need ARP but could use any out-of-band signaling mechanism. Alternatively, a central system (orchestration tool) could propagate that information using a routing protocol or a FIB update protocol (ex: OpenFlow).

Core-network-based flooding like today's VLAN-based virtual networks is either too coarse (resulting in wasted bandwidth and NVE resources), or requires per-tenant state in the core (example: IP multicast-based flooding in VXLAN). Both approaches have scalability problems once the data center becomes big enough.

Am I missing something?
Ivan

Hi David,

A pre-associate with resource reservation essentially ensures that the switch can perform a future associate with the same parameters (or indicates a failure).  It reserves the resources needed so that a future associate with the same parameters will be able to complete, but it does not necessarily configure those parameters into the switch; i.e., a pre-associate has no impact on the currently running configuration.  An associate actually configures the parameters so that when the new VM becomes active, everything is ready to go.  This is the part that can be a bit slow, so we want it to be completed before the VM comes alive in the new location.  However, we can't update forwarding at this point since the VM is still active in the old location.  Forwarding gets updated through the normal learning process which ensures that forwarding is not impacted until the VM is alive in the new location.

Joe Pelissier

Joe,

We're in clear agreement that both operations (set up parameters, activate forwarding) are needed, and I had been using "pre-associate" as the term used in the VDP spec for the in-advance operation.

Thanks,
--David

My $0.02 worth...
Joe Pelissier

Maria,

We may be close to agreement.  I don't see ARP/RARP as viable on all topologies between End Device
and NVE, but it clearly works fine over a single L2 link.  The moment the access has to cross an
L3 forwarding hop, ARP/RARP gets much more involved, and an IP-based solution is probably a better
idea.

I'm also not advocating "trying to be forever backwards compatible", but ARP/RARP sure looks like
a useful mechanism to preserve for some transition scenarios.  The alternative of complete
("forklift") upgrades of everything at once gets ugly fast in large facilities.

Thanks,
--David

:-)

If this "access" stretches out beyond ToR (not even mentioning hypervisor, which, IMO, it is the most optimal place to initiate the virtual network including the encapsulation), then I am not sure what value such solution will bring in terms of addressing one of the main requirements for virtualized data centers which is the independence of the data-center switching infrastructure.

Maria

Ivan,

> Obviously we'll have to re-architect the existing "squashed complexity sausage"
> Evidently this approach led us nowhere (or we wouldn't have this working group)
> and yesterday's news told us VMware clearly recognized simplistic hypervisor
> just might admit that current bad practices are just that - bad.

That's a fine "greenfield" or "blank sheet of paper" design approach, but I think
you're ignoring the "running code" that uses ARP/RARP for data plane learning.
It would be good to have a means of coping with that "running code" while we
transition to this brave new world ...

For clarity, I don't think this is an either/or situation, but rather one in which
both approaches have merits and both should be pursued.  Specifically, I'd expect
better results from an explicit associate message (e.g., it provides an explicit
check that the interface [VSI or vNIC] that is associating is the one that is
expected based on the pre-associate), and would agree that this should be the
primary design approach.  OTOH, the ability to accommodate existing ARP/RARP usage
as an implicit associate is likely to make what we design easier to incrementally
deploy into existing data centers, especially enterprise data centers, whose admins
tend to show the door to anyone who talks about complete ("forklift") upgrades ;-).

For IPv6, ND can be used, but I'm not sure how much of that is deployed for this
data-plane learning purpose, and I wouldn't advocate this unless we have a
significant deployed base of "running code" to deal with.

Thanks,
--David

*** Provided that *** we have a management channel that can send pre-associate message from the orchestration tool to the NVE, that same management channel can also send the associate message (including the MAC address of the VM). Don't use two protocols for two steps of the same process, and don't rely on protocols that (as Truman noted) work for IPv4 only (see also http://tools.ietf.org/html/draft-george-ipv6-support-01).

Obviously we'll have to re-architect the existing "squashed complexity sausage" ways of doing things, in which the networking industry tries to implement all sorts of kludges to accommodate simplistic VLAN/L2-only hypervisor switches. Evidently this approach led us nowhere (or we wouldn't have this working group) and yesterday's news told us VMware clearly recognized simplistic hypervisor switches are not the way to go. While re-architecting the existing mess, we just might admit that current bad practices are just that - bad.

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]<mailto:%5bmailto:[email-masked]%5d> On Behalf Of Kireeti Kompella
Sent: Saturday, July 14, 2012 8:16 PM
To: Black, David
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] Comments on draft-kompella-nvo3-server2nve

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Tue, 24 Jul 2012 21:17:34 +0000

Peter,
Are you talking about L2 or L3, or both?
Are you talking about overlay Virtual Network or underlay infrastructure, assume overlay?

From: AshwoodsmithPeter [mailto:[email-masked]]
Sent: Tuesday, July 24, 2012 3:54 PM
To: Luyuan Fang (lufang); Ivan Pepelnjak; [email-masked]; [email-masked]
Subject: RE: [nvo3] Role of ARP/RARP

Peter

You nailed it.
Luyuan

Best,
Ivan

Ivan,

Thanks,
--David

Joe,

Am I missing something?
Ivan

Hi David,

Joe Pelissier

Joe,

Thanks,
--David

My $0.02 worth...
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: Lucy yong <[email-masked]>
Date: Tue, 24 Jul 2012 21:15:42 +0000

> bridging doesn’t scale, IP does

This is misleading comment for this discussion. IP has its usage place, bridging has too.
I am not saying we should use the bridging at virtual access. But the bridging at virtual access does not have the scalability issue.

Lucy

From: [email-masked] [mailto:[email-masked]] On Behalf Of Luyuan Fang (lufang)
Sent: Tuesday, July 24, 2012 2:27 PM
To: Ivan Pepelnjak; [email-masked]; [email-masked]
Subject: Re: [nvo3] Role of ARP/RARP

You nailed it.
Luyuan

Best,
Ivan

Ivan,

Thanks,
--David

Joe,

Am I missing something?
Ivan

Hi David,

Joe Pelissier

Joe,

Thanks,
--David

My $0.02 worth…
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Tue, 24 Jul 2012 21:18:10 +0000

Hi David,

To be clear that we are discussing the same thing, and not muddy the discussion with VDP terminology and semantics, I would like to ask for clarification using the slides I presented at the NVO3 BOF in Paris.  If you look at slides 10 to 12 in
http://www.ietf.org/proceedings/83/slides/slides-83-nvo3-4.pptx
you will see that I have identified two different types of "attach".  One is a VN Attach (slide 10) and one is a VM VNIC Attach (slides 11 and 12).  The VN Attach is used to populate the NVE with information required for participating in the VN, and to negotiate a locally significant VLAN tag to use on the link between the hypervisor end device and the NVE for that VN.  The VM VNIC Attach is used to inform the NVE that a given VNIC's MAC is present on the interface within the attached VN.

Given that, an ARP sent with the VM VNICs IP/MAC could only serve as the VNIC Attach, and could not replace the VN Attach, so a protocol is still needed for the VN Attach regardless of whether ARP/ND is used to as VN Attach…right?

 - Larry

From: David Black <[email-masked]<mailto:[email-masked]>>
Date: Tuesday, July 24, 2012 6:03 AM
To: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [nvo3] Role of ARP/RARP

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

From:[email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Linda Dunbar
Sent: Monday, July 23, 2012 5:59 PM
To: Xuxiaohu; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>

My two cents.

Linda Dunbar

From:[email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Xuxiaohu

Best regards,
Xiaohu

<skipped>

Thanks,
--David

From:[email-masked]<mailto:[email-masked]> [mailto:[email-masked]]<mailto:[mailto:[email-masked]]> On Behalf Of Kireeti Kompella
Sent: Saturday, July 14, 2012 8:16 PM
To: Black, David
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] Comments on draft-kompella-nvo3-server2nve

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Tue, 24 Jul 2012 21:50:38 +0000

The point we were making was not bridging does not have place. In fact, it has and it will, for long time if not forever. We are talking about scaling, including operational simplicity.
A lot of places may not need to scale huge in reality, either way would work, if they like to stay the way it is, fine.
A few others need huge scale, they may not need to worry about backward compatibility or follow the old path, especially when building green field.
We have different sets of operators, the goals/requirements/solutions would not be the same, I think Pedro pointed this out a while ago.

Luyuan

From: Lucy yong [mailto:[email-masked]]
Sent: Tuesday, July 24, 2012 5:16 PM
To: Luyuan Fang (lufang); Ivan Pepelnjak; [email-masked]; [email-masked]
Subject: RE: [nvo3] Role of ARP/RARP

Lucy

You nailed it.
Luyuan

Best,
Ivan

Ivan,

Thanks,
--David

Joe,

Am I missing something?
Ivan

Hi David,

Joe Pelissier

Joe,

Thanks,
--David

My $0.02 worth…
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: <[email-masked]>
Date: Tue, 24 Jul 2012 18:31:52 -0400

Yes.

From: Larry Kreeger (kreeger) [mailto:[email-masked]]
Sent: Tuesday, July 24, 2012 5:18 PM
To: Black, David; [email-masked]
Cc: [email-masked]
Subject: Re: [nvo3] Role of ARP/RARP

Hi David,

Given that, an ARP sent with the VM VNICs IP/MAC could only serve as the VNIC Attach, and could not replace the VN Attach, so a protocol is still needed for the VN Attach regardless of whether ARP/ND is used to as VN Attach...right?

 - Larry

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: David Allan I <[email-masked]>
Date: Tue, 24 Jul 2012 19:10:56 -0400

Lets not throw the baby out with the bathwater.

The current IPv4 address is a combination of both location and identity and we have a need to, at least a local level, render the location part irrelevant. The location limitations can be at least partially circumvented with some underlying level of indirection. If you try to work around this without it, you end up with a flat unaggregatable set of host routes as useful geographic or virtualized clustering of all members in a prefix is not possible.

IP scales, partially because of the notion of subnet, which is an enabler (not the only one) of aggregatable addresses. The VLAN as a virtual subnet also has the property of providing that useful level of indirection and geographic independence. The limitation is the practical size of the community that can be served by any single virtualized broadcast domain.

So I get a bit nervous when "IP scales" starts to sound like a mantra. The statement should not be "IP scales" but "we need to use IP in a way that does not undermine its scaling properties" and frankly that will involve bridging IMHO.

cheers
Dave

________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Luyuan Fang (lufang)
Sent: Tuesday, July 24, 2012 2:51 PM
To: Lucy yong; Ivan Pepelnjak; [email-masked]; [email-masked]
Subject: Re: [nvo3] Role of ARP/RARP

Luyuan

Lucy

You nailed it.
Luyuan

Best,
Ivan

Ivan,

Thanks,
--David

Joe,

Am I missing something?
Ivan

Hi David,

Joe Pelissier

Joe,

Thanks,
--David

My $0.02 worth...
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 25 Jul 2012 01:47:31 +0000

Hi Dave,

IP-based solution (i.e., IP-in-IP) scales better that MAC-based solution (i.e., MAC-in-IP) is not just due to address aggregation.  With the IP-based solution which uses host routes, the flooding domain and the MAC learning domain associated with the subnet/VLAN which has been extended across multiple racks, pods and even data center locations can be confined within a limited scope, and therefore the solution can scale very well. For more details, pls see Virtual Subnet (http://tools.ietf.org/html/draft-xu-virtual-subnet-08).

Best regards,
Xiaohu

发件人: [email-masked] [mailto:[email-masked]] 代表 David Allan I
发送时间: 2012年7月25日 7:11
收件人: Luyuan Fang (lufang); Lucy yong; Ivan Pepelnjak; [email-masked]; [email-masked]
主题: Re: [nvo3] Role of ARP/RARP

cheers
Dave

Luyuan

Lucy

You nailed it.
Luyuan

Best,
Ivan

Ivan,

Thanks,
--David

Joe,

Am I missing something?
Ivan

Hi David,

Joe Pelissier

Joe,

Thanks,
--David

My $0.02 worth…
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 25 Jul 2012 02:00:10 +0000

Hi David,

In the case where there is an Ethernet bridge or network between a given L2 NVE and its corresponding TESes, upon receiving the notification of VM attachment event from a remote NVE through a more scalable way as you mentioned, the NVE needs to translate this notification into a customer broadcast/multicast frame (e.g., ARP broadcast) and send it towards the attached Ethernet bridge or network so as to update the corresponding forwarding table entry for the moved VM. In this case, it’s still worthwhile to flood the gratuitous ARP packets and even the unknown uncast frames as a normal bridge does, IMHO.

Best regards,
Xiaohu

发件人: [email-masked] [mailto:[email-masked]] 代表 [email-masked]
发送时间: 2012年7月25日 2:55
收件人: [email-masked]; [email-masked]
主题: Re: [nvo3] Role of ARP/RARP

Missing word (oops) ...

Thanks,
--David

Ivan,

Thanks,
--David

Joe,

Am I missing something?
Ivan

Hi David,

Joe Pelissier

Joe,

Thanks,
--David

My $0.02 worth…
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Wed, 25 Jul 2012 02:16:07 +0000

Subnet works well as long as you don't have a lot of traffic traversing the subnets.
In many DC cases, there are more traffic crossing IP subnet (VLANs) than traffic stays within a subnet. The L2 solution does not address traffic crossing IP subnets.

From: David Allan I [mailto:[email-masked]]
Sent: Tuesday, July 24, 2012 7:11 PM
To: Luyuan Fang (lufang); Lucy yong; Ivan Pepelnjak; [email-masked]; [email-masked]
Subject: RE: [nvo3] Role of ARP/RARP

cheers
Dave

Luyuan

Lucy

You nailed it.
Luyuan

Best,
Ivan

Ivan,

Thanks,
--David

Joe,

Am I missing something?
Ivan

Hi David,

Joe Pelissier

Joe,

Thanks,
--David

My $0.02 worth...
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 25 Jul 2012 02:39:36 +0000

Hi Joe,

Thanks a lot for this professional clarification.  BTW, another benefit of using gratuitous ARP, rather than any frame received with the source MAC or IP address of the VM,  as a notification of VM attachments is that the former is applicable not only in the L2NVE case but also in the L3NVE case, and more importantly, it has no requirement for any new implementation on L3NVEs.

Best regards,
Xiaohu

发件人: [email-masked] [mailto:[email-masked]] 代表 Joe Pelissier (jopeliss)
发送时间: 2012年7月25日 1:22
收件人: [email-masked]
主题: Re: [nvo3] Role of ARP/RARP

My $0.02 worth…
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Wed, 25 Jul 2012 04:08:36 +0000

Xiaohu,

Are you assuming that a gratuitous ARP is snooped by the NVE, “turned” into a host route and propagated via a control plane to other NVEs? If yes, this is not sufficient to assure the required minimal disruption for “live” VM move because the route propagation is not going to happen in 10s or even 100s of milliseconds. What is needed is (1) decoupling of the instantiation of the end-point (i.e., the VM IP address becoming active) from its advertisement to the network as the preferred route to that IP address; and (2) the ability to tunnel traffic via VM’s old (NVE) location during the routing convergence.

Maria

From: [email-masked] [mailto:[email-masked]] On Behalf Of Xuxiaohu
Sent: Tuesday, July 24, 2012 10:40 PM
To: Joe Pelissier (jopeliss); [email-masked]
Subject: Re: [nvo3] Role of ARP/RARP

Hi Joe,

Best regards,
Xiaohu

发件人: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] 代表 Joe Pelissier (jopeliss)
发送时间: 2012年7月25日 1:22
收件人: [email-masked]<mailto:[email-masked]>
主题: Re: [nvo3] Role of ARP/RARP

My $0.02 worth…
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: "Ivan Pepelnjak" <[email-masked]>
Date: Wed, 25 Jul 2012 07:11:47 +0200

Totally agree. Thanks for being way more precise than I was.

Ivan

From: David Allan I [mailto:[email-masked]] 
Sent: Wednesday, July 25, 2012 1:11 AM
To: Luyuan Fang (lufang); Lucy yong; Ivan Pepelnjak; [email-masked]; [email-masked]
Subject: RE: [nvo3] Role of ARP/RARP

cheers

Dave

  _____  

Luyuan

Lucy

You nailed it.

Luyuan

Best,

Ivan

Ivan,

Thanks,
--David

Joe,

Am I missing something?

Ivan

Hi David,

Joe Pelissier

Joe,

Thanks,
--David

My $0.02 worth…

Joe Pelissier

Maria,

idea.

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,

Ivan

Thanks,
--David

Ivan

Truman

Cc: [email-masked]

My two cents. 

Linda Dunbar

Best regards,

Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Kireeti.

Thanks,
--David

 

-- 
Kireeti

---

From: "Ivan Pepelnjak" <[email-masked]>
Date: Wed, 25 Jul 2012 07:25:12 +0200

Maria,

Are there any documented requirements for maximum disruption time during the live VM migration event? 

Based on simple tests (like ping) people usually think the packet loss is minimal, but in a high-volume non-TCP environment you might lose hundreds of packets before the second VM becomes operational.

Ivan 

From: [email-masked] [mailto:[email-masked]] On Behalf Of NAPIERALA, MARIA H
Sent: Wednesday, July 25, 2012 6:09 AM
To: Xuxiaohu; Joe Pelissier (jopeliss); [email-masked]
Subject: Re: [nvo3] Role of ARP/RARP

Xiaohu,

 

Maria

Hi Joe,

Best regards,

Xiaohu

My $0.02 worth…

Joe Pelissier

Maria,

idea.

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,

Ivan

Thanks,
--David

Ivan

Truman

Cc: [email-masked]

My two cents. 

Linda Dunbar

Best regards,

Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Kireeti.

Thanks,
--David

 

-- 
Kireeti

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Wed, 25 Jul 2012 06:18:00 +0000

Ivan,

the transport sessions would typically timeout in few seconds but some interactive applications have stringent latency requirements of less than a second. The network restoration should not be a bottleneck in the VM migration. I believe that the goal for network related disruption should be no more than few hundreds of milliseconds. This is a general requirement for any network these days ;-)

Maria

From: Ivan Pepelnjak [mailto:[email-masked]]
Sent: Wednesday, July 25, 2012 1:25 AM
To: NAPIERALA, MARIA H; 'Xuxiaohu'; 'Joe Pelissier (jopeliss)'; [email-masked]
Subject: RE: [nvo3] Role of ARP/RARP

Maria,

Ivan

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of NAPIERALA, MARIA H
Sent: Wednesday, July 25, 2012 6:09 AM
To: Xuxiaohu; Joe Pelissier (jopeliss); [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] Role of ARP/RARP

Xiaohu,

Maria

Hi Joe,

Best regards,
Xiaohu

My $0.02 worth…
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Wed, 25 Jul 2012 07:10:41 +0000

David,

When virtualization is done correctly, the data center transport network is unaware of VPNs/VLANs. I think this is what you are assuming also.
In L3VPN approach to data center, the VM route distribution to NVEs (which, ideally, should be the hypervisors) will be constrained based on the VPN membership. Today, there are WAN deployments of MPLS/BGP VPNs supporting tens of thousands of VPNs and several (7+) millions of routes.
A L3VPN system can be configured to suppress more specific routes than a specified aggregate by advertising the aggregate from a specific PE where the more specific routes are present.

Maria

From: [email-masked] [mailto:[email-masked]] On Behalf Of David Allan I
Sent: Tuesday, July 24, 2012 7:11 PM
To: Luyuan Fang (lufang); Lucy yong; Ivan Pepelnjak; [email-masked]; [email-masked]
Subject: Re: [nvo3] Role of ARP/RARP

cheers
Dave

________________________________
From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Luyuan Fang (lufang)
Sent: Tuesday, July 24, 2012 2:51 PM
To: Lucy yong; Ivan Pepelnjak; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] Role of ARP/RARP
The point we were making was not bridging does not have place. In fact, it has and it will, for long time if not forever. We are talking about scaling, including operational simplicity.
A lot of places may not need to scale huge in reality, either way would work, if they like to stay the way it is, fine.
A few others need huge scale, they may not need to worry about backward compatibility or follow the old path, especially when building green field.
We have different sets of operators, the goals/requirements/solutions would not be the same, I think Pedro pointed this out a while ago.

Luyuan

From: Lucy yong [mailto:[email-masked]]
Sent: Tuesday, July 24, 2012 5:16 PM
To: Luyuan Fang (lufang); Ivan Pepelnjak; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: RE: [nvo3] Role of ARP/RARP

Lucy

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Luyuan Fang (lufang)
Sent: Tuesday, July 24, 2012 2:27 PM
To: Ivan Pepelnjak; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] Role of ARP/RARP

You nailed it.
Luyuan

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Ivan Pepelnjak
Sent: Tuesday, July 24, 2012 3:02 PM
To: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] Role of ARP/RARP

Best,
Ivan

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]
Sent: Tuesday, July 24, 2012 8:47 PM
To: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: RE: [nvo3] Role of ARP/RARP

Ivan,

Thanks,
--David

Joe,

Am I missing something?
Ivan

Hi David,

Joe Pelissier

Joe,

Thanks,
--David

My $0.02 worth...
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 25 Jul 2012 07:13:35 +0000

Shouldn’t the timeout threshold for a TCP session be a few minutes, rather than a few seconds?

Best regards,
Xiaohu

发件人: NAPIERALA, MARIA H [mailto:[email-masked]]
发送时间: 2012年7月25日 14:18
收件人: Ivan Pepelnjak; Xuxiaohu; 'Joe Pelissier (jopeliss)'; [email-masked]
主题: RE: [nvo3] Role of ARP/RARP

Ivan,

Maria

Maria,

Ivan

Xiaohu,

Maria

Hi Joe,

Best regards,
Xiaohu

My $0.02 worth…
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 25 Jul 2012 07:44:39 +0000

Hi Maria,

Yes, in Virtual Subnet, upon receiving an ARP from a given local TES (e.g., a VM), the NVE (i.e., PE router) would generate a corresponding host route for that TES and then propagate that route to remote PE routers.

As for “this is not sufficient to assure the required minimal disruption for “live” VM move because the route propagation is not going to happen in 10s or even 100s of milliseconds”, I know that there is a limitation for VM live move, that is: the maximum round-trip latency between the source and destination hypervisors cannot exceed a few milliseconds (e.g., 5 to 10 millisecond), but I don’t understand the sessions to/from the moved VM would timeout due to a destination unreachability of a few milliseconds. If so, did that mean any DC VPN approach which uses control-plane to propagate MAC or IP reachability info of VMs between NVEs is not available for supporting live VM migration?

Best regards,
Xiaohu

发件人: NAPIERALA, MARIA H [mailto:[email-masked]]
发送时间: 2012年7月25日 12:09
收件人: Xuxiaohu; Joe Pelissier (jopeliss); [email-masked]
主题: RE: [nvo3] Role of ARP/RARP

Xiaohu,

Maria

Hi Joe,

Best regards,
Xiaohu

My $0.02 worth…
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Wed, 25 Jul 2012 08:36:27 +0000

Xiaohu,

I did not say that the solutions using control-plane to propagate VM reachability would not support VM-live migration. I said that those solutions must assure that the network connectivity to VM’s new location is restored fast enough so that the applications (e.g., with sub-second latency requirement for their correct operation) are not impacted. The VM's network connectivity restoration time is critical to a successful VM move. Hence, you cannot just rely on the “plain” convergence of the control plane to the VM’s new location because it is not going to be fast enough for all applications. You need some mechanisms (such as I listed) to assure minimal disruption in network connectivity to the moved VM.

Maria

From: Xuxiaohu [mailto:[email-masked]]
Sent: Wednesday, July 25, 2012 3:45 AM
To: NAPIERALA, MARIA H; Joe Pelissier (jopeliss); [email-masked]
Subject: 答复: [nvo3] Role of ARP/RARP

Hi Maria,

Best regards,
Xiaohu

发件人: NAPIERALA, MARIA H [mailto:[email-masked]]
发送时间: 2012年7月25日 12:09
收件人: Xuxiaohu; Joe Pelissier (jopeliss); [email-masked]<mailto:[email-masked]>
主题: RE: [nvo3] Role of ARP/RARP

Xiaohu,

Maria

Hi Joe,

Best regards,
Xiaohu

My $0.02 worth…
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 25 Jul 2012 10:04:42 +0000

Hi Maria,

As far as I know,  the “plain” convergence of most modern routing protocols (e.g., OSPF, ISIS and BGP) could already be able to achieve sub-second convergence.

Best regards,

Xiaohu

发送时间: 2012年7月25日 16:36

Xiaohu,

Maria

Hi Maria,

Best regards,
Xiaohu

Xiaohu,

Maria

Hi Joe,

Best regards,
Xiaohu

My $0.02 worth…
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Wed, 25 Jul 2012 14:48:21 +0000

Xiaohu,

In any decent size data center the routing updates (in virtual networks) will need be propagated (and processed) via multiple control elements to reach remote PEs. Also, there will be a lot of routing churn due to VM moves, including maintenance events where thousands of VMs can be simultaneously moved between server racks. The convergence of a control plane supporting such system will not be in sub-second (not to mention 10s or few 100s of milliseconds)..
Remember that we have also to account for the time needed in the final phase of migration which pauses the virtual machine in its old location and copies any remaining modified memory pages to the VM’s new destination. This implies that the time that can be allocated to network convergence to the VM’s new location is limited.

Maria

From: Xuxiaohu [mailto:[email-masked]]
Sent: Wednesday, July 25, 2012 6:05 AM
To: NAPIERALA, MARIA H; Joe Pelissier (jopeliss); [email-masked]
Subject: re: [nvo3] Role of ARP/RARP

Hi Maria,

Best regards,

Xiaohu

Xiaohu,

Maria

From: Xuxiaohu [mailto:[email-masked]]
Sent: Wednesday, July 25, 2012 3:45 AM
To: NAPIERALA, MARIA H; Joe Pelissier (jopeliss); [email-masked]<mailto:[email-masked]>
Subject: 答复: [nvo3] Role of ARP/RARP

Hi Maria,

Best regards,
Xiaohu

Xiaohu,

Maria

Hi Joe,

Best regards,
Xiaohu

My $0.02 worth…
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: David Allan I <[email-masked]>
Date: Wed, 25 Jul 2012 11:33:53 -0400

HI Xiaohu:

IMO IPinIP will degenerate into host routes carried over a layer that has aggregatable addresses. So the Customer-IP (for lack of better term) layer will run effectively flat. A small amount of address aggregation will be possible when a bunch of VMs in a common subnet are hosted on a common server but for a hypervisor based solution that's about the limit. A server is the maximum scope of a subnet and any VMs moved away from the server are simply longer match exceptions.

So the part that actually might improve scaling is the complete elimination of broadcast domains. But if I have to route between subnets anyway, I'd observe that a useful tool is being discarded. IMO a real subnet as a unit to which admin/policy etc. can be applied is administratively much simpler than at the level of individual host. Far less keyboard fondling by ops. and far fewer touch points if the subnet is not spread all over the map as a set of various length match prefixes.

cheers
Dave

________________________________
From: Xuxiaohu [mailto:[email-masked]]
Sent: Tuesday, July 24, 2012 6:48 PM
To: David Allan I; Luyuan Fang (lufang); Lucy yong; Ivan Pepelnjak; [email-masked]; [email-masked]
Subject: re: [nvo3] Role of ARP/RARP

Hi Dave,

Best regards,
Xiaohu

cheers
Dave

Luyuan

Lucy

You nailed it.
Luyuan

Best,
Ivan

Ivan,

Thanks,
--David

Joe,

Am I missing something?
Ivan

Hi David,

Joe Pelissier

Joe,

Thanks,
--David

My $0.02 worth…
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: David Allan I <[email-masked]>
Date: Wed, 25 Jul 2012 11:40:16 -0400

I agree with your points at a 50000 foot level.

But you are contrasting the capabilities of a portion of a silicon chip in a server with some multi-terabit switching platform located in the WAN. I would expect it to scale better.

IMO at the end of the day, the approximate amount of state in a hypervisor for MACinIP or IPinIP will be a wash. And the question is simply "how complex is the system that populates that state?"

cheers
Dave

Sent: Wednesday, July 25, 2012 12:11 AM

David,

Maria

cheers
Dave

Luyuan

Lucy

You nailed it.
Luyuan

Best,
Ivan

Ivan,

Thanks,
--David

Joe,

Am I missing something?
Ivan

Hi David,

Joe Pelissier

Joe,

Thanks,
--David

My $0.02 worth...
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Wed, 25 Jul 2012 15:48:20 +0000

David,

When referring to millions of routes I am referring to control plane "capacity" (which is outside of the hypervisor, or TOR if you like) and not to a single hypervisor forwarding capacity. A hypervisor will only need to store forwarding tables of VPNs that have local attachments.
IPinIP solutions do not have to maintain MAC-to-IP mappings across a virtual network which is a big scaling advantage.

Maria

From: David Allan I [mailto:[email-masked]]
Sent: Wednesday, July 25, 2012 11:40 AM
To: NAPIERALA, MARIA H; Luyuan Fang (lufang); Lucy yong; Ivan Pepelnjak; [email-masked]; [email-masked]
Subject: RE: [nvo3] Role of ARP/RARP

cheers
Dave

To: David Allan I; Luyuan Fang (lufang); Lucy yong; Ivan Pepelnjak; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>

Maria

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of David Allan I
Sent: Tuesday, July 24, 2012 7:11 PM
To: Luyuan Fang (lufang); Lucy yong; Ivan Pepelnjak; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] Role of ARP/RARP

cheers
Dave

Luyuan

Lucy

You nailed it.
Luyuan

Best,
Ivan

Ivan,

Thanks,
--David

Joe,

Am I missing something?
Ivan

Hi David,

Joe Pelissier

Joe,

Thanks,
--David

My $0.02 worth...
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: AshwoodsmithPeter <[email-masked]>
Date: Wed, 25 Jul 2012 16:04:59 +0000

> IMO at the end of the day, the approximate amount of state in a hypervisor for MACinIP or IPinIP will be a wash.
> And the question is simply "how complex is the system that populates that state?"
> cheers
> Dave

Exactly. It does not matter if the inner address is 48 bits or 32, the fact that its flat and that control mechanisms are required to either distribute all (inner/outer) relationships pro-actively or perform re-active queries is what causes the challenges to unicast scale. Distributed storage mechanisms can help the re-active mechanisms a bit by improving scale linearly by the distribution factor but there is no substitute for hierarchy.

Peter

Maria

cheers
Dave

Luyuan

Lucy

You nailed it.
Luyuan

Best,
Ivan

Ivan,

Thanks,
--David

Joe,

Am I missing something?
Ivan

Hi David,

Joe Pelissier

Joe,

Thanks,
--David

My $0.02 worth...
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: Truman Boyes <[email-masked]>
Date: Wed, 25 Jul 2012 12:10:06 -0400

On Wed, Jul 25, 2012 at 3:44 AM, Xuxiaohu <[email-masked]> wrote:

>  Hi Maria,****
>
> ** **
>
> Yes, in Virtual Subnet, upon receiving an ARP from a given local TES
> (e.g., a VM), the NVE (i.e., PE router) would generate a corresponding host
> route for that TES and then propagate that route to remote PE routers. ***
> *
>
> ** **
>
> As for “this is not sufficient to assure the required minimal disruption
> for “live” VM move because the route propagation is not going to happen in
> 10s or even 100s of milliseconds”, I know that there is a limitation for VM
> live move, that is: the maximum round-trip latency between the source and
> destination hypervisors cannot exceed a few milliseconds (e.g., 5 to 10
> millisecond), but I don’t understand the sessions to/from the moved VM
> would timeout due to a destination unreachability of a few milliseconds. If
> so, did that mean any DC VPN approach which uses control-plane to propagate
> MAC or IP reachability info of VMs between NVEs is not available for
> supporting live VM migration?****
>
> ** **
>
> Best regards,****
>
> Xiaohu
>

Hi Xiaohu,

As you referenced Virtual Subnet, I wanted to share a couple comments and
thoughts on this approach in DC:

   - The document discusses one of the motivations for layer 2 extensions
   is to avoid IP renumbering of servers (ie. VMs). I think the landscape is
   changing dramatically regarding renumbering of virtual machines;
   Cloud-based approaches are making addressing dynamic (DHCP assigned
   servers) and effectively abstracting service from servers. OpenStack,
   CloudStack, and other such technologies make renumbering of machines
   unnecessary. Essentially, when a VM needs to move to another location; a
   new instance is created in Site B, and traffic is "mapped" to the instance
   instead of dealing with complexities of network renumbering.
   - I think the draft (draft-xu-virtual-subnet-08) provides a very useful
   approach to enabling layer 2 (by way of Layer 3 technologies) for Data
   Centers, but there are some considerations that should be detailed:
      - The CE Host Discovery methods are prone to inefficiencies or
      failure. The size of the subnet would have a   significant
impact on ARP or
      ICMP scans. LLDP would not exist on each VM OS; if this type of signaling
      occurred at the Hypervisor level it might be more feasible, but then
      VDP/LLDP might not be the best protocol for this purpose.
   - In section 3.5 (CE Host Mobility), the gratuitous ARP packet
   generation will be OS dependent. In many implementations (Linux, some
   BSDs), the GARP packet is sent when the interface transitions or addressing
   changes on the interface. In order to accommodate this, we would need to
   have guest VMs perform this action, which may in turn cause application
   interference.
   - In section 3.6.2, section 2, there should be a consideration on
   routing propagation impacts as there is a potential impact on security and
   network resilience. Effectively, there is now a linkage between host ARP
   requests, FIB updates, MP-BGP update messages, and Route Reflectors, and
   control plan processing on PEs.  Let's say for example that a guest VM that
   belongs to 192.168/16 sends ARP requests to all 65k addresses; would this
   have a significant impact on the control plane of numerous PE and MP-BGP
   speakers?

 Kind regards,
Truman

****
>
> ** **
>
> *发件人:* NAPIERALA, MARIA H [mailto:[email-masked]]
> *发送时间:* 2012年7月25日 12:09
> *收件人:* Xuxiaohu; Joe Pelissier (jopeliss); [email-masked]
> *主题:* RE: [nvo3] Role of ARP/RARP****
>
> ** **
>
> Xiaohu,****
>
> ** **
>
> Are you assuming that a gratuitous ARP is snooped by the NVE, “turned”
> into a host route and propagated via a control plane to other NVEs? If yes,
> this is not sufficient to assure the required minimal disruption for “live”
> VM move because the route propagation is not going to happen in 10s or even
> 100s of milliseconds. What is needed is (1) decoupling of the
> instantiation of the end-point (i.e., the VM IP address becoming active)
> from its advertisement to the network as the preferred route to that IP
> address; and (2) the ability to tunnel traffic via VM’s old (NVE) location
> during the routing convergence. ****
>
>  ****
>
> ** **
>
> Maria****
>
> ** **
>
> ** **
>
> Of *Xuxiaohu
> *Sent:* Tuesday, July 24, 2012 10:40 PM
> *To:* Joe Pelissier (jopeliss); [email-masked]
> *Subject:* Re: [nvo3] Role of ARP/RARP****
>
> ** **
>
> Hi Joe,****
>
> ** **
>
> Thanks a lot for this professional clarification.  BTW, another benefit of
> using gratuitous ARP, rather than any frame received with the source MAC or
> IP address of the VM,  as a notification of VM attachments is that the
> former is applicable not only in the L2NVE case but also in the L3NVE case,
> and more importantly, it has no requirement for any new implementation on
> L3NVEs.****
>
> ** **
>
> Best regards,****
>
> Xiaohu****
>
> ** **
>
> *发件人**:* [email-masked] [mailto:[email-masked]<[email-masked]>]
> *代表** *Joe Pelissier (jopeliss)
> *发送时间**:* 2012年7月25日 1:22
> *收件人**:* [email-masked]
> *主**题**:* Re: [nvo3] Role of ARP/RARP****
>
> ** **
>
> I think there is a subtle difference between an associate and what the
> ARP/RARP (or ND) does.  There are potentially two steps:****
>
> ** **
>
> **1)      **Associate: this sets up all of the network parameters in the
> switch/router/NVE but does not update the forwarding.  This can be a
> relatively slow process since it sets up things like ACLs, etc., and may
> require fetching data from other databases.  This process can take place
> while the VM is still operating in the old location.****
>
> **2)      ** ARP/RARP/ND: This indicates that the VM is now becoming
> active in the new location (and that it is shut down in the old location)
> and forwarding tables need to be updated.  The nice thing about an
> ARP/RARP/ND approach is that it updates all of the tables in the L2 LAN,
> and is therefore a fundamental part of a VM move.  The ARP/RARP/ND
> essentially needs to be done anyway to quickly converge forwarding in the
> L2, so there may not be a lot of point in duplicating this.  The
> ARP/RARP/NDs could be snooped by an NVE to indicate that the VM is now
> active.****
>
> ** **
>
> IAW: the ARP/RARP/ND is not an implicit associate; the associate and
> ARP/RARP/ND perform different functions.  The critical time is the period
> during which the VM is suspended during the move, so it is important to
> have all of the associate work ready to go prior to the move.  The
> ARP/RARP/ND indicates that the move is now complete and to activate the new
> forwarding, which can be done very quickly.****
>
> ** **
>
> (One other observation: it really does not need to be an ARP/RARP/ND.
> Essentially, any frame received with the source MAC or IP address of the VM
> (which is known as part of the associate process), is indicative of the VM
> becoming active.  ARP/RARP/ND approach is widely deployed since it has the
> effect of updating the L2 forwarding everywhere in the L2 network.)****
>
> ** **
>
> My $0.02 worth…****
>
> Joe Pelissier****
>
> ** **
>
> *On Behalf Of [email-masked]
> *Sent:* Tuesday, July 24, 2012 9:46 AM
> *To:* [email-masked]
> *Cc:* [email-masked]
> *Subject:* Re: [nvo3] Role of ARP/RARP****
>
> ** **
>
> Maria,****
>
> ** **
>
> We may be close to agreement.  I don’t see ARP/RARP as viable on all
> topologies between End Device****
>
> and NVE, but it clearly works fine over a single L2 link.  The moment the
> access has to cross an****
>
> L3 forwarding hop, ARP/RARP gets much more involved, and an IP-based
> solution is probably a better****
>
> idea.****
>
> ** **
>
> I’m also not advocating “trying to be forever backwards compatible”, but
> ARP/RARP sure looks like****
>
> a useful mechanism to preserve for some transition scenarios.  The
> alternative of complete****
>
> (“forklift”) upgrades of everything at once gets ugly fast in large
> facilities.****
>
> ** **
>
> Thanks,
> --David****
>
> ** **
>
> *From:* NAPIERALA, MARIA H [mailto:[email-masked] <[email-masked]>]
> *Sent:* Tuesday, July 24, 2012 12:37 PM
> *To:* Black, David; [email-masked]
> *Cc:* [email-masked]
> *Subject:* RE: [nvo3] Role of ARP/RARP****
>
> ** **
>
> David> Also, at the risk of really attracting responses ;-), this
> simplification of****
>
> ** **
>
> :-)****
>
> ** **
>
> David> incremental deployment is a reason to consider ARP/RARP as an
> additional implicit****
>
> David> associate primitive for IP-only solutions (and see previous
> discussion about this****
>
> David> L2 approach working well on a single L2 link but less well as the
> L2 and L3****
>
> David> topology between End Device and NVE gets more complex).****
>
> ** **
>
> If this “access” stretches out beyond ToR (not even mentioning hypervisor,
> which, IMO, it is the most optimal place to initiate the virtual network
> including the encapsulation), then I am not sure what value such solution
> will bring in terms of addressing one of the main requirements for
> virtualized data centers which is the independence of the data-center
> switching infrastructure.****
>
> ** **
>
> When the technology moves forwards, the upgrades of hypervisor and
> switching equipment will be required. Trying to be forever backward
> compatible with an old technology will not be productive and will have to
> be abandoned.  This happened many times in, e.g., large Service Providers
> networks. ****
>
> ** **
>
> Maria****
>
> ** **
>
> ** **
>
> *On Behalf Of [email-masked]
> *Sent:* Tuesday, July 24, 2012 9:03 AM
> *To:* [email-masked]
> *Cc:* [email-masked]
> *Subject:* Re: [nvo3] Role of ARP/RARP****
>
> ** **
>
> Ivan,****
>
> ** **
>
> > Obviously we’ll have to re-architect the existing “squashed complexity
> sausage”****
>
> > ways of doing things, in which the networking industry tries to
> implement all****
>
> > sorts of kludges to accommodate simplistic VLAN/L2-only hypervisor
> switches.****
>
> > Evidently this approach led us nowhere (or we wouldn’t have this working
> group)****
>
> > and yesterday’s news told us VMware clearly recognized simplistic
> hypervisor****
>
> > switches are not the way to go. While re-architecting the existing mess,
> we****
>
> > just might admit that current bad practices are just that – bad.****
>
> ** **
>
> That’s a fine “greenfield” or “blank sheet of paper” design approach, but
> I think****
>
> you’re ignoring the “running code” that uses ARP/RARP for data plane
> learning.****
>
> It would be good to have a means of coping with that “running code” while
> we****
>
> transition to this brave new world ...****
>
> ** **
>
> For clarity, I don’t think this is an either/or situation, but rather one
> in which****
>
> both approaches have merits and both should be pursued.  Specifically, I’d
> expect****
>
> better results from an explicit associate message (e.g., it provides an
> explicit****
>
> check that the interface [VSI or vNIC] that is associating is the one that
> is****
>
> expected based on the pre-associate), and would agree that this should be
> the****
>
> primary design approach.  OTOH, the ability to accommodate existing
> ARP/RARP usage****
>
> as an implicit associate is likely to make what we design easier to
> incrementally****
>
> deploy into existing data centers, especially enterprise data centers,
> whose admins****
>
> tend to show the door to anyone who talks about complete (“forklift”)
> upgrades ;-).****
>
> ** **
>
> For IPv6, ND can be used, but I’m not sure how much of that is deployed
> for this****
>
> significant deployed base of “running code” to deal with.****
>
> ** **
>
> Also, at the risk of really attracting responses ;-), this simplification
> of****
>
> incremental deployment is a reason to consider ARP/RARP as an additional
> implicit****
>
> associate primitive for IP-only solutions (and see previous discussion
> about this****
>
> topology between End Device and NVE gets more complex).****
>
> ** **
>
> Thanks,
> --David****
>
> ** **
>
> *On Behalf Of *Ivan Pepelnjak
> *Sent:* Tuesday, July 24, 2012 3:41 AM
> *To:* Black, David; [email-masked]
> *Cc:* [email-masked]
> *Subject:* Re: [nvo3] Role of ARP/RARP (was: Comments on
> draft-kompella-nvo3-server2nve)****
>
> ** **
>
> *** Provided that *** we have a management channel that can send
> pre-associate message from the orchestration tool to the NVE, that same
> management channel can also send the associate message (including the MAC
> address of the VM). Don’t use two protocols for two steps of the same
> process, and don’t rely on protocols that (as Truman noted) work for IPv4
> only (see also http://tools.ietf.org/html/draft-george-ipv6-support-01). *
> ***
>
> ** **
>
> Obviously we’ll have to re-architect the existing “squashed complexity
> sausage” ways of doing things, in which the networking industry tries to
> implement all sorts of kludges to accommodate simplistic VLAN/L2-only
> hypervisor switches. Evidently this approach led us nowhere (or we wouldn’t
> have this working group) and yesterday’s news told us VMware clearly
> recognized simplistic hypervisor switches are not the way to go. While
> re-architecting the existing mess, we just might admit that current bad
> practices are just that – bad.****
>
> ** **
>
> Best,****
>
> Ivan****
>
> ** **
>
> *On Behalf Of [email-masked]
> *Sent:* Monday, July 23, 2012 11:49 PM
> *To:* [email-masked]
> *Cc:* [email-masked]
> *Subject:* [nvo3] Role of ARP/RARP (was: Comments on
> draft-kompella-nvo3-server2nve)****
>
> ** **
>
> **Provided that** the pre-associate operation that selects the virtual
> network****
>
> for the VM and sets up access it (e.g., VLAN between End Device and NVE),*
> ***
>
> is done via other (suitably secured) means, a gratuitous ARP or RARP on
> that****
>
> pre-configured access can be used to signal the VM’s arrival to use the
> network****
>
> that’s been preconfigured for it (i.e., the associate event).  As Linda
> noted,****
>
> there’s lots of running code that does this, but (as Truman comments)
> ARP/RARP****
>
> (that has to be done via some other means).****
>
> ** **
>
> Thanks,
> --David****
>
> ** **
>
> *From:* Truman Boyes [mailto:[email-masked] <[email-masked]>]
>
> *Sent:* Monday, July 23, 2012 1:03 PM
> *To:* Ivan Pepelnjak
> *Cc:* Linda Dunbar; Xuxiaohu; Black, David; [email-masked];
> [email-masked]
> On Mon, Jul 23, 2012 at 12:22 PM, Ivan Pepelnjak <[email-masked]>
> wrote:****
>
> Ivan****
> challenges if we had a reliance on  IPv4 address resolution protocol. ****
>
> ** **
>
> Truman****
>
>  ****
>
>   ****
>
> *To:* Xuxiaohu; [email-masked]; [email-masked]****
>
>
> *Cc:* [email-masked]

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Wed, 25 Jul 2012 16:09:35 +0000

David,

I don’t think we can make any assumptions about subnet locality/scope since a VM can move anywhere across a data center (this is how you make a data center cost effective). We have to assume that there will be collection of host routes randomly spread across a large area.

Maria

From: [email-masked] [mailto:[email-masked]] On Behalf Of David Allan I
Sent: Wednesday, July 25, 2012 11:34 AM
To: Xuxiaohu; Luyuan Fang (lufang); Lucy yong; Ivan Pepelnjak; [email-masked]; [email-masked]
Subject: Re: [nvo3] Role of ARP/RARP

HI Xiaohu:

cheers
Dave

Best regards,
Xiaohu

发件人: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] 代表 David Allan I
发送时间: 2012年7月25日 7:11
收件人: Luyuan Fang (lufang); Lucy yong; Ivan Pepelnjak; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
主题: Re: [nvo3] Role of ARP/RARP

cheers
Dave

Luyuan

Lucy

You nailed it.
Luyuan

Best,
Ivan

Ivan,

Thanks,
--David

Joe,

Am I missing something?
Ivan

Hi David,

Joe Pelissier

Joe,

Thanks,
--David

My $0.02 worth…
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: David Allan I <[email-masked]>
Date: Wed, 25 Jul 2012 12:11:28 -0400

I think we are saying the exact same thing. I was simply trying to explore best case vs, (at least to me) the obvious worst case.

cheers
Dave

________________________________
From: NAPIERALA, MARIA H [mailto:[email-masked]]
Sent: Wednesday, July 25, 2012 9:10 AM
To: David Allan I; Xuxiaohu; Luyuan Fang (lufang); Lucy yong; Ivan Pepelnjak; [email-masked]; [email-masked]
Subject: RE: [nvo3] Role of ARP/RARP

David,

Maria

HI Xiaohu:

cheers
Dave

Best regards,
Xiaohu

cheers
Dave

Luyuan

Lucy

You nailed it.
Luyuan

Best,
Ivan

Ivan,

Thanks,
--David

Joe,

Am I missing something?
Ivan

Hi David,

Joe Pelissier

Joe,

Thanks,
--David

My $0.02 worth…
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: David Allan I <[email-masked]>
Date: Wed, 25 Jul 2012 12:21:07 -0400

I would agree that we effective have a 48+32 bit mapping to server IP vs. simply a 32 bit mapping to server IP in the IPv4 case.

Where I would expect economy of MACinIP over IPinIP is the default gateway as a single entry for everything outside the subnet vs. flat operation so fewer but for all intents and purposes bigger entries for MACinIP vs. more entries for IPinIP.

Where the breakpoint would be would depend on the scale we were working to and the design of virtualization of the networking to interconnect the VMs.

cheers
Dave

Sent: Wednesday, July 25, 2012 8:48 AM

David,

Maria

cheers
Dave

Maria

cheers
Dave

Luyuan

Lucy

You nailed it.
Luyuan

Best,
Ivan

Ivan,

Thanks,
--David

Joe,

Am I missing something?
Ivan

Hi David,

Joe Pelissier

Joe,

Thanks,
--David

My $0.02 worth...
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: "Ivan Pepelnjak" <[email-masked]>
Date: Wed, 25 Jul 2012 19:17:15 +0200

Truman, as I explained yesterday, the GARP sent after live VM move is actually generated (faked) by the hypervisor soft switch. It always happens regardless of what OS VM uses (unless you disable it).

From: [email-masked] [mailto:[email-masked]] On Behalf Of Truman Boyes
Sent: Wednesday, July 25, 2012 6:10 PM
To: Xuxiaohu
Cc: Joe Pelissier (jopeliss); [email-masked]; NAPIERALA, MARIA H
Subject: Re: [nvo3] ??: Role of ARP/RARP

Hi Maria,

Best regards,

Xiaohu

Hi Xiaohu,

As you referenced Virtual Subnet, I wanted to share a couple comments and thoughts on this approach in DC:

*	The document discusses one of the motivations for layer 2 extensions is to avoid IP renumbering of servers (ie. VMs). I think the landscape is changing dramatically regarding renumbering of virtual machines; Cloud-based approaches are making addressing dynamic (DHCP assigned servers) and effectively abstracting service from servers. OpenStack, CloudStack, and other such technologies make renumbering of machines unnecessary. Essentially, when a VM needs to move to another location; a new instance is created in Site B, and traffic is "mapped" to the instance instead of dealing with complexities of network renumbering.
*	I think the draft (draft-xu-virtual-subnet-08) provides a very useful approach to enabling layer 2 (by way of Layer 3 technologies) for Data Centers, but there are some considerations that should be detailed:

*	The CE Host Discovery methods are prone to inefficiencies or failure. The size of the subnet would have a   significant impact on ARP or ICMP scans. LLDP would not exist on each VM OS; if this type of signaling occurred at the Hypervisor level it might be more feasible, but then VDP/LLDP might not be the best protocol for this purpose. 

*	In section 3.5 (CE Host Mobility), the gratuitous ARP packet generation will be OS dependent. In many implementations (Linux, some BSDs), the GARP packet is sent when the interface transitions or addressing changes on the interface. In order to accommodate this, we would need to have guest VMs perform this action, which may in turn cause application interference.
*	In section 3.6.2, section 2, there should be a consideration on routing propagation impacts as there is a potential impact on security and network resilience. Effectively, there is now a linkage between host ARP requests, FIB updates, MP-BGP update messages, and Route Reflectors, and control plan processing on PEs.  Let's say for example that a guest VM that belongs to 192.168/16 sends ARP requests to all 65k addresses; would this have a significant impact on the control plane of numerous PE and MP-BGP speakers?

 Kind regards,

Truman

Xiaohu,

 

Maria

Hi Joe,

Best regards,

Xiaohu

My $0.02 worth…

Joe Pelissier

Maria,

idea.

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,

Ivan

Thanks,
--David

Ivan

Truman

Cc: [email-masked]

My two cents. 

Linda Dunbar

Best regards,

Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Kireeti.

Thanks,
--David

 

-- 
Kireeti

---

From: Truman Boyes <[email-masked]>
Date: Wed, 25 Jul 2012 14:08:32 -0400

Ivan, yes I followed your clarification yesterday. Would the GARP packets
from VMs be filtered or acted upon at the vswitch level?

On Wed, Jul 25, 2012 at 1:17 PM, Ivan Pepelnjak <[email-masked]>wrote:

> Truman, as I explained yesterday, the GARP sent after live VM move is
> actually generated (faked) by the hypervisor soft switch. It always happens
> regardless of what OS VM uses (unless you disable it).****
>
> ** **
>
> Of *Truman Boyes
> *Sent:* Wednesday, July 25, 2012 6:10 PM
> *To:* Xuxiaohu
> *Cc:* Joe Pelissier (jopeliss); [email-masked]; NAPIERALA, MARIA H
> *Subject:* Re: [nvo3] ??: Role of ARP/RARP****
>
> ** **
>
> On Wed, Jul 25, 2012 at 3:44 AM, Xuxiaohu <[email-masked]> wrote:****
>
> Hi Maria,****
>
>  ****
>
> Hi Xiaohu,****
> thoughts on this approach in DC:****
>
> ** **
>
>    instead of dealing with complexities of network renumbering.****
>    - I think the draft (draft-xu-virtual-subnet-08) provides a very
>    useful approach to enabling layer 2 (by way of Layer 3 technologies) for
>    Data Centers, but there are some considerations that should be detailed:
>    ****
>
>
>       failure. The size of the subnet would have a   significant impact on ARP or
>       VDP/LLDP might not be the best protocol for this purpose. ****
>
>
>    interference.****
>    speakers?****
>  Kind regards,****
> *发件人**:* NAPIERALA, MARIA H [mailto:[email-masked]]
> *发送时间**:* 2012年7月25日 12:09
> *收件人**:* Xuxiaohu; Joe Pelissier (jopeliss); [email-masked]
> *主**题**:* RE: [nvo3] Role of ARP/RARP****
>
>  ****
>
> Xiaohu,****
>
>  ****
>
> 1)      Associate: this sets up all of the network parameters in the
> 2)       ARP/RARP/ND: This indicates that the VM is now becoming active
> in the new location (and that it is shut down in the old location) and
> forwarding tables need to be updated.  The nice thing about an ARP/RARP/ND
> approach is that it updates all of the tables in the L2 LAN, and is
> therefore a fundamental part of a VM move.  The ARP/RARP/ND essentially
> needs to be done anyway to quickly converge forwarding in the L2, so there
> may not be a lot of point in duplicating this.  The ARP/RARP/NDs could be
> snooped by an NVE to indicate that the VM is now active.****
>
>  ****
>

---

From: <[email-masked]>
Date: Wed, 25 Jul 2012 14:35:12 -0400

VM-issued ARPs need to work because there’s an ARP implementation in each VM ... the software virtual switch in the hypervisor (vSwitch) behaves like an L2 bridge in current implementations.

Thanks,
--David

From: [email-masked] [mailto:[email-masked]] On Behalf Of Truman Boyes
Sent: Wednesday, July 25, 2012 2:09 PM
To: Ivan Pepelnjak
Cc: Joe Pelissier (jopeliss); Xuxiaohu; [email-masked]; NAPIERALA, MARIA H
Subject: Re: [nvo3] ??: Role of ARP/RARP

Ivan, yes I followed your clarification yesterday. Would the GARP packets from VMs be filtered or acted upon at the vswitch level?

On Wed, Jul 25, 2012 at 1:17 PM, Ivan Pepelnjak <[email-masked]<mailto:[email-masked]>> wrote:
Truman, as I explained yesterday, the GARP sent after live VM move is actually generated (faked) by the hypervisor soft switch. It always happens regardless of what OS VM uses (unless you disable it).

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Truman Boyes
Sent: Wednesday, July 25, 2012 6:10 PM
To: Xuxiaohu
Cc: Joe Pelissier (jopeliss); [email-masked]<mailto:[email-masked]>; NAPIERALA, MARIA H
Subject: Re: [nvo3] ??: Role of ARP/RARP

On Wed, Jul 25, 2012 at 3:44 AM, Xuxiaohu <[email-masked]<mailto:[email-masked]>> wrote:
Hi Maria,

Best regards,
Xiaohu

Hi Xiaohu,

发件人: NAPIERALA, MARIA H [mailto:[email-masked]<mailto:[email-masked]>]
发送时间: 2012年7月25日 12:09
收件人: Xuxiaohu; Joe Pelissier (jopeliss); [email-masked]<mailto:[email-masked]>
主题: RE: [nvo3] Role of ARP/RARP

Xiaohu,

Maria

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Xuxiaohu
Sent: Tuesday, July 24, 2012 10:40 PM
To: Joe Pelissier (jopeliss); [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] Role of ARP/RARP

Hi Joe,

Best regards,
Xiaohu

My $0.02 worth…
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: Xuxiaohu <[email-masked]>
Date: Thu, 26 Jul 2012 03:54:00 +0000

Hi Maria,

Pls see my responses inline.

发送时间: 2012年7月25日 22:48

Xiaohu,

In any decent size data center the routing updates (in virtual networks) will need be propagated (and processed) via multiple control elements to reach remote PEs. Also, there will

[Xiaohu] Since today’s data center networks are becoming more and more flat (e.g., CLOS topology), the route update would not traverse too many routing nodes before reaching remote NVEs (no more than 3 nodes in most cases, IMHO).

be a lot of routing churn due to VM moves, including maintenance events where thousands of VMs can be simultaneously moved between server racks. The convergence of a control plane

[Xiaohu] Will the VM movement cause route churn or route flapping? In other words, will the route for a moved VM change many times during the VM migration process? If so, could you help explain the detailed reasons for such route flapping?

supporting such system will not be in sub-second (not to mention 10s or few 100s of milliseconds)..

[Xiaohu] Thousands of routes is not a big issue for a modern routing protocol (e.g., BGP) to achieve sub-second convergence performance indeed.

[Xiaohu] This doc (http://www.vmware.com/files/pdf/vmotion-perf-vsphere5.pdf) is about Vmotion performance in different environments and you can see that the downtime of VM live migration is about a few seconds, rather than a few millisecond. Hence, the sub-second convergence performance is good enough compared to the time period of stop-and-copy phase, IMHO.

Best regards,
Xiaohu

Maria

Hi Maria,

Best regards,

Xiaohu

Xiaohu,

Maria

Hi Maria,

Best regards,
Xiaohu

Xiaohu,

Maria

Hi Joe,

Best regards,
Xiaohu

My $0.02 worth…
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: "Ivan Pepelnjak" <[email-masked]>
Date: Thu, 26 Jul 2012 08:41:47 +0200

Xiaohu, the VMware paper you quote talks about performance degradation (of SQL sessions running over TCP sessions), not VM downtime. 

The graphs are clearly labeled with “Orders per second” and the only time they talk about disruption is “The duration of the switchover time, during which the application throughput was zero, was less than 1 second”, which is clearly useless for our discussion. 

From: [email-masked] [mailto:[email-masked]] On Behalf Of Xuxiaohu
Sent: Thursday, July 26, 2012 5:54 AM
To: NAPIERALA, MARIA H; Joe Pelissier (jopeliss); [email-masked]
Subject: Re: [nvo3] Role of ARP/RARP

Hi Maria,

Xiaohu, 

Best regards,

Xiaohu

Maria 

Hi Maria,

Best regards,
Xiaohu  

Xiaohu,

Maria

Hi Maria,

Best regards,

Xiaohu

Xiaohu,

 

Maria

Hi Joe,

Best regards,

Xiaohu

My $0.02 worth…

Joe Pelissier

Maria,

idea.

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,

Ivan

Thanks,
--David

Ivan

Truman

Cc: [email-masked]

My two cents. 

Linda Dunbar

Best regards,

Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Kireeti.

Thanks,
--David

 

-- 
Kireeti

---

From: Xuxiaohu <[email-masked]>
Date: Thu, 26 Jul 2012 07:58:13 +0000

The following may be more useful:

http://www.oracle.com/technetwork/server-storage/vm/ovm-sparc-livemigration-1522412.pdf (search for “SUSPENSION TIME”)
http://en.wikipedia.org/wiki/Live_migration (it said “The time between stopping VM on source and resuming it on destination is called "down-time". Down-time of a VM during a live migration could be a few millisecond to seconds according to the size of memory and applications running on the VM.”)

By the way, if the downtime needs to be reduced further, the network convergence can actually be performed as soon as the “stop-and-copy” phase starts (i.e., the target hypervisor could send a GARP as the beginning of the stop-and-copy phase), as long as the new VM doesn’t process the incoming traffic until the migration is completed. As such, Maris’s claim “This implies that the time that can be allocated to network convergence to the VM’s new location is limited” needs no long to be concerned.

Best regards,
Xiaohu

发件人: [email-masked] [mailto:[email-masked]] 代表 Ivan Pepelnjak
发送时间: 2012年7月26日 14:42
收件人: Xuxiaohu; 'NAPIERALA, MARIA H'; 'Joe Pelissier (jopeliss)'; [email-masked]
主题: Re: [nvo3] Role of ARP/RARP

Hi Maria,

Xiaohu,

Best regards,
Xiaohu

Maria

Hi Maria,

Best regards,

Xiaohu

Xiaohu,

Maria

Hi Maria,

Best regards,
Xiaohu

Xiaohu,

Maria

Hi Joe,

Best regards,
Xiaohu

My $0.02 worth…
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: Xuxiaohu <[email-masked]>
Date: Thu, 26 Jul 2012 09:29:36 +0000

Hi Truman,

Thanks for your comments and please see my response inline.

发件人: Truman Boyes [mailto:[email-masked]]
发送时间: 2012年7月26日 0:10
收件人: Xuxiaohu
抄送: NAPIERALA, MARIA H; Joe Pelissier (jopeliss); [email-masked]
主题: Re: [nvo3] 答复: Role of ARP/RARP

Best regards,
Xiaohu

Hi Xiaohu,

  *   The document discusses one of the motivations for layer 2 extensions is to avoid IP renumbering of servers (ie. VMs). I think the landscape is changing dramatically regarding renumbering of virtual machines; Cloud-based approaches are making addressing dynamic (DHCP assigned servers) and effectively abstracting service from servers. OpenStack, CloudStack, and other such technologies make renumbering of machines unnecessary. Essentially, when a VM needs to move to another location; a new instance is created in Site B, and traffic is "mapped" to the instance instead of dealing with complexities of network renumbering.
[Xiaohu] It seems we both agree that renumbering the VM being migrated should be avoid. Correct?

     *   The CE Host Discovery methods are prone to inefficiencies or failure. The size of the subnet would have a   significant impact on ARP or ICMP scans. LLDP would not exist on each VM OS; if this type of signaling occurred at the Hypervisor level it might be more feasible, but then VDP/LLDP might not be the best protocol for this purpose.
[Xiaohu] Your observation is reasonable. In fact, the ARP or ICMP scan is only needed in the case where the PE router is rebooted so that the PE router could obtain the information of its attached CE hosts (e.g., VMs) as soon as possible. Once the IP to MAC mappings of local CE hosts has been learnt by some means (e.g., receiving an ARP from a given local CE host, interaction with the orchestration system, or using VDP or LLDP), the PE router just needs to refresh them through unicast ARP before these ARP entries expire accordingly which is a normal behavior of most routers.

  *   In section 3.5 (CE Host Mobility), the gratuitous ARP packet generation will be OS dependent. In many implementations (Linux, some BSDs), the GARP packet is sent when the interface transitions or addressing changes on the interface. In order to accommodate this, we would need to have guest VMs perform this action, which may in turn cause application interference.
[Xiaohu] As Ivan has points out, such GARP is sent by the target hypervisor, rather than the guest VM, after the VM migration.

  *   In section 3.6.2, section 2, there should be a consideration on routing propagation impacts as there is a potential impact on security and network resilience. Effectively, there is now a linkage between host ARP requests, FIB updates, MP-BGP update messages, and Route Reflectors, and control plan processing on PEs.  Let's say for example that a guest VM that belongs to 192.168/16 sends ARP requests to all 65k addresses; would this have a significant impact on the control plane of numerous PE and MP-BGP speakers?
[Xiaohu] In the example you made, the PE router only needs to learn one ARP entry (i.e., the ARP entry of the ARP requesting host). Hence, it would not cause any significant impact on the control plane.
Best regards,
Xiaohu
 Kind regards,
Truman

Xiaohu,

Maria

Hi Joe,

Best regards,
Xiaohu

My $0.02 worth…
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Thu, 26 Jul 2012 13:32:43 +0000

Xiaohu,

This is about convergence of the control plane of the virtual networks, not the convergence of the transport network.

Maria

From: Xuxiaohu [mailto:[email-masked]]
Sent: Wednesday, July 25, 2012 11:54 PM
To: NAPIERALA, MARIA H; Joe Pelissier (jopeliss); [email-masked]
Subject: re: [nvo3] Role of ARP/RARP

Hi Maria,

Xiaohu,

Best regards,
Xiaohu

Maria

Hi Maria,

Best regards,

Xiaohu

Xiaohu,

Maria

Hi Maria,

Best regards,
Xiaohu

Xiaohu,

Maria

Hi Joe,

Best regards,
Xiaohu

My $0.02 worth…
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Thu, 26 Jul 2012 14:08:13 +0000

David,

> Where I would expect economy of MACinIP over IPinIP is the default gateway as a single entry for everything outside
> the subnet vs. flat operation

It was true in traditional data centers but in the virtualized data centers the concept of L2/L3 boundary not only loses its value (because a VM can be located/moved anywhere in the network) but causes VM traffic to cross the core network at least twice which would not be acceptable.

Maria

From: David Allan I [mailto:[email-masked]]
Sent: Wednesday, July 25, 2012 12:21 PM
To: NAPIERALA, MARIA H; Luyuan Fang (lufang); Lucy yong; Ivan Pepelnjak; [email-masked]; [email-masked]
Subject: RE: [nvo3] Role of ARP/RARP

cheers
Dave

Maria

From: David Allan I [mailto:[email-masked]]
Sent: Wednesday, July 25, 2012 11:40 AM
To: NAPIERALA, MARIA H; Luyuan Fang (lufang); Lucy yong; Ivan Pepelnjak; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: RE: [nvo3] Role of ARP/RARP

cheers
Dave

Maria

cheers
Dave

Luyuan

Lucy

You nailed it.
Luyuan

Best,
Ivan

Ivan,

Thanks,
--David

Joe,

Am I missing something?
Ivan

Hi David,

Joe Pelissier

Joe,

Thanks,
--David

My $0.02 worth...
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: Truman Boyes <[email-masked]>
Date: Thu, 26 Jul 2012 10:15:27 -0400

Hi Xiaohu, Thanks for your clarifications and response. Just a few comments
on  your comments, inline ;)

On Thu, Jul 26, 2012 at 5:29 AM, Xuxiaohu <[email-masked]> wrote:

>  Hi Truman,****
>
> ** **
>
> Thanks for your comments and please see my response inline.****
>
> ** **
>
> *发件人:* Truman Boyes [mailto:[email-masked]]
> *发送时间:* 2012年7月26日 0:10
> *收件人:* Xuxiaohu
> *抄送:* NAPIERALA, MARIA H; Joe Pelissier (jopeliss); [email-masked]
> *主题:* Re: [nvo3] 答复: Role of ARP/RARP****
>
> ** **
>

[Truman] Correct, we agree.

> [Xiaohu] Your observation is reasonable. In fact, the ARP or ICMP scan is
> only needed in the case where the PE router is rebooted so that the PE
> router could obtain the information of its attached CE hosts (e.g., VMs) as
> soon as possible. Once the IP to MAC mappings of local CE hosts has been
> learnt by some means (e.g., receiving an ARP from a given local CE host,
> interaction with the orchestration system, or using VDP or LLDP), the PE
> router just needs to refresh them through unicast ARP before these ARP
> entries expire accordingly which is a normal behavior of most routers.
>

[Truman] This issue surrounding rebooting might not be a significant
problem in implementation. Take for example large scale broadband networks
where BNG devices need to keep track of DHCP leases. They preserve the
leases in a file that is state synchronized between control plane/routing
engines. The same could happen for ARP cache.

****
> [Xiaohu] As Ivan has points out, such GARP is sent by the target
> hypervisor, rather than the guest VM, after the VM migration.****
>
> [Xiaohu] In the example you made, the PE router only needs to learn one
> ARP entry (i.e., the ARP entry of the ARP requesting host). Hence, it would
> not cause any significant impact on the control plane.****
>
>
[Truman] Makes sense. But what about a host that spoofs MAC addresses in
an iterative stepping process. ie. SRC-MAC++ sends ARP request? I suppose
the standard VPLS/L3VPN controls would need to be in place to protect the
control plane (ie. max prefixes, or size of L2 table).

> *发件人:* [email-masked] [mailto:[email-masked]<[email-masked]>]
> *代表 *Joe Pelissier (jopeliss)
> *发送时间:* 2012年7月25日 1:22
> *收件人:* [email-masked]
> *主题:* Re: [nvo3] Role of ARP/RARP****
>
>  ****
>

---

From: David Allan I <[email-masked]>
Date: Thu, 26 Jul 2012 14:23:06 -0400

Can you provide an example of why you think this is true...?

I have to admit I'm not seeing what you describe as a mandatory consequence...

thanks
Dave

Sent: Thursday, July 26, 2012 7:08 AM

David,

Maria

cheers
Dave

Maria

cheers
Dave

Maria

cheers
Dave

Luyuan

Lucy

You nailed it.
Luyuan

Best,
Ivan

Ivan,

Thanks,
--David

Joe,

Am I missing something?
Ivan

Hi David,

Joe Pelissier

Joe,

Thanks,
--David

My $0.02 worth...
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Thu, 26 Jul 2012 20:06:27 +0000

Building a large scale data center with an underlying assumption that inter-subnet traffic will need to traverse data center fabric multiple times is a non-starter due to fabric cost.
It might be acceptable in small data center environment but this is not the one I am addressing.

If desired/appropriate a concept of a "gateway" supporting route aggregation can be easily introduced in a L3VPN system. As I mentioned before L3VPN network can be configured to suppress more specific routes than a specified aggregate by advertising the aggregate from one or more "gateway" PEs where the more specific routes are present.
L3VPN technology allows for creating various topologies by manipulating VRF import and export configuration, including "hub-and-spoke" topologies and overlapping VPNs.

Maria

From: David Allan I [mailto:[email-masked]]
Sent: Thursday, July 26, 2012 2:23 PM
To: NAPIERALA, MARIA H; Luyuan Fang (lufang); Lucy yong; Ivan Pepelnjak; [email-masked]; [email-masked]
Subject: RE: [nvo3] Role of ARP/RARP

thanks
Dave

Maria

cheers
Dave

Maria

cheers
Dave

Maria

cheers
Dave

Luyuan

Lucy

You nailed it.
Luyuan

Best,
Ivan

Ivan,

Thanks,
--David

Joe,

Am I missing something?
Ivan

Hi David,

Joe Pelissier

Joe,

Thanks,
--David

My $0.02 worth...
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: David Allan I <[email-masked]>
Date: Thu, 26 Jul 2012 16:22:39 -0400

HI Maria:

You were misunderstanding my question. That inter-subnet traffic needs to traverse a DC fabric multiple times would be an artifact of how it the fabric is architected, not a mandatory feature. I 'm looking to understand WHY you think it has to be this way, not the economic consequences of assuming it MUST be this way.

cheers
Dave

Sent: Thursday, July 26, 2012 1:06 PM

Maria

thanks
Dave

Maria

cheers
Dave

Maria

cheers
Dave

Maria

cheers
Dave

Luyuan

Lucy

You nailed it.
Luyuan

Best,
Ivan

Ivan,

Thanks,
--David

Joe,

Am I missing something?
Ivan

Hi David,

Joe Pelissier

Joe,

Thanks,
--David

My $0.02 worth...
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: Xuxiaohu <[email-masked]>
Date: Fri, 27 Jul 2012 01:00:36 +0000

Hi Truman,

Please see my reply in line.

发件人: Truman Boyes [mailto:[email-masked]]
发送时间: 2012年7月26日 22:15
收件人: Xuxiaohu
抄送: NAPIERALA, MARIA H; Joe Pelissier (jopeliss); [email-masked]
主题: Re: 答复: [nvo3] 答复: Role of ARP/RARP

Hi Xiaohu, Thanks for your clarifications and response. Just a few comments on  your comments, inline ;)
On Thu, Jul 26, 2012 at 5:29 AM, Xuxiaohu <[email-masked]<mailto:[email-masked]>> wrote:
Hi Truman,

发件人: Truman Boyes [mailto:[email-masked]<mailto:[email-masked]>]
发送时间: 2012年7月26日 0:10
收件人: Xuxiaohu
抄送: NAPIERALA, MARIA H; Joe Pelissier (jopeliss); [email-masked]<mailto:[email-masked]>
主题: Re: [nvo3] 答复: Role of ARP/RARP

Best regards,
Xiaohu

Hi Xiaohu,

[Truman] This issue surrounding rebooting might not be a significant problem in implementation. Take for example large scale broadband networks where BNG devices need to keep track of DHCP leases. They preserve the leases in a file that is state synchronized between control plane/routing engines. The same could happen for ARP cache.

[Xiaohu] That’s a good idea. After rebooting up, the PE router could verify the availability of each recorded local CE host by sending a unicast ARP or ICMP. Thus, there is no need to perform ARP scan on these hosts.

[Truman] Makes sense. But what about a host that spoofs MAC addresses in an iterative stepping process. ie. SRC-MAC++ sends ARP request? I suppose the standard VPLS/L3VPN controls would need to be in place to protect the control plane (ie. max prefixes, or size of L2 table).

[Xiaohu] You are right.

Best regards,
Xiaohu
Best regards,
Xiaohu
 Kind regards,
Truman

Xiaohu,

Maria

Hi Joe,

Best regards,
Xiaohu

My $0.02 worth…
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: Xuxiaohu <[email-masked]>
Date: Fri, 27 Jul 2012 02:49:49 +0000

Hi Maria,

Did you mean the convergence of the underlying network by saying the convergence of the transport network? If so, do you believe the VM migration event (i.e., VM attachment/detachment) would impacts the convergence of the underlying network?

Best regards,
Xiaohu

发送时间: 2012年7月26日 21:33

Xiaohu,

Maria

Hi Maria,

Xiaohu,

Best regards,
Xiaohu

Maria

Hi Maria,

Best regards,

Xiaohu

Xiaohu,

Maria

Hi Maria,

Best regards,
Xiaohu

Xiaohu,

Maria

Hi Joe,

Best regards,
Xiaohu

My $0.02 worth…
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Fri, 27 Jul 2012 03:44:20 +0000

David,

This is a wrong question to ask since the problem of path optimality has to be solved by the overlay not by the underlay network.

Maria

From: David Allan I [mailto:[email-masked]]
Sent: Thursday, July 26, 2012 4:23 PM
To: NAPIERALA, MARIA H; Luyuan Fang (lufang); Lucy yong; Ivan Pepelnjak; [email-masked]; [email-masked]
Subject: RE: [nvo3] Role of ARP/RARP

HI Maria:

cheers
Dave

Maria

thanks
Dave

Maria

cheers
Dave

Maria

cheers
Dave

Maria

cheers
Dave

Luyuan

Lucy

You nailed it.
Luyuan

Best,
Ivan

Ivan,

Thanks,
--David

Joe,

Am I missing something?
Ivan

Hi David,

Joe Pelissier

Joe,

Thanks,
--David

My $0.02 worth...
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Fri, 27 Jul 2012 03:51:41 +0000

>Did you mean the convergence of the underlying network by saying the convergence of the transport network?

Yes, Underlay Network = Transport Network

>If so, do you believe the VM migration event (i.e., VM attachment/detachment) would impacts the convergence of the underlying network?

Underlay/transport network must be decoupled/unaware of the virtual networks (which includes VM moves).

Maria

From: [email-masked] [mailto:[email-masked]] On Behalf Of Xuxiaohu
Sent: Thursday, July 26, 2012 10:50 PM
To: NAPIERALA, MARIA H; Joe Pelissier (jopeliss); [email-masked]
Subject: [nvo3] 答复: Role of ARP/RARP

Hi Maria,

Best regards,
Xiaohu

Xiaohu,

Maria

Hi Maria,

Xiaohu,

Best regards,
Xiaohu

Maria

Hi Maria,

Best regards,

Xiaohu

Xiaohu,

Maria

Hi Maria,

Best regards,
Xiaohu

Xiaohu,

Maria

Hi Joe,

Best regards,
Xiaohu

My $0.02 worth…
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: Xuxiaohu <[email-masked]>
Date: Fri, 27 Jul 2012 04:46:06 +0000

Since you know that Underlay/transport network must be decoupled/unaware of the virtual networks (which includes VM moves), what did you mean by saying “This is about convergence of the control plane of the virtual networks, not the convergence of the transport network.”

Xiaohu

发件人: NAPIERALA, MARIA H [mailto:[email-masked]]
发送时间: 2012年7月27日 11:52
收件人: Xuxiaohu; Joe Pelissier (jopeliss); [email-masked]
主题: RE: [nvo3] 答复: Role of ARP/RARP

Maria

Hi Maria,

Best regards,
Xiaohu

Xiaohu,

Maria

Hi Maria,

Xiaohu,

Best regards,
Xiaohu

Maria

Hi Maria,

Best regards,

Xiaohu

Xiaohu,

Maria

Hi Maria,

Best regards,
Xiaohu

Xiaohu,

Maria

Hi Joe,

Best regards,
Xiaohu

My $0.02 worth…
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Fri, 27 Jul 2012 04:49:42 +0000

You lost me there..

From: Xuxiaohu [mailto:[email-masked]]
Sent: Friday, July 27, 2012 12:46 AM
To: NAPIERALA, MARIA H; Joe Pelissier (jopeliss); [email-masked]
Subject: 答复: [nvo3] 答复: Role of ARP/RARP

Xiaohu

Maria

Hi Maria,

Best regards,
Xiaohu

Xiaohu,

Maria

Hi Maria,

Xiaohu,

Best regards,
Xiaohu

Maria

Hi Maria,

Best regards,

Xiaohu

Xiaohu,

Maria

Hi Maria,

Best regards,
Xiaohu

Xiaohu,

Maria

Hi Joe,

Best regards,
Xiaohu

My $0.02 worth…
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: Xuxiaohu <[email-masked]>
Date: Fri, 27 Jul 2012 05:02:30 +0000

Sorry for confusing you. Let me give a concrete example: In the typical Two-tier data center network topology, the ToR switches are configured as PE routers while the core switches are configured as RRs, the L3VPN BGP route updates would not be processed by more than three BGP nodes. Hence, I believe the process of L3VPN route update propagation and processing would not take too much time and therefore it could easily achieve sub-second convergence performance.

Best regards,
Xiaohu

发送时间: 2012年7月27日 12:50

You lost me there..

Xiaohu

Maria

Hi Maria,

Best regards,
Xiaohu

Xiaohu,

Maria

Hi Maria,

Xiaohu,

Best regards,
Xiaohu

Maria

Hi Maria,

Best regards,

Xiaohu

Xiaohu,

Maria

Hi Maria,

Best regards,
Xiaohu

Xiaohu,

Maria

Hi Joe,

Best regards,
Xiaohu

My $0.02 worth…
Joe Pelissier

Maria,

Thanks,
--David

:-)

Maria

Ivan,

Thanks,
--David

Best,
Ivan

Thanks,
--David

Ivan

Truman

My two cents.

Linda Dunbar

Best regards,
Xiaohu

<skipped>

Thanks,
--David

Hi David,

Thanks!

More on this below.

--- Detailed comments by section ---

-- Section 2.2.2

Good point.  Will fix.

-- Section 2.2.3

See separate email.

-- Section 2.2.4

Not exactly ;-).

Okay, something like.

-- Section 3

Thanks,
--David

--
Kireeti

---

From: Paul Unbehagen <[email-masked]>
Date: Fri, 27 Jul 2012 09:51:45 -0600

Many web apps require cross VM communications, eg Web server to App server to DB server back to App server back to web server then finally back to user browser. Thus cross fabric flows are typical in many applications

Size, scale, and design may affect this a bit, but that's a general app flow commonality that exists. This is why interVLAN routing is used heavily in DC's of many different sizes. 

--
Paul Unbehagen

On Jul 26, 2012, at 2:22 PM, David Allan I <[email-masked]> wrote:

> If desired/appropriate a concept of a “gateway” supporting route aggregation can be easily introduced in a L3VPN system. As I mentioned before L3VPN network can be configured to suppress more specific routes than a specified aggregate by advertising the aggregate from one or more “gateway” PEs where the more specific routes are present.
> When referring to millions of routes I am referring to control plane “capacity” (which is outside of the hypervisor, or TOR if you like) and not to a single hypervisor forwarding capacity. A hypervisor will only need to store forwarding tables of VPNs that have local attachments.

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 27 Jul 2012 17:59:29 +0200

Paul,

 > Many web apps require cross VM communications, eg Web server to App
 > server to DB server back to App server back to web server then
 > finally back to user browser. Thus cross fabric flows are typical in
 > many applications

Absolutely.

That's why avoiding creation of VLANs in the first place in any part of 
the DC network where East-West traffic is of non negligible amount is 
highly recommended.

L3VPN over no service aware pure IP transport works very nicely and 
addresses the above application model pretty well.

Best,
R.

> Many web apps require cross VM communications, eg Web server to App
> server to DB server back to App server back to web server then finally
> back to user browser. Thus cross fabric flows are typical in many
> applications
>
> Size, scale, and design may affect this a bit, but that's a general app
> flow commonality that exists. This is why interVLAN routing is used
> heavily in DC's of many different sizes.
>
> --
> Paul Unbehagen

---

From: Truman Boyes <[email-masked]>
Date: Fri, 27 Jul 2012 12:06:16 -0400

On Fri, Jul 27, 2012 at 11:51 AM, Paul Unbehagen <[email-masked]> wrote:

> Many web apps require cross VM communications, eg Web server to App server
> to DB server back to App server back to web server then finally back to
> user browser. Thus cross fabric flows are typical in many applications
>
> flow commonality that exists. This is why interVLAN routing is used heavily
> in DC's of many different sizes.
>
> --
> Paul Unbehagen
>

Hi Paul, absolutely this is how most SOA apps are designed. This type of
app flow should not require interVLAN connectivity but unfortunately a
model of tiers has been developed. Security models should not force us to
use VLANs.

Truman

> You were misunderstanding my question. That inter-subnet traffic needs to
> traverse a DC fabric multiple times would be an artifact of how
> it the fabric is architected, not a mandatory feature. I 'm looking to
> understand *WHY* you think it has to be this way, not the economic
> consequences of assuming it *MUST* be this way.
>
> cheers
> Dave
>
>  ------------------------------
> *From:* NAPIERALA, MARIA H [mailto:[email-masked]]
> *Sent:* Thursday, July 26, 2012 1:06 PM
> *To:* David Allan I; Luyuan Fang (lufang); Lucy yong; Ivan Pepelnjak;
> [email-masked]; [email-masked]
> *Subject:* RE: [nvo3] Role of ARP/RARP
>
>  Building a large scale data center with an underlying assumption that
> inter-subnet traffic will need to traverse data center fabric multiple
> times is a non-starter due to fabric cost.****
>
> It might be acceptable in small data center environment but this is not
> the one I am addressing.****
>
> ** **
>
> If desired/appropriate a concept of a “gateway” supporting route
> aggregation can be easily introduced in a L3VPN system. As I mentioned
> before L3VPN network can be configured to suppress more specific routes
> than a specified aggregate by advertising the aggregate from one or more
> “gateway” PEs where the more specific routes are present.****
>
> L3VPN technology allows for creating various topologies by manipulating
> VRF import and export configuration, including "hub-and-spoke" topologies
> and overlapping VPNs.****
>
> ** **
>
> Maria****
>
> ** **
>
> ** **
>
> *From:* David Allan I [mailto:[email-masked]]
> *Sent:* Thursday, July 26, 2012 2:23 PM
> *To:* NAPIERALA, MARIA H; Luyuan Fang (lufang); Lucy yong; Ivan
> Pepelnjak; [email-masked]; [email-masked]
> Can you provide an example of why you think this is true...?****
>
>  ****
>
> I have to admit I'm not seeing what you describe as a mandatory
> consequence...****
>
>  ****
>
> thanks****
>
> Dave****
>
> ** **
>  ------------------------------
>
> *Sent:* Thursday, July 26, 2012 7:08 AM
> *To:* David Allan I; Luyuan Fang (lufang); Lucy yong; Ivan Pepelnjak;
> [email-masked]; [email-masked]
> David,****
> > Where I would expect economy of MACinIP over IPinIP is the default
> gateway as a single entry for everything outside ****
>
> > the subnet vs. flat operation ****
>
> ** **
>
> It was true in traditional data centers but in the virtualized data centers the concept of L2/L3 boundary not only loses its value (because a VM can be located/moved anywhere in the network) but causes VM traffic to cross the core network at least twice which would not be acceptable. ****
>
> ** **
>
> Maria****
>
> ** **
>
> ** **
>
> *From:* David Allan I [mailto:[email-masked]<[email-masked]>]
>
> *Sent:* Wednesday, July 25, 2012 12:21 PM
> *To:* NAPIERALA, MARIA H; Luyuan Fang (lufang); Lucy yong; Ivan
> Pepelnjak; [email-masked]; [email-masked]
> I would agree that we effective have a 48+32 bit mapping to server IP vs.
> simply a 32 bit mapping to server IP in the IPv4 case. ****
>
>  ****
>
> Where I would expect economy of MACinIP over IPinIP is the default gateway
> as a single entry for everything outside the subnet vs. flat operation so
> fewer but for all intents and purposes bigger entries for MACinIP vs. *
> more* entries for IPinIP. ****
>
>  ****
>
> Where the breakpoint would be would depend on the scale we were working to
> and the design of virtualization of the networking to interconnect the VMs.
> ****
>
>  ****
>
> cheers****
>
> Dave****
>
>  ****
>
>  ****
>
> ** **
>  ------------------------------
>
> *Sent:* Wednesday, July 25, 2012 8:48 AM
> *To:* David Allan I; Luyuan Fang (lufang); Lucy yong; Ivan Pepelnjak;
> [email-masked]; [email-masked]
> David,****
> When referring to millions of routes I am referring to control plane
> “capacity” (which is outside of the hypervisor, or TOR if you like) and not
> to a single hypervisor forwarding capacity. A hypervisor will only need to
> store forwarding tables of VPNs that have local attachments.****
>
> IPinIP solutions do not have to maintain MAC-to-IP mappings across a
> virtual network which is a big scaling advantage.****
>
> ** **
>
> Maria****
>
> ** **
>
> *From:* David Allan I [mailto:[email-masked]<[email-masked]>]
>
> *Sent:* Wednesday, July 25, 2012 11:40 AM
> *To:* NAPIERALA, MARIA H; Luyuan Fang (lufang); Lucy yong; Ivan
> Pepelnjak; [email-masked]; [email-masked]
> I agree with your points at a 50000 foot level. ****
>
>  ****
>
> But you are contrasting the capabilities of a portion of a silicon chip in
> a server with some multi-terabit switching platform located in the WAN. I
> would expect it to scale better.****
>
>  ****
>
> IMO at the end of the day, the approximate amount of state in a hypervisor
> for MACinIP or IPinIP will be a wash. And the question is simply "how
> complex is the system that populates that state?"****
>
>  ****
>
> cheers****
>
> Dave****
>
> ** **
>  ------------------------------
>
> *Sent:* Wednesday, July 25, 2012 12:11 AM
> *To:* David Allan I; Luyuan Fang (lufang); Lucy yong; Ivan Pepelnjak;
> [email-masked]; [email-masked]
> David,****
> When virtualization is done correctly, the data center transport network
> is unaware of VPNs/VLANs. I think this is what you are assuming also. ****
>
> In L3VPN approach to data center, the VM route distribution to NVEs
> (which, ideally, should be the hypervisors) will be constrained based on
> the VPN membership. Today, there are WAN deployments of MPLS/BGP VPNs
> supporting tens of thousands of VPNs and several (7+) millions of routes.
>                             ****
>
> A L3VPN system can be configured to suppress more specific routes than a
> specified aggregate by advertising the aggregate from a specific PE where
> the more specific routes are present.****
>
> ** **
>
> Maria****
>
> ** **
>
> *On Behalf Of *David Allan I
> *Sent:* Tuesday, July 24, 2012 7:11 PM
> *To:* Luyuan Fang (lufang); Lucy yong; Ivan Pepelnjak; [email-masked];
> [email-masked]
> Lets not throw the baby out with the bathwater.****
>
>  ****
>
> The current IPv4 address is a combination of both location and identity
> and we have a need to, at least a local level, render the location part
> irrelevant. The location limitations can be at least partially circumvented
> with some underlying level of indirection. If you try to work around this
> without it, you end up with a flat unaggregatable set of host routes as
> useful geographic or virtualized clustering of all members in a prefix is
> not possible. ****
>
>  ****
>
> IP scales, partially because of the notion of subnet, which is an enabler
> (not the only one) of aggregatable addresses. The VLAN as a virtual subnet
> also has the property of providing that useful level of indirection and
> geographic independence. The limitation is the practical size of the
> community that can be served by any single virtualized broadcast domain. *
> ***
>
>  ****
>
> So I get a bit nervous when "IP scales" starts to sound like a mantra. The
> statement should not be "IP scales" but "we need to use IP in a way that
> does not undermine its scaling properties" and frankly that will involve
> bridging IMHO.****
>
>  ****
>
> cheers****
>
> Dave****
>
> ** **
>  ------------------------------
>
> *On Behalf Of *Luyuan Fang (lufang)
> *Sent:* Tuesday, July 24, 2012 2:51 PM
> *To:* Lucy yong; Ivan Pepelnjak; [email-masked]; [email-masked]
> The point we were making was not bridging does not have place. In fact, it
> has and it will, for long time if not forever. We are talking about
> scaling, including operational simplicity.****
>
> A lot of places may not need to scale huge in reality, either way would
> work, if they like to stay the way it is, fine.****
>
> A few others need huge scale, they may not need to worry about backward
> compatibility or follow the old path, especially when building green field.
> ****
>
> We have different sets of operators, the goals/requirements/solutions
> would not be the same, I think Pedro pointed this out a while ago.****
>
> ** **
>
> Luyuan****
>
> ** **
>
> *From:* Lucy yong [mailto:[email-masked] <[email-masked]>]
> *Sent:* Tuesday, July 24, 2012 5:16 PM
> *To:* Luyuan Fang (lufang); Ivan Pepelnjak; [email-masked];
> [email-masked]
> > bridging doesn’t scale, IP does****
>
> ** **
>
> This is misleading comment for this discussion. IP has its usage place,
> bridging has too. ****
>
> I am not saying we should use the bridging at virtual access. But the
> bridging at virtual access does not have the scalability issue.****
>
> ** **
>
> Lucy****
>
> ** **
>
> *On Behalf Of *Luyuan Fang (lufang)
> *Sent:* Tuesday, July 24, 2012 2:27 PM
> *To:* Ivan Pepelnjak; [email-masked]; [email-masked]
> > bridging doesn’t scale, IP does****
>
> You nailed it.****
>
> Luyuan****
>
> ** **
>
> *Sent:* Tuesday, July 24, 2012 3:02 PM
> *To:* [email-masked]; [email-masked]
> So glad to hear we’re past end-to-end L2 flooding.****
>
> ** **
>
> I’m probably missing something else at this moment (physical device
> integration comes to mind), but the only reason I see why we’re discussing
> ARP, VLANs (and potentially VDP) today is the idea to retain access network
> (including hypervisor switches) intact in the dumb L2-only stage and
> implement some complex mechanism in ToR switch (or whatever other device
> acts as NVO3 PE-router) to cope with the existing signaling mechanisms
> (gratuitous ARP) used by hypervisor switches.****
>
> ** **
>
> If that’s the case, we’re missing the boat. All the hypervisor vendors got
> the “bridging doesn’t scale, IP does” message loud and clear a year ago.
> VMware has VXLAN (currently with Nexus 1000V, but very probably they won’t
> stay there, and they bought Nicira for a reason), Hyper-V will have NVGRE
> with Windows Server 2012 and Xen/KVM have GRE (or CAPWAP and eventually
> VXLAN)  support through OVS. Let’s acknowledge that reality and move on.**
> **
>
> ** **
>
> Best,****
>
> Ivan****
>
> ** **
>
> *From:* [email-masked] [mailto:[email-masked]<[email-masked]>]
>
> *Sent:* Tuesday, July 24, 2012 8:47 PM
> *To:* [email-masked]; [email-masked]
> You’ve missed an important twist:****
>
> ** **
>
> > However, the use of ARP (or any other L2-broadcast-based mechanism) to
> propagate the MAC****
>
> > forwarding across the L2 virtual network has a severe scalability
> limitation – it relies on flooding. ****
>
> ** **
>
> The idea is to have the NVE intercept the ARP and something more scalable
> than flooding to get the job done.  There are multiple -nvo3- drafts that
> envision this sort of approach to replace flooding at scale.****
>
> ** **
>
> Thanks,
> --David****
>
> ** **
>
> *Sent:* Tuesday, July 24, 2012 2:43 PM
> *To:* 'Joe Pelissier (jopeliss)'; [email-masked]
> Joe,****
> Thanks for a very concise explanation – I’m probably too focused on the
> hypervisor-based solutions and overlooked the *associate action might
> take a while *reality.****
>
> ** **
>
> However, the use of ARP (or any other L2-broadcast-based mechanism) to
> propagate the MAC forwarding across the L2 virtual network has a severe
> scalability limitation – it relies on flooding. ****
>
> ** **
>
> Flooding can be implemented with head-end replication, in which case we
> have to assume the NVE knows L3 addresses of all NVEs participating in the
> same virtual network. If that information is already available in NVE, we
> don’t need ARP but could use any out-of-band signaling mechanism.
> Alternatively, a central system (orchestration tool) could propagate that
> information using a routing protocol or a FIB update protocol (ex:
> OpenFlow).****
>
> ** **
>
> Core-network-based flooding like today’s VLAN-based virtual networks is
> either too coarse (resulting in wasted bandwidth and NVE resources), or
> requires per-tenant state in the core (example: IP multicast-based flooding
> in VXLAN). Both approaches have scalability problems once the data center
> becomes big enough.****
>
> ** **
>
> Am I missing something?****
>
> Ivan****
>
> ** **
>
> *On Behalf Of *Joe Pelissier (jopeliss)
> *Sent:* Tuesday, July 24, 2012 7:58 PM
> *To:* [email-masked]
> Just to clarify, in the VDP specification, there are two types of
> pre-associate: with and without resource reservation.  What you are
> referring to is the pre-associate with resource reservation.  However, it
> does not operate quite the way you describe.****
>
> ** **
>
> A pre-associate with resource reservation essentially ensures that the
> switch can perform a future associate with the same parameters (or
> indicates a failure).  It reserves the resources needed so that a future
> associate with the same parameters will be able to complete, but it does
> not necessarily configure those parameters into the switch; i.e., a
> pre-associate has no impact on the currently running configuration.  An
> associate actually configures the parameters so that when the new VM
> becomes active, everything is ready to go.  This is the part that can be a
> bit slow, so we want it to be completed before the VM comes alive in the
> new location.  However, we can’t update forwarding at this point since the
> VM is still active in the old location.  Forwarding gets updated through
> the normal learning process which ensures that forwarding is not impacted
> until the VM is alive in the new location.****
>
> ** **
>
> If associate updated the forwarding in the local switch, then we would be
> forced to do the associate during the period that the VM is suspended which
> increases the time that the VM must be suspended.  This is undesirable.  By
> separating the associate from the learning, the associate is removed from
> the critical path.****
>
> ** **
>
> Even if we used the associate to update the forwarding, it is
> insufficient.  It would only update the forwarding in the local switch, not
> the entire L2 LAN.  Therefore, the ARP/RARP/ND is still highly desirable.
> Since we need the ARP/RARP/ND anyway for the rest of the L2 LAN, it might
> as well be used to update forwarding in the local switch also, with the
> benefit of moving associate out of the critical path.****
>
> ** **
>
> Joe Pelissier****
>
> ** **
>
> *From:* [email-masked] [mailto:[email-masked]<[email-masked]>]
>
> *Sent:* Tuesday, July 24, 2012 10:26 AM
> Joe,****
> My assumption behind this has been that **if** all the network parameters
> can be set up in advance using the pre-associate, then ARP/RARP/ND can
> serve as the indication to put them into service and update the forwarding.
> ****
>
> ** **
>
> We’re in clear agreement that both operations (set up parameters, activate
> forwarding) are needed, and I had been using “pre-associate” as the term
> used in the VDP spec for the in-advance operation.****
>
> ** **
>
> Thanks,
> --David****
>
> ** **
>
> *On Behalf Of *Joe Pelissier (jopeliss)
> *Sent:* Tuesday, July 24, 2012 1:22 PM
> *To:* [email-masked]
> +1 (508) 293-7953 <%2B1%20%28508%29%20293-7953>             FAX: +1 (508)
> 293-7786 <%2B1%20%28508%29%20293-7786>
> [email-masked]        Mobile: +1 (978) 394-7754<%2B1%20%28978%29%20394-7754>

---

From: "Ivan Pepelnjak" <[email-masked]>
Date: Fri, 27 Jul 2012 18:06:52 +0200

Usually one would have a firewall between Web->App and App->DB server or it makes no sense to have them in different subnets.

With a firewall between app stack layers, L3VPN buys you almost nothing - the traffic has to go through the firewall anyway. 

The other option is a hypervisor-embedded firewall sitting in front of a VM - yet again, having VMs in different subnets is a cosmetic exercise.

Alternatively, one could install per-flow shortcut entries after a flow is inspected by a traditional firewall (ex: Cisco VSG, although it's not an inter-subnet FW), for example through OpenFlow, but that's (in my opinion) way beyond simple L3VPN.

Ivan

> From: Robert Raszuk [mailto:[email-masked]]
> Sent: Friday, July 27, 2012 5:59 PM
> To: Paul Unbehagen
> Cc: David Allan I; Ivan Pepelnjak; [email-masked]; [email-masked];
> Lucy yong; NAPIERALA, MARIA H; Luyuan Fang (lufang)
>  > Many web apps require cross VM communications, eg Web server to App  >
> server to DB server back to App server back to web server then  > finally
> back to user browser. Thus cross fabric flows are typical in  > many
> applications
> 
> Absolutely.
> 
> > Size, scale, and design may affect this a bit, but that's a general
> > app flow commonality that exists. This is why interVLAN routing is
> > used heavily in DC's of many different sizes.
> >
> > --
> > Paul Unbehagen

---

From: Paul Unbehagen <[email-masked]>
Date: Fri, 27 Jul 2012 10:10:44 -0600

Not sure you can avoid it completely, nor do I know that everyone else in the industry wants to avoid it either.   As avoiding VLANs doesn't address the LANs needed to support clusters of servers acting as any single tier of the application model described below.  Web clusters, app clusters, etc...

It's a fine balance that is flexible today because we can use the VLANs and IP subnets appropriately depending on the need and scale.  We should be sure not to box ourselves into a one size fits all mentality. InterVLAN routing inside a VRF is very appropriate for many DC's for example and used today. 

--
Paul Unbehagen

Sent from my iPad

On Jul 27, 2012, at 9:59 AM, Robert Raszuk <[email-masked]> wrote:

> That's why avoiding creation of VLANs in the first place in any part of the DC network where East-West traffic is of non negligible amount is highly recommended.
> 
> L3VPN over no service aware pure IP transport works very nicely and addresses the above application model pretty well.
> 
> Best,
> R.
>

---

From: Paul Unbehagen <[email-masked]>
Date: Fri, 27 Jul 2012 10:15:38 -0600

Not sure I see why security models should preclude this as one of the deployment models. Especially since its used heavily today. 

--
Paul Unbehagen

Sent from my iPad

On Jul 27, 2012, at 10:06 AM, Truman Boyes <[email-masked]> wrote:

> Hi Paul, absolutely this is how most SOA apps are designed. This type of app flow should not require interVLAN connectivity but unfortunately a model of tiers has been developed. Security models should not force us to use VLANs. 
> 
> Truman 
>  
> 
> 
>

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 27 Jul 2012 18:21:34 +0200

Hi Ivan,

 > With a firewall between app stack layers, L3VPN buys you almost
 > nothing - the traffic has to go through the firewall anyway.

Completely disagree.

L3VPN I have in mind is really not "traditional" L3VPN - it is just 
mechanism we could be reusing.

So if the tenant requires firewall between his subnets you simply model 
such tenant as two L3VPNs and with two lines of configuration 
interconnect such two VPNs via firewall appliance sitting _anywhere_ in 
the DC fabric.

That's the point of reusing L3VPN technology in DC .. not to provide WAN 
like analogy that customer get's single VPN. (However even there as you 
very well know it is pretty trivial to create arbitrary topologies by 
simply playing with RTs - example full mesh vs hub and spoke)

Best,
R.

> Usually one would have a firewall between Web->App and App->DB server
> or it makes no sense to have them in different subnets.
>
> With a firewall between app stack layers, L3VPN buys you almost
> nothing - the traffic has to go through the firewall anyway.
>
> The other option is a hypervisor-embedded firewall sitting in front
> of a VM - yet again, having VMs in different subnets is a cosmetic
> exercise.
>
> Alternatively, one could install per-flow shortcut entries after a
> flow is inspected by a traditional firewall (ex: Cisco VSG, although
> it's not an inter-subnet FW), for example through OpenFlow, but
> that's (in my opinion) way beyond simple L3VPN.
>
> Ivan
>
>> -----Original Message----- From: Robert Raszuk
>> [mailto:[email-masked]] Sent: Friday, July 27, 2012 5:59 PM To:
>> Paul Unbehagen Cc: David Allan I; Ivan Pepelnjak;
>> [email-masked]; [email-masked]; Lucy yong; NAPIERALA, MARIA H;
>> Luyuan Fang (lufang) Subject: Re: [nvo3] Role of ARP/RARP
>>
>> Paul,
>>
>> server to DB server back to App server back to web server then  >
>> finally back to user browser. Thus cross fabric flows are typical
>> in  > many applications
>>
>> Absolutely.
>>
>> That's why avoiding creation of VLANs in the first place in any
>> part of the DC network where East-West traffic is of non negligible
>> amount is highly recommended.
>>
>> L3VPN over no service aware pure IP transport works very nicely
>> and addresses the above application model pretty well.
>>
>> Best, R.
>>
>>> Many web apps require cross VM communications, eg Web server to
>>> App server to DB server back to App server back to web server
>>> then finally back to user browser. Thus cross fabric flows are
>>> typical in many applications
>>>
>>> Size, scale, and design may affect this a bit, but that's a
>>> general app flow commonality that exists. This is why interVLAN
>>> routing is used heavily in DC's of many different sizes.
>>>
>>> -- Paul Unbehagen
>
>
>

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 27 Jul 2012 18:27:15 +0200

Hi Paul,

Clearly if you have bunch of legacy applications which require L2 and 
are not capable of crossing L3 you need to provide them the L2 network.

But beyond that I really do not see a need for VLANs. Subnet - of course 
- but within a given tier ... more ... withing given TOR.

What would be any other reasons to be stack with L2 for DC transport ?

Best,
R.

 > Not sure you can avoid it completely, nor do I know that everyone 
else in the industry wants to avoid it either.   As avoiding VLANs 
doesn't address the LANs needed to support clusters of servers acting as 
any single tier of the application model described below.  Web clusters, 
app clusters, etc...

---

From: Paul Unbehagen <[email-masked]>
Date: Fri, 27 Jul 2012 10:38:02 -0600

There are many reasons, MTU, Architecture, flexibility. Many DC's do similar work by doing interISID or interVlan routing and extend the ISID or VLAN under the IP layer for VM moves without needing to jump through hoops to make sure they can provide optimum routing flows in and out of the DC as well as cross app tier subnets. 

Overlays can be very useful, but aren't a panacea in of themselves.

--
Paul Unbehagen

Sent from my iPad

On Jul 27, 2012, at 10:27 AM, Robert Raszuk <[email-masked]> wrote:

> Hi Paul,
> 
> Clearly if you have bunch of legacy applications which require L2 and are not capable of crossing L3 you need to provide them the L2 network.
> 
> But beyond that I really do not see a need for VLANs. Subnet - of course - but within a given tier ... more ... withing given TOR.
>