Subject: [armd] Why need GRE tunnel for overlay network to scale ARP/ND in DC?

From: Linda Dunbar <[email-masked]>
Date: Wed, 17 Aug 2011 20:22:52 +0000

Warren,

Thank you very much for posting "draft-wkumari-dcops-l3-vmmobility-00". It shows a possible approach to scale Address Resolution for large number of hosts in data center.

Your draft proposes an overlay network boarded by Hypervisor (or virtual switch within a server in IEEE802.1 Data Center terminology).  The boarder node, i.e. Hypervisor in your draft, can intercept ARP/ND requests, pass it to a "mapping server" (or directory server), and cache the response. The response from the "mapping server" not only has the target host's MAC but also the physical server's IP where target host resides). When the Source host sends a data frame to the target host, the "Hypervisor" encapsulate the data packet with the IP header (with SA = IP of the physical server where Source host resides, and DA= IP of the physical server where target host resides).

My question is why need GRE tunnel for the overlay network? Why can't it be simple IP network?

The ARP/ND statistics work done by Merit Network showed that the pain points of large Layer 2 network is at the L2/L3 Gateway (or boundary) node.  Your draft describes the way for  "overlay Edge node" (i.e. Hypervisor in your draft) to intercept ARP/ND requests and reroute them to "mapping server". This method can definitely reduce the amount of ARP/ND requests to be processed by L2/L3 gateway nodes. However, L2/L3 Gateway nodes still have to maintain a huge mapping table (target host address & Customer ID <-> Server Address). Are you suggesting L2/L3 Gateway also using external "mapping server"?

I didn't see any description in your draft on how servers/guest OS get notification when some VMs have moved. Can you give some explanation on how to avoid traffic being forwarded to the wrong servers when target VMs have moved?

Thank you very much.

Linda Dunbar

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 17 Aug 2011 17:16:34 -0400

Linda, I must have missed a set of statistics.
The earlier descriptions I had heard, and the descriptions from folks at 
Yahoo (I will let Warren speak for Google) have stated that the problem 
is at the end device and its virtual switch, and the process of mapping 
to the actual MAC addresses to use.

If instead the key point is where external traffic (either from users, 
or other families of servers) enter the subnet where the virtual machine 
is located, then I would appreciate the pointers to those statistics. 
(Sorting out this kind of confusion is exactly what ARMD was chartered 
for, so the fact of confusion comes as no surprise.)

Yours,
Joel

On 8/17/2011 4:22 PM, Linda Dunbar wrote:
> Warren,
>
> Thank you very much for posting “draft-wkumari-dcops-l3-vmmobility-00”.
> It shows a possible approach to scale Address Resolution for large
> number of hosts in data center.
>
> Your draft proposes an overlay network boarded by Hypervisor (or virtual
> switch within a server in IEEE802.1 Data Center terminology). The
> boarder node, i.e. Hypervisor in your draft, can intercept ARP/ND
> requests, pass it to a "mapping server" (or directory server), and cache
> the response. The response from the "mapping server" not only has the
> target host's MAC but also the physical server's IP where target host
> resides). When the Source host sends a data frame to the target host,
> the "Hypervisor" encapsulate the data packet with the IP header (with SA
> = IP of the physical server where Source host resides, and DA= IP of the
> physical server where target host resides).
>
> My question is why need GRE tunnel for the overlay network? Why can’t it
> be simple IP network?
>
> The ARP/ND statistics work done by Merit Network showed that the pain
> points of large Layer 2 network is at the L2/L3 Gateway (or boundary)
> node. Your draft describes the way for “overlay Edge node” (i.e.
> Hypervisor in your draft) to intercept ARP/ND requests and reroute them
> to “mapping server”. This method can definitely reduce the amount of
> ARP/ND requests to be processed by L2/L3 gateway nodes. However, L2/L3
> Gateway nodes still have to maintain a huge mapping table (target host
> address & Customer ID <-> Server Address). Are you suggesting L2/L3
> Gateway also using external “mapping server”?
>
> I didn’t see any description in your draft on how servers/guest OS get
> notification when some VMs have moved. Can you give some explanation on
> how to avoid traffic being forwarded to the wrong servers when target
> VMs have moved?
>
> Thank you very much.
>
> Linda Dunbar
>
>
>
> _______________________________________________
> armd mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/armd

---

From: Linda Dunbar <[email-masked]>
Date: Wed, 17 Aug 2011 21:43:54 +0000

Joel, 

Merit Networks' Jim Rees and Manish Karir (CC'ed) presented their ARP/ND statistics at  81st IETF ARMD WG session. 

Their slides can be found at https://datatracker.ietf.org/meeting/81/materials.html under ARMD Address Resolution Statistics. 

Their draft is https://datatracker.ietf.org/doc/draft-karir-armd-statistics/

Linda

> -----Original Message-----
> From: Joel M. Halpern [mailto:[email-masked]]
> Sent: Wednesday, August 17, 2011 4:17 PM
> To: Linda Dunbar
> Cc: [email-masked]; [email-masked]
> Subject: Re: [armd] Why need GRE tunnel for overlay network to scale
> ARP/ND in DC?
> 
> > Thank you very much for posting "draft-wkumari-dcops-l3-vmmobility-
> 00".
> > My question is why need GRE tunnel for the overlay network? Why can't
> it
> > be simple IP network?
> >
> > node. Your draft describes the way for "overlay Edge node" (i.e.
> > to "mapping server". This method can definitely reduce the amount of
> > Gateway also using external "mapping server"?
> >
> > I didn't see any description in your draft on how servers/guest OS
> get