Subject: Re: [armd] In-Service VM Move requirement and questions

From: Linda Dunbar <[email-masked]>
Date: Tue, 12 Oct 2010 11:05:07 -0500

Rakesh, 

Thank you very much for the clarification. It is good to know that
In-Service VM Move is a key requirement for Data Center. That makes ARMD
problem statements even more solid. 

p.s. I copied to [email-masked] to let the people who are interested in the
subject be aware of this key requirement. If any of them have difference of
opinions or questions, they can voice them too. 

Linda Dunbar

  _____  

From: Rakesh Sharma [mailto:[email-masked]] 
Sent: Tuesday, October 12, 2010 10:30 AM
To: Linda Dunbar
Cc: 'Bob Sultan'; 'changyue 00118872'; Inder Gopal; Jeffrey Lynch; 'Peter
Ashwood-Smith'; Renato Recio; 'Hares Susan'; 'Ben Mack-Crane'; 'Vijoy
Pandey'; 'guoli'
Subject: RE: ARP optimization for large Data Cener

Hi Linda:

In-service VM Move (Live VM Mobility) is key value proposition for
Enterprise Data Centers due to:

1. Load balancing across servers.
2. Opening server HW maintenance window without disruption
3. Clouds

So, it is critical. 

Rakesh Sharma
STSM and Power Systems Communications and I/O Architect
Internal ZIP: 045-3D128
11400 Burnet Road, Austin, TX 78758
Ph: (512)286-8514

Inactive hide details for Linda Dunbar ---10/12/2010 09:49:21
AM---Rakesh,Linda Dunbar ---10/12/2010 09:49:21 AM---Rakesh,

From:

Linda Dunbar <[email-masked]>

To:

Rakesh Sharma/Austin/IBM@IBMUS

Cc:

"'Bob Sultan'" <[email-masked]>, "'changyue 00118872'"
<[email-masked]>, Inder Gopal/Somers/IBM@IBMUS, Jeffrey
Lynch/Raleigh/IBM@IBMUS, "'Peter Ashwood-Smith'"
<[email-masked]>, Renato Recio/Austin/IBM@IBMUS, "'Hares
Susan'" <[email-masked]>, "'Ben Mack-Crane'" <[email-masked]>,
"'Vijoy Pandey'" <[email-masked]>, "'guoli'"
<[email-masked]>

Date:

10/12/2010 09:49 AM

Subject:

RE: Seeking your input on our initiative at IETF for ARP optimization for
large Data Cener

  _____  

Rakesh, 

How often do you see In-Service VM move? 
I have heard conflicting requirement. Some people say most VM moving are
off-line. Is it true? 

Linda 

  _____  

From: Rakesh Sharma [mailto:[email-masked]] 
Sent: Tuesday, October 12, 2010 7:47 AM
To: Linda Dunbar
Cc: 'Bob Sultan'; 'changyue 00118872'; Inder Gopal; Jeffrey Lynch; 'Peter
Ashwood-Smith'; Renato Recio; 'Hares Susan'; 'Ben Mack-Crane'; 'Vijoy
Pandey'; 'guoli'
Subject: RE: ARP optimization for large Data Cener

Renato, Linda:

One of the key requirement for VM Mobility and VM HA is that we maintain TCP
connections, i.e. do not let them
time out. That is why hypervisors today move MAC addresses with VM move. If
not, all clients will need to get
their ARP entries refreshed.

Rakesh Sharma
STSM and Power Systems Communications and I/O Architect
Internal ZIP: 045-3D128
11400 Burnet Road, Austin, TX 78758
Ph: (512)286-8514

Inactive hide details for Linda Dunbar ---10/11/2010 05:52:48 PM---Do you
mean a VM gets a new MAC when move to a new location?Linda Dunbar
---10/11/2010 05:52:48 PM---Do you mean a VM gets a new MAC when move to a
new location? That would be nice. 

From:

Linda Dunbar <[email-masked]>

To:

Renato Recio/Austin/IBM@IBMUS

Cc:

"'Bob Sultan'" <[email-masked]>, "'changyue 00118872'"
<[email-masked]>, Inder Gopal/Somers/IBM@IBMUS, Jeffrey
Lynch/Raleigh/IBM@IBMUS, "'Peter Ashwood-Smith'"
<[email-masked]>, Rakesh Sharma/Austin/IBM@IBMUS, "'Hares
Susan'" <[email-masked]>, "'Ben Mack-Crane'" <[email-masked]>,
"'Vijoy Pandey'" <[email-masked]>, "'guoli'"
<[email-masked]>

Date:

10/11/2010 05:52 PM

Subject:

RE: Seeking your input on our initiative at IETF for ARP optimization for
large Data Cener

  _____  

Do you mean a VM gets a new MAC when move to a new location? That would be
nice. 

However, if VM maintains its MAC while moving from Rack 1 to Rack 5, then
all the TORs in the data center need to know that the VM is now under TOR 5.
That is what we want ARMD to do, i.e. an Address Server giving the new TOR
switch information as extended ARP reply. Any suggestion on this? 

Linda 

  _____  

From: Renato Recio [ <mailto:[email-masked]> mailto:[email-masked]] 
Sent: Monday, October 11, 2010 4:27 PM
To: Linda Dunbar
Cc: 'Bob Sultan'; 'changyue 00118872'; Inder Gopal; Jeffrey Lynch; 'Peter
Ashwood-Smith'; Rakesh Sharma; 'Hares Susan'; 'Ben Mack-Crane'; 'Vijoy
Pandey'; 'guoli'
Subject: RE: ARP optimization for large Data Cener

Right, sooo... if the MAC Addresses where provided by a DHCP server, then
the issue with (2) is gone and the issues (ARPs, MAC Table sizes going
crazy) you're trying to solve are too. 

Renato J Recio
DE, IBM DC Networking CTO
PMI PMP Certified
Member IBM Academy of Technology
Tel 512-286-8188, T/L 363-8188 

From: 

Linda Dunbar <[email-masked]> 

To: 

Renato Recio/Austin/IBM@IBMUS 

Cc: 

"'Vijoy Pandey'" <[email-masked]>, Jeffrey
Lynch/Raleigh/IBM@IBMUS, Rakesh Sharma/Austin/IBM@IBMUS, Inder
Gopal/Somers/IBM@IBMUS, "'guoli'" <[email-masked]>, "'changyue
00118872'" <[email-masked]>, "'Peter Ashwood-Smith'"
<[email-masked]>, "'Hares Susan'" <[email-masked]>, "'Bob
Sultan'" <[email-masked]>, "'Ben Mack-Crane'" <[email-masked]> 

Date: 

10/11/2010 03:18 PM 

Subject: 

RE: Seeking your input on our initiative at IETF for ARP optimization for
large Data Cener

  _____  

Hi Renato, 

Questions and comments are inserted below: 

  _____  

From: Renato Recio [ <mailto:[email-masked]> mailto:[email-masked]] 
Sent: Monday, October 11, 2010 6:47 AM
To: Linda Dunbar
Cc: 'Vijoy Pandey'; Jeffrey Lynch; Rakesh Sharma; Inder Gopal; 'guoli';
'changyue 00118872'; Peter Ashwood-Smith
Subject: RE: ARP optimization for large Data Cener 

Linda, 

An additional approach is to replace ARPs with a DHCP like service for MAC
Addresses (see section E of
<http://i-teletraffic.org/fileadmin/ITC21_files/DC-CAVES/DC-CAVES_-_Automate
dEthernetVirtualBridging.pdf>
http://i-teletraffic.org/fileadmin/ITC21_files/DC-CAVES/DC-CAVES_-_Automated
EthernetVirtualBridging.pdf. This service could be done by the Hypervisor/OS
or by the nearest neighbor switch. 
[Linda] Do you mean VM getting MAC address from an ARP server? Like host
getting IP address from DHCP? I think how a VM gets its MAC address is less
of a concern. But bigger concern is when a VM moves to a new rack or row,
the TOR or End of Row switch need to know how to forward data frames from
this VM to targets and the old TOR needs to know how to re-direct or drop
frames. 
Your paper stated that today's MAC being used to associate VM with a port
profile is an issue when VM moves. Are we talking about the same issue? It
is especially painful when there is another layer of encapsulation (like
MPLS or MAC-in-MAC) by the TOR switch. The work we are proposing at ARMD is
to expand the ARP server with more network information, so that when VM
moves, the new TOR can get proper network information associated with this
MAC. 
Do you agree with us? 

When this service is combined with a MOOSE (
<http://i-teletraffic.org/fileadmin/ITC21_files/DC-CAVES/DC-CAVES_-_MOOSE.pd
f>
http://i-teletraffic.org/fileadmin/ITC21_files/DC-CAVES/DC-CAVES_-_MOOSE.pdf
) like approach to MAC Addresses, additional, much more significant benefits
accrue. 
[Linda] The inventor of MOOSE from Cambridge University presented their
solution at our bar BOF in 78th IETF. The general feedback is that MOOSE
requires change to all switches. Instead of MAC forwarding, MOOSE requires
Prefix forwarding, very much like IP. 
Another problem with MOOSE is that the first switch changes the Source
Address, which impact DHCP behavior. 

Linda 

Thoughts? 

Thanks,

Renato J Recio
DE, IBM DC Networking CTO
PMI PMP Certified
Member IBM Academy of Technology
Tel 512-286-8188, T/L 363-8188 

From: 

Linda Dunbar <[email-masked]> 

To: 

Renato Recio/Austin/IBM@IBMUS 

Date: 

10/09/2010 11:26 AM 

Subject: 

RE: Seeking your input on our initiative at IETF for ARP optimization for
large Data Cener

  _____