Subject: [armd] FW: Updated charter statement for ARMD

From: T Sridhar <[email-masked]>
Date: Tue, 12 Oct 2010 18:14:31 -0700

My earlier message to the ARMD list was blocked indicating "Too many recipients"... Apologize for the resend....

From: T Sridhar
Sent: Tuesday, October 12, 2010 6:07 PM
To: 'Linda Dunbar'; [email-masked]
Cc: 'Hamilton, James'; 'changyue 00118872'; 'Rakesh Sharma'; 'Peter Ashwood-Smith'; 'Inder Gopal'; 'Hares Susan'; 'Bob Sultan'; 'Vijoy Pandey'; [email-masked]; 'Ben Mack-Crane'; 'Renato Recio'; 'guoli'; Geoff Arnold; 'Jeffrey Lynch'
Subject: RE: [armd] Updated charter statement for ARMD

Linda,

A diff with the previous charter would be useful.. :)

Some comments/suggested text:

1.       Suggest ARMD expand to "Address Resolution to Massive Number of Hosts in Data Center Environments". In the same context, we could universally change "amount of hosts" to "number of hosts" across the entire document.

2.      The text in the "Next Gen or Cloud Data Center have to handle massive amount of subnets (or Closed User Groups)" section is not very clear. Here's some draft text for consideration:

a.      Cloud Data Centers offer multitenant services which may require each tenant (and their VMs) to be in their own VPN environment, equivalent to a Closed User Group.  The number of such tenants/groups might be much larger than the VLAN space (4095).

3.       Would suggest rephrasing the next bullet item "The topology of subnet changes...". Some suggested text:

a.       VM migration in Layer 2 environments will require updating the Layer 2 (MAC) tables in the individual switches in the data center to ensure accurate forwarding. Consider a case where a VM migrates across racks.  The migrated VM often sends out a gratuitous ARP broadcast when it comes up at the new location. This is forwarded by the top of rack switch at the new rack across the entire network. The individual switches learn the new location of the migrated VM through the source address of this broadcast frame. The top of rack switch at the old rack is not aware of the migration until it receives this gratuitous ARP. So it continues to forward frames to the port where it  learnt the VM's MAC address from earlier, leading to black holing of traffic.  The duration of this black holing period may depend upon the topology. It may be longer if the VM has moved to  a rack in a different data center connected to this data center over Layer 2.

Thanks,
Sridhar

From: [email-masked] [mailto:[email-masked]] On Behalf Of Linda Dunbar
Sent: Tuesday, October 12, 2010 11:47 AM
To: [email-masked]
Cc: 'Hamilton, James'; 'changyue 00118872'; 'Rakesh Sharma'; 'Peter Ashwood-Smith'; 'Inder Gopal'; 'Hares Susan'; 'Bob Sultan'; 'Vijoy Pandey'; [email-masked]; 'Ben Mack-Crane'; 'Renato Recio'; 'guoli'; Geoff Arnold; 'Jeffrey Lynch'
Subject: [armd] Updated charter statement for ARMD

When we did ARP222 bar BOF at 78th IETF, the focus was more on large broadcast domain (ARP storm bombarding application servers, and FDB size getting too big. As we dive in more discussions with various data centers operators, we see more clearly of three major problems facing in Next Gen Data Centers:

 *   Massive amount of hosts
 *   Large number of subnets
 *   VM migration causing their corresponding subnet topology changing.

ARP in general is a problem, especially IPv4. But the VM migration just makes the problem even more severe. Therefore, we made some changes to the Charter Statements. Your comments are appreciated

Regards, Linda Dunbar

ARMD: Address Resolution for Massive amount of hosts in Data Center

Description of Working Group:

*         Server virtualization introduces massive amount of hosts in a data center

As server virtualization is introduced to data centers, the number of hosts in a data center can grow dramatically.  Each physical server, which used to host one end-station, now can host hundreds of end-stations or virtual machines (VMs). Virtual machines (VMs) can be easily added and/or deleted and moved among physical servers.  The VM's flexible addition, deletion, and (or) migration make it possible to combine pool of servers into one huge single resource entity for ease of management and better resource utilization. It also provides foundation for virtual hosts and virtual desktops offered by Cloud Computing services.

This rapid growth of virtual hosts could tremendously impact networks and servers. One major issue is frequent ARP (IPv4) or neighbor discovery (IPv6) requests from hosts. All hosts send out those requests frequently due to host ARP caches being aged out in minutes. With tens of thousands of hosts (each with a distinct MAC address) in one Data Center, the number of ARP (or ND) packets per second from all hosts could potentially be as high as 1,000 to 10,000/second. This rate imposes tremendous computational burden on servers.

*         Next Gen or Cloud Data Center have to handle massive amount of subnets (or Closed User Groups)

When Telecom service providers extend their existing VPN to let VMs in their own data centers to house some of the client VPN hosts, their data center would need to support massive amount of subnets, with each subnet matching one VPN. The number could be exceeding the number of VLANs (4095) available.

*         The topology of subnet changes as virtual machines migrate from one location to another

One key characteristic of a next-generation Data Center is the possibility of one subnet spanning across multiple sites and its topology changes. For example, when VMs move from one rack to another, their associated subnets topology will change accordingly.  Therefore, the paths from other hosts to the hosts on the newly moved VMs are moving as well. Part of address resolution in traditional Layer 2 network is for all intermediate switches to learn the proper path between requesting host and target host. When path changes, there will be more dropped data frames or more occurrences of re-directed traffic.

Traditional subnet (VLAN) partitioning no longer works well when servers are virtualized. When each server only has one host, there is only one VLAN enabled for the switch port towards the server. The switch can block all broadcast messages from other subnets (VLANs) from going to the server. When one physical server is supporting >100 Virtual Machines, i.e. >100 hosts, most likely the virtual hosts on one server are on different subnets (VLANs). If there are 50 subnets (VLANs) enabled on the switch port to the server, the server has to handle all the ARP broadcast messages on all 50 subnets (VLANs). The amount of ARP to be processed by each server is still too much. Another issue with VLAN partitioning is that there could be more subnets than available VLANs (4095) in a Data Center.

The goal of this working group is to develop interoperable solutions to solve the above problems within a Data Center and multi-site Data Center environments.  The multiple sites could be connected by any types of network, like L2VPN, L3VPN, Ethernet network, or simple IP networks. This could involve multi-domain (multi-company), multi-site and multi-vendor environments. Efficient address resolution across multiple sites should be considered by this working group. Since VMs allocation and migration within a data center is managed by data center's central resource management, directory based approach should also be considered as well.
The design should consider the following properties:

 *   All solutions developed by ARMD WG should not require any behavior changes on hosts, applications, or Virtual Machines being deployed in the market.
 *   All solutions developed should not break DHCP, or any other broadcast/multicast mechanism used by applications.
 *   Evaluating the impact to IPv6 ND, and develop solutions accordingly if needed.
 *   Should consider variety of solutions, including directory based, proxy based, or cache based solutions.
 *   Include analysis of security concerns of IPv4 ARP requests from malicious users. Evaluating potential security solutions and conclude if the security threat can justify solutions.
 *   ARMD assumes the direct links to individual hosts and virtual machines are IEEE802.3 Ethernet links.
 *   Should consider scenarios of one Ethernet network being interconnected by another network, which can be L2VPN, pure IP, Ethernet, or others.
 *   Should consider a performance analysis of proposed solutions.

Here are the items which should not be in the scope of the working group:

 *   Re-define DHCP behavior
 *   Re-define IPv6 ND security model
 *   Solution developed should not expect any behavior changes for the guest OS/applications running on the VMs
 *   Direct links from hosts and virtual hosts being non-Ethernet links.

Goals and Milestones:

 *   Problem statement
 *   Gap Analysis Statement within Data Center
    *   Gaps within ARP
    *   Gaps within IPv6 ND or autoconfiguration
 *   Survey of Existing Solutions
    *   Survey of NHRP (RFC2332) & SCSP (RFC2334),  and their applicability to Ethernet networks;
    *   Survey of TRILL work as a potential solution;
    *   Survey of  other potential solutions, like MOOSE or SEATTLE; and
    *   Other proposals
 *   Security Analysis of Existing Solutions
    *   Study existing solutions, like Dynamic ARP Inspection, Egress ARP Inspection, etc, and evaluate if any of them or others should be included in the solutions specified by this working group.
 *   Architectural Design for Next-Generation Data Center
 *   Protocol Documents

Time line for BOF presentations

Initial Problem Statement                                        August
GAP analysis                                                          October
Survey of Existing Solutions                                    October
Security Analysis of Existing Solutions                     2011
Architectural Design for NG                                    October
Final Problem statement for BOF                            October
Protocol Documents                                               2011

---

From: "So, Ning" <[email-masked]>
Date: Thu, 14 Oct 2010 15:53:45 +0000

Hypervisor should be off the scope for now.  It can be added later in case a solution cannot be found without Hypervisor being a part of it.  

Best regrads,

Ning So

Network Evolution Planning

Verizon, Inc.

(office) 972-729-7905

(Cell) 972-955-0914

*	From: Linda Dunbar <ldunbar at huawei.com <mailto:[email-masked]> > 
*	To: wang.jun17 at zte.com.cn <mailto:[email-masked]>  
*	Cc: 'Renato Recio' <recio at us.ibm.com <mailto:[email-masked]> >, 'Rakesh Sharma' <rakesh at us.ibm.com <mailto:[email-masked]> >, armd at ietf.org <mailto:[email-masked]>  
*	Date: Thu, 14 Oct 2010 09:41:38 -0500 
*	In-reply-to: <OF07C47F3C.8DCCD487-ON482577BC.002B44BD-482577BC.002C2952 at zte.com.cn <mailto:[email-masked]> > 
*	References: <00b701cb6ae6$ef86ead0$4b0c7c0a at china.huawei.com <mailto:[email-masked]> > <OF07C47F3C.8DCCD487-ON482577BC.002B44BD-482577BC.002C2952 at zte.com.cn <mailto:[email-masked]> > 
*	List-id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." <armd.ietf.org> 

________________________________

size=2 width="100%" align=center> 
I believe L2/L3 should be in the ARMD scope too. When TOR does the encapsulation, it is very possible to have some types of L3VPN encapsulation. 

But all those behavior is on switches, not really on hypervisors unless there is virtual switch embedded on hypervisor. When Hypervisor has embedded virtual switch, the protocol to be developed by ARMD should work too. 

I copy this to some IBM folks. They probably have more accurate information on how likely Hypervisors have embedded virtual switch. 

Linda

________________________________

From: wang.jun17 at zte.com.cn [mailto:wang.jun17 at zte.com.cn] 
Sent: Thursday, October 14, 2010 3:02 AM
To: Linda Dunbar
Cc: armd at ietf.org
Subject: RE: [armd] Updated charter statement for ARMD

Yes, the OS and hypervisor is out of scope, but I think that the L2/L3 protocols inside are in the scope. 

If we take consideration of VM migration especially the In-service migration, I'm wondering whether it can be solved without hypervisor's involvement.

Linda Dunbar <ldunbar at huawei.com> 

2010-10-13 22:57 

收件人

wang.jun17 at zte.com.cn 

抄送

armd at ietf.org 

主题

RE: [armd] Updated charter statement for ARMD

I think that guest OS and hypervisor behavior is out of the scope of IETF. Therefore, ARMD doesn’t not assume any behavior changes to guest OS and hypervisor. 
  
Linda Dunbar 

________________________________

From: armd-bounces at ietf.org [mailto:armd-bounces at ietf.org] On Behalf Of wang.jun17 at zte.com.cn
Sent: Tuesday, October 12, 2010 7:53 PM
To: Linda Dunbar
Cc: 'Hamilton, James'; 'changyue 00118872'; 'Ben Mack-Crane'; 'Rakesh Sharma'; armd at ietf.org; 'Peter Ashwood-Smith'; 'Inder Gopal'; armd-bounces at ietf.org; 'Bob Sultan'; 'Vijoy Pandey'; dmaltz at microsoft.com; Geoff Arnold; 'Hares Susan'; 'Renato Recio'; 'guoli'; 'Jeffrey Lynch'
Subject: Re: [armd] Updated charter statement for ARMD 

Does it mean we can't modify either the guest OS or the hypervisor? 

Linda Dunbar <ldunbar at huawei.com> 
发件人:  armd-bounces at ietf.org 

2010-10-13 02:47 

收件人

armd at ietf.org 

抄送

"'Hamilton, James'" <james at amazon.com>, 'changyue 00118872' <changyue at huawei.com>, 'Rakesh Sharma' <rakesh at us.ibm.com>, 'Peter Ashwood-Smith' <Peter.AshwoodSmith at huawei.com>, 'Inder Gopal' <igopal at us.ibm.com>, 'Hares Susan' <shares at huawei.com>, 'Bob Sultan' <bsultan at huawei.com>, 'Vijoy Pandey' <vijoy.pandey at bladenetwork.net>, dmaltz at microsoft.com, 'Ben Mack-Crane' <tmackcrane at huawei.com>, 'Renato Recio' <recio at us.ibm.com>, 'guoli' <yinguoli at huawei.com>, Geoff Arnold <gma at yahoo-inc.com>, 'Jeffrey Lynch' <jjlynch at us.ibm.com> 

主题

[armd] Updated charter statement for ARMD

  

*         

When we did ARP222 bar BOF at 78th IETF, the focus was more on large broadcast domain (ARP storm bombarding application servers, and FDB size getting too big. As we dive in more discussions with various data centers operators, we see more clearly of three major problems facing in Next Gen Data Centers: 
o        Massive amount of hosts 
o        Large number of subnets 
o        VM migration causing their corresponding subnet topology changing. 

Regards, Linda Dunbar 

 

As server virtualization is introduced to data centers, the number of hosts in a data center can grow dramatically.  Each physical server, which used to host one end-station, now can host hundreds of end-stations or virtual machines (VMs). Virtual machines (VMs) can be easily added and/or deleted and moved among physical servers.  The VM’s flexible addition, deletion, and (or) migration make it possible to combine pool of servers into one huge single resource entity for ease of management and better resource utilization. It also provides foundation for virtual hosts and virtual desktops offered by Cloud Computing services. 

The goal of this working group is to develop interoperable solutions to solve the above problems within a Data Center and multi-site Data Center environments.  The multiple sites could be connected by any types of network, like L2VPN, L3VPN, Ethernet network, or simple IP networks. This could involve multi-domain (multi-company), multi-site and multi-vendor environments. Efficient address resolution across multiple sites should be considered by this working group. Since VMs allocation and migration within a data center is managed by data center’s central resource management, directory based approach should also be considered as well. 
The design should consider the following properties: 
o        All solutions developed by ARMD WG should not require any behavior changes on hosts, applications, or Virtual Machines being deployed in the market. 
o        All solutions developed should not break DHCP, or any other broadcast/multicast mechanism used by applications. 
o        Evaluating the impact to IPv6 ND, and develop solutions accordingly if needed. 
o        Should consider variety of solutions, including directory based, proxy based, or cache based solutions. 
o        Include analysis of security concerns of IPv4 ARP requests from malicious users. Evaluating potential security solutions and conclude if the security threat can justify solutions. 
o        ARMD assumes the direct links to individual hosts and virtual machines are IEEE802.3 Ethernet links.   
o        Should consider scenarios of one Ethernet network being interconnected by another network, which can be L2VPN, pure IP, Ethernet, or others. 
o        Should consider a performance analysis of proposed solutions.   
  
Here are the items which should not be in the scope of the working group: 
o        Re-define DHCP behavior 
o        Re-define IPv6 ND security model 
o        Solution developed should not expect any behavior changes for the guest OS/applications running on the VMs 
o        Direct links from hosts and virtual hosts being non-Ethernet links. 
  
Goals and Milestones: 
o        Problem statement 
o        Gap Analysis Statement within Data Center 
o        Gaps within ARP 
o        Gaps within IPv6 ND or autoconfiguration 
o        Survey of Existing Solutions 
o        Survey of NHRP (RFC2332) & SCSP (RFC2334),  and their applicability to Ethernet networks; 
o        Survey of TRILL work as a potential solution; 
o        Survey of  other potential solutions, like MOOSE or SEATTLE; and 
o        Other proposals 
o        Security Analysis of Existing Solutions 
o        Study existing solutions, like Dynamic ARP Inspection, Egress ARP Inspection, etc, and evaluate if any of them or others should be included in the solutions specified by this working group.   
o        Architectural Design for Next-Generation Data Center 
o        Protocol Documents 

 
_______________________________________________
armd mailing list
armd at ietf.org
https://www.ietf.org/mailman/listinfo/armd 

-------------------------------------------------------- 
ZTE Information Security Notice: The information contained in this mail is solely property of the sender's organization. This mail communication is confidential. Recipients named above are obligated to maintain secrecy and are not permitted to disclose the contents of this communication to others. 
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the originator of the message. Any views expressed in this message are those of the individual sender. 
This message has been scanned for viruses and Spam by ZTE Anti-Spam system. 

--------------------------------------------------------
ZTE Information Security Notice: The information contained in this mail is solely property of the sender's organization. This mail communication is confidential. Recipients named above are obligated to maintain secrecy and are not permitted to disclose the contents of this communication to others.
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the originator of the message. Any views expressed in this message are those of the individual sender.
This message has been scanned for viruses and Spam by ZTE Anti-Spam system.
________________________________

size=2 width="100%" align=center> 

*	References: 

	*	Re: [armd] Updated charter statement for ARMD <http://www.ietf.org/mail-archive/web/armd/current/msg00022.html>  

		*	From: Linda Dunbar

	*	Re: [armd] Updated charter statement for ARMD <http://www.ietf.org/mail-archive/web/armd/current/msg00024.html>  

		*	From: wang . jun17

*	Prev by Date: Re: [armd] Updated charter statement for ARMD <http://www.ietf.org/mail-archive/web/armd/current/msg00024.html>  
*	Next by Date: [armd] FW: New Version Notification for draft-mackcrane-armd-ipv6-nd-scaling-00 <http://www.ietf.org/mail-archive/web/armd/current/msg00026.html>  
*	Previous by thread: Re: [armd] Updated charter statement for ARMD <http://www.ietf.org/mail-archive/web/armd/current/msg00024.html>  
*	Next by thread: Re: [armd] Updated charter statement for ARMD <http://www.ietf.org/mail-archive/web/armd/current/msg00020.html>  
*	Index(es): 

	*	Date <http://www.ietf.org/mail-archive/web/armd/current/maillist.html#00025>  
	*	Thread <http://www.ietf.org/mail-archive/web/armd/current/threads.html#00025>  

Note Well: Messages sent to this mailing list are the opinions of the senders and do not imply endorsement by the IETF.