Subject: Re: [armd] IP over IP solution for data center interconnect

From: Xuxiaohu <[email-masked]>
Date: Fri, 23 Dec 2011 06:38:40 +0000

> -----邮件原件-----
> 发件人: Xuxiaohu
> 发送时间: 2011年12月22日 16:28
> 收件人: 'Eggert, Lars'; [email-masked]; [email-masked].'
> 主题: IP over IP solution for data center interconnect
> 
> Hi all,
> 
> There has been a lot of L2 over L3 solutions or proposals for data center
> network and data center interconnect till now. However, it seems recently
> there are increasing interests on L3 over L3 (e.g., IP over IP) solutions for data
> center network and data center interconnect, please see the following text
> quoted from IETF82 L2VPN minutes
> ( https://www.ietf.org/proceedings/82/minutes/l2vpn.txt). It's a general belief
> that L3 is more scalable than L2. Especially when considering the data center
> interconnection case, the L3 over L3 solutions can bring DC operators more
> benefits compared to the L2 over L3 solutions, such as path optimization,
> active-active data centers, MAC table reduction on DC switches and broadcast
> flood suppression etc .

By the way, the ARP table scaling issue on DC gateways, which is deemed by the ARMD WG as the only worthy ARP problem in data center networks, could also be solved with the IP over IP solution.

Best regards,
Xiaohu
 
> 
> Although Layer 2 connectivity is still required for some high-availability clusters
> which use non-IP and link-local multicast for communication, more and more
> cluster vendors are either already able to support cluster services at Layer3 or
> will support it in the near future. In addition, the GSLB mechanism (e.g., DNS
> based GSLB) works very well in most cases, is geo-cluster service still a
> common requirement for data center interconnect? If not, could we spend any
> time on exploring the possibility of using L3 over L3 solution for the most data
> center interconnection scenarios where geo-clustering, especially non-IP based
> geo-clustering is not needed?
> 
> Best regards,
> Xiaohu
> 
> ****************************************************************
> **********************************************************
> Kireeti: I keep seeing ISID, PBB, VLANs.  We have to stop conceding to layer 2
> and start moving to layer 3 (applause) and lots of problems will go away.
> 
> Florin: We should put VPN on the same page to modify the priorities. You need
> to expand so can look at L3 over L3 as well as L2 over L3. For item number 2 we
> need to look at L3 transport as well as Ethernet and MPLS.  And for number 3
> we need to work on this as there's a lot of expertise in L2, L3 and other WGs.
> 
> 
> Marc: The problem statement needs to include more of the L3 issues around
> sending L2 over L3, as the L2 traffic already contains L3.  Issue is just framing.
> 
> Eric Nordmark; "Yes, Yes and Yes".  We need to focus from the DC prospective
> and not from the VPN view.  don't just think about Ethernet over IP, but also IP
> over IP.  Hypervisor may get decoupled over time.
> ****************************************************************
> **********************************************************
> > -----邮件原件-----
> > 发件人: [email-masked] [mailto:[email-masked]] 代表 Eggert,
> Lars
> > 发送时间: 2011年12月16日 16:24
> > 收件人: [email-masked]
> > 主题: [dc] IRTF datacenter research group discussion list
> >
> > Hi,
> >
> > I wanted to make you all aware of the IRTF's dcrg-interest mailing list, which
> > was set up following a face-to-face meeting at SIGCOMM this year to discuss
> a
> > possible IRTF research group on datacenter networking:
> > http://irtf.org/mailman/listinfo/dcrg-interest
> >
> > There has not been much activity towards the formation of an IRTF RG since
> > SIGCOMM, but I am hopeful that the high energy level demonstrated in the
> > various Taipei meetings on the topic will inject some energy here - several of
> > the topics that may be too early for the IETF to standardize around would fit
> an
> > IRTF RG very well. I'm certainly supportive of this.
> >
> > Lars
> > IRTF Chair
> >
> >
> > --
> > Mobile number during December:    +358 46 5215582
> > Mobile number starting January:  +49 151 12055791

---

From: Benson Schliesser <[email-masked]>
Date: Fri, 23 Dec 2011 11:44:48 -0600

> By the way, the ARP table scaling issue on DC gateways, which is deemed by the ARMD WG as the only worthy ARP problem in data center networks, could also be solved with the IP over IP solution.

Two comments:

First, if anybody is interested in exploring additional address resolution issues in the datacenter, please speak up now on the ARMD mailing list. We are attempting to finish our Problem Statement soon. The Problem Statement will only include those issues that are described by WG contributors. So if you think we should deem other issues to be worthy problems, please contribute to the discussion.

Second, at a high-level I think that a number of ARMD participants will agree with your comment about ARP scale issues being solved by "IP over IP" solutions. In fact, it's not just IP-over-IP that can help. Any map-and-encap scheme, including L2-over-IP, can help by shifting the burden to a mapping mechanism. Local ARP/ND proxy functions (e.g. in the VM v-switch, TOR, etc) could take advantage of the mapping mechanism to facilitate address resolution.

In my personal opinion: The real problem is figuring out how to do the mapping, what the right mapping approach is for a datacenter, etc.  This is the topic of the NVO3 effort that was discussed during L2VPN in Taipei.

Cheers,
-Benson

---

From: Xuxiaohu <[email-masked]>
Date: Sat, 24 Dec 2011 01:27:09 +0000

> -----邮件原件-----
> 发件人: Benson Schliesser [mailto:[email-masked]]
> 发送时间: 2011年12月24日 1:45
> 收件人: Xuxiaohu
> 抄送: Eggert, Lars; [email-masked]; [email-masked]; [email-masked]
> 主题: Re: [dcrg-interest] IP over IP solution for data center interconnect
> 
> 
> > By the way, the ARP table scaling issue on DC gateways, which is deemed by
> the ARMD WG as the only worthy ARP problem in data center networks, could
> also be solved with the IP over IP solution.
> 
> Two comments:
> 
> First, if anybody is interested in exploring additional address resolution issues in
> the datacenter, please speak up now on the ARMD mailing list. We are
> attempting to finish our Problem Statement soon. The Problem Statement will
> only include those issues that are described by WG contributors. So if you think
> we should deem other issues to be worthy problems, please contribute to the
> discussion.
> 
> Second, at a high-level I think that a number of ARMD participants will agree
> with your comment about ARP scale issues being solved by "IP over IP"
> solutions. In fact, it's not just IP-over-IP that can help. Any map-and-encap
> scheme, including L2-over-IP, can help by shifting the burden to a mapping
> mechanism. Local ARP/ND proxy functions (e.g. in the VM v-switch, TOR, etc)
> could take advantage of the mapping mechanism to facilitate address
> resolution.

As for L2 over IP solutions, I agree that the ARP broadcast flood impact on the servers/routers and networks could be alleviated by using the ARP cache mechanism (tools.ietf.org/pdf/draft-shah-armd-arp-reduction-02.txt) or some ARP mapping system.

However, since the DC gateways usually need to forward traffic to so many hosts at most time, they need a very large ARP cache table to contain so many ARP entries. Otherwise, the ARP cache aging-out timer should be set short enough. In the way, the DC gateways would have to send ARP requests more frequently, in addition, the gateways are required to have a much large packet cache to cache the received packets destined for the hosts before the ARP responses for them are returned, otherwise the received packets would have to be discarded when the cache is overflowed. In a word, I don't know how the mapping system could help on this ARP table scaling issue.

Best regards,
Xiaohu

> In my personal opinion: The real problem is figuring out how to do the mapping,
> what the right mapping approach is for a datacenter, etc.  This is the topic of
> the NVO3 effort that was discussed during L2VPN in Taipei.
> 
> Cheers,
> -Benson
>

---

From: "Ashish Dalela (adalela)" <[email-masked]>
Date: Tue, 27 Dec 2011 11:43:15 +0530

For all the proponents of L3, here is a thought.

The biggest concern in cloud is data protection. That is just make or break. Data can't move when the VM moves. It's just too expensive and time consuming. Further, to reduce disk wastage you have to put data on network storage. That's just easy for data backup, and cheap in terms of total disk space you need. There is a lot going on here in terms of data de-duplication.

Now given network storage, you have two options - NAS and SAN.

On the cloud each VM administrator has root privileges. And IP and MAC addresses can be duplicated (if you are root just configure any IP and MAC on a virtual interface of choice). So, you have to protect the network storage from multiple tenants who can have the same IP address and MAC address. NAS assumes your id on the UNIX is globally unique and your permissions are unique - not true anymore. So, you have IP, MAC and ID duplication.

That means that if you have to protect data, you have to know somehow which host is allowed to see which disks. Doing this end-to-end using TCP/IP is problematic because the NAS controller doesn't know about tenant id, and whether the MAC and IP are being duplicated across customers. To make the storage aware of segmentation, you have to carry some segment (GRE, MPLS, VLAN, etc.) into the storage box. That's not true today.

Network level isolation happens in SAN, but not in NAS. NAS just runs end-to-end over TCP. In SAN you can say which host will see which disk. The FC security is high because MAC and FC-ID is assigned by the network and not in the control of the host. The network knows which port is which host/MAC. So, if you only know which port is which customer, you know which MAC can see which disk. Binding a port to a customer is not hard. But mapping all of that to data security and disk visibility is not trivial. E.g. you have to do GRE tunnel mapping to a set of disk permissions. Who needs that?

SAN does not use IP. It uses Fiber Channel. With convergence, we are moving to FC over Ethernet (FCoE). FCIP and iSCSI have not worked very well. Security for data comes from FC and lowering of cost comes from use of FCoE. That needs native Ethernet (no IP). 

When you have site-to-site VM mobility, data is not moving across sites. E.g. an application server will connect to a set of load-balanced set of database servers within a VLAN. The application and database servers can be different sites, and they connect over a VLAN. Alternately, the application and database servers can be in the same location and the DB server accesses the SAN over the Internet. Even if data is replicated across many sites, load-balancing across application or db servers uses VLANs. In HPC as well, there is a lot of broadcast and discovery, and this may use IP or not, the fact is that it depends on broadcast. The HPC case externalizes the memory on a host to other hosts through RDMA. Everything to do with Big Data will rely on this, and besides disk externalization we will also see increase in memory externalization. I just went to a IEEE HPC cloud conference :-)

L3 assumes that all data (disk and memory) is inside the host. That assumption changed a decade ago. Cloud just increases the distribution. Just as compute is optimized by muxing the VMs on a physical machine, storage is optimized by consolidating data separately on storage devices and memory by hosting it in different hosts. Network has to deal with all types of traffic and security/isolation (imagine someone changing memory over the network). Storage and memory traffic uses L2. To discover each other, hosts use L2. These L2 domains need to span across sites and within the site.

This are not corner cases. This is just central to what cloud means in the long run. 

Thanks, Ashish

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Xuxiaohu
Sent: Friday, December 23, 2011 12:09 PM
To: Eggert, Lars; [email-masked]; [email-masked]
Cc: [email-masked]
Subject: Re: [armd] IP over IP solution for data center interconnect

Best regards,
Xiaohu
 
> geo-clustering is not needed?

_______________________________________________
armd mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/armd