Subject: [Int-area] FW: New Version Notification for draft-nachum-sarp-05.txt

From: Linda Dunbar <[email-masked]>
Date: Thu, 11 Jul 2013 21:49:40 +0000

draft-nachum-sarp has been presented at InterArea WG 3 times. As the motivation for the scheme wasn't clearly stated, many people in the WG have been debating on the necessity of the scheme during the meeting and over the mailing list. 

We updated the draft to address the issues that WG have brought up: http://www.ietf.org/internet-drafts/draft-nachum-sarp-05.txt

The proposed scheme is more like proxy gateway to address the following issues facing Data Centers with large number virtual machines that don't have fixed association with access switches:

1)	intermediate switches’ MAC address table (FDB) explosion:
When hosts in a VLAN (or subnet) span across many locations (or Access Switches) and each Access Switch has many VLANs enabled, the Access switches are exposed to all MAC addresses among all the VLANs enabled. 
For example, for an Access switch with 40 physical servers attached, and each server has 100 VMs, there are 4000 hosts under the Access Switch. If it is truly that hosts/VMs can be moved anywhere, the worst case for the Access Switch is when all those 4000 VMs belong to different VLANs, i.e. the access switch has 4000 VLANs enabled. If each of VLANs has 200 hosts, this access switch’s MAC table potentially has 200*4000 = 800,000 entries. 
Although above case is rare, but could happen regardless IPv4 or IPv6. 
This is worse than what today’s L2/3 Gateway has to face. In today’s environment where each subnet is limited to a few access switches, only the limited ports facing those access switches are exposed to MAC address of the subnet. 

2)	the ARP/ND processing load impact on the L2/L3 boundary routers; 
All VMs periodically send NDs to their corresponding Gateway nodes to get gateway nodes’ MAC addresses. When the combined VMs across all the VLANs are large, processing the responses to the ND requests from those VMs can easily knock down the gateway’s CPU utilization. 
A L2/L3 boundary router could be hit with ARP/ND twice when the originating and destination stations are in different subnets attached to the same router and when those hosts don’t communicate with external peers very often. The first hit is when the originating station in subnet-A initiates ARP/ND request to the L2/L3 boundary router if the router’s MAC is not in the host’s cache; and the second hit is when the L2/L3 boundary router initiates ARP/ND requests to the target in subnet-B if the target is not in router’s ARP/ND cache.

3)	In IPv4, every end station in a subnet receives ARP broadcast messages from all other end stations in the subnet. IPv6 ND has eliminated this issue by using multicast.  
However, most devices have limited number of Multicast filtering. Once the number of multicast addresses goes beyond the multicast filter limit, the multicast addresses have to be processed by devices’ CPU (i.e. the slow path). 
It is less of an issue in DC without VM mobility because each port is only dedicated to one (or a few number of) VLANs. So the number of multicast addresses hitting each port is much less. 

4)	the ARP/ND messages being flooded to many physical link segments which can reduce bandwidth utilization for user traffic; 

ARP/ND flooding is probably an insignificant issue in today’s data center because the majority of data center servers are moving towards 1G or 10G ports. The bandwidth taken by ARP/ND, even with them flooded to all physical links, becomes negligible comparing with the link bandwidth. In addition, the IGMP/MLD snooping [RFC4541] can further reduce the ND multicast traffic to some physical link segments.

Hope those points are more clear. 

Linda 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Thursday, July 11, 2013 3:13 PM
To: Linda Dunbar; Youval Nachum; Ilan Yerushalmi; Tal Mizrahi
Subject: New Version Notification for draft-nachum-sarp-05.txt

A new version of I-D, draft-nachum-sarp-05.txt
has been successfully submitted by Youval Nachum and posted to the
IETF repository.

Filename:	 draft-nachum-sarp
Revision:	 05
Title:		 Scaling the Address Resolution Protocol for Large Data Centers (SARP)
Creation date:	 2013-07-11
Group:		 Individual Submission
Number of pages: 20
URL:             http://www.ietf.org/internet-drafts/draft-nachum-sarp-05.txt
Status:          http://datatracker.ietf.org/doc/draft-nachum-sarp
Htmlized:        http://tools.ietf.org/html/draft-nachum-sarp-05
Diff:            http://www.ietf.org/rfcdiff?url2=draft-nachum-sarp-05

Abstract:
   This document introduces SARP, an architecture that uses proxy
   gateways and allows to scale data center networks. SARP is based on
   fast proxies that significantly reduce switches' FDB (MAC table)
   sizes and ARP/ND impact on network elements in an environment that
   hosts within one subnet (or VLAN) can spread over various locations.
   SARP supports smooth and fast virtual machine (VM) mobility without
   any modification to the VM, while keeping the connection up and
   running efficiently.  SARP is targeted for massive scaling data
   centers with a significant number of VMs that can move across various
   physical locations.

The IETF Secretariat

---

From: Ted Lemon <[email-masked]>
Date: Thu, 11 Jul 2013 22:37:33 +0000

How does this relate to the work being done in the trill and nvo3 working groups?

---

From: Linda Dunbar <[email-masked]>
Date: Thu, 11 Jul 2013 22:48:24 +0000

Ted, 

Glad you ask this question. 

TRILL and NVO3 provides an overlay mechanism so that the hosts (or VMs) addresses are hidden from the switches in the core. 

However, all the overlay edge switches are still exposed to all the hosts addresses on the VLANs that are enabled on the edge. If the number of VMs attached to an edge is very small (e.g. overlay edge in the server), then the number of VLANs enabled on the edge is limited. In this scenario, there could be very large number of edges for large DC, which sometimes makes IGP challenging. 

For a typical Access switch (Top of Rack)  with 40 physical servers attached, where each server has 100 VMs, there are 4000 hosts under the Access Switch. If indeed hosts/VMs can be moved anywhere, the worst case for the Access Switch is when all those 4000 VMs belong to different VLANs, i.e. the access switch has 4000 VLANs enabled. If each VLAN has 200 hosts, this access switch's MAC table potentially has 200*4000 = 800,000 entries. 
It is important to note that the example above is relevant regardless of whether IPv4 or IPv6 are used. 
The example illustrates a scenario that is worse than what today's L2/3 Gateway has to face. In today's environment where each subnet is limited to a few access switches, the number of MAC addresses the gateway has to learn is of a significantly smaller scale.

Linda 

> From: Ted Lemon [mailto:[email-masked]]
> Sent: Thursday, July 11, 2013 5:38 PM
> To: Linda Dunbar
> Cc: Internet Area; Suresh Krishnan
> Subject: Re: [Int-area] FW: New Version Notification for draft-nachum-
> sarp-05.txt
>

---

From: Ted Lemon <[email-masked]>
Date: Thu, 11 Jul 2013 23:02:44 +0000

> However, all the overlay edge switches are still exposed to all the hosts addresses on the VLANs that are enabled on the edge. If the number of VMs attached to an edge is very small (e.g. overlay e…

True.   However, they do have various strategies for addressing this problem, and I was wondering how you think your proposal fits in.   For example, the trill working group recently submitted a document for publication that addresses the issue of small, scattered virtual links with frequently migrating VMs in a very large campus using a directory service.   I reviewed an nvo3 problem statement document for the telechat two weeks ago that talked about the same issue as well.

---

From: Linda Dunbar <[email-masked]>
Date: Fri, 12 Jul 2013 15:28:37 +0000

Ted, 

You have really good questions. 

Answers to your questions are inserted below: 

> True.   However, they do have various strategies for addressing this
> problem, and I was wondering how you think your proposal fits in.   For
> example, the trill working group recently submitted a document for
> publication that addresses the issue of small, scattered virtual links
> with frequently migrating VMs in a very large campus using a directory
> service.   

[Linda] TRILL Directory Service, which I am one of the authors, allows ingress edge nodes to resolve the target egress edge node(s) without flooding data frames (unknown DA or ARP/ND) across the core. The SARP mechanism is to reduce the amount of hosts (VMs) addresses exposed to edge nodes: all remote hosts (VMs) are represented by their corresponding GW. 

Directory Service can make SARP run more smoothly. When remote VMs are moved/added/deleted, the Directory Service can quickly inform the relevant edge nodes on the change. 

I reviewed an nvo3 problem statement document for the
> telechat two weeks ago that talked about the same issue as well.

[Linda] If NVO3 edges (NVE) are on Top of Rack switch or End of Row switch and there are switches between hosts (VMs) and Overlay Edges, all those switches/overlay edges will also run into MAC table explosion problem. 

I once saw a NvGRE presentation by Microsoft which has a section on overlay edge node behaving as proxy Gateway for all the hosts (VMs) attached. SARP is along the same principle, i.e. making VMs use their corresponding Edge node's MAC address as MAC-DA instead of the MAC address of their Layer 2 peers' that are attached to different edge nodes. 

Please ask more questions!  It will help us to make the draft more clear. 

Linda

---

From: Ted Lemon <[email-masked]>
Date: Sun, 14 Jul 2013 18:08:39 +0000

> You have really good questions. 

What these are are questions by your AD who is trying to figure out how this stuff all fits together.   It's kind of you to say that they are good questions, but really I'm just trying to fill in knowledge of things about which I am fairly ignorant.

The question I was trying to answer was how this differs from the techniques described in the two drafts to which I was referring.   I had read the document fairly carefully at the beginning, but didn't really understand what it was trying to do, and it seemed to be doing the same thing that the trill and nvo3 drafts are talking about, hence the questions.   I hadn't noticed that you were a co-author of the trill draft—sorry about that.   I notice now that you also refer to the nvo3 draft in this draft.

So one thing that I think this draft fails to do right now that you need to fix is that I didn't actually know what technique was being described in the draft until reading section 3.2.   It would really help to have something in the introduction that says something like "this document describes a technique for reducing the size of arp tables on L2/L3 switches by substituting the MAC address of the switch to which a host is connected for the MAC address of the host when forwarding packets to other switches hosting the same VLAN to which the source host is connected."

I realize that this text isn't exactly right, but something like this would help.   It would really help to have two or three paragraphs after the initial section 1 header that talk very briefly about the motivation for the work, and then describe the technique at the level of overview that I used in my example.   Having to get through a page and a half of motivation, a page and a half of deployment scenario, and still not know what the draft is actually going to tell me how to do makes it difficult to get motivated to go any deeper.

The draft also over-uses abbreviations.   For instance:

   When a VM transmits packets to a destination VM that is located at
   the same site, there is no change in the data plane. The packets are
   sent from (IP-S, MAC-S) to (IP-D, MAC-D).

Why is it necessary to abbreviate like this?  Why not say:

   When a VM transmits packets to a destination VM that is located at
   the same site, there is no change in the data plane. The packets are
   sent from the source host IP and MAC address to the destination host
   IP and MAC address?

Or if you want to keep the tuples:

   When a VM transmits packets to a destination VM that is located at
   the same site, there is no change in the data plane. The packets are
   sent from (IP Source, MAC Source) to (IP Dest, MAC Dest).

This isn't all that important in the example I'm giving, but it was a bit painful to read the rest of this section because of all the abbreviations.

It would also help the readability of this document prior to its arrival at the RFC Editor, assuming it gets there, if you could clean up some of the indentation problems in the introduction—these make the document look rough, which again discourages reading it.   Using xml2rfc hanging indent for the numbered items in 1.1 and making sure there's a blank line between paragraphs would neaten it up.   I realize this is a really minor nit, but when you're asking for review making your document as clean as possible really helps.

Anyway, I think the answer to the main question that I was asking about this draft has been answered.   The question I was trying to figure out was why you were proposing this in intarea and not trill or nvo3; the answer, I presume, is that the solution applies to both?

---

From: Linda Dunbar <[email-masked]>
Date: Mon, 15 Jul 2013 14:18:55 +0000

Ted, 

Thank you very much for the suggested text. 
I tweaked a little bit. How about the following text added to the Introduction section?

"This document describes a proxy gateway technique, which is called SARP throughout this document, to reduce switches' FDB (MAC table) sizes and ARP/ND impact on network elements in an environment where hosts within one subnet (or VLAN) can spread over various access domains in data centers. 

The main idea of SARP is to make all VMs (or hosts) under each access domain be represented by their corresponding access (or aggregation) node's MAC address regardless if the access (or aggregation) node is the VMs (hosts)' gateway or not. For example, when a host "a" under access domain "S" needs to communicate with peers in the same VLAN but located in other access domains, SARP requires "a" to use remote access node's MAC address instead of their remote Layer 2 peer's MAC address. By doing so, all the switches' FDB don't need to keep remote hosts MAC addresses."   

> "this document describes a technique for reducing the size of arp
> tables on L2/L3 switches by substituting the MAC address of the switch
> to which a host is connected for the MAC address of the host when
> forwarding packets to other switches hosting the same VLAN to which the
> source host is connected."

> 
> Anyway, I think the answer to the main question that I was asking about
> this draft has been answered.   The question I was trying to figure out
> was why you were proposing this in intarea and not trill or nvo3; the
> answer, I presume, is that the solution applies to both?

[Linda] Yes, the proposed solution applies to both. 

Thank you. 

Linda

---

From: Ted Lemon <[email-masked]>
Date: Mon, 15 Jul 2013 15:17:10 +0000

> The main idea of SARP is to make all VMs (or hosts) under each access domain be represented by their corresponding access (or aggregation) node's MAC address regardless if the access (or aggregatio…

How about:

   This document describes a proxy gateway technique called Scalable               
   Address Resolution Protocol (SARP), which reduces switches' Friends             
   Database (FDB) table size and limits and ARP/Neighbor Discovery                 
   impact on network elements in an environment where hosts within one          
   subnet (or VLAN) can spread over various access domains in data                                 
   centers.                                                                     

   The main idea of SARP is to represent all VMs (or hosts) under each             
   access domain by their corresponding access (or aggregation) node's             
   MAC address, regardless of whether the access (or aggregation) node             
   is the VMs (hosts)' gateway or not. For example, when a host "a"             
   under access domain "S" needs to communicate with a peer on the              
   same VLAN but connected to a different access domain, SARP requires          
   "a" to use the remote access node's MAC address, rather than the             
   peer's MAC address. By doing so, each switch in the VLAN does not
   need to maintain a list of MAC addresses for all each node on the 
   VLAN in its FDB.

I don't know what FDB stands for, and the acronym is never expanded in the document, so I just said "Friends Database" but obviously you should substitute in whatever FDB _actually_ stands for.    You should make sure that there are no unexpanded acronyms in the document.   Even if acronyms are generally understood by people who are working on the document, excessive use of acronyms and use of unexpanded acronyms creates an very high barrier to entry for new readers of a document.

BTW, can you speak briefly to the question of how this actually saves significant space in an FDB?   The reason I ask is that it seems to me that you still need to keep a complete list of all IP addresses in the FDB, even if they refer to the same MAC address.   So this represents a space savings of maybe 27%, assuming a very efficient data representation.   The win is much bigger for IPv4 addresses, but it's still only 60%.

---

From: Linda Dunbar <[email-masked]>
Date: Mon, 15 Jul 2013 16:02:33 +0000

Ted, 

Thank you very much for the suggested text. We will change the draft before uploading today. We will also spell out all the acronyms and reduce using them as much as possible. 

FDB: Filtering Data Base, which is a term used for Layer 2 switches (IEEE802.1Q). 
Layer 2 switches flood data frames when MAC-DA is not in FDB, whereas routers drop data frames when the DA is not in the Forwarding Information Base (FIB). 

That is why Filtering Data Base (FDB) is used for Layer 2 switches. 

> BTW, can you speak briefly to the question of how this actually saves
> significant space in an FDB?   The reason I ask is that it seems to me
> that you still need to keep a complete list of all IP addresses in the
> FDB, even if they refer to the same MAC address.   So this represents a
> space savings of maybe 27%, assuming a very efficient data
> representation.   The win is much bigger for IPv4 addresses, but it's
> still only 60%.

[Linda] Good question. When remote VMs (hosts) are represented by their corresponding access (or aggregation) node's MAC address, all switches only need to keep the MAC addresses of their local VMs (hosts) and the MAC addresses of remote access nodes in their FDB. They don't need to keep the MAC addresses of remote VMs (hosts). That is how all switches' FDB size is significantly reduced no matter how wide VLANs are spread. 

The SARP scheme also includes MAC address translation at egress gateway. I.e. when an egress SARP gateway receives a data frame with destination address equal to itself, the egress gateway will replace the destination MAC to the local host's MAC based on the inner IP destination address. 

Hope it helps. 

Linda

---

From: Ted Lemon <[email-masked]>
Date: Mon, 15 Jul 2013 16:05:38 +0000

> [Linda] Good question. When remote VMs (hosts) are represented by their corresponding access (or aggregation) node's MAC address, all switches only need to keep the MAC addresses of their local VMs…

The name "filtering database" clarified it, actually.   You don't need to store the IP addresses at all, right?

---

From: Linda Dunbar <[email-masked]>
Date: Mon, 15 Jul 2013 16:07:47 +0000

> to store the IP addresses at all, right?

[Linda] Correct.