Subject: [armd] my understanding of ARMD - comments/corrections welcomed

From: Bob Sultan <[email-masked]>
Date: Thu, 09 Dec 2010 09:44:27 -0500

ARMD participants:

I have been following the ARMD discussions on the mailing list.  I
participate in the 802.1 Data Center Bridging group but don't generally
attend IETF.  However, I do think that the burden of ARP processing the Data
Center is a significant problem that deserves a robust standard solution
interoperable across equipment from multiple vendors.  I wanted to make sure
that my understanding of the problem to be solved by the ARMD group is
correct, so I created the description below.  Comments/corrections, even
minor ones, are invited. or if you think this description is really off in
left-field, feel free to provide an alternative.  

Thanks,

Bob Sultan

My understanding of the problem to be solved by ARMD

A subnetwork within a Data Center Network or other dense network may be
required to support very large numbers of attached hosts.  Alternatively, a
region of a network may be required to support significant numbers of
subnetworks, where each subnetwork supports significant numbers of hosts.
The number of hosts may be particularly large when the hosts are
instantiated as virtual machines.  Processing of conventional
broadcast-based ARP (rfc 826) in such subnetworks can place excessive load
on the processor associated with a host or a gateway.  The problem is
exacerbated when multiple hosts share processor resources, as when the hosts
are instantiated as virtual machines.  ARMD is intended to reduce processor
load to acceptable levels in such environments.  Convincing statistical
evidence for the existence of this problem is provided by Figure 4 and
accompanying text on page 4 of
http://www.cs.cmu.edu/~hzhang/papers/hotnets04-ethernet.pdf. Although the
primary problem is that of excessive processor load, a secondary problem is
that of network bandwidth consumption due to broadcast and/or flooding.  

NOTE:  I'm using the word 'gateway' here to mean 'router' which I think is
standard IETF terminology.  I have included 'gateways' (i.e. routers) in the
problem statement because these devices might be most vulnerable to the ARP
processing burden.  The hosts (servers) are designed to handle lots of
processing, but the routers may be more limited in their processing
capabilities than servers and their processors must additionally support a
wide variety of routing control functions.

---

From: Linda Dunbar <[email-masked]>
Date: Mon, 13 Dec 2010 17:06:36 -0600

Thanks to Bob for the clear wording of the problem statements. 

Any more comments? 

I plan to re-word the Charter Statement based on many emails I have received
so far. Hopefully we can reach consensus when the New Year arrives. 

Happy holiday to you and your family!

Best Regards, Linda Dunbar

  _____  

From: [email-masked] [mailto:[email-masked]] On Behalf Of Bob
Sultan
Sent: Thursday, December 09, 2010 8:44 AM
To: [email-masked]
Cc: 'Bob Sultan'
Subject: [armd] my understanding of ARMD - comments/corrections welcomed

ARMD participants:

Thanks,

Bob Sultan

---

From: Anoop Ghanwani <[email-masked]>
Date: Tue, 14 Dec 2010 10:40:36 -0800

I'm not sure if I agree that the bandwidth issue is secondary.
One may choose to have subnets that are no more larger than today's
subnets in which case the issue of CPU utilization for broadcast
processing doesn't exist.  However, simply by virtue of stretching
subnets across the entire data center, one will run into the bandwidth
issue.

So I think both problems should be addressed, but I'm actually
more interest in a solution for the bandwidth utilization problem.

Anoop

From: [email-masked] [mailto:[email-masked]] On Behalf Of Linda Dunbar
Sent: Monday, December 13, 2010 3:07 PM
To: 'Bob Sultan'; [email-masked]
Subject: Re: [armd] my understanding of ARMD - comments/corrections welcomed

Any more comments?

I plan to re-word the Charter Statement based on many emails I have received so far. Hopefully we can reach consensus when the New Year arrives.

________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Bob Sultan
Sent: Thursday, December 09, 2010 8:44 AM
To: [email-masked]
Cc: 'Bob Sultan'
Subject: [armd] my understanding of ARMD - comments/corrections welcomed

ARMD participants:

I have been following the ARMD discussions on the mailing list.  I participate in the 802.1 Data Center Bridging group but don't generally attend IETF.  However, I do think that the burden of ARP processing the Data Center is a significant problem that deserves a robust standard solution interoperable across equipment from multiple vendors.  I wanted to make sure that my understanding of the problem to be solved by the ARMD group is correct, so I created the description below.  Comments/corrections, even minor ones, are invited... or if you think this description is really off in left-field, feel free to provide an alternative.

Thanks,
Bob Sultan

A subnetwork within a Data Center Network or other dense network may be required to support very large numbers of attached hosts.  Alternatively, a region of a network may be required to support significant numbers of subnetworks, where each subnetwork supports significant numbers of hosts.  The number of hosts may be particularly large when the hosts are instantiated as virtual machines.  Processing of conventional broadcast-based ARP (rfc 826) in such subnetworks can place excessive load on the processor associated with a host or a gateway.  The problem is exacerbated when multiple hosts share processor resources, as when the hosts are instantiated as virtual machines.  ARMD is intended to reduce processor load to acceptable levels in such environments.  Convincing statistical evidence for the existence of this problem is provided by Figure 4 and accompanying text on page 4 of http://www.cs.cmu.edu/~hzhang/papers/hotnets04-ethernet.pdf. Although the primary problem is that of excessive processor load, a secondary problem is that of network bandwidth consumption due to broadcast and/or flooding.

NOTE:  I'm using the word 'gateway' here to mean 'router' which I think is standard IETF terminology.  I have included 'gateways' (i.e. routers) in the problem statement because these devices might be most vulnerable to the ARP processing burden.  The hosts (servers) are designed to handle lots of processing, but the routers may be more limited in their processing capabilities than servers and their processors must additionally support a wide variety of routing control functions.

---

From: Bob Sultan <[email-masked]>
Date: Wed, 15 Dec 2010 13:08:33 -0500

Hi Anoop,

You make a good point.  My intuition has been that the arp hit to processor
resources is more significant than the arp hit to link bandwidth.  The cmu
paper makes the argument in terms bandwidth consumption, supporting your
assertion.  I don't have the evidence to conclude that one problem is more
significant than the other.  It seems reasonable that both problems are
within the scope of ARMD.  So, I propose the following change to what I
previously wrote:

A subnetwork within a Data Center Network or other dense network may be
required to support very large numbers of attached hosts.  Alternatively, a
region of a network may be required to support significant numbers of
subnetworks, where each subnetwork supports significant numbers of hosts.
The number of hosts may be particularly large when the hosts are
instantiated as virtual machines.  Use of conventional broadcast-based ARP
(rfc 826) in such subnetworks can result in excessive bandwidth consumption
on links within the subnetworks.  Similarly, processing of broadcast-based
ARP can place excessive load on the processor associated with a host or a
gateway.  The problem of excessive processor load is exacerbated when
multiple hosts share processor resources, as when the hosts are instantiated
as virtual machines.  ARMD is intended to reduce link bandwidth consumption
and processor load to acceptable levels in such environments.  Convincing
statistical evidence for the existence of this problem is provided by Figure
4 and the accompanying text on page 4 of
http://www.cs.cmu.edu/~hzhang/papers/hotnets04-ethernet.pdf. 

Regards,

Bob

  _____  

From: Anoop Ghanwani [mailto:[email-masked]] 
Sent: Tuesday, December 14, 2010 1:41 PM
To: Linda Dunbar; 'Bob Sultan'; [email-masked]
Subject: RE: [armd] my understanding of ARMD - comments/corrections welcomed

issue.

Anoop

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Linda Dunbar
Sent: Monday, December 13, 2010 3:07 PM
To: 'Bob Sultan'; [email-masked]
Subject: Re: [armd] my understanding of ARMD - comments/corrections welcomed

Any more comments? 

  _____  

ARMD participants:

Thanks,

Bob Sultan

---

From: Anoop Ghanwani <[email-masked]>
Date: Wed, 15 Dec 2010 10:34:08 -0800

intuition has been that the arp hit to processor resources is more significant than the arp hit to link bandwidth.  

That is certainly true of networks today because:
(a) They run IPv4
(b) They are not designed in such as way that 
    subnets stretch across the data center.

When running IPv6, the CPU utilization problem
will be mitigated because adapters can filter the
uninteresting multicasts in hardware.

The paper you cite would not have been able
to see the bandwidth utilization problem because
they were studying data centers as they are
architected today (or rather at the time that
the paper was written).

Anoop

----

From: Bob Sultan [mailto:[email-masked]] 
Sent: Wednesday, December 15, 2010 10:09 AM
To: Anoop Ghanwani
Cc: [email-masked]
Subject: RE: [armd] my understanding of ARMD - comments/corrections welcomed

Hi Anoop,

You make a good point.  My intuition has been that the arp hit to processor resources is more significant than the arp hit to link bandwidth.  The cmu paper makes the argument in terms bandwidth consumption, supporting your assertion.  I don't have the evidence to conclude that one problem is more significant than the other.  It seems reasonable that both problems are within the scope of ARMD.  So, I propose the following change to what I previously wrote:

A subnetwork within a Data Center Network or other dense network may be required to support very large numbers of attached hosts.  Alternatively, a region of a network may be required to support significant numbers of subnetworks, where each subnetwork supports significant numbers of hosts.  The number of hosts may be particularly large when the hosts are instantiated as virtual machines.  Use of conventional broadcast-based ARP (rfc 826) in such subnetworks can result in excessive bandwidth consumption on links within the subnetworks.  Similarly, processing of broadcast-based ARP can place excessive load on the processor associated with a host or a gateway.  The problem of excessive processor load is exacerbated when multiple hosts share processor resources, as when the hosts are instantiated as virtual machines.  ARMD is intended to reduce link bandwidth consumption and processor load to acceptable levels in such environments.  Convincing statistical evidence for the existence of this problem is provided by Figure 4 and the accompanying text on page 4 of http://www.cs.cmu.edu/~hzhang/papers/hotnets04-ethernet.pdf. 

Regards,
Bob

________________________________________

Anoop

Any more comments? 

________________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Bob Sultan
Sent: Thursday, December 09, 2010 8:44 AM
To: [email-masked]
Cc: 'Bob Sultan'
Subject: [armd] my understanding of ARMD - comments/corrections welcomed

ARMD participants:

I have been following the ARMD discussions on the mailing list.  I participate in the 802.1 Data Center Bridging group but don't generally attend IETF.  However, I do think that the burden of ARP processing the Data Center is a significant problem that deserves a robust standard solution interoperable across equipment from multiple vendors.  I wanted to make sure that my understanding of the problem to be solved by the ARMD group is correct, so I created the description below.  Comments/corrections, even minor ones, are invited. or if you think this description is really off in left-field, feel free to provide an alternative.  

Thanks,
Bob Sultan

---

From: Bob Sultan <[email-masked]>
Date: Wed, 15 Dec 2010 14:03:20 -0500

Hi Anoop,

So, unless I'm mistaken, we're in agreement.  I was saying that my
'intuition' (vs. evidence) was that processor cycles is the significant
problem (at least in v4), but the evidence presented in the cmu paper
suggests that the problem is bandwidth (i.e., the paper agrees with you).
The cmu paper cites experimental results and extrapolates them to larger
data centers.

So the real question is, does the revised statement below (copied from my
last email) describe your understanding of the problem?  The revision says
that bandwidth and/or processor resources can cause a problem.  It lists
bandwidth first, but it does not make any claim about which is more
significant.

-------------------------

---------------------------------------

Regards,

Bob

-----Original Message-----
From: Anoop Ghanwani [mailto:[email-masked]] 
Sent: Wednesday, December 15, 2010 1:34 PM
To: Bob Sultan
Cc: [email-masked]
Subject: RE: [armd] my understanding of ARMD - comments/corrections welcomed

intuition has been that the arp hit to processor resources is more
significant than the arp hit to link bandwidth.  

(a) They run IPv4

the paper was written).

Anoop

----

To: Anoop Ghanwani

Cc: [email-masked]

Hi Anoop,

Regards,

Bob

issue.

Anoop

Any more comments? 

To: [email-masked]

Cc: 'Bob Sultan'

ARMD participants:

Thanks,

Bob Sultan

---

From: Anoop Ghanwani <[email-masked]>
Date: Wed, 15 Dec 2010 11:08:00 -0800

Bob,

I think it fails to mention that a subnet may have members distributed
across the data center (rather than localized to, e.g. one ToR or
one aggregation switch) which could result in excessive broadcast/
multicast traffic in the data center backbone.

Anoop

From: Bob Sultan [mailto:[email-masked]]
Sent: Wednesday, December 15, 2010 11:03 AM
To: Anoop Ghanwani
Cc: [email-masked]
Subject: RE: [armd] my understanding of ARMD - comments/corrections welcomed

Hi Anoop,

So, unless I'm mistaken, we're in agreement.  I was saying that my 'intuition' (vs. evidence) was that processor cycles is the significant problem (at least in v4), but the evidence presented in the cmu paper suggests that the problem is bandwidth (i.e., the paper agrees with you).  The cmu paper cites experimental results and extrapolates them to larger data centers.

So the real question is, does the revised statement below (copied from my last email) describe your understanding of the problem?  The revision says that bandwidth and/or processor resources can cause a problem.  It lists bandwidth first, but it does not make any claim about which is more significant.

Regards,
Bob

-----Original Message-----
From: Anoop Ghanwani [mailto:[email-masked]]
Sent: Wednesday, December 15, 2010 1:34 PM
To: Bob Sultan
Cc: [email-masked]
Subject: RE: [armd] my understanding of ARMD - comments/corrections welcomed

(a) They run IPv4

the paper was written).

Anoop

----

To: Anoop Ghanwani

Cc: [email-masked]

Hi Anoop,

Regards,

Bob

issue.

Anoop

Any more comments?

To: [email-masked]

Cc: 'Bob Sultan'

ARMD participants:

Thanks,

Bob Sultan

---

From: Bob Sultan <[email-masked]>
Date: Wed, 15 Dec 2010 15:59:34 -0500

Anoop,

I agree.  I added a sentence to cover this.  

I also made a minor change to the sentence preceding the added sentence.  I
changed "on links within the subnetworks" to "on links associated with the
subnetworks".  The original version seems to imply that the link is
dedicated to a particular subnetwork.  'associated with' doesn't imply this
and allows for the most common case in which a link is shared by multiple
subnets (e.g., VLANs).   

I also changed the sentence following the added sentence to make it clear
that the 'similarity' is to the sentence "Use of conventional
broadcast-based ARP (rfc 826) in such subnetworks can result in excessive
bandwidth consumption on links associated with the subnetworks", not the
newly added sentence.

Feel free to suggest other changes in wording.

Thanks,

Bob

A subnetwork within a Data Center Network or other dense network may be
required to support very large numbers of attached hosts.  Alternatively, a
region of a network may be required to support significant numbers of
subnetworks, where each subnetwork supports significant numbers of hosts.
The number of hosts may be particularly large when the hosts are
instantiated as virtual machines.  Use of conventional broadcast-based ARP
(rfc 826) in such subnetworks can result in excessive bandwidth consumption
on links associated with the subnetworks.  The problem of link bandwidth
consumption may be particularly severe in the backbone portion of a network,
as in the case of a backbone interconnecting Top of Rack bridges in a Data
Center Network, when subnetworks extend across the backbone. Similar to the
problem of excessive bandwidth consumption, processing of broadcast-based
ARP can place excessive load on the processor associated with a host or a
gateway.  The problem of excessive processor load is exacerbated when
multiple hosts share processor resources, as when the hosts are instantiated
as virtual machines.  ARMD is intended to reduce link bandwidth consumption
and processor load to acceptable levels in such environments.  Convincing
statistical evidence for the existence of this problem is provided by Figure
4 and the accompanying text on page 4 of
http://www.cs.cmu.edu/~hzhang/papers/hotnets04-ethernet.pdf. 

  _____  

From: Anoop Ghanwani [mailto:[email-masked]] 
Sent: Wednesday, December 15, 2010 2:08 PM
To: Bob Sultan
Cc: [email-masked]
Subject: RE: [armd] my understanding of ARMD - comments/corrections welcomed

Bob,

Anoop

Hi Anoop,

Regards,

Bob

-----Original Message-----
From: Anoop Ghanwani [mailto:[email-masked]] 
Sent: Wednesday, December 15, 2010 1:34 PM
To: Bob Sultan
Cc: [email-masked]
Subject: RE: [armd] my understanding of ARMD - comments/corrections welcomed

(a) They run IPv4

the paper was written).

Anoop

----

To: Anoop Ghanwani

Cc: [email-masked]

Hi Anoop,

Regards,

Bob

issue.

Anoop

Any more comments? 

To: [email-masked]

Cc: 'Bob Sultan'

ARMD participants:

Thanks,

Bob Sultan

---

From: Linda Dunbar <[email-masked]>
Date: Thu, 16 Dec 2010 14:54:31 -0600

Actually there is one more problem caused by today's ARP besides causing
extra processing load to servers (IPv4) and overtaking link bandwidth (IPv4
& v6): 

When VMs are spread across multiple TOR switches, all the ARP broadcast
messages and ND multicast messages are sent to all TOR switches in the
broadcast domain. This will cause TOR switch to refresh its FDB even for the
MAC not used by its local hosts. Most likely, the frequency of ARP/ND by
hosts is shorter then TOR FDB MAC aging time. That means TOR FDB will have
all the MAC addresses even if most of them are not used by local hosts. 

When VMs migrate, the number of broadcast domains enabled on each TOR will
increase. The TOR FDB has to hold all MACs on all broadcast domains enabled
on the TOR. In large data centers, the number can be very large. 

Linda Dunbar

  _____  

From: [email-masked] [mailto:[email-masked]] On Behalf Of Bob
Sultan
Sent: Wednesday, December 15, 2010 3:00 PM
To: 'Anoop Ghanwani'
Cc: [email-masked]
Subject: Re: [armd] my understanding of ARMD - comments/corrections welcomed

Anoop,

Thanks,

Bob

 

  _____  

 

Bob,

Anoop

Hi Anoop,

Regards,

Bob

-----Original Message-----
From: Anoop Ghanwani [mailto:[email-masked]] 
Sent: Wednesday, December 15, 2010 1:34 PM
To: Bob Sultan
Cc: [email-masked]
Subject: RE: [armd] my understanding of ARMD - comments/corrections welcomed

(a) They run IPv4

the paper was written).

Anoop

----

To: Anoop Ghanwani

Cc: [email-masked]

Hi Anoop,

Regards,

Bob

issue.

Anoop

Any more comments? 

To: [email-masked]

Cc: 'Bob Sultan'

ARMD participants:

Thanks,

Bob Sultan