Subject: [armd] ARMD BOF follow up

From: Linda Dunbar <[email-masked]>
Date: Tue, 16 Nov 2010 15:21:56 -0600

Thanks to everyone who participated in the ARMD BOF discussion last Friday.
There were two major comments we heard from the audience. One comment is
wishing to see some statistics (by Internet AD Ralph), and another comment
is suggesting that we should go to "Data Center Operation" working group to
discuss with other "data center operation issues". 

Suggestion 1: Need ARP statistics: 

At the BOF, I forgot to mention that ARP statistics was quoted in our first
problem statement draft for ARP222 (back when it was only at the bar BOF
stage).  As time goes on, we modified the draft several times by several
authors to include more important stuff, those numbers get eliminated from
the draft. 

The study done by CMU shows
(http://www.cs.cmu.edu/~hzhang/papers/hotnets04-ethernet.pdf): 

the number of ARP queries received at a workstation on CMU's School of
Computer Science LAN over a 12 hour period on August 9, 2004. At peak, the
host received 1150 ARPs per second, and on average, the host received 89
ARPs per second, which corresponds to 45 kbps of traffic. During the data
collection, 2,456 hosts were observed sending ARP queries. 

CMU expected that the amount of ARP traffic will scale linearly with the
number of hosts on the LAN. For 1 million hosts, they would expect 468,240
ARPs per second or 239 Mbps of ARP traffic to arrive at each host at peak,
which is more than enough to overwhelm a standard 100 Mbps LAN connection.
Ignoring the link capacity, forcing hosts to handle an extra half million
packets per second to inspect each ARP packet would impose a prohibitive
computational burden.

As Cisco's Claudio DeSanti pointed out during the BOF, large number of
client subnets is effectively causing network to have the same problem as
one large broadcast domain. 

Question: What other statistics do people would like to see before being
convinced that ARP optimization is indeed necessary to make network scale to
support next gen mega data center? 

Suggestion 2: To lump the ARP improvement discussion with Data Center
Operation working group (by Ross Callon)

Chatting with Ross Callon after the BOF, I learnt that Ross is referring to
the potential Cloud Operation group (as the result of Cloud bar BOF). 

 Actually, I initially did present the ARP issue at the Cloud bar BOF at the
78th IETF. But I learned that the audience for Cloud is too diverse to
converge on network problems. The problems which Cloud bar BOF has presented
cover Applications APIs, Logging, Security, etc. They are not even on
network problems. 

I am afraid of mixing network problems with application problems will just
get many people more confused. I remember during 78th IETF Cloud Bar BOF,
the people who focus on Application APIs think network problem is too
trivial to be even discussed at the same place with cloud applications
problems. 

Question: Should we go towards this direction? 

What other refinement we need to consider? 

Thanks, Linda Dunbar

---

From: "So, Ning" <[email-masked]>
Date: Tue, 16 Nov 2010 21:33:12 +0000

Linda,

I concur with your concern on your suggestion #2.  Cloud Data Center WG
(name will likely change) and Cloud AppsBoF were both being proposed
during the meeting last week.  One potential solution is to have network
related ARMD drafts go to the new WG, while non-network related SRMD
drafts go to the new Cloud AppsBoF.  

Best regrads,

Ning So

Network Evolution Planning

Verizon, Inc.

(office) 972-729-7905

(Cell) 972-955-0914

________________________________

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Linda Dunbar
Sent: Tuesday, November 16, 2010 3:22 PM
To: [email-masked]
Cc: [email-masked]; [email-masked]; 'Jari Arkko'; 'Claudio
DeSanti'
Subject: [armd] ARMD BOF follow up

Thanks to everyone who participated in the ARMD BOF discussion last
Friday. There were two major comments we heard from the audience. One
comment is wishing to see some statistics (by Internet AD Ralph), and
another comment is suggesting that we should go to "Data Center
Operation" working group to discuss with other "data center operation
issues". 

At the BOF, I forgot to mention that ARP statistics was quoted in our
first problem statement draft for ARP222 (back when it was only at the
bar BOF stage).  As time goes on, we modified the draft several times by
several authors to include more important stuff, those numbers get
eliminated from the draft. 

the number of ARP queries received at a workstation on CMU's School of
Computer Science LAN over a 12 hour period on August 9, 2004. At peak,
the host received 1150 ARPs per second, and on average, the host
received 89 ARPs per second, which corresponds to 45 kbps of traffic.
During the data collection, 2,456 hosts were observed sending ARP
queries. 

CMU expected that the amount of ARP traffic will scale linearly with the
number of hosts on the LAN. For 1 million hosts, they would expect
468,240 ARPs per second or 239 Mbps of ARP traffic to arrive at each
host at peak, which is more than enough to overwhelm a standard 100 Mbps
LAN connection. Ignoring the link capacity, forcing hosts to handle an
extra half million packets per second to inspect each ARP packet would
impose a prohibitive computational burden.

As Cisco's Claudio DeSanti pointed out during the BOF, large number of
client subnets is effectively causing network to have the same problem
as one large broadcast domain. 

Question: What other statistics do people would like to see before being
convinced that ARP optimization is indeed necessary to make network
scale to support next gen mega data center? 

Chatting with Ross Callon after the BOF, I learnt that Ross is referring
to the potential Cloud Operation group (as the result of Cloud bar BOF).

 Actually, I initially did present the ARP issue at the Cloud bar BOF at
the 78th IETF. But I learned that the audience for Cloud is too diverse
to converge on network problems. The problems which Cloud bar BOF has
presented cover Applications APIs, Logging, Security, etc. They are not
even on network problems. 

I am afraid of mixing network problems with application problems will
just get many people more confused. I remember during 78th IETF Cloud
Bar BOF, the people who focus on Application APIs think network problem
is too trivial to be even discussed at the same place with cloud
applications problems. 

 

Thanks, Linda Dunbar

---

From: Vishwas Manral <[email-masked]>
Date: Tue, 16 Nov 2010 17:34:41 -0800

Hi Linda,

I recently joined the group. So forgive me if I repeat something that
has been discussed earlier.

> For 1 million hosts, they would expect 468,240
> ARPs per second or 239 Mbps of ARP traffic to arrive at
> each host at peak, which is more than enough to
> overwhelm a standard 100 Mbps LAN connection.
As ARP messages may not always be  broadcast and the broadcast domain,
is 239 Mbps the total ARP packet on a network or is it per host?

If it is not per host we should not talk about the overwhelming the 100Mpbs LAN.

Thanks,
Vishwas

On Tue, Nov 16, 2010 at 1:21 PM, Linda Dunbar <[email-masked]> wrote:
> is suggesting that we should go to “Data Center Operation” working group to
> discuss with other “data center operation issues”.
>
>
>
> the number of ARP queries received at a workstation on CMU’s School of
> As Cisco’s Claudio DeSanti pointed out during the BOF, large number of
> _______________________________________________
> armd mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/armd
>
>

---

From: "Adrian Farrel" <[email-masked]>
Date: Wed, 17 Nov 2010 10:40:26 -0000

Hi,
 
I don't believe Ops Area WGs are allowed to do protocol extensions (except for
management protocols), so option 2 is not an option if the ARMD work requires
protocol changes.
 
OTOH, if the ARMD work collapses to how to operate ARP in a specific
environment, then this would be clear for Ops.
 
Adrian
 
From: So, Ning [mailto:[email-masked]] 
Sent: 16 November 2010 21:33
To: Linda Dunbar; [email-masked]
Cc: [email-masked]; [email-masked]; Jari Arkko; Claudio DeSanti
Subject: RE: [armd] ARMD BOF follow up
 
Linda,
 
I concur with your concern on your suggestion #2.  Cloud Data Center WG (name
will likely change) and Cloud AppsBoF were both being proposed during the
meeting last week.  One potential solution is to have network related ARMD
drafts go to the new WG, while non-network related SRMD drafts go to the new
Cloud AppsBoF.  

Best regrads,
 
Ning So
 
  _____  

From: [email-masked] [mailto:[email-masked]] On Behalf Of Linda
Dunbar
Sent: Tuesday, November 16, 2010 3:22 PM
To: [email-masked]
Cc: [email-masked]; [email-masked]; 'Jari Arkko'; 'Claudio DeSanti'
Subject: [armd] ARMD BOF follow up
 
Thanks to everyone who participated in the ARMD BOF discussion last Friday.
There were two major comments we heard from the audience. One comment is wishing
to see some statistics (by Internet AD Ralph), and another comment is suggesting
that we should go to "Data Center Operation" working group to discuss with other
"data center operation issues". 
 
Suggestion 1: Need ARP statistics: 
At the BOF, I forgot to mention that ARP statistics was quoted in our first
problem statement draft for ARP222 (back when it was only at the bar BOF stage).
As time goes on, we modified the draft several times by several authors to
include more important stuff, those numbers get eliminated from the draft. 

the number of ARP queries received at a workstation on CMU's School of Computer
Science LAN over a 12 hour period on August 9, 2004. At peak, the host received
1150 ARPs per second, and on average, the host received 89 ARPs per second,
which corresponds to 45 kbps of traffic. During the data collection, 2,456 hosts
were observed sending ARP queries. 
 
CMU expected that the amount of ARP traffic will scale linearly with the number
of hosts on the LAN. For 1 million hosts, they would expect 468,240 ARPs per
second or 239 Mbps of ARP traffic to arrive at each host at peak, which is more
than enough to overwhelm a standard 100 Mbps LAN connection. Ignoring the link
capacity, forcing hosts to handle an extra half million packets per second to
inspect each ARP packet would impose a prohibitive computational burden.

As Cisco's Claudio DeSanti pointed out during the BOF, large number of client
subnets is effectively causing network to have the same problem as one large
broadcast domain. 

Suggestion 2: To lump the ARP improvement discussion with Data Center Operation
working group (by Ross Callon)
Chatting with Ross Callon after the BOF, I learnt that Ross is referring to the
potential Cloud Operation group (as the result of Cloud bar BOF). 
 
 Actually, I initially did present the ARP issue at the Cloud bar BOF at the
78th IETF. But I learned that the audience for Cloud is too diverse to converge
on network problems. The problems which Cloud bar BOF has presented cover
Applications APIs, Logging, Security, etc. They are not even on network
problems. 
I am afraid of mixing network problems with application problems will just get
many people more confused. I remember during 78th IETF Cloud Bar BOF, the people
who focus on Application APIs think network problem is too trivial to be even
discussed at the same place with cloud applications problems. 

 
Thanks, Linda Dunbar

---

From: Ross Callon <[email-masked]>
Date: Wed, 17 Nov 2010 09:41:24 -0500

I have CC'd Ron Bonica on this, as he is the operations AD.

My understanding of the results of the BOF is that there is clearly a problem to be solved, an urgency to begin work, and several possible solutions.

There is also a need to clearly document what the problem is, the requirements for a solution, and the range of possible solutions. This is work that could be done in a data center operations working group.

It is possible that some potential solutions may involve care of how existing protocols are deployed (an operational issue).

It is possible that some potential solutions may involve extensions or changes to existing IETF protocols. I am not an AD (and no longer play one on TV or anywhere else). My understanding is that these extensions to existing IETF protocols would need to be done in other IETF WGs - presumably the ones responsible for whatever protocol is being extended. However, it will be much easier to charter this work once there is agreement on an initial draft documenting what the problem is and requirements of a solution.

Ross

From: Adrian Farrel [mailto:[email-masked]]
Sent: Wednesday, November 17, 2010 5:40 AM
To: 'So, Ning'; 'Linda Dunbar'; [email-masked]
Cc: Ross Callon; 'Jari Arkko'; 'Claudio DeSanti'
Subject: RE: [armd] ARMD BOF follow up

Hi,

I don't believe Ops Area WGs are allowed to do protocol extensions (except for management protocols), so option 2 is not an option if the ARMD work requires protocol changes.

OTOH, if the ARMD work collapses to how to operate ARP in a specific environment, then this would be clear for Ops.

Adrian

Linda,

I concur with your concern on your suggestion #2.  Cloud Data Center WG (name will likely change) and Cloud AppsBoF were both being proposed during the meeting last week.  One potential solution is to have network related ARMD drafts go to the new WG, while non-network related SRMD drafts go to the new Cloud AppsBoF.

Best regrads,

Ning So

From: [email-masked] [mailto:[email-masked]] On Behalf Of Linda Dunbar

Thanks to everyone who participated in the ARMD BOF discussion last Friday. There were two major comments we heard from the audience. One comment is wishing to see some statistics (by Internet AD Ralph), and another comment is suggesting that we should go to "Data Center Operation" working group to discuss with other "data center operation issues".

Suggestion 1: Need ARP statistics:
At the BOF, I forgot to mention that ARP statistics was quoted in our first problem statement draft for ARP222 (back when it was only at the bar BOF stage).  As time goes on, we modified the draft several times by several authors to include more important stuff, those numbers get eliminated from the draft.

The study done by CMU shows (http://www.cs.cmu.edu/~hzhang/papers/hotnets04-ethernet.pdf):

the number of ARP queries received at a workstation on CMU's School of Computer Science LAN over a 12 hour period on August 9, 2004. At peak, the host received 1150 ARPs per second, and on average, the host received 89 ARPs per second, which corresponds to 45 kbps of traffic. During the data collection, 2,456 hosts were observed sending ARP queries.

CMU expected that the amount of ARP traffic will scale linearly with the number of hosts on the LAN. For 1 million hosts, they would expect 468,240 ARPs per second or 239 Mbps of ARP traffic to arrive at each host at peak, which is more than enough to overwhelm a standard 100 Mbps LAN connection. Ignoring the link capacity, forcing hosts to handle an extra half million packets per second to inspect each ARP packet would impose a prohibitive computational burden.

As Cisco's Claudio DeSanti pointed out during the BOF, large number of client subnets is effectively causing network to have the same problem as one large broadcast domain.

Question: What other statistics do people would like to see before being convinced that ARP optimization is indeed necessary to make network scale to support next gen mega data center?

Suggestion 2: To lump the ARP improvement discussion with Data Center Operation working group (by Ross Callon)
Chatting with Ross Callon after the BOF, I learnt that Ross is referring to the potential Cloud Operation group (as the result of Cloud bar BOF).

 Actually, I initially did present the ARP issue at the Cloud bar BOF at the 78th IETF. But I learned that the audience for Cloud is too diverse to converge on network problems. The problems which Cloud bar BOF has presented cover Applications APIs, Logging, Security, etc. They are not even on network problems.
I am afraid of mixing network problems with application problems will just get many people more confused. I remember during 78th IETF Cloud Bar BOF, the people who focus on Application APIs think network problem is too trivial to be even discussed at the same place with cloud applications problems.

Thanks, Linda Dunbar

---

From: Linda Dunbar <[email-masked]>
Date: Wed, 17 Nov 2010 09:23:32 -0600

Vishwas, 

See answers inserted below:

-----Original Message-----
From: Vishwas Manral [mailto:[email-masked]] 
Sent: Tuesday, November 16, 2010 7:35 PM
To: Linda Dunbar
Cc: [email-masked]; [email-masked]; [email-masked]; Jari Arkko;
Claudio DeSanti
Subject: Re: [armd] ARMD BOF follow up

Hi Linda,

[Linda] It is 239 Mbps towards each host. The traffic in the network is much
higher. 

If it is not per host we should not talk about the overwhelming the 100Mpbs
LAN.

Thanks,
Vishwas

> Thanks to everyone who participated in the ARMD BOF discussion last
Friday.
> At the BOF, I forgot to mention that ARP statistics was quoted in our
first
> Chatting with Ross Callon after the BOF, I learnt that Ross is referring
to
>  Actually, I initially did present the ARP issue at the Cloud bar BOF at
the

---

From: Vishwas Manral <[email-masked]>
Date: Wed, 17 Nov 2010 10:36:16 -0800

Hi Linda,

So is the assumption that of a million hosts 1/7 of the ARP traffic
will be bound to 1 particular host - to get the number of 239 Mbps?

I was wondering how the calculations were done.

Thanks,
Vishwas

---

From: Linda Dunbar <[email-masked]>
Date: Wed, 17 Nov 2010 10:39:43 -0600

Ross and Adrian, 

Thank you very much for the valuable input. Glad to learn that everyone
agrees that there is clearly a problem to be solved and the urgency to begin
work. The discussion is on where the work should be chartered. The detailed
problems for ARMD has been described by
<http://trac.tools.ietf.org/bof/trac/attachment/wiki/WikiStart/ARMD-Problem%
20Statement.pdf>
http://trac.tools.ietf.org/bof/trac/attachment/wiki/WikiStart/ARMD-Problem%2
0Statement.pdf

Please provide comments if the problems stated in the document are not clear
enough. 

There are 4 potential solutions presented at the BOF. Clearly protocol
extension has been identified. 

I am very concerned of lumping ARMD problems with Cloud Data Center
Operation. As of now, in addition to Cloud Application extension (such as
APIs, Logging, and Security), the Cloud Data Center operations already
include the following areas for network protocol extension:

*        VPN extension for Cloud

*        Virtual Resource management

*        Cloud hosted UE

*        Service mobility for virtualized networks

ARMD is a very tightly scoped problem domain for large data center. Even
though it is part of total large data center problem domain, the ARMD itself
can be very easily carved out from other problems associated with Cloud Data
Center. I thought that IETF always wants to create a small working group
focusing on one problem at a time. 

Best Regards

Linda Dunbar

 

  _____  

From: Ross Callon [mailto:[email-masked]] 
Sent: Wednesday, November 17, 2010 8:41 AM
To: [email-masked]; 'So, Ning'; 'Linda Dunbar'; [email-masked]
Cc: 'Jari Arkko'; 'Claudio DeSanti'; Ronald Bonica
Subject: RE: [armd] ARMD BOF follow up

My understanding of the results of the BOF is that there is clearly a
problem to be solved, an urgency to begin work, and several possible
solutions.  

There is also a need to clearly document what the problem is, the
requirements for a solution, and the range of possible solutions. This is
work that could be done in a data center operations working group.

It is possible that some potential solutions may involve care of how
existing protocols are deployed (an operational issue). 

It is possible that some potential solutions may involve extensions or
changes to existing IETF protocols. I am not an AD (and no longer play one
on TV or anywhere else). My understanding is that these extensions to
existing IETF protocols would need to be done in other IETF WGs - presumably
the ones responsible for whatever protocol is being extended. However, it
will be much easier to charter this work once there is agreement on an
initial draft documenting what the problem is and requirements of a
solution. 

Ross

Hi,

I don't believe Ops Area WGs are allowed to do protocol extensions (except
for management protocols), so option 2 is not an option if the ARMD work
requires protocol changes.

Adrian

Linda,

I concur with your concern on your suggestion #2.  Cloud Data Center WG
(name will likely change) and Cloud AppsBoF were both being proposed during
the meeting last week.  One potential solution is to have network related
ARMD drafts go to the new WG, while non-network related SRMD drafts go to
the new Cloud AppsBoF.  

Best regrads,

Ning So

Verizon, Inc.

(office) 972-729-7905

(Cell) 972-955-0914

  _____  

 

 

Thanks, Linda Dunbar

---

From: Vishwas Manral <[email-masked]>
Date: Sat, 20 Nov 2010 13:59:19 -0800

Hi Linda,

Reading one of the papers on Data center flows done by MS Research, it
seems 80% of the cloud data center data stays within the rack. It
would also mean that in such cases ARP would probably not need to be
resolved in most of these cases.

Thanks,
Vishwas

---

From: Linda Dunbar <[email-masked]>
Date: Mon, 22 Nov 2010 17:58:49 -0600

Vishwas: 

Can you point out which paper you are referring to? 

Many Cloud research papers, like this one from MS:
http://portal.acm.org/citation.cfm?id=1496103 are arguing that Network
constraints (like VLAN, ARPs, ACL) are the major barrier to Data Center
agility even though network is only 15% of total data center cost. 

The trend of data center is to scale out, instead of scale up, which means
there would be more and more servers in data center. To be able to provide
host service anywhere on any servers is the key in reducing overall data
center cost. 

Linda Dunbar

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Vishwas Manral
Sent: Saturday, November 20, 2010 3:59 PM
To: Linda Dunbar
Cc: [email-masked]; Jari Arkko; [email-masked]
Subject: Re: [armd] ARMD BOF follow up

Hi Linda,

Thanks,
Vishwas

the
>>> number of hosts on the LAN. For 1 million hosts, they would expect
468,240
>>> client subnets is effectively causing network to have the same problem
as
>>> convinced that ARP optimization is indeed necessary to make network
scale
>>> I am afraid of mixing network problems with application problems will
just

---

From: Vishwas Manral <[email-masked]>
Date: Mon, 22 Nov 2010 16:57:33 -0800

Hi Linda,

I am refering to the paper
http://research.microsoft.com/pubs/138279/DC-Network-Characterization-imc2010.pdf

I am not saying that ARP scalability is not required (I have noticed
that with even a few thousand ARP the CPU can get overwhelmed). I am
just trying to understand the constraints under which the numbers
being used are defined.

Thanks,
Vishwas

---

From: Linda Dunbar <[email-masked]>
Date: Tue, 30 Nov 2010 17:34:13 -0600

During afterwards discussion on how to move ARMD forward, we were pointed
out that there are two different areas being addressed by ARMD problem
statements:

One is the general ARP problem associated with massive number of hosts

*	The massive number of client subnets in Data Center will cause
similar problems to networks in data center with virtualized servers. 

Another one is ARP can actually be used as solution to quickly update
intermediate switches' FDB when hosts (VMs) migrate from one location to
another. 

*	(our original thought is about there will be more ARPs when VMs
migrate. The bullet listed in the ARMD Problem Statement is to justify
urgency to enhance ARP). 

We'd like to hear people's opinion on if ARMD problem domain should include
only the first one or both (i.e. ARP needing extension to be used in data
center with large number of hosts and ARP as solution for Layer 2 switches
quick convergence)? 

Your input is appreciated. 

Linda Dunbar

  _____  

From: [email-masked] [mailto:[email-masked]] On Behalf Of Ross
Callon
Sent: Wednesday, November 17, 2010 8:41 AM
To: [email-masked]; 'So, Ning'; 'Linda Dunbar'; [email-masked]
Cc: Ronald Bonica; 'Jari Arkko'; 'Claudio DeSanti'
Subject: Re: [armd] ARMD BOF follow up

Ross

Hi,

Adrian

Linda,

 

Best regrads,

Ning So

Verizon, Inc.

(office) 972-729-7905

(Cell) 972-955-0914

  _____  

 

 

Thanks, Linda Dunbar