Subject: Re: [armd] Need to reach concensus on ARMD problem domain

From: Anoop Ghanwani <[email-masked]>
Date: Wed, 1 Dec 2010 14:14:16 -0800

I was not at the meeting so I may be missing some
critical information with respect to the problems
and solutions that were discussed.  That said...

I view the first problem of being one where I want
to stretch a subnet across multiple ToR switches in the
data center, and where I want to avoid having ARPs go across
the backbone to the extent that it is possible to avoid it.  I
definitely think we should do something to address that
problem.  If we don't address it in a working group,
folks are going to address it in a non-interoperable way.

I'm not clear on what the second problem is and how we
would go about solving it.

Anoop

From: [email-masked] [mailto:[email-masked]] On Behalf Of Linda Dunbar
Sent: Tuesday, November 30, 2010 3:34 PM
To: [email-masked]; [email-masked]
Cc: 'Jari Arkko'; 'Ralph Droms'
Subject: [armd] Need to reach concensus on ARMD problem domain

During afterwards discussion on how to move ARMD forward, we were pointed out that there are two different areas being addressed by ARMD problem statements:

One is the general ARP problem associated with massive number of hosts

    *   The massive number of client subnets in Data Center will cause similar problems to networks in data center with virtualized servers.
Another one is ARP can actually be used as solution to quickly update intermediate switches' FDB when hosts (VMs) migrate from one location to another.

    *   (our original thought is about there will be more ARPs when VMs migrate. The bullet listed in the ARMD Problem Statement is to justify urgency to enhance ARP).

We'd like to hear people's opinion on if ARMD problem domain should include only the first one or both (i.e. ARP needing extension to be used in data center with large number of hosts and ARP as solution for Layer 2 switches quick convergence)?

Your input is appreciated.

Linda Dunbar

________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Ross Callon
Sent: Wednesday, November 17, 2010 8:41 AM
To: [email-masked]; 'So, Ning'; 'Linda Dunbar'; [email-masked]
Cc: Ronald Bonica; 'Jari Arkko'; 'Claudio DeSanti'
Subject: Re: [armd] ARMD BOF follow up

I have CC'd Ron Bonica on this, as he is the operations AD.

My understanding of the results of the BOF is that there is clearly a problem to be solved, an urgency to begin work, and several possible solutions.

There is also a need to clearly document what the problem is, the requirements for a solution, and the range of possible solutions. This is work that could be done in a data center operations working group.

It is possible that some potential solutions may involve care of how existing protocols are deployed (an operational issue).

It is possible that some potential solutions may involve extensions or changes to existing IETF protocols. I am not an AD (and no longer play one on TV or anywhere else). My understanding is that these extensions to existing IETF protocols would need to be done in other IETF WGs - presumably the ones responsible for whatever protocol is being extended. However, it will be much easier to charter this work once there is agreement on an initial draft documenting what the problem is and requirements of a solution.

Ross

From: Adrian Farrel [mailto:[email-masked]]
Sent: Wednesday, November 17, 2010 5:40 AM
To: 'So, Ning'; 'Linda Dunbar'; [email-masked]
Cc: Ross Callon; 'Jari Arkko'; 'Claudio DeSanti'
Subject: RE: [armd] ARMD BOF follow up

Hi,

I don't believe Ops Area WGs are allowed to do protocol extensions (except for management protocols), so option 2 is not an option if the ARMD work requires protocol changes.

OTOH, if the ARMD work collapses to how to operate ARP in a specific environment, then this would be clear for Ops.

Adrian

From: So, Ning [mailto:[email-masked]]
Sent: 16 November 2010 21:33
To: Linda Dunbar; [email-masked]
Cc: [email-masked]; [email-masked]; Jari Arkko; Claudio DeSanti
Subject: RE: [armd] ARMD BOF follow up

Linda,

I concur with your concern on your suggestion #2.  Cloud Data Center WG (name will likely change) and Cloud AppsBoF were both being proposed during the meeting last week.  One potential solution is to have network related ARMD drafts go to the new WG, while non-network related SRMD drafts go to the new Cloud AppsBoF.

Best regrads,

Ning So
Network Evolution Planning
Verizon, Inc.
(office) 972-729-7905
(Cell) 972-955-0914

________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Linda Dunbar
Sent: Tuesday, November 16, 2010 3:22 PM
To: [email-masked]
Cc: [email-masked]; [email-masked]; 'Jari Arkko'; 'Claudio DeSanti'
Subject: [armd] ARMD BOF follow up

Thanks to everyone who participated in the ARMD BOF discussion last Friday. There were two major comments we heard from the audience. One comment is wishing to see some statistics (by Internet AD Ralph), and another comment is suggesting that we should go to "Data Center Operation" working group to discuss with other "data center operation issues".

Suggestion 1: Need ARP statistics:
At the BOF, I forgot to mention that ARP statistics was quoted in our first problem statement draft for ARP222 (back when it was only at the bar BOF stage).  As time goes on, we modified the draft several times by several authors to include more important stuff, those numbers get eliminated from the draft.

The study done by CMU shows (http://www.cs.cmu.edu/~hzhang/papers/hotnets04-ethernet.pdf):

the number of ARP queries received at a workstation on CMU's School of Computer Science LAN over a 12 hour period on August 9, 2004. At peak, the host received 1150 ARPs per second, and on average, the host received 89 ARPs per second, which corresponds to 45 kbps of traffic. During the data collection, 2,456 hosts were observed sending ARP queries.

CMU expected that the amount of ARP traffic will scale linearly with the number of hosts on the LAN. For 1 million hosts, they would expect 468,240 ARPs per second or 239 Mbps of ARP traffic to arrive at each host at peak, which is more than enough to overwhelm a standard 100 Mbps LAN connection. Ignoring the link capacity, forcing hosts to handle an extra half million packets per second to inspect each ARP packet would impose a prohibitive computational burden.

As Cisco's Claudio DeSanti pointed out during the BOF, large number of client subnets is effectively causing network to have the same problem as one large broadcast domain.

Question: What other statistics do people would like to see before being convinced that ARP optimization is indeed necessary to make network scale to support next gen mega data center?

Suggestion 2: To lump the ARP improvement discussion with Data Center Operation working group (by Ross Callon)
Chatting with Ross Callon after the BOF, I learnt that Ross is referring to the potential Cloud Operation group (as the result of Cloud bar BOF).

 Actually, I initially did present the ARP issue at the Cloud bar BOF at the 78th IETF. But I learned that the audience for Cloud is too diverse to converge on network problems. The problems which Cloud bar BOF has presented cover Applications APIs, Logging, Security, etc. They are not even on network problems.
I am afraid of mixing network problems with application problems will just get many people more confused. I remember during 78th IETF Cloud Bar BOF, the people who focus on Application APIs think network problem is too trivial to be even discussed at the same place with cloud applications problems.

Question: Should we go towards this direction?

What other refinement we need to consider?

Thanks, Linda Dunbar

---

From: Yizhou Li <[email-masked]>
Date: Fri, 03 Dec 2010 11:52:09 +0800

I would think the second problem is a special use case of which needs to be
taken care under the scope of the first problem. 

VM migration is one of the reasons that makes the subnet stretching across
multiple ToRs. And it is a dynamic change instead of static. Therefore we
may need some different handling of ARP in this case. E.g. certain ARP
should not be avoided going to backbone if it signals a migration rather
than a new node starting up.

Yizhou

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Anoop Ghanwani
Sent: Thursday, December 02, 2010 6:14 AM
To: Linda Dunbar; [email-masked]; [email-masked]
Cc: 'Jari Arkko'; 'Ralph Droms'
Subject: Re: [armd] Need to reach concensus on ARMD problem domain

Anoop

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Linda Dunbar

During afterwards discussion on how to move ARMD forward, we were pointed
out that there are two different areas being addressed by ARMD problem
statements:

*	The massive number of client subnets in Data Center will cause
similar problems to networks in data center with virtualized servers. 

Another one is ARP can actually be used as solution to quickly update
intermediate switches' FDB when hosts (VMs) migrate from one location to
another. 

*	(our original thought is about there will be more ARPs when VMs
migrate. The bullet listed in the ARMD Problem Statement is to justify
urgency to enhance ARP). 

We'd like to hear people's opinion on if ARMD problem domain should include
only the first one or both (i.e. ARP needing extension to be used in data
center with large number of hosts and ARP as solution for Layer 2 switches
quick convergence)? 

Linda Dunbar

  _____  

From: [email-masked] [mailto:[email-masked]] On Behalf Of Ross
Callon

My understanding of the results of the BOF is that there is clearly a
problem to be solved, an urgency to begin work, and several possible
solutions.  

There is also a need to clearly document what the problem is, the
requirements for a solution, and the range of possible solutions. This is
work that could be done in a data center operations working group.

It is possible that some potential solutions may involve care of how
existing protocols are deployed (an operational issue). 

It is possible that some potential solutions may involve extensions or
changes to existing IETF protocols. I am not an AD (and no longer play one
on TV or anywhere else). My understanding is that these extensions to
existing IETF protocols would need to be done in other IETF WGs - presumably
the ones responsible for whatever protocol is being extended. However, it
will be much easier to charter this work once there is agreement on an
initial draft documenting what the problem is and requirements of a
solution. 

Ross

Hi,

I don't believe Ops Area WGs are allowed to do protocol extensions (except
for management protocols), so option 2 is not an option if the ARMD work
requires protocol changes.

OTOH, if the ARMD work collapses to how to operate ARP in a specific
environment, then this would be clear for Ops.

Adrian

Linda,

I concur with your concern on your suggestion #2.  Cloud Data Center WG
(name will likely change) and Cloud AppsBoF were both being proposed during
the meeting last week.  One potential solution is to have network related
ARMD drafts go to the new WG, while non-network related SRMD drafts go to
the new Cloud AppsBoF.  

Best regrads,

Ning So

Verizon, Inc.

(office) 972-729-7905

(Cell) 972-955-0914

  _____  

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Linda Dunbar
Sent: Tuesday, November 16, 2010 3:22 PM
To: [email-masked]
Cc: [email-masked]; [email-masked]; 'Jari Arkko'; 'Claudio
DeSanti'
Subject: [armd] ARMD BOF follow up

Thanks to everyone who participated in the ARMD BOF discussion last Friday.
There were two major comments we heard from the audience. One comment is
wishing to see some statistics (by Internet AD Ralph), and another comment
is suggesting that we should go to "Data Center Operation" working group to
discuss with other "data center operation issues". 

At the BOF, I forgot to mention that ARP statistics was quoted in our first
problem statement draft for ARP222 (back when it was only at the bar BOF
stage).  As time goes on, we modified the draft several times by several
authors to include more important stuff, those numbers get eliminated from
the draft. 

The study done by CMU shows
(http://www.cs.cmu.edu/~hzhang/papers/hotnets04-ethernet.pdf): 

the number of ARP queries received at a workstation on CMU's School of
Computer Science LAN over a 12 hour period on August 9, 2004. At peak, the
host received 1150 ARPs per second, and on average, the host received 89
ARPs per second, which corresponds to 45 kbps of traffic. During the data
collection, 2,456 hosts were observed sending ARP queries. 

CMU expected that the amount of ARP traffic will scale linearly with the
number of hosts on the LAN. For 1 million hosts, they would expect 468,240
ARPs per second or 239 Mbps of ARP traffic to arrive at each host at peak,
which is more than enough to overwhelm a standard 100 Mbps LAN connection.
Ignoring the link capacity, forcing hosts to handle an extra half million
packets per second to inspect each ARP packet would impose a prohibitive
computational burden.

As Cisco's Claudio DeSanti pointed out during the BOF, large number of
client subnets is effectively causing network to have the same problem as
one large broadcast domain. 

Question: What other statistics do people would like to see before being
convinced that ARP optimization is indeed necessary to make network scale to
support next gen mega data center? 

Suggestion 2: To lump the ARP improvement discussion with Data Center
Operation working group (by Ross Callon)

Chatting with Ross Callon after the BOF, I learnt that Ross is referring to
the potential Cloud Operation group (as the result of Cloud bar BOF). 

 Actually, I initially did present the ARP issue at the Cloud bar BOF at the
78th IETF. But I learned that the audience for Cloud is too diverse to
converge on network problems. The problems which Cloud bar BOF has presented
cover Applications APIs, Logging, Security, etc. They are not even on
network problems. 

I am afraid of mixing network problems with application problems will just
get many people more confused. I remember during 78th IETF Cloud Bar BOF,
the people who focus on Application APIs think network problem is too
trivial to be even discussed at the same place with cloud applications
problems. 

 

Thanks, Linda Dunbar

---

From: Vishwas Manral <[email-masked]>
Date: Thu, 2 Dec 2010 20:50:40 -0800

Hi Linda,

I think both the problems are critical and need to be tackled.

Thanks,
Vishwas

On Tue, Nov 30, 2010 at 3:34 PM, Linda Dunbar <[email-masked]> wrote:
> The massive number of client subnets in Data Center will cause similar
> problems to networks in data center with virtualized servers.
>
> intermediate switches’ FDB when hosts (VMs) migrate from one location to
> another.
>
> (our original thought is about there will be more ARPs when VMs migrate. The
> bullet listed in the ARMD Problem Statement is to justify urgency to enhance
> ARP).
>
>
>
> We’d like to hear people’s opinion on if ARMD problem domain should include
> I have CC’d Ron Bonica on this, as he is the operations AD.
>
>
>
> existing IETF protocols would need to be done in other IETF WGs – presumably
> is suggesting that we should go to “Data Center Operation” working group to
> discuss with other “data center operation issues”.
>
>
>
> the number of ARP queries received at a workstation on CMU’s School of
> As Cisco’s Claudio DeSanti pointed out during the BOF, large number of
> _______________________________________________
> armd mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/armd
>
>

---

From: Shehzad Merchant <[email-masked]>
Date: Thu, 2 Dec 2010 20:50:37 -0800

Yes on the first one.

I think the second one is something that should be addressed, otherwise different implementations may do things differently. Whether ARP is the right mechanism is open to discussion, but certainly in the absence of any other standard mechanism one could consider this as a mechanism or at least make it part of the problem statement.
-S

From: [email-masked] [mailto:[email-masked]] On Behalf Of Yizhou Li
Sent: Thursday, December 02, 2010 7:52 PM
To: 'Anoop Ghanwani'; 'Linda Dunbar'; [email-masked]; [email-masked]
Cc: 'Jari Arkko'; 'Ralph Droms'
Subject: Re: [armd] Need to reach concensus on ARMD problem domain

I would think the second problem is a special use case of which needs to be taken care under the scope of the first problem.

VM migration is one of the reasons that makes the subnet stretching across multiple ToRs. And it is a dynamic change instead of static. Therefore we may need some different handling of ARP in this case. E.g. certain ARP should not be avoided going to backbone if it signals a migration rather than a new node starting up.

Yizhou

From: [email-masked] [mailto:[email-masked]] On Behalf Of Anoop Ghanwani

Anoop

Linda Dunbar

Ross

Hi,

Adrian

Linda,

Best regrads,

Ning So

Thanks, Linda Dunbar

________________________________
DISCLAIMER:
This e-mail and any attachments to it may contain confidential and proprietary material and is solely for the use of the intended recipient. Any review, use, disclosure, distribution or copying of this transmittal is prohibited except by or on behalf of the intended recipient. If you have received this transmittal in error, please notify the sender and destroy this e-mail and any attachments and all copies, whether electronic or printed.

---

From: "So, Ning" <[email-masked]>
Date: Fri, 03 Dec 2010 15:04:58 +0000

Linda and all,

I agree both problems are real and urgent problems that need resolution.
The 1st problem is mostly contained within ARMD domain.  The 2nd problem
(VM Mobility induced ARP problem) may have to be worked within ARMD
domain as well as other WGs.  For example, the problem and requirement
may fall in the Cloud Data Center WG (not established) as a part of the
general VM Mobility Requirement draft.  The solution draft may be joint
work between ARMD and L2VPN if L2VPN protocols are impacted by the
solution. 

Best regrads,

Ning So

Verizon, Inc.

(office) 972-729-7905

(Cell) 972-955-0914

From: Linda Dunbar [mailto:[email-masked]] 
Sent: Tuesday, November 30, 2010 5:34 PM
To: [email-masked]; [email-masked]
Cc: 'Jari Arkko'; 'Laganier, Julien'; 'Ralph Droms'; So, Ning
Subject: Need to reach concensus on ARMD problem domain

During afterwards discussion on how to move ARMD forward, we were
pointed out that there are two different areas being addressed by ARMD
problem statements:

	*	The massive number of client subnets in Data Center will
cause similar problems to networks in data center with virtualized
servers. 

	*	(our original thought is about there will be more ARPs
when VMs migrate. The bullet listed in the ARMD Problem Statement is to
justify urgency to enhance ARP). 

We'd like to hear people's opinion on if ARMD problem domain should
include only the first one or both (i.e. ARP needing extension to be
used in data center with large number of hosts and ARP as solution for
Layer 2 switches quick convergence)? 

Linda Dunbar

Ross Callon

There is also a need to clearly document what the problem is, the
requirements for a solution, and the range of possible solutions. This
is work that could be done in a data center operations working group.

It is possible that some potential solutions may involve extensions or
changes to existing IETF protocols. I am not an AD (and no longer play
one on TV or anywhere else). My understanding is that these extensions
to existing IETF protocols would need to be done in other IETF WGs -
presumably the ones responsible for whatever protocol is being extended.
However, it will be much easier to charter this work once there is
agreement on an initial draft documenting what the problem is and
requirements of a solution. 

Ross

Hi,

I don't believe Ops Area WGs are allowed to do protocol extensions
(except for management protocols), so option 2 is not an option if the
ARMD work requires protocol changes.

Adrian

Cc: [email-masked]; [email-masked]; Jari Arkko; Claudio
DeSanti

Linda,

I concur with your concern on your suggestion #2.  Cloud Data Center WG
(name will likely change) and Cloud AppsBoF were both being proposed
during the meeting last week.  One potential solution is to have network
related ARMD drafts go to the new WG, while non-network related SRMD
drafts go to the new Cloud AppsBoF.  

Best regrads,

Ning So

Verizon, Inc.

(office) 972-729-7905

(Cell) 972-955-0914

Thanks to everyone who participated in the ARMD BOF discussion last
Friday. There were two major comments we heard from the audience. One
comment is wishing to see some statistics (by Internet AD Ralph), and
another comment is suggesting that we should go to "Data Center
Operation" working group to discuss with other "data center operation
issues". 

At the BOF, I forgot to mention that ARP statistics was quoted in our
first problem statement draft for ARP222 (back when it was only at the
bar BOF stage).  As time goes on, we modified the draft several times by
several authors to include more important stuff, those numbers get
eliminated from the draft. 

the number of ARP queries received at a workstation on CMU's School of
Computer Science LAN over a 12 hour period on August 9, 2004. At peak,
the host received 1150 ARPs per second, and on average, the host
received 89 ARPs per second, which corresponds to 45 kbps of traffic.
During the data collection, 2,456 hosts were observed sending ARP
queries. 

CMU expected that the amount of ARP traffic will scale linearly with the
number of hosts on the LAN. For 1 million hosts, they would expect
468,240 ARPs per second or 239 Mbps of ARP traffic to arrive at each
host at peak, which is more than enough to overwhelm a standard 100 Mbps
LAN connection. Ignoring the link capacity, forcing hosts to handle an
extra half million packets per second to inspect each ARP packet would
impose a prohibitive computational burden.

As Cisco's Claudio DeSanti pointed out during the BOF, large number of
client subnets is effectively causing network to have the same problem
as one large broadcast domain. 

Question: What other statistics do people would like to see before being
convinced that ARP optimization is indeed necessary to make network
scale to support next gen mega data center? 

Chatting with Ross Callon after the BOF, I learnt that Ross is referring
to the potential Cloud Operation group (as the result of Cloud bar BOF).

 Actually, I initially did present the ARP issue at the Cloud bar BOF at
the 78th IETF. But I learned that the audience for Cloud is too diverse
to converge on network problems. The problems which Cloud bar BOF has
presented cover Applications APIs, Logging, Security, etc. They are not
even on network problems. 

I am afraid of mixing network problems with application problems will
just get many people more confused. I remember during 78th IETF Cloud
Bar BOF, the people who focus on Application APIs think network problem
is too trivial to be even discussed at the same place with cloud
applications problems. 

 

Thanks, Linda Dunbar