Subject: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts

From: Linda Dunbar <[email-masked]>
Date: Sat, 17 Sep 2011 18:35:17 +0000

There are many drafts recently on various Overlay Network schemes for data centers with large number of tenants/hosts. When Overlay Network is deployed in a data center, local switches/routers within the data center network don't see the hosts addresses anymore except the nodes which perform the network address encapsulation/decapsulation.

However, when a large data center has tens of thousands of tenants/hosts which communicate to peers outside the data center, all those tenants/hosts send data packets destined towards external peers to the Gateway routers of the data center. Each tenant has at least several hosts which need to communicate with external peers.

For each host (which runs on a VM) which need to communicate with external world, it frequently sends ARP/ND requests to its default Gateway router. For an application with dual stacks on, it sends both ARP & ND requests to its default gateway router.

Routers are built to forward large amount of traffic. Traffic forwarded from line card to line card don't go through CPU. But ARP/ND requests to the gateway routers have to be processed by the router's CPU. That is why we see Router's CPU utilization reaching 40% (Merit Network statistics) or higher  in many data centers.

I think that this kind of problems should be captured in the ARMD WG problem statement. What do you think?

Linda Dunbar

---

From: Donald Eastlake <[email-masked]>
Date: Sun, 18 Sep 2011 12:02:06 -0400

This sort of problem seems on target for the ARMD charter and should
be verified and documented.

Thanks,
Donald

On Sat, Sep 17, 2011 at 2:35 PM, Linda Dunbar <[email-masked]> wrote:
>
>
> There are many drafts recently on various Overlay Network schemes for data
> centers with large number of tenants/hosts. When Overlay Network is deployed
> in a data center, local switches/routers within the data center network
> don’t see the hosts addresses anymore except the nodes which perform the
> network address encapsulation/decapsulation.
>
>
> However, when a large data center has tens of thousands of tenants/hosts
> which communicate to peers outside the data center, all those tenants/hosts
> send data packets destined towards external peers to the Gateway routers of
> the data center. Each tenant has at least several hosts which need to
> communicate with external peers.
>
>
> For each host (which runs on a VM) which need to communicate with external
> world, it frequently sends ARP/ND requests to its default Gateway router.
> For an application with dual stacks on, it sends both ARP & ND requests to
> its default gateway router.
>
>
> Routers are built to forward large amount of traffic. Traffic forwarded from
> line card to line card don’t go through CPU. But ARP/ND requests to the
> gateway routers have to be processed by the router’s CPU. That is why we see
> Router’s CPU utilization reaching 40% (Merit Network statistics) or higher
>  in many data centers.
>
>

---

From: "He, Peng" <[email-masked]>
Date: Mon, 19 Sep 2011 09:02:06 -0400

Agreed.. nice description for the problem statement.

Regards,
Peng

-----Original Message-----
>From: [email-masked] [mailto:[email-masked]] On Behalf Of
>Donald Eastlake
>Sent: Sunday, September 18, 2011 12:02 PM
>To: Linda Dunbar
>Cc: [email-masked]
>Subject: Re: [armd] Bottleneck of Overlay network in a data center with
>large number of tenants/hosts
>
>> don't see the hosts addresses anymore except the nodes which perform
>the
>> line card to line card don't go through CPU. But ARP/ND requests to
>the
>> gateway routers have to be processed by the router's CPU. That is why
>we see
>> Router's CPU utilization reaching 40% (Merit Network statistics) or
>higher
>>  in many data centers.
>>
>>
>> I think that this kind of problems should be captured in the ARMD WG
>problem
>> statement. What do you think?
>>
>>
>> Linda Dunbar
>_______________________________________________
>armd mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/armd

---

From: "So, Ning" <[email-masked]>
Date: Mon, 19 Sep 2011 09:58:19 -0400

+1.

Best regards,
 
Ning So
Verizon Corporate Technology
(office) 972-729-7905
(Cell) 972-955-0914

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Donald Eastlake
Sent: Sunday, September 18, 2011 11:02 AM
To: Linda Dunbar
Cc: [email-masked]
Subject: Re: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts

This sort of problem seems on target for the ARMD charter and should be verified and documented.

Thanks,
Donald

On Sat, Sep 17, 2011 at 2:35 PM, Linda Dunbar <[email-masked]> wrote:
>
>
> There are many drafts recently on various Overlay Network schemes for 
> data centers with large number of tenants/hosts. When Overlay Network 
> is deployed in a data center, local switches/routers within the data 
> center network don't see the hosts addresses anymore except the nodes 
> which perform the network address encapsulation/decapsulation.
>
>
> However, when a large data center has tens of thousands of 
> tenants/hosts which communicate to peers outside the data center, all 
> those tenants/hosts send data packets destined towards external peers 
> to the Gateway routers of the data center. Each tenant has at least 
> several hosts which need to communicate with external peers.
>
>
> For each host (which runs on a VM) which need to communicate with 
> external world, it frequently sends ARP/ND requests to its default Gateway router.
> For an application with dual stacks on, it sends both ARP & ND 
> requests to its default gateway router.
>
>
> Routers are built to forward large amount of traffic. Traffic 
> forwarded from line card to line card don't go through CPU. But ARP/ND 
> requests to the gateway routers have to be processed by the router's 
> CPU. That is why we see Router's CPU utilization reaching 40% (Merit 
> Network statistics) or higher
>  in many data centers.
>
>
> problem statement. What do you think?
>
>
> Linda Dunbar

---

From: Vishwas Manral <[email-masked]>
Date: Tue, 20 Sep 2011 15:12:29 -0700

Hi Linda,

An overlay may not originate at the host, it could originate at the router
or any other device. I am sure
as the specs eveolve they will clearly seperate the functionality and packet
formats (specified by IETF) from the location of the functionality (can be
informational but not necessarily something we need to write something
against).

However if you have issues with the encapsulation (too much overhead/ hard
to hash flows/ fragmentation or whatever else) that is something we could
discuss.

Also regarding ARP/ ND I think there are well knows ways to get over
broadcast issues, which may require standards track work.

Thanks,
Vishwas
On Sat, Sep 17, 2011 at 11:35 AM, Linda Dunbar <[email-masked]>wrote:

>  ** **
> network address encapsulation/decapsulation. ****
>
> ** **
>
> communicate with external peers.****
>
>
> ****
>
> its default gateway router. ****
>
> ** **
>
> Routers are built to forward large amount of traffic. Traffic forwarded
> from line card to line card don’t go through CPU. But ARP/ND requests to the
>  in many data centers. ****
>
> ** **
>
> problem statement. What do you think?  ****
>
> ** **
>
> Linda Dunbar****
>

---

From: Anoop Ghanwani <[email-masked]>
Date: Mon, 26 Sep 2011 12:10:37 -0700

> network address encapsulation/decapsulation.
...
> its default gateway router.

I think the issue of whether or not an overlay network is in use is orthogonal
to the issue of ARP/ND traffic that ARMD is trying to address.
If we have an overlay network built over a data center core,
we still run into the problem where ARP/ND traffic from a VM needs to
be broadcast/multicast over the core (even though it uses the
overlay).  So unless
some other mechanism is used to mitigate the ARP/ND traffic, we would
be no better than a regular L2 fabric that spans the entire data center (at
least from the standpoint of the volume of ARP/ND traffic traversing
the core).

---

From: Vishwas Manral <[email-masked]>
Date: Mon, 26 Sep 2011 13:31:19 -0700

Totally agree with Anoop here!!!

-Vishwas

On Mon, Sep 26, 2011 at 12:10 PM, Anoop Ghanwani <[email-masked]>wrote:

> data
> external

---

From: Gary Berger <[email-masked]>
Date: Tue, 27 Sep 2011 10:36:05 -0400

+1
From:  Vishwas Manral <[email-masked]>
Date:  Mon, 26 Sep 2011 13:31:19 -0700
To:  Anoop Ghanwani <[email-masked]>
Cc:  "[email-masked]" <[email-masked]>
Subject:  Re: [armd] Bottleneck of Overlay network in a data center with
large number of tenants/hosts

-Vishwas
 
On Mon, Sep 26, 2011 at 12:10 PM, Anoop Ghanwani <[email-masked]>
wrote:
>> > don¹t see the hosts addresses anymore except the nodes which perform the

_______________________________________________ armd mailing list
[email-masked] https://www.ietf.org/mailman/listinfo/armd

---

From: John Menerick <[email-masked]>
Date: Tue, 27 Sep 2011 09:49:39 -0700

Agreed!

John Menerick

---

From: Benson Schliesser <[email-masked]>
Date: Tue, 27 Sep 2011 10:35:47 -0700

I also agree with Anoop¹s comment. But architecturally speaking, what we
achieve with an overlay mechanism is a form of compartmentalization (or
³partitioning², ³isolation², etc, if you prefer), e.g. of a given set of
hosts across the datacenter.  We can also achieve compartmentalization of
the network by dividing it into regions or clusters, and we can use both
regions and overlays together to get even more compartmentalization.
Whatever mechanism we use, it might allow us to distribute the address
resolution function. For instance, we can distribute L3 gateways across
multiple physical nodes or distribute ARP/ND proxy functions.

Thus, it strikes me that there may be an architectural issue for ARMD to
explore.  What is the relationship between compartmentalization and address
resolution scale? Is it only ³scale² that is affected by
compartmentalization, or are there other address resolution attributes also
affected?

Cheers,
-Benson

---

From: Gary Berger <[email-masked]>
Date: Tue, 27 Sep 2011 17:44:34 -0400

Isn't that the purpose of the network-layer?  Why do we need
compartmentalization if we have a functioning network layer and proper
address-bindings? The whole issue is in the scaling function of dynamic
learning which is a side-effect of the mobility requirement in todays data
centers.

From:  Benson Schliesser <[email-masked]>
Date:  Tue, 27 Sep 2011 10:35:47 -0700
To:  "[email-masked]" <[email-masked]>
Subject:  Re: [armd] Bottleneck of Overlay network in a data center with
large number of tenants/hosts

Re: [armd] Bottleneck of Overlay network in a data center with large number
of tenants/hosts

Cheers,
-Benson

---

From: Linda Dunbar <[email-masked]>
Date: Tue, 27 Sep 2011 22:10:59 +0000

Overlay network enables operators to move VMs from one rack to another without re-configuring IP addresses.

Linda

From: [email-masked] [mailto:[email-masked]] On Behalf Of Gary Berger
Sent: Tuesday, September 27, 2011 4:45 PM
To: Benson Schliesser; [email-masked]
Subject: Re: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts

Isn't that the purpose of the network-layer?  Why do we need compartmentalization if we have a functioning network layer and proper address-bindings? The whole issue is in the scaling function of dynamic learning which is a side-effect of the mobility requirement in todays data centers.

From: Benson Schliesser <[email-masked]<mailto:[email-masked]>>
Date: Tue, 27 Sep 2011 10:35:47 -0700
To: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts

I also agree with Anoop's comment. But architecturally speaking, what we achieve with an overlay mechanism is a form of compartmentalization (or "partitioning", "isolation", etc, if you prefer), e.g. of a given set of hosts across the datacenter.  We can also achieve compartmentalization of the network by dividing it into regions or clusters, and we can use both regions and overlays together to get even more compartmentalization.  Whatever mechanism we use, it might allow us to distribute the address resolution function. For instance, we can distribute L3 gateways across multiple physical nodes or distribute ARP/ND proxy functions.

Thus, it strikes me that there may be an architectural issue for ARMD to explore.  What is the relationship between compartmentalization and address resolution scale? Is it only "scale" that is affected by compartmentalization, or are there other address resolution attributes also affected?

Cheers,
-Benson

John Menerick

-Vishwas

On Mon, Sep 26, 2011 at 12:10 PM, Anoop Ghanwani <[email-masked]> wrote:
On Sat, Sep 17, 2011 at 11:35 AM, Linda Dunbar <[email-masked]> wrote:
> don't see the hosts addresses anymore except the nodes which perform the
...

_______________________________________________ armd mailing list [email-masked] https://www.ietf.org/mailman/listinfo/armd

________________________________
_______________________________________________
armd mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/armd
_______________________________________________ armd mailing list [email-masked]<mailto:[email-masked]> https://www.ietf.org/mailman/listinfo/armd

---

From: Anoop Ghanwani <[email-masked]>
Date: Tue, 27 Sep 2011 16:07:25 -0700

On Tue, Sep 27, 2011 at 10:35 AM, Benson Schliesser <[email-masked]> wrote:
> I also agree with Anoop’s comment. But architecturally speaking, what we
> “partitioning”, “isolation”, etc, if you prefer), e.g. of a given set of

Can you clarify what you mean by "cluster"?

Regardless, in the absence of any other protocol mechanisms,
having an overlay won't help ARP/ND in any way whatsoever.

Are you using cluster to mean a physical section of the network
that will help keep the overlays themselves contained in order to
limit the propagation of ARP/ND messages?  If so, I agree.  However,
at the limit, if we make each ToR its own cluster, then we don't
even need the overlay.

> multiple physical nodes or distribute ARP/ND proxy functions.

It is easier for me to understand distribution of the ARP/ND
proxy functions because that is purely a control-plane problem.
I have trouble with the distributed L3 gateway because that
is a data plane problem and we will hav to worry about
optimality of the forwarding paths.  Can you shed more
light on the distributed L3 gateway solution?

> resolution scale? Is it only “scale” that is affected by

What other attributes did you have in mind?  The scale issue
is clear with respect to number of MAC addresses, the
the MAC:IP bindings, and the volume of broadcast/multicast
traffic that gets sent across core links.

---

From: Christopher Morrow <[email-masked]>
Date: Wed, 28 Sep 2011 01:12:18 -0400

> without re-configuring IP addresses.

Or, to put it slightly differently....

  o Build a large L3 enabled datacenter(s) network
  o overlay L2 across this (vpls, for one example)
  o feel free to move about the datacenter(s) with your vm-image

There's still the problem at the edge nodes understanding what/where
the nodes are at L2, there's addtionaly the operations problem of
managing a changing VPLS (in this example) overlay (perhaps vlan 77
did not exist at rack 22, but you need it there now... quick, add it
to the local network devices!!)

It's intriguing, I'm not sure how well it fixes/compartmentalizes the
problem, I'm also not clear on where the cost just got shifted to :)
It also assumes that the network gear (potentially the
bottom-of-the-barrel TOR device we were dealing with before) can
perform VPLS functions.

this could due with some study it seems, to me at least.

-chris

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 28 Sep 2011 07:06:15 +0000

> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代表
> Christopher Morrow
> 发送时间: 2011年9月28日 13:12
> 收件人: Linda Dunbar
> 抄送: [email-masked]
> 主题: Re: [armd] Bottleneck of Overlay network in a data center with large
> number of tenants/hosts
> 

VPLS could actually be considered as a overlay network (L2 across L3) solution for multi-tenant data centers due to its many scaling capabilities. In fact, some vendors are proposing VPLS as a practical data center network solution to cloud data center providers. As for how to reduce the configuration and maintenance burdens associated with the legacy VPLS solutions (i.e., LDP VPLS or BGP VPLS ), IS-IS VPLS (http://tools.ietf.org/html/draft-xu-l2vpn-vpls-isis-01) is an ongoing attempt towards that direction.

Best regards,
Xiaohu

> Gary
> > Berger
> > On Sat, Sep 17, 2011 at 11:35 AM, Linda Dunbar
> <[email-masked]>
> > wrote:
> >>
> >>

---

From: Joel jaeggli <[email-masked]>
Date: Wed, 28 Sep 2011 00:28:22 -0700

On 9/28/11 00:06 , Xuxiaohu wrote:
> 
>> -----邮件原件----- 发件人: [email-masked]
>> [mailto:[email-masked]] 代表 Christopher Morrow 发送时间: 2011年9月
>> 28日 13:12 收件人: Linda Dunbar 抄送: [email-masked] 主题: Re: [armd]
>> Bottleneck of Overlay network in a data center with large number of
>> tenants/hosts
>> 
>>> Overlay network enables operators to move VMs from one rack to
>>> another without re-configuring IP addresses.
>> 
>> o Build a large L3 enabled datacenter(s) network o overlay L2
>> across this (vpls, for one example) o feel free to move about the
>> datacenter(s) with your vm-image
>> 
>> There's still the problem at the edge nodes understanding
>> what/where the nodes are at L2, there's addtionaly the operations
>> problem of managing a changing VPLS (in this example) overlay
>> (perhaps vlan 77 did not exist at rack 22, but you need it there
>> now... quick, add it to the local network devices!!)
> 
> VPLS could actually be considered as a overlay network (L2 across L3)
> solution for multi-tenant data centers due to its many scaling
> capabilities. In fact, some vendors are proposing VPLS as a practical
> data center network solution to cloud data center providers. As for
> how to reduce the configuration and maintenance burdens associated
> with the legacy VPLS solutions (i.e., LDP VPLS or BGP VPLS ), IS-IS
> VPLS (http://tools.ietf.org/html/draft-xu-l2vpn-vpls-isis-01) is an
> ongoing attempt towards that direction.

assume for a moment that I have 5000 (PE)  devices supporting 100,000
(CE) physical machines with 500,000 VMs on them. takes a few paths to
build my mesh. I presuming I end up with more that one vpls per DC as well.

> Best regards, Xiaohu
> 
>> It's intriguing, I'm not sure how well it fixes/compartmentalizes
>> the problem, I'm also not clear on where the cost just got shifted
>> to :) It also assumes that the network gear (potentially the 
>>> Berger Sent: Tuesday, September 27, 2011 4:45 PM To: Benson
>>> Schliesser; [email-masked]
>>> 
>>> Subject: Re: [armd] Bottleneck of Overlay network in a data
>>> center with large number of tenants/hosts
>>> 
>>> 
>>> 
>>> compartmentalization if we have a functioning network layer and
>>> proper address-bindings? The whole issue is in the scaling
>>> function of dynamic learning which is a side-effect of the
>>> mobility requirement in todays data centers.
>>> 
>>> 
>>> 
>>> From: Benson Schliesser <[email-masked]> Date: Tue, 27 Sep
>>> 2011 10:35:47 -0700 To: "[email-masked]" <[email-masked]> Subject:
>>> Re: [armd] Bottleneck of Overlay network in a data center with 
>>> I also agree with Anoop’s comment. But architecturally speaking,
>>> what we achieve with an overlay mechanism is a form of
>>> compartmentalization (or “partitioning”, “isolation”, etc, if you
>>> prefer), e.g. of a given set of hosts across the datacenter.  We
>>> can also achieve compartmentalization of the network by dividing
>>> it into regions or clusters, and we can use both regions and
>>> overlays together to get even more compartmentalization. Whatever
>>> mechanism we use, it might allow us to distribute the address 
>>> resolution function. For instance, we can distribute L3 gateways
>>> across multiple physical nodes or distribute ARP/ND proxy
>>> functions.
>>> 
>>> Thus, it strikes me that there may be an architectural issue for
>>> ARMD to explore.  What is the relationship between
>>> compartmentalization and
>> address
>>> compartmentalization, or are there other address resolution
>>> attributes also affected?
>>> 
>>> Cheers, -Benson
>>> 
>>> 
>>> +1 From: Vishwas Manral <[email-masked]> Date: Mon, 26
>>> Sep 2011 13:31:19 -0700 To: Anoop Ghanwani
>>> <[email-masked]> Cc: "[email-masked]" <[email-masked]> 
>>> Subject: Re: [armd] Bottleneck of Overlay network in a data
>>> center with large number of tenants/hosts
>>> 
>>>> centers with large number of tenants/hosts. When Overlay
>>>> Network is deployed in a data center, local switches/routers
>>>> within the data center network don’t see the hosts addresses
>>>> anymore except the nodes which perform
>> the
>>>> world, it frequently sends ARP/ND requests to its default
>>>> Gateway router. For an application with dual stacks on, it
>>>> sends both ARP & ND requests to its default gateway router.
>>> 
>>> I think the issue of whether or not an overlay network is in use
>>> is orthogonal to the issue of ARP/ND traffic that ARMD is trying
>>> to address. If we have an overlay network built over a data
>>> center core, we still run into the problem where ARP/ND traffic
>>> from a VM needs to be broadcast/multicast over the core (even
>>> though it uses the overlay).  So unless some other mechanism is
>>> used to mitigate the ARP/ND traffic, we would be no better than a
>>> regular L2 fabric that spans the entire data center (at least
>>> from the standpoint of the volume of ARP/ND traffic traversing 
>>> the core). _______________________________________________ armd
>>> mailing list [email-masked] 
>>> _______________________________________________ armd mailing
>>> list [email-masked] https://www.ietf.org/mailman/listinfo/armd
>>> 
>>> _______________________________________________ armd mailing
>>> list [email-masked] https://www.ietf.org/mailman/listinfo/armd
>>> 
>>> 
>>> 
>>> _______________________________________________ armd mailing
>>> list [email-masked] https://www.ietf.org/mailman/listinfo/armd
>>> 
>>> _______________________________________________ armd mailing
>>> list [email-masked] https://www.ietf.org/mailman/listinfo/armd
>>> 
>>> _______________________________________________ armd mailing
>>> list [email-masked] https://www.ietf.org/mailman/listinfo/armd
>>> 
>>>

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 28 Sep 2011 07:59:25 +0000

> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代表 Anoop
> Ghanwani
> 发送时间: 2011年9月27日 3:11
> 收件人: Linda Dunbar
> 抄送: [email-masked]

Absolutely agree. 

Besides, if the real reason for the Router’s CPU utilization reaching 40% (Merit Network statistics) or higher in many data centers is because there are too many ARP/ND requests to the gateway routers, the matured ARP cache mechanism could be deployed on the edge network devices (e.g., ToR switches) to solve the above issue. if the above reason is not true or not the major reason for the high router CPU utilization, for example, it is due to the reason that the routers that need to periodically send ARP/ND requests to keep their ARP/ND cache entries from timing out, we need to consider other solutions.

Best regards,
Xiaohu

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 28 Sep 2011 08:03:23 +0000

> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代表 Joel
> jaeggli
> 发送时间: 2011年9月28日 15:28
> 收件人: Xuxiaohu
> 抄送: [email-masked]

Can you explain more about your point?

Xiaohu