Subject: [armd] soliciting typical network designs for ARMD

From: Linda Dunbar <[email-masked]>
Date: Tue, 9 Aug 2011 12:22:01 -0500

During the 81st IETF ARMD WG discussion, it was suggested that it is
necessary to document typical data center network designs so that address
resolution scaling issues can be properly described. Many data center
operators have expressed that they can't openly reveal their detailed
network designs. Therefore, we only want to document anonymous designs
without too much detail. During the journey of establishing ARMD, we have
come across the following typical data center network designs:

   1. layer 3 all the way to TOR (Top of Rack switches),
   2. large layer 2 with hundreds (or thousands) of ToRs being
   interconnected by Layer 2. This design will have thousands of hosts under
   the L2/L3 boundary router (s)
   3. CLOS design  with thousands of switches. This design will have
   thousands of hosts under the L2/L3 boundary router(s)

We have heard that each of the designs above has its own problems. ARMD
problem statements might need to document DC problems under each typical
design.
Please send feedback to us (either to the armd email list  or to the ARMD
chair Benson & Linda) to indicate if we have missed any typical Data Center
network designs.

Your contribution can greatly accelerate the progress of ARMD WG.

Thank you very much.

Linda & Benson

---

From: Vishwas Manral <[email-masked]>
Date: Tue, 9 Aug 2011 10:58:30 -0700

Hi Linda,
I am unsure what you mean by this, but:

We can also have a heirarchical network, with the core totally Layer-3 (and
having seperate routing), from the hosts still in a large Layer-3 subnet.
Another aspect could be to have a totally Layer-3 network.

The difference between them is the link between the servers and the ToR.

Thanks,
Vishwas

---

From: Linda Dunbar <[email-masked]>
Date: Tue, 9 Aug 2011 13:51:49 -0500

Vishwas,

In my mind the bullet 1) in the list refers to ToR switches downstream ports
(facing servers) running Layer 2 and ToR uplinks ports run IP Layer 3.

Have you seen data center networks with ToR switches downstream ports (i.e.
facing servers) enabling IP routing, even though the physical links are
Ethernet?
If yes, we should definitely include it in the ARMD draft.

Thanks,
Linda
On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral <[email-masked]>wrote:

>   On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar <[email-masked]>wrote:
>

---

From: Vishwas Manral <[email-masked]>
Date: Tue, 9 Aug 2011 13:27:05 -0700

Hi Linda,

The data packets can be tunnelled at the ToR over say a GRE packet and the
core is a Layer-3 core (except for the downstream ports). So we could have
encapsulation/ decapsulation of L2 over GRE at the ToR.

The very same thing can be done at the hypervisor layer too, in which case
the entire DC network would look like a Layer-3 flat network including the
ToR to server link and the hypervisor would do the tunneling.

I am not sure if you got the points above or not. I know cloud OS companies
that provide the service and have big announced customers.

Thanks,
Vishwas
On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar <[email-masked]> wrote:

> Vishwas,
>
> In my mind the bullet 1) in the list refers to ToR switches downstream
> ports (facing servers) running Layer 2 and ToR uplinks ports run IP Layer 3.
>
>
>> We can also have a heirarchical network, with the core totally Layer-3
>> (and having seperate routing), from the hosts still in a large Layer-3
>> subnet. Another aspect could be to have a totally Layer-3 network.
>>

---

From: Vishwas Manral <[email-masked]>
Date: Tue, 9 Aug 2011 14:41:59 -0700

Hi Anoop,

>From what I know they do not use Multicast GRE (I hear the extra 4 bytes in
the GRE header is a proprietery extension).

I think a directory based mechanism is what is used (though I think if there
was a standard way to map Multicast MAC to Multicast IP, they could probably
use such a standard mechanisms).

Thanks,
Vishwas
On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani <[email-masked]>wrote:

> Hi Vishwas,
>
> How do they get multicast through the network in that case?
> Are they planning to use multicast GRE, or just use directory
> based lookups and not worry about multicast applications
> for now?
>
> Anoop
>
>   On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral <[email-masked]>wrote:
>
>>   Hi Linda,
>>
>> I am not sure if you got the points above or not. I know cloud OS
>> companies that provide the service and have big announced customers.
>>
>> Thanks,
>> Vishwas
>>   On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar <[email-masked]>wrote:
>>
>>> Vishwas,
>>>
>>> Have you seen data center networks with ToR switches downstream ports
>>> (i.e. facing servers) enabling IP routing, even though the physical links
>>> are Ethernet?
>>>   On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral <
>>> [email-masked]> wrote:
>>>
>>>> Hi Linda,
>>>>> Please send feedback to us (either to the armd email list  or to the
>>>>> ARMD chair Benson & Linda) to indicate if we have missed any typical Data
>>>>> Center network designs.
>>>>>

---

From: Anoop Ghanwani <[email-masked]>
Date: Tue, 9 Aug 2011 14:03:56 -0700

Hi Vishwas,

Anoop

On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral <[email-masked]>wrote:

>>   On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral <[email-masked]
>> > wrote:
>>
>>> Hi Linda,
> _______________________________________________
> armd mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/armd
>
>

---

From: Anoop Ghanwani <[email-masked]>
Date: Tue, 9 Aug 2011 14:50:07 -0700

(though I think if there was a standard way to map Multicast MAC to
Multicast IP, they could probably use such a standard mechanisms).

They can do that, but then this imposes requirements on the
equipment to be able to do multicast forwarding, and even if does,
because of pruning requirements the number of groups would be
very large.  The average data center switch probably won't handle
that many groups.

On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral <[email-masked]>wrote:

> I think a directory based mechanism is what is used (though I think if
> there was a standard way to map Multicast MAC to Multicast IP, they could
> probably use such a standard mechanisms).
>
> Thanks,
> Vishwas
>>> The data packets can be tunnelled at the ToR over say a GRE packet and
>>> the core is a Layer-3 core (except for the downstream ports). So we could
>>> have encapsulation/ decapsulation of L2 over GRE at the ToR.
>>>
>>> The very same thing can be done at the hypervisor layer too, in which
>>> case the entire DC network would look like a Layer-3 flat network including
>>> the ToR to server link and the hypervisor would do the tunneling.
>>>
>>>>>> We have heard that each of the designs above has its own problems.
>>>>>> ARMD problem statements might need to document DC problems under each
>>>>>> typical design.

---

From: Vishwas Manral <[email-masked]>
Date: Thu, 11 Aug 2011 20:40:44 -0700

Hi Linda/ Anoop,

Here is the example of the design I was talking about, as defined by google.
http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt

Thanks,
Vishwas
On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani <[email-masked]>wrote:

>> From what I know they do not use Multicast GRE (I hear the extra 4 bytes
>> in the GRE header is a proprietery extension).
>>
>>>   On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral <[email-masked]
>>> > wrote:
>>>
>>>>   Hi Linda,
>>>>

---

From: Warren Kumari <[email-masked]>
Date: Wed, 31 Aug 2011 15:16:09 -0400

> Here is the example of the design I was talking about, as defined by google.

Just a clarification -- s/as defined by google/as described by someone who happens to work for google/

W

> On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani <[email-masked]> wrote:
> 
> >>>>
> (though I think if there was a standard way to map Multicast MAC to Multicast IP, they could probably use such a standard mechanisms).
> >>>>
> 
> On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral <[email-masked]> wrote:
> Hi Anoop,
>  
> From what I know they do not use Multicast GRE (I hear the extra 4 bytes in the GRE header is a proprietery extension).
>  
> I think a directory based mechanism is what is used (though I think if there was a standard way to map Multicast MAC to Multicast IP, they could probably use such a standard mechanisms).
>  
> Thanks,
> Vishwas
> On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani <[email-masked]> wrote:
> Hi Vishwas,
> 
> On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral <[email-masked]> wrote:
> Hi Linda,
>  
> The data packets can be tunnelled at the ToR over say a GRE packet and the core is a Layer-3 core (except for the downstream ports). So we could have encapsulation/ decapsulation of L2 over GRE at the ToR.
>  
> The very same thing can be done at the hypervisor layer too, in which case the entire DC network would look like a Layer-3 flat network including the ToR to server link and the hypervisor would do the tunneling.
>  
> I am not sure if you got the points above or not. I know cloud OS companies that provide the service and have big announced customers.
>  
> Thanks,
> Vishwas
> In my mind the bullet 1) in the list refers to ToR switches downstream ports (facing servers) running Layer 2 and ToR uplinks ports run IP Layer 3.
>  
> Have you seen data center networks with ToR switches downstream ports (i.e. facing servers) enabling IP routing, even though the physical links are Ethernet?  
> On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral <[email-masked]> wrote:
> Hi Linda,
> 	• layer 3 all the way to TOR (Top of Rack switches),
> We can also have a heirarchical network, with the core totally Layer-3 (and having seperate routing), from the hosts still in a large Layer-3 subnet. Another aspect could be to have a totally Layer-3 network.
>  
> During the 81st IETF ARMD WG discussion, it was suggested that it is necessary to document typical data center network designs so that address resolution scaling issues can be properly described. Many data center operators have expressed that they can't openly reveal their detailed network designs. Therefore, we only want to document anonymous designs without too much detail. During the journey of establishing ARMD, we have come across the following typical data center network designs:
> 	• layer 3 all the way to TOR (Top of Rack switches),
> 	• large layer 2 with hundreds (or thousands) of ToRs being interconnected by Layer 2. This design will have thousands of hosts under the L2/L3 boundary router (s)
> 	• CLOS design  with thousands of switches. This design will have thousands of hosts under the L2/L3 boundary router(s)
> We have heard that each of the designs above has its own problems. ARMD problem statements might need to document DC problems under each typical design.
> Please send feedback to us (either to the armd email list  or to the ARMD chair Benson & Linda) to indicate if we have missed any typical Data Center network designs.
>

---

From: Linda Dunbar <[email-masked]>
Date: Wed, 31 Aug 2011 19:36:43 +0000

Warren, 

Does the referenced design map the Multicast MAC to Multicast IP?  Or map one multicast in MAC to multiple uni-cast messages? 

Linda

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Warren Kumari
> Sent: Wednesday, August 31, 2011 2:16 PM
> To: Vishwas Manral
> Cc: [email-masked]
> Subject: Re: [armd] soliciting typical network designs for ARMD
> 
> 
> > From what I know they do not use Multicast GRE (I hear the extra 4
> bytes in the GRE header is a proprietery extension).
> >
> > I think a directory based mechanism is what is used (though I think
> if there was a standard way to map Multicast MAC to Multicast IP, they
> could probably use such a standard mechanisms).
> >
> > Thanks,
> > Vishwas
> > The data packets can be tunnelled at the ToR over say a GRE packet
> and the core is a Layer-3 core (except for the downstream ports). So we
> could have encapsulation/ decapsulation of L2 over GRE at the ToR.
> >
> case the entire DC network would look like a Layer-3 flat network
> including the ToR to server link and the hypervisor would do the
> tunneling.
> >
> > In my mind the bullet 1) in the list refers to ToR switches
> downstream ports (facing servers) running Layer 2 and ToR uplinks ports
> run IP Layer 3.
> >
> (i.e. facing servers) enabling IP routing, even though the physical
> links are Ethernet?
> > 	* layer 3 all the way to TOR (Top of Rack switches),
> > We can also have a heirarchical network, with the core totally Layer-
> 3 (and having seperate routing), from the hosts still in a large Layer-
> 3 subnet. Another aspect could be to have a totally Layer-3 network.
> >
> necessary to document typical data center network designs so that
> address resolution scaling issues can be properly described. Many data
> center operators have expressed that they can't openly reveal their
> detailed network designs. Therefore, we only want to document anonymous
> designs without too much detail. During the journey of establishing
> ARMD, we have come across the following typical data center network
> designs:
> > 	* layer 3 all the way to TOR (Top of Rack switches),
> > 	* large layer 2 with hundreds (or thousands) of ToRs being
> interconnected by Layer 2. This design will have thousands of hosts
> under the L2/L3 boundary router (s)
> > 	* CLOS design  with thousands of switches. This design will have
> ARMD chair Benson & Linda) to indicate if we have missed any typical
> Data Center network designs.
> >