Subject: Re: [armd] Data center issues

From: Patrick Frejborg <[email-masked]>
Date: Tue, 1 Feb 2011 17:23:45 +0200

Hi Susan,

my intention was not to interfere with your work at the WG.

I have been involved in some large DC network architectures and ARP
scalability has never been an issue. My point is that we should try to
find ways to move away from L2 topologies towards L3 topologies as
they have at Yahoo (that is the impression I got from the link you
posted).

MPTCP is evolving and that itself provides a session layer for the
application. If we apply theoretically MPTCP on the Yahoo
architecture, then the first subflow is connected to the load balancer
(LB) - user is identified, MPTCP token exchanged and the IP address
(to setup a second subflow to the real server) is exchanged with the
client. The LB is sending the token via a trusted backdoor to the real
server so the client and real server can establish a new subflow for
the application - after that the client  uses the application without
traversing the LB anymore.

Same with VM migration - the active server freezes the application and
send the DRAM image and MPTCP tokens (session identifiers) over to the
standby server. At the same time the active server informs all his
active clients that his IP address is going to change because the
active server changes location (though the physical server is not
changing location - the VM is moving to another place). Once the
standby  server is ready it starts to accept client sessions and the
sessions are accepted once the tokens and other parameters matches.
DNS needs to be updated so new clients can find the server, since it
changed location recently.

It MPTCP goes mainstream, L2 issues in data centers would be history -
but it requires upgrades of the client stack and studies what kind of
scalability issues there are with MPTCP enabled LBs, VM migration with
MPTCP etc. My guess is that MPTCP will happen, it will be very useful
for smartphones to roam from one network to another.

Perhaps this approach is something that not belongs to IETF - perhaps
IRTF is the right forum or perhaps it do not belong here at all?

So my apologizes for interfering the WG

Patrick

On Mon, Jan 31, 2011 at 10:38 PM, Susan Hares <[email-masked]> wrote:
> Patrick:
>
> Yes, I agree that Data Center problem is bigger than just ARP/ND. However,
> to make progress on the ARP/ND portion we need to focus our work tightly.
>
> I agree that the problem needs the technologies of 100Gb Ethernet, IP/OTN or
> WDWM, 802.1aq/SPB and TRILL plus new TCP/IP stack options (STCP or new TCP
> or IPv6 options), session concepts and NM topologies to provide cross-layer
> optimization for data centers. I'm actually participating with IETF folks in
> all these areas on protocols or NM.
>
> Networks/DC are focused on expanding the L2 (see the presentation below:
> http://www.nanog.org/meetings/nanog51/presentations/Monday/NANOG51.Talk45.na
> nog51-Schaumann.pdf)
>
> IETF WG do best by tackling one part of the total solution. We are staying
> tightly focused on the ARP/ND problem, and proposing a very tight working
> group. We are focused on collecting publishable ARP/ND statistics, getting
> specific requirements, creating models for discussion, and creating
> prototype that the community can access.
>
> We have demonstrated at our BOF in Beijing that there are many solutions to
> the ARP/ND issues.
>
> Linda and I have talked to many people providing DC technology or VM
> technology or DC services. However, the topic is so hot that people do not
> want to publish their statistics due to competitive pressures. We are
> investigating anonymous data from Research networks with similar DC
> topologies.
>
> If you want to suggest a WG for session layer, I will definitely attend that
> group. I'm already tracking the rest of this work, plus the IEEE VLAN group.
>
>
> If you disagree with the tight focus, please send your comments to our ADs
> (Ralph Droms and Jari Arkko).
>
> Sue
>
>
> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Patrick Frejborg
> Sent: Monday, January 31, 2011 1:20 PM
> To: [email-masked]
> Subject: [armd] Data center issues
>
> Hi all,
>
> I'm new to the list and I just read through  the "Updated ARMD Charter and
> Milestones" thread.
>
> I do agree with Thomas Narten in his post on 27th of January, this is a much
> bigger problem than only ARP. It is also about VLAN scalability, how many
> MAC-addresses needs to be supported in the L2 domain, spanning-tree issues,
> failure domain etc
>
> Much of these challenges can today be solved by implementing VPLS, tomorrow
> they can be solved by TRILL/SPB - but a L2 broadcast domain is also IMHO a
> failure domain since a bridge floods all unknown
> packets out on all ports when a router only forward packets to known
> destinations.
>
> Thus I think the root cause to have large L2 domains is the TCP/IP stack
> itself - there is no session layer in the current stack. If the
> TCP/IP stack contained a session layer, by implementing a locator/identifier
> split, then the application could be moved from one server to another
> (physical or virtual, it doesn't matter) on the fly.
>
>
> Then we could design data center networks according to best practices, i.e.
> implement routing whenever you can and the failure domains would be much
> smaller - also ARP, VLAN, MAC scalability issues are solved.
>
> And with a session layer in the TCP/IP stack , the customer could move his
> applications from one cloud service provider to another cloud -
> over a routed network. The security nodes should not use IP addresses to
> verify the users/servers, instead the decoupled identifier should
> be used to grant/deny access to a resource.
>
> Examples of  locator/identifier split architectures are Host Identity
> Protocol (HIP), Identfier/Locator Network Protocol (ILNP), Name Based
> Sockets (NBS).
>
> My 2 cents...
>
> Patrick
> _______________________________________________
> armd mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/armd
>
>

---

From: "So, Ning" <[email-masked]>
Date: Tue, 01 Feb 2011 15:38:10 +0000

All,

I just want to point out that L3 solution may work for some DC design such as YAHOO, but it is not a universal case.  For example, TOS/COS/QoS are widely deployed in many service providers' networks, and highly valued by Enterprise/Government customers (and their applications).  Scaling L3 network may not be any easier than L2 in that operational condition.  I do see the needs to keep the options open becasue one size fits all may not work. 

Best regards,
 
Ning So
Network Evolution Planning
Verizon, Inc.
(office) 972-729-7905
(Cell) 972-955-0914
 
-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Patrick Frejborg
Sent: Tuesday, February 01, 2011 9:24 AM
To: Susan Hares
Cc: Ralph Droms; [email-masked]
Subject: Re: [armd] Data center issues

Hi Susan,

Patrick

---

From: Jari Arkko <[email-masked]>
Date: Tue, 01 Feb 2011 07:42:35 -0800

Patrick,

You are not "interfering" with the work! I think the issues that you and 
Thomas raise are important ones. These reasons are partly behind my 
desire to limit the working group to documenting the problem and 
providing operational guidance, before proceeding to development of new 
solutions.

I do think that the group should be limited to ARP and ND scaling, but 
it should not be limited in describing what factors are driving the 
scaling problems, or talking about operational ways to avoid those 
problems (different network architectures etc). I need to think if 
there's something in the charter text that should change because of 
this. Thomas, do you have any ideas about that?

Jari

---

From: Benson Schliesser <[email-masked]>
Date: Tue, 1 Feb 2011 15:40:48 -0600

I agree with Ning. One of the basic issues that advocates of all-L3 datacenter networks typically overlook (i.e. because it doesn't apply to them), is multi-tenancy.  It can be done with L3-centric designs, of course - with MPLS, tunnel overlays, etc - but it's more complex and costly.  There is a reason datacenter networks have been predominantly L2.

Cheers,
-Benson

---

From: Kathy McEwen <[email-masked]>
Date: Tue, 1 Feb 2011 16:33:00 -0600

A layer 7 approach I tightly working in co-operation with layers 0 to 7 works too. 

via mobile ...

---

From: Patrick Frejborg <[email-masked]>
Date: Wed, 2 Feb 2011 14:42:20 +0200

Hi Benson and Ning,

some thoughts on your responses:

First, it is very true that there isn't a size that fits all - but
that is currently the case we have today. We have to  implement L2
solution regardless if it is small, mid-size, large, extra-large or
extra-extra large data center.
And the reason is due to current enterprise applications and tools -
it is the only way if you can't change the application (which you
can't). My point here is that the networking people are forced to
implement L2, not because that we like to have very large L2 network
around, but we do not have any other option due to the applications an
enterprise is using.

The very large content providers, such as Yahoo and Google, seems to
have realized that large L2 networks means operational challenges -
that's my conclusion from the Yahoo presentation and Google's debate
with VMware  around cloud
http://www.infoworld.com/d/cloud-computing/google-vmware-argue-over-private-clouds-198
It seems that in order to achieve better scalability,  they are
applying changes at applications to be used over L3 networks - but I
could be wrong.

About multi-tenancy solutions, it is very common that MPLS gets
involved to solve both L2 and L3 scaling issues - and actually the L2
MPLS based solutions are way more complex to implement and operate
than a MPLS L3 solution.

The most important design criteria to keep in mind in order to scale
L2 domains are "logical ports"and "virtual ports", if those numbers
are exceeded then the network admin is sailing on deep and dark
waters. AFAIK, there is only one vendor providing these kinds of
parameters - it would be nice if every switch vendor came up with
similar guidelines because with these parameters you can determine
your L2 topology and when to move towards VPLS/TRILL/SPB.

But whatever L2 technology used it will not remove the fact that a L2
domain is also a failure domain - would like to have some hands on
TRILL to see how well TTL header dampens loops though. If there is
loop in the L2 domain then the domain is lost. So it is a business
question, do you put all your eggs in one basket or several baskets??

If you prefer to have several baskets then the operational expenditure
and issues between the isolated L2 domains becomes  much higher - here
a common L3 domain, and if the applications support L3 endpoint
mobility, would be easier and less expensive to operate. There is a
tipping point when it makes sense to switch over to a L3 based DC
network, if the applications just would allow us to do it.....so
application architects, can we soon have applications that better
support endpoint mobility, please....

Patrick

---

From: "Susan Hares" <[email-masked]>
Date: Wed, 2 Feb 2011 07:49:25 -0500

Ning, Linda, Benson:

I think Yahoo's implementation represents choices (hacks) that deny the use
of QoS/CoS/ToS bits that many industries (ex. Government) will not accept.

Yet, Yahoo made these changes because the type of L2/ARMD did not exist.  I
think this tells us that we need to hasten the timeline on the ARMD work. 

So.. I strongly agree with all of you. We need to really get moving on the
ARMD definitions and work.

Sue 

-----Original Message-----
From: Linda Dunbar [mailto:[email-masked]] 
Sent: Tuesday, February 01, 2011 5:27 PM
To: Benson Schliesser; So, Ning; Patrick Frejborg; Susan Hares; Ralph Droms;
[email-masked]
Subject: RE: [armd] Data center issues

Yahoo solution also requires changes to applications (hosts) and stealing
the DSCP bits to carry IP information. 

Linda 
-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Benson Schliesser
Sent: Tuesday, February 01, 2011 3:41 PM
To: So, Ning; Patrick Frejborg; Susan Hares; Ralph Droms; [email-masked]
Subject: Re: [armd] Data center issues

I agree with Ning. One of the basic issues that advocates of all-L3
datacenter networks typically overlook (i.e. because it doesn't apply to
them), is multi-tenancy.  It can be done with L3-centric designs, of course
- with MPLS, tunnel overlays, etc - but it's more complex and costly.  There
is a reason datacenter networks have been predominantly L2.

Cheers,
-Benson

On Feb 1, 2011, at 9:38 AM, So, Ning wrote:

> All,
> 
> I just want to point out that L3 solution may work for some DC design such
as YAHOO, but it is not a universal case.  For example, TOS/COS/QoS are
widely deployed in many service providers' networks, and highly valued by
Enterprise/Government customers (and their applications).  Scaling L3
network may not be any easier than L2 in that operational condition.  I do
see the needs to keep the options open becasue one size fits all may not
work. 
Patrick Frejborg

---

From: Lars Eggert <[email-masked]>
Date: Wed, 2 Feb 2011 15:13:38 +0200

Patrick, Costin (CC'ed) and others have recently played quite a bit with MPTCP in datacenters and I have heard there are huge gains.

---

From: Linda Dunbar <[email-masked]>
Date: Wed, 2 Feb 2011 09:16:30 -0600

Hi Patrick,

When you design large DC, do you always confine hosts belonging to one
subnet as close as possible?

The approach you described is at application layer. So is Yahoo's L3DSR
approach. Yahoo's approach requires changes to applications and also changes
to IP layer (to steal the DSCP bits to carry the IP information).
I am sure that application layer will continuously re-inventing itself to
fit into today's rigid Layer 2 boundary.
But there are a lot of applications being deployed which is hard to
change. If Layer 2 can re-inventing itself to eliminate all those
restrictions, then existing applications can have more freedom on where
they can be placed in Data Center.

Linda Dunbar

---

From: "So, Ning" <[email-masked]>
Date: Wed, 02 Feb 2011 15:40:45 +0000

All,

I see a lot of people are addressing the different needs and wants, and
what can or cannot be done.  So here is one more piece of "needs" from a
Cloud Service Provider and Data Center Operator's perspective.  Whatever
the new direction we collectively are taking, it has to have backward
compatibility and seamless transition from the current operating
environment.  Please keeping mind operators like flexibility, a lot.

Best regards,

Ning So

Verizon, Inc.

(office) 972-729-7905

(Cell) 972-955-0914

________________________________

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Linda Dunbar
Sent: Wednesday, February 02, 2011 9:17 AM
To: Patrick Frejborg; [email-masked]
Subject: Re: [armd] Data center issues

Hi Patrick, 

The approach you described is at application layer. So is Yahoo's L3DSR
approach. Yahoo's approach requires changes to applications and also
changes to IP layer (to steal the DSCP bits to carry the IP
information). 

I am sure that application layer will continuously re-inventing itself
to fit into today's rigid Layer 2 boundary. 

Linda Dunbar

On Tue, Feb 1, 2011 at 9:23 AM, Patrick Frejborg <[email-masked]>
wrote:

Hi Susan,

Patrick

> Networks/DC are focused on expanding the L2 (see the presentation
below:
http://www.nanog.org/meetings/nanog51/presentations/Monday/NANOG51.Talk4
5.na

---

From: Phil Bedard <[email-masked]>
Date: Wed, 02 Feb 2011 11:50:05 -0500

I agree with this.  I think we could solve almost all of the issues at the
application layer if it was really a choice, but it's not.

Yahoo did not have a host mobility problem, which is one of the bigger
problems identified thus far.   They had a problem of L2 domain scale and
the use of STP, with existing equipment.   Standards and vendors are moving
towards eliminating STP to increase efficiency and greatly increase the
scale of L2 domains.    Larger L2 networks are going to be the norm in the
future, whether we like it or not.

I am all for pushing application vendors to making their applications more
friendly to mobility with location identifiers and transport connectivity
which is dynamically configured.   However, people have been talking about
this for years and not much to date has materialized.

Phil     

From:  "So, Ning" <[email-masked]>
Date:  Wed, 02 Feb 2011 15:40:45 +0000
To:  Linda Dunbar <[email-masked]>, Patrick Frejborg
<[email-masked]>, <[email-masked]>
Subject:  Re: [armd] Data center issues

All,
 
I see a lot of people are addressing the different needs and wants, and what
can or cannot be done.  So here is one more piece of ³needs² from a Cloud
Service Provider and Data Center Operator¹s perspective.  Whatever the new
direction we collectively are taking, it has to have backward compatibility
and seamless transition from the current operating environment.  Please
keeping mind operators like flexibility, a lot.

Best regards,
 
Ning So

Hi Patrick, 

But there are a lot of applications being deployed which is hard to change.
If Layer 2 can re-inventing itself to eliminate all those restrictions, then
existing applications can have more freedom on where they can be placed in
Data Center. 

Linda Dunbar

Patrick

_______________________________________________ armd mailing list
[email-masked] https://www.ietf.org/mailman/listinfo/armd

---

From: Vishwas Manral <[email-masked]>
Date: Thu, 3 Feb 2011 09:41:40 -0800

Hi Patrik,

Though I agree to the argument that the main reason to use a flat
Layer-2 network is because of the applications. I do not the link you
have provided talks about the Layer-2 versus Layer-3 model. It seems
to talk about the fact that hosting applications in Public cloud is
better then in your own cloud.

Thanks,
Vishwas

---

From: Patrick Frejborg <[email-masked]>
Date: Fri, 4 Feb 2011 10:24:02 +0200

Hi Vishwas,

yes - I might conclude too much on the article, this statement:

"In the virtualization approach of private datacenters, a company
takes a server and subdivides it into many servers to increase
efficiency," Sheth writes. "We do the opposite by taking a large set
of low-cost commodity systems and tying them together into one large
supercomputer.... Enterprise hardware components are designed to be
very reliable, but they can never be 100 percent reliable, so
enterprises spend a lot of time and money on maintenance. In contrast,
we expect the hardware to fail, and design for reliability in the
software such that, when the hardware does fail, customers are just
shifted to another server."

It is unclear if the software is acting upon L2 or L3 - but they have
made some changes on the application(?) and not trying to build a
state of the art  HA infrastructure - when making changes on
applications I would aim for L3 mobility.
However, I have no facts how they do it.....

Patrick

---

From: "Susan Hares" <[email-masked]>
Date: Wed, 16 Feb 2011 09:37:06 -0500

Patrick:

I was re-reading some of my ARMD mail this morning, and I wanted to also
acknowledge that a a part of the solution can be changes to applications for
DC.  I think the DC platforms (Amazon, Yahoo, Google, Microsoft, etc) show
people are moving to this. 

I also agree that a whole suite of solutions is necessary for the community
right now. 

Sue 
-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Patrick Frejborg
Sent: Wednesday, February 02, 2011 7:42 AM
To: Benson Schliesser; [email-masked]
Cc: Ralph Droms; Susan Hares; [email-masked]
Subject: Re: [armd] Data center issues

Hi Benson and Ning,

http://www.infoworld.com/d/cloud-computing/google-vmware-argue-over-private-
clouds-198

Patrick

>> I just want to point out that L3 solution may work for some DC design
such as YAHOO, but it is not a universal case.  For example, TOS/COS/QoS are

---

From: "Susan Hares" <[email-masked]>
Date: Thu, 17 Feb 2011 17:11:27 -0500

Lars, Patrick and Costin: 

I'd like to dig into the ARMD in preparation for IETF 80.  Can you send any
additional pointers on your work? 

Many thanks,
Sue Hares

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Lars
Eggert
Sent: Wednesday, February 02, 2011 8:14 AM
To: Patrick Frejborg; Costin Raiciu
Cc: Ralph Droms; Susan Hares; [email-masked]
Subject: Spam:******, Re: [armd] Data center issues

Patrick, Costin (CC'ed) and others have recently played quite a bit with
MPTCP in datacenters and I have heard there are huge gains.

---

From: Patrick Frejborg <[email-masked]>
Date: Fri, 18 Feb 2011 20:46:42 +0200

Hi Susan,

my work is not really suitable for ARMD since it is produced within
IRTF/RRG and considered controversial
http://tools.ietf.org/html/draft-frejborg-hipv4-12
I'm currently looking into combine it with MPTCP and VL2, a use case
study for data centers

You might find the VL2 paper interesting and suitable for ARMD, "VL2:
A Scalable and Flexible Data Center Network"

Patrick

---

From: David Meyer <[email-masked]>
Date: Fri, 18 Feb 2011 12:36:19 -0800

FWIW there is quite an active research community in this area. I've put up a few
related things on http://www.1-4-5.net/~dmm/DC for other purposes but folks
might find some of it interesting...

Dave