Subject: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt

From: Warren Kumari <[email-masked]>
Date: Thu, 11 Aug 2011 19:44:09 -0700

Hi there all,

At two previous meetings (IETF in Quebec and NANOG in Denver) I threw a bit of a hissy fit about how VM mobility can be implemented without the need for L2 between the hosts.

So, here is a very drafty draft, outlining the principle at a high level…

I have no real plans for this draft, it's more just an explanation of the idea…

W

Begin forwarded message:

> From: [email-masked]
> Date: August 11, 2011 5:53:35 PM PDT
> To: [email-masked]
> Cc: [email-masked], [email-masked]
> Subject: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt
> 
> A new version of I-D, draft-wkumari-dcops-l3-vmmobility-00.txt has been successfully submitted by Warren Kumari and posted to the IETF repository.
> 
> Filename:	 draft-wkumari-dcops-l3-vmmobility
> Revision:	 00
> Title:		 Virtual Machine mobility in L3 Networks.
> Creation date:	 2011-08-11
> WG ID:		 Individual Submission
> Number of pages: 8
> 
> Abstract:
>   This document outlines how Virtual Machine mobility can be
>   accomplished in datacenter networks that are based on L3
>   technologies.  It is not really intended to solve (or fully define)
>   the problem, but rather to outline it at a very high level to
>   determine if standardization within the IETF makes sense.
> 
> 
> 
> 
> The IETF Secretariat
>

---

From: Vishwas Manral <[email-masked]>
Date: Thu, 11 Aug 2011 21:05:44 -0700

Hi Warren,

It is good to see the draft. I guess you can position this as an
informational draft of how network operators are using techniques to
overcome big layer-2 issues (the ARP issues themselves are resolved by the
directory mechanism). I think this approach has been discussed in VNRG for
some time now. May be you can use the terms defined there.

Also if I understand right, we are still using a big layer-2 network, only
thing by using a heirarchical overlay network we are doing away with the
actual network devices switches/ routers using Layer-2?

As is well known "Overlay networks" have issues with scalability, which
become worse as the number of overlays increases (think of all IPsec VPN's).
How do we deal with the same? Also what about issues with fragmentation/
management of hypervisor based shim etc?

It would be good if you could give all such details.

Thanks,
Vishwas
On Thu, Aug 11, 2011 at 7:44 PM, Warren Kumari <[email-masked]> wrote:

> Hi there all,
>
> At two previous meetings (IETF in Quebec and NANOG in Denver) I threw a bit
> of a hissy fit about how VM mobility can be implemented without the need for
> L2 between the hosts.
>
> > A new version of I-D, draft-wkumari-dcops-l3-vmmobility-00.txt has been
> successfully submitted by Warren Kumari and posted to the IETF repository.
> >
> _______________________________________________
> armd mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/armd
>

---

From: Warren Kumari <[email-masked]>
Date: Thu, 11 Aug 2011 21:33:20 -0700

On Aug 11, 2011, at 9:05 PM, Vishwas Manral wrote:

> Hi Warren,
>  
> It is good to see the draft.

And it's good to see that someone has already read it :-P

> I guess you can position this as an informational draft of how network operators are using techniques to overcome big layer-2 issues (the ARP issues themselves are resolved by the directory mechanism). I think this approach has been discussed in VNRG for some time now. May be you can use the terms defined there.

Yup.

>  
> Also if I understand right, we are still using a big layer-2 network, only thing by using a heirarchical overlay network we are doing away with the actual network devices switches/ routers using Layer-2?

Kinda -- the very high level view is that datacenters can be built with L3 designs (L2 designs, or whatever design happens to exists), and then separate overlay networks, one per customer.

>  
> As is well known "Overlay networks" have issues with scalability, which become worse as the number of overlays increases (think of all IPsec VPN's).

Yes, but as the overlay is only built between the VM hosts (and only *those* that are actually communicating), the number of overlays that needs to be tracked *per device* is fairly limited. Also, the overlays are built on general purpose servers (and are invisible to the network) and so you don't have all of the standard scaling issues that happen with network devices...

> How do we deal with the same? Also what about issues with fragmentation/ management of hypervisor based shim etc?

Yup, there is still lots of work to be done on specifying things like that -- who does fragmentation / reassembly, a standard protocol for populating the directory, how the DS informs the shims that a VM has moved, etc are all (currently) coverd by hand-waving -- I do actually have answers on most of that, but I don't have them written down….

---

From: Vishwas Manral <[email-masked]>
Date: Thu, 11 Aug 2011 22:38:16 -0700

Hi Warren,

Great to read the stuff in draft, which we all know has existed in Google.
You rightly mentioned that a lot of the parts still need to be standardized.
Am not sure whre that will be done as I hear ARMD has a very narrow focus.

BTW I know people use L2TP tunnels instead of GRE in a lot of cases.
Theinteroperability needs to be looked at too even in such cases. Also
things like QoS need to be mentioned and looked at and mentioned.

Thanks,
Vishwas
On Thu, Aug 11, 2011 at 9:33 PM, Warren Kumari <[email-masked]> wrote:

> > I guess you can position this as an informational draft of how network
> operators are using techniques to overcome big layer-2 issues (the ARP
> issues themselves are resolved by the directory mechanism). I think this
> approach has been discussed in VNRG for some time now. May be you can use
> the terms defined there.
>
> Yup.
>
> >
> > Also if I understand right, we are still using a big layer-2 network,
> only thing by using a heirarchical overlay network we are doing away with
> the actual network devices switches/ routers using Layer-2?
>
> Kinda -- the very high level view is that datacenters can be built with L3
> designs (L2 designs, or whatever design happens to exists), and then
> separate overlay networks, one per customer.
>
> >
> Yes, but as the overlay is only built between the VM hosts (and only
> *those* that are actually communicating), the number of overlays that needs
> to be tracked *per device* is fairly limited. Also, the overlays are built
> on general purpose servers (and are invisible to the network) and so you
> don't have all of the standard scaling issues that happen with network
> devices...
>
> Yup, there is still lots of work to be done on specifying things like that
> -- who does fragmentation / reassembly, a standard protocol for populating
> the directory, how the DS informs the shims that a VM has moved, etc are all
> (currently) coverd by hand-waving -- I do actually have answers on most of
> that, but I don't have them written down….
>
> >
> > At two previous meetings (IETF in Quebec and NANOG in Denver) I threw a
> bit of a hissy fit about how VM mobility can be implemented without the need
> for L2 between the hosts.
> >

---

From: <[email-masked]>
Date: Fri, 12 Aug 2011 09:08:59 -0400

Hi Warren,

> And it's good to see that someone has already read it :-P

Make that at least two someones ;-).  Thanks for getting the conversation started.

A detailed example of related technology can be found in draft-hasmit-otv-03
(http://www.ietf.org/id/draft-hasmit-otv-03.txt).  This L2-in-L3 functionality
(MAC-in-IP) is targeted at carrier/provider networks rather than data center
networks, and uses UDP encapsulation (e.g., as opposed to GRE).  Courtesy of
the focus on carrier/provider networks, one of the areas of design emphasis
is avoidance of flooding. 

Please don't ask me about OTV details - the draft authors (e.g., Dino) would
be a better resource for those sorts of questions.

Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Warren Kumari
> Sent: Friday, August 12, 2011 12:33 AM
> To: Vishwas Manral
> Cc: [email-masked]
> Subject: Re: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt
> 
> 
> > I guess you can position this as an informational draft of how network operators are using
> techniques to overcome big layer-2 issues (the ARP issues themselves are resolved by the directory
> mechanism). I think this approach has been discussed in VNRG for some time now. May be you can use the
> terms defined there.
> 
> Yup.
> 
> >
> > Also if I understand right, we are still using a big layer-2 network, only thing by using a
> heirarchical overlay network we are doing away with the actual network devices switches/ routers using
> Layer-2?
> 
> Kinda -- the very high level view is that datacenters can be built with L3 designs (L2 designs, or
> whatever design happens to exists), and then separate overlay networks, one per customer.
> 
> >
> > As is well known "Overlay networks" have issues with scalability, which become worse as the number
> of overlays increases (think of all IPsec VPN's).
> 
> Yes, but as the overlay is only built between the VM hosts (and only *those* that are actually
> communicating), the number of overlays that needs to be tracked *per device* is fairly limited. Also,
> the overlays are built on general purpose servers (and are invisible to the network) and so you don't
> have all of the standard scaling issues that happen with network devices...
> 
> Yup, there is still lots of work to be done on specifying things like that -- who does fragmentation /
> reassembly, a standard protocol for populating the directory, how the DS informs the shims that a VM
> has moved, etc are all (currently) coverd by hand-waving -- I do actually have answers on most of
> that, but I don't have them written down..
> 
> >
> > At two previous meetings (IETF in Quebec and NANOG in Denver) I threw a bit of a hissy fit about how
> VM mobility can be implemented without the need for L2 between the hosts.
> >
> > So, here is a very drafty draft, outlining the principle at a high level.
> >
> > I have no real plans for this draft, it's more just an explanation of the idea.
> >
> > W
> >
> >
> > Begin forwarded message:
> >
> > > A new version of I-D, draft-wkumari-dcops-l3-vmmobility-00.txt has been successfully submitted by
> Warren Kumari and posted to the IETF repository.
> > >

---

From: Gary Berger <[email-masked]>
Date: Fri, 12 Aug 2011 10:01:45 -0400

I would hope that we can find a more elegant approach to the problem of
dealing with dynamic address learning and mobility instead of adding
overlays. 

We have created the L2 scaling problem (which shows itself in mobility and
control plane scalability) because of the fact that the data-link address,
IP Address and what we use to identify the host (I.e. The DNS name) are all
bound to the point of attachment. This created the desire for the "flat"
network which as we know doesn't scale. It would be great if part of this
working group can look from the point of view that the Internet is an
experiment and we never evolved to a clean implementation. Decoupling the
service (I.e. Where a socket call connects to) from the node address and the
point of attachment will go a long way to fixing the scaling challenges. A
better description  can be found in Saltzer, et al  RFC-1498
<http://tools.ietf.org/html/rfc1498> . Also, one of the chief OSI developers
John Day has an interesting perspective on this here
http://pouzin.pnanetworks.com/images/KoreaNamingFund100218.pdf

-g

On 8/12/11 9:08 AM, "[email-masked]" <[email-masked]> wrote:

>>> operators are using
>>> only thing by using a
>> designs (L2 designs, or
>>> become worse as the number
>> -- who does fragmentation /
>>> bit of a hissy fit about how

---

From: Warren Kumari <[email-masked]>
Date: Fri, 12 Aug 2011 08:43:21 -0700

> is avoidance of flooding. 

Oooh, thank you, sir…
So far I have only skimmed it, but so far it looks really good as a solution…..

> be a better resource for those sorts of questions.

Cool.

Thanks again,
W

---

From: Vishwas Manral <[email-masked]>
Date: Fri, 12 Aug 2011 09:50:04 -0700

Hi Gary,

Thanks for the links. I will read through them.

Is OTV as pointed out by David Black also not an overlay?

Thanks,
Vishwas
On Fri, Aug 12, 2011 at 7:01 AM, Gary Berger <[email-masked]> wrote:

> better description  can be found in Saltzer, et al  RFC-1498<http://tools.ietf.org/html/rfc1498>.
> Also, one of the chief OSI developers John Day has an interesting
> perspective on this here
> From: [email-masked] [mailto:[email-masked]<[email-masked]>]
> On Behalf Of Warren Kumari
> *those* that are actually
>  > How do we deal with the same? Also what about issues with
> fragmentation/ management of hypervisor
> based shim etc?
> successfully submitted by

---

From: Gary Berger <[email-masked]>
Date: Fri, 12 Aug 2011 12:52:16 -0400

From:  Vishwas Manral <[email-masked]>
Date:  Fri, 12 Aug 2011 09:50:04 -0700
To:  Gary Berger <[email-masked]>
Cc:  <[email-masked]>, <[email-masked]>, <[email-masked]>
Subject:  Re: [armd] Fwd: New Version Notification for
draft-wkumari-dcops-l3-vmmobility-00.txt

Hi Gary,

 
@gaberger: I would say anything with encapsulation is utterly an overlay..
This is certainly a workable solution within scope. The question is if you
want to address the core architectural flaws or paint over them with sexy
implementations which incur their own side-effects..

Thanks,
Vishwas
On Fri, Aug 12, 2011 at 7:01 AM, Gary Berger <[email-masked]> wrote:
> dealing with dynamic address learning and mobility instead of adding overlays.
> 
> control plane scalability) because of the fact that the data-link address, IP
> Address and what we use to identify the host (I.e. The DNS name) are all bound
> to the point of attachment. This created the desire for the "flat" network
> which as we know doesn't scale. It would be great if part of this working
> group can look from the point of view that the Internet is an experiment and
> we never evolved to a clean implementation. Decoupling the service (I.e. Where
> a socket call connects to) from the node address and the point of attachment
> will go a long way to fixing the scaling challenges. A better description  can
> be found in Saltzer, et al  RFC-1498 <http://tools.ietf.org/html/rfc1498> .
> Also, one of the chief OSI developers John Day has an interesting perspective
> on this here http://pouzin.pnanetworks.com/images/KoreaNamingFund100218.pdf
> 
> -g
> 
>> +1 (508) 293-7953 <tel:%2B1%20%28508%29%20293-7953>              FAX: +1
>> (508) 293-7786 <tel:%2B1%20%28508%29%20293-7786>
>> <tel:%2B1%20%28978%29%20394-7754>

---

From: Phil Bedard <[email-masked]>
Date: Fri, 12 Aug 2011 13:06:36 -0400

I've read it as well and have heard of the approach or similar approaches
in the past using both GRE and L2TP.  My main concerns are around the
burden it places on the hypervisor, especially in cases of very fragmented
clusters of VMs where there may be a lot of encap/decap going on.  Also
the process of external traffic reaching the destination host.

OTV extends L2 domains in the same way VPLS does and places the burden of
configuring the edge network devices to make sure the network to
Hypervisor switch connection is in the right L2 domain.  There is no
mapping service and instead uses a dynamic control plane to disseminate
MAC and location information (IS-IS).  Ethernet VPN,
http://tools.ietf.org/html/draft-raggarwa-sajassi-l2vpn-evpn-02 is similar
only using BGP to distribute MAC information and uses MPLS to encapsulate
frames.   Both could work for L3 mobility but require the network devices
be configured as part of the mobility process.  And of course means you
need L3 devices supporting the aforementioned protocols.   The Ethernet
VPN concept using BGP (or any other control plane protocol) could be
extended to tunneling mechanisms other than MPLS.

I think the only way to move away from overlays is more of a paradigm
shift where we are no longer using IP addresses or MAC addresses to send
packets to end hosts.

Phil

---

From: <[email-masked]>
Date: Fri, 12 Aug 2011 13:37:28 -0400

Hi Gary,

Are you aware of the IETF work already in progress on that sort of decoupling, e.g., LISP, HIP and ILNP?

I don't understand why another effort in this space would be useful.

Thanks,
--David

From: Gary Berger [mailto:[email-masked]]
Sent: Friday, August 12, 2011 10:02 AM
To: Black, David; [email-masked]
Cc: [email-masked]
Subject: Re: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt

I would hope that we can find a more elegant approach to the problem of dealing with dynamic address learning and mobility instead of adding overlays.

We have created the L2 scaling problem (which shows itself in mobility and control plane scalability) because of the fact that the data-link address, IP Address and what we use to identify the host (I.e. The DNS name) are all bound to the point of attachment. This created the desire for the "flat" network which as we know doesn't scale. It would be great if part of this working group can look from the point of view that the Internet is an experiment and we never evolved to a clean implementation. Decoupling the service (I.e. Where a socket call connects to) from the node address and the point of attachment will go a long way to fixing the scaling challenges. A better description  can be found in Saltzer, et al  RFC-1498<http://tools.ietf.org/html/rfc1498>. Also, one of the chief OSI developers John Day has an interesting perspective on this here http://pouzin.pnanetworks.com/images/KoreaNamingFund100218.pdf

-g

On 8/12/11 9:08 AM, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>> wrote:

Hi Warren,

Thanks,
--David
[email-masked]<mailto:[email-masked]>        Mobile: +1 (978) 394-7754

-----Original Message-----
From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Warren Kumari
Sent: Friday, August 12, 2011 12:33 AM
To: Vishwas Manral
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt
> On Thu, Aug 11, 2011 at 7:44 PM, Warren Kumari <[email-masked]<mailto:[email-masked]>> wrote:
> Hi there all,
>
> > From: [email-masked]<mailto:[email-masked]>
> > To: [email-masked]<mailto:[email-masked]>
> > Cc: [email-masked]<mailto:[email-masked]>, [email-masked]<mailto:[email-masked]>
> [email-masked]<mailto:[email-masked]>
_______________________________________________
armd mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/armd

_______________________________________________
armd mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/armd

---

From: Gary Berger <[email-masked]>
Date: Fri, 12 Aug 2011 13:51:08 -0400

I am not advocating any such thing.. I merely am trying to point to a bit of
history and the problem at hand; as noted in the ARMD Call for
Investigation:

One such aspect,being investigated by the ARMD working group, is the scaling
of address resolution between the network (L3) and link (L2) layers of
modern datacenter networks.

You cannot decouple the resolution and binding of addresses without
understanding the limitations in the current architecture. LISP doesn't
scale because of the path liveness problem, I am not familiar enough with
EIP or ILNP but I am sure that this issue is long-lived and unsolved and at
the root of our scaling challenges not just in DC but across the Internet..

-g

From:  <[email-masked]>
Date:  Fri, 12 Aug 2011 13:37:28 -0400
To:  Gary Berger <[email-masked]>
Cc:  <[email-masked]>
Subject:  RE: [armd] Fwd: New Version Notification for
draft-wkumari-dcops-l3-vmmobility-00.txt

Hi Gary,
 
Are you aware of the IETF work already in progress on that sort of
decoupling, e.g., LISP, HIP and ILNP?
 
I don¹t understand why another effort in this space would be useful.

Thanks,
--David

-g

On 8/12/11 9:08 AM, "[email-masked]" <[email-masked]> wrote:

>>> thing by using a
>>> management of hypervisor

---

From: "So, Ning" <[email-masked]>
Date: Fri, 12 Aug 2011 14:12:03 -0400

Gary,

Can you elaborate a bit on LISP's path liveness problem?  Is it documented anywhere?

Best regards,

Ning So
Verizon Corporate Technology
(office) 972-729-7905
(Cell) 972-955-0914

From: [email-masked] [mailto:[email-masked]] On Behalf Of Gary Berger
Sent: Friday, August 12, 2011 12:51 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt

I am not advocating any such thing.. I merely am trying to point to a bit of history and the problem at hand; as noted in the ARMD Call for Investigation:

One such aspect,being investigated by the ARMD working group, is the scaling of address resolution between the network (L3) and link (L2) layers of modern datacenter networks.

You cannot decouple the resolution and binding of addresses without understanding the limitations in the current architecture. LISP doesn't scale because of the path liveness problem, I am not familiar enough with EIP or ILNP but I am sure that this issue is long-lived and unsolved and at the root of our scaling challenges not just in DC but across the Internet..

-g

From: <[email-masked]<mailto:[email-masked]>>
Date: Fri, 12 Aug 2011 13:37:28 -0400
To: Gary Berger <[email-masked]<mailto:[email-masked]>>
Cc: <[email-masked]<mailto:[email-masked]>>
Subject: RE: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt

Hi Gary,

Thanks,
--David

To: Black, David; [email-masked]<mailto:[email-masked]>

-g

Hi Warren,

Thanks,
--David

---

From: Steven Blake <[email-masked]>
Date: Fri, 12 Aug 2011 14:30:14 -0400

On Fri, 2011-08-12 at 11:12 -0700, So, Ning wrote:

> Gary,
> 
>  
> 
> Can you elaborate a bit on LISP’s path liveness problem?  Is it
> documented anywhere?

http://tools.ietf.org/html/draft-meyer-loc-id-implications-01

Note that the problem is not specific to loc/id separation; it is a
general problem for multi-addressed hosts.

Regards,

/////////////////////////////////////////////
Steven Blake       [email-masked]
Extreme Networks              +1 919-884-3211

---

From: Patrick Frejborg <[email-masked]>
Date: Sat, 13 Aug 2011 10:41:01 +0300

On Fri, Aug 12, 2011 at 5:01 PM, Gary Berger <[email-masked]> wrote:
> better description  can be found in Saltzer, et al  RFC-1498. Also, one of
> the chief OSI developers John Day has an interesting perspective on this
> here http://pouzin.pnanetworks.com/images/KoreaNamingFund100218.pdf

Have also a look on the SCAFFOLD paper
ftp://ftp.cs.princeton.edu/reports/2010/885.pdf
That would remove the L2 scaling problem for online services

The approach requires a hierarchical addressing scheme, check out
RFC6306, which is focusing on scalability for the routing architecture
of the Internet.

The identifier mechanism described in SCAFFOLD can be applied with the
packet header described in RFC6306 though it is not discussed in
detail in the document. This combination of the two separate works
would introduce a session identifier (by MPTCP/SCTP) and service
identifier (by SCAFFOLD) that are decoupled from the addressing scheme
(locators).

Patrick

---

From: "Gary Berger (gaberger)" <[email-masked]>
Date: Sat, 13 Aug 2011 09:27:29 -0400

Hi Patrick,

Yes I have read SCAFFOLD and I like the approach. Fundamentally you either solve lookup through exhaustive search or through hinting and a hierarchal address is critical to this. It doesn't solve the problem completely but it's a good vision of where we can reformulate our view of binding layers.  From an appdev point of view being able to target an abstract topology seems incredibly beneficial. The topology can be geographically bounded, segmented by service or customer. There is a group lead by Martin Sustrik  working on a scalability protocol to abstract the network by communication patterns I.e pub/sub, req/rep, etc. but these proposals breakdown because we route on the point of attachment not on the node. You can side-step with a proxy  pattern (SCAFFOLD) if you are willing to add latency and indirection.  

Tks

G

Sent from my iPhone

---

From: Jim Rees <[email-masked]>
Date: Fri, 12 Aug 2011 07:09:43 -0400

Warren Kumari wrote:

  Hi there all,
  
  At two previous meetings (IETF in Quebec and NANOG in Denver) I threw a
  bit of a hissy fit about how VM mobility can be implemented without the
  need for L2 between the hosts.

This reminds me a bit of Mobile IP (rfc5944).

---

From: Patrick Frejborg <[email-masked]>
Date: Mon, 15 Aug 2011 09:26:15 +0300

Hi Gary,

I'm interested to hear what you think is missing from the SCAFFOLD
paper, think there are some minor issues but they can be improved.

I also believe that routing always has to be applied against
attachment points, i.e. locators are used for routing. Thus you to do
a binding of locators and identifiers (nodes) somewhere, how the
binding (of Loc/ID to produce the services) is designed is the real
challenge - you could end up with scalability, privacy and latency
issues - some additional latency is imposed but the later the binding
is applied the less latency there is going to be.

IMHO, if you do routing based on identifiers then there is really no
decoupling of locators and identifiers - of course this depends on
one's definition of Loc/ID split.

Patrick

---

From: Gary Berger <[email-masked]>
Date: Tue, 16 Aug 2011 09:24:34 -0400

Patrick,

Where to start.. Ugh.. Let me just say I think the approach Scaffold takes
is innovative especially in the light of bringing together applications,
services and networking.. Certainly we have learned that topology is a lot
more complicated than just a physical graph and moving towards decoupling
the physical topology will go along way with providing scalable services.

As per SCAFFOLD, I have to say I have not thought through exhaustively the
design but I can give you some things that just give me pause.
* Some of the implementation issues are scary.. I.e. Changing the data
plane, control plane, the network stack and programming model..
> * These are HUGE changes, not necessarily impossible but definitely a
> challenge
> * Kudos for the innovative thinking. I think there is growing practice in
> looking at the network stack especially from a messaging point of view instead
> of an opaque streaming interface[1] but it still is very scary and if we are
> going to go to this extent maybe we should look at the entire protocol stack..
* Pushing the bindings into the server are creative and maybe a first
attempt to look at these alternatives while they mature possibly migrating
into the dataplane of the network. Operationally people are going to get a
bit burned without the proper OAM tools.
*  I do like that you have added the elusive application service. Having a
proper binding for this is important as well as the appropriate lookup
services. Ip's "well-known" ports was a hack because we had no such
directory. You do answer Saltzers first objective:
1. A given service may run at one or more nodes, and may need to move from
one node to another without losing its identity as a service.[3]
* But again we are missing the Node Address. Saltzer missed this, if you
believe the conjecture it was because multi-homed node systems were not
envisioned at this point. The IP Address is naming the subnetwork attachment
point, this is ultimately the scaling problem..[2] What is the "identity as
a node" he is talking about? An identity needs to be unique within the scope
of the layer (we don't have this with MAC or IP today).
2. A given node may be connected to one or more network attachment points,
and may need to move from one attachment point to another without losing its
identity as a node.[3]
* In the paper it doesn't seem you have the concept of an identity in fact
under VM migration you talk about cycling the network interface to get a new
Scaffold address.. In which case where is the identity of the node?
* There is also overloading the hostAddr name with ASAddr and sockID in the
example implementation.. This in practice may be challenging but I assume
this is just an experimental viewpoint.

"IMHO, if you do routing based on identifiers then there is really no
decoupling of locators and identifiers - of course this depends on one's
definition of Loc/ID split"

I don't think this has to do with loc/id split, that in fact maybe a false
path[2].. But you do routing on a "name/address" relative to the layer.
Mux/Demux is just composition/decomposition in the network domain.

If we take the perspective that devices are multi-homed as the canonical
model;  examples being  a device moving from AP to AP, Wi-fi to 3G/4G or DC
to DC. Scaffold goes to a lot of effort to make this seamless but the
complexity should be an indicator something is wrong.

I put together a short example here:
http://www.slideshare.net/gaberger/scaffold-8866328 which describes
Saltzer's model as well as Scaffold. I also took a stab at what it would
actually look like if you incorporated a Node Identifier in the address
architecture. I am not saying that this is an easy topic, its been 40 years
in the making, the question is where do we concentrate our efforts in
properly designing the address architecture and managing the layer
bindings.. The latter as a charter in this group "the what", the former as a
necessity in understanding "the why".

1. http://www.250bpm.com/concepts
2. http://pouzin.pnanetworks.com/images/LocIDSplit090309.pdf
3. http://tools.ietf.org/html/rfc1498

On 8/15/11 2:26 AM, "Patrick Frejborg" <[email-masked]> wrote:

>>  Yes I have read SCAFFOLD and I like the approach. Fundamentally you either
>> solve lookup through exhaustive search or through hinting and a hierarchal
>> address is critical to this. It doesn't solve the problem completely but it's
>> a good vision of where we can reformulate our view of binding layers.  From
>> an appdev point of view being able to target an abstract topology seems
>> incredibly beneficial. The topology can be geographically bounded, segmented
>> by service or customer. There is a group lead by Martin Sustrik  working on a
>> scalability protocol to abstract the network by communication patterns I.e
>> pub/sub, req/rep, etc. but these proposals breakdown because we route on the
>> point of attachment not on the node. You can side-step with a proxy  pattern
>> (SCAFFOLD) if you are willing to add latency and indirection.
>> 
>>  Tks
>> 
>>  G
>> 
>>  Sent from my iPhone
>>

---

From: Patrick Frejborg <[email-masked]>
Date: Tue, 16 Aug 2011 19:52:14 +0300

Hi Gary,

our discussion is way off the topic for ARMD, not sure if we can continue
our discussion on the list - but I continue a while, and if somebody feel
that we shouldn't have the discussion, please drop a note and I take our
discussion off the list.

First of all, I'm not the inventor behind SCAFFOLD. I have focused on RFC
6303 which is a controversial research work from RGG and focusing on routing
scalability. I stumbled on SCAFFOLD somewhere in February-April, found it
interesting and also "compatible" with RFC 6306 - then I found out what to
do with the identifier.

Comments inline

On Tue, Aug 16, 2011 at 4:24 PM, Gary Berger <[email-masked]> wrote:

>    - Some of the implementation issues are scary.. I.e. Changing the data
>       - These are HUGE changes, not necessarily impossible but definitely
>       a challenge
>
> If you have a look on RFC 6306, you'll notice that the current data plane
is compatible - no changes are needed but the middleboxes and some
additional nodes needs to be changed/added. Only one change is needed to the
control plane, i.e.to add support for ICMP extensions, but that shouldn't be
a major showstopper for experiments. New extensions are needed to the
networking stack, some applications will suffer (but what ever is done, some
applications will suffer). My knowledge about the programming model is very
limited, thus no comments on that.

>
>    - Kudos for the innovative thinking. I think there is growing practice
>       in looking at the network stack especially from a messaging point of view
>       instead of an opaque streaming interface[1] but it still is very scary and
>       if we are going to go to this extent maybe we should look at the entire
>       protocol stack..
>    - Pushing the bindings into the server are creative and maybe a first
>    -  I do like that you have added the elusive application service.
>    Having a proper binding for this is important as well as the appropriate
>    lookup services. Ip's "well-known" ports was a hack because we had no such
>    - But again we are missing the Node Address. Saltzer missed this, if
>    you believe the conjecture it was because multi-homed node systems were not
> Hmm. here I disagree :-)

This is the thinking from the past - what we have been trying to do in the
data centers during the past +10 years is to break up this host-to-host
architecture. When I have designed data centers you get to a point when you
have to decide what to do with the most important services that need an
availibilty of five nines. There is desire to break the host-to-host concept
and replace it with a host-to-many-hosts concept - thus you implement some
kind of load balancing solution (SLB, DNS, etc) in front of your real
servers so you can have a host-to-many-hosts solution. But what I realized,
after reading the SCAFFOLD paper, is what we have been trying to accomplish
that past +10 years is to create a host-to-service solution with help of
these load balancing kludges - e.g. writing this e-mail I don't care on
which node the mail gets composed, the most important thing is that the
service is available!
Did Saltzer think of online services, where there online services on the
Internet at that time? I believe that mainframes (node) to produce a high
available service was mainstream at that time - the online service concept
(of the magnitude we see today) has arrived later. The demand for
mission-critical services are growing, and here we sit with an outdated
stack in our hands, trying to respond to the new requirements.

Also, if you read RFC 6306 (e.g. Appendix C) you'll notice that the node has
a two level "attachment point" - one level describing how the node is
connected to the local network and how the local network is connected to the
Internet. This model is like the highway where you have several exits to a
destination and the nodes can leverage this information to reach another
node. The current model that we use, is more or less like the railway -
nodes have no choice to make a decision which path is used to reach the
destination, instead the railway (network) tries to figure out which path
works best for the nodes. If we could use a two-level addressing scheme we
could replace multi-homing with multi-pathing and let the transport protocol
figure out which path works best for them (like the driver on the highway).
Oh, and the transport protocol is pretty good on liveness detection also.

>    -
>    In the paper it doesn't seem you have the concept of an identity in fact under VM migration you talk about cycling the network interface to get a new Scaffold address.. In which case where is the identity of the node?
>
>
>
>    - There is also overloading the hostAddr name with ASAddr and sockID in
>    the example implementation.. This in practice may be challenging but I
>    assume this is just an experimental viewpoint.
>
>
Yes, drop this packet header structure and use the one described in RFC 6306
- it is better suited for the current Internet and there is placeholder for
an identifier.

MPTCP do have a token that can be used as a session identifier - then the
application can be moved from one node to another - and I don't care if this
e-mail gets composed in Finland or Belgium, as long as the service is
available.

For these two

>
> "IMHO, if you do routing based on identifiers then there is really
> no decoupling of locators and identifiers - of course this depends on one's

I don't think SCAFFOLD address this issue - this is better suited for MPTCP
and they have work in progress to solve roaming issues.

> necessity in understanding "the why".

I'll have a look on that - I do have a presentation about hIPv4 and uses
cases but it need to be updated before releasing it.

Patrick

---

From: Erik Nordström <[email-masked]>
Date: Tue, 16 Aug 2011 17:08:33 -0400

Hi All,

I am one of the authors of the SCAFFOLD paper and I learned of this discussion just a few days ago. I thought I'd chip in with a few comments, updates and clarifications regarding our work. I hope I can cover most of the SCAFFOLD-related issues that have popped up. My apologies if some of this is out of the list's scope, but I hoped to also add more context for other list subscribers.

First of all, SCAFFOLD has been superseded by our new, less radical, approach to service-centric networking, called Serval (for Service Access Layer). You'll find a link to a manuscript of a recent Serval paper below.

Serval realizes most features of SCAFFOLD (e.g., service naming with a group abstraction, service-layer anycast, and so forth) while sitting on top of IP instead of redefining the entire stack. This makes Serval easier to deploy and integrate with existing infrastructure and data centers. In fact, we have a running prototype implementation for the Linux kernel (as a module) that runs also on Android. We can hence deploy and run Serval today on existing hosts by simply inserting our kernel module. The existing TCP/IP stack can, of course, continue to run in parallel. We also have an in-network translator that allows legacy hosts to connect to Serval networks, and have implemented legacy NAT traversal through UDP encapsulation (as somewhat standard). I think this addresses many of the complexity and deployment issues that have been raised in relation to our earlier work on SCAFFOLD.

While SCAFFOLD focused on replicated services, Serval takes a broader view of the interest in handling 'multiplicity'. As such, it also adds support for multi-homed / multi-interfaced hosts, supporting interface and (virtual host) migration (allowing IP addresses to change without breaking TCP connections) and multi-path TCP (although the latter has not yet been implemented).  Because the IP addresses of ongoing flows can change, we've demo'd migrating VMs across layer-3 boundaries.

Serval realizes the above functionality by defining a Service Access Layer (SAL) in-between the network and transport layer. The SAL implements primitives for building scalable service resolution schemes that allow mapping a service name to an IP address (i.e., service instance) based on a service name in the first packet of a connection. This resolution allows efficient load balancing of connections on this first packet only. Serval provides flexible service resolution primitives instead of a one-size-fits-all solution. In fact, we explicitly designed Serval so that it is easy to build and use multiple resolution schemes concurrently (e.g., broadcast in the local segment, directory services or static configuration in the data center, or leveraging DNS or service routing schemes similar to BGP or LISP+ALT in the wide area). 

As pointed out earlier on the list, SCAFFOLD and Serval bear resemblance to HIP and other layer 3.5s. The main difference, in terms of naming, is that Serval names services (i.e., a group of processes) and not hosts. Serval also untangles the overloaded use of ports and addresses by introducing serviceIDs and flowIDs (previously called sockIDs) that explicitly name services and flows, respectively. (I guess our flowID would work as the sessionID that was mentioned earlier in this thread.) We actually decided against adding an explicit host identifier to our architecture. The main reason was that we found no compelling need for such an identifier that would warrant an additional name/ID/address mapping scheme. In most cases, end hosts can track each other across mobility/migration events via in-band signaling without explicit host identities. In some corner cases (e.g., simultaneous migration) one can use rendezvous points, or similar solutions. For those hosts that for some reason need their own globally unique name/ID, it is perfectly fine to assign them their own serviceID (i.e., forming a singleton group).

Below you will find the link to the manuscript of our most recent Serval paper. We encourage those interested to take a look at this paper that accurately reflects our current work. We are happy to continue the discussion regarding the use of Serval in ARMD settings and we also welcome suggestions and feedback.

http://sns.cs.princeton.edu/projects/serval/

Best regards,

--Erik

On Aug 16, 2011, at 12:52 PM, Patrick Frejborg wrote:

> Hi Gary,
> 
> our discussion is way off the topic for ARMD, not sure if we can continue our discussion on the list - but I continue a while, and if somebody feel that we shouldn't have the discussion, please drop a note and I take our discussion off the list.
> 
> First of all, I'm not the inventor behind SCAFFOLD. I have focused on RFC 6303 which is a controversial research work from RGG and focusing on routing scalability. I stumbled on SCAFFOLD somewhere in February-April, found it interesting and also "compatible" with RFC 6306 - then I found out what to do with the identifier.
> 
> Comments inline
> 
> Where to start.. Ugh.. Let me just say I think the approach Scaffold takes is innovative especially in the light of bringing together applications, services and networking.. Certainly we have learned that topology is a lot more complicated than just a physical graph and moving towards decoupling the physical topology will go along way with providing scalable services.
> 
> As per SCAFFOLD, I have to say I have not thought through exhaustively the design but I can give you some things that just give me pause.
> Some of the implementation issues are scary.. I.e. Changing the data plane, control plane, the network stack and programming model.. 
> These are HUGE changes, not necessarily impossible but definitely a challenge
> If you have a look on RFC 6306, you'll notice that the current data plane is compatible - no changes are needed but the middleboxes and some additional nodes needs to be changed/added. Only one change is needed to the control plane, i.e.to add support for ICMP extensions, but that shouldn't be a major showstopper for experiments. New extensions are needed to the networking stack, some applications will suffer (but what ever is done, some applications will suffer). My knowledge about the programming model is very limited, thus no comments on that.
>  
> Kudos for the innovative thinking. I think there is growing practice in looking at the network stack especially from a messaging point of view instead of an opaque streaming interface[1] but it still is very scary and if we are going to go to this extent maybe we should look at the entire protocol stack..
> Pushing the bindings into the server are creative and maybe a first attempt to look at these alternatives while they mature possibly migrating into the dataplane of the network. Operationally people are going to get a bit burned without the proper OAM tools.
>  I do like that you have added the elusive application service. Having a proper binding for this is important as well as the appropriate lookup services. Ip's "well-known" ports was a hack because we had no such directory. You do answer Saltzers first objective:
> 1. A given service may run at one or more nodes, and may need to move from one node to another without losing its identity as a service.[3]
> But again we are missing the Node Address. Saltzer missed this, if you believe the conjecture it was because multi-homed node systems were not envisioned at this point. The IP Address is naming the subnetwork attachment point, this is ultimately the scaling problem..[2] What is the "identity as a node" he is talking about? An identity needs to be unique within the scope of the layer (we don't have this with MAC or IP today).
> Hmm. here I disagree :-)
> 
> This is the thinking from the past - what we have been trying to do in the data centers during the past +10 years is to break up this host-to-host architecture. When I have designed data centers you get to a point when you have to decide what to do with the most important services that need an availibilty of five nines. There is desire to break the host-to-host concept and replace it with a host-to-many-hosts concept - thus you implement some kind of load balancing solution (SLB, DNS, etc) in front of your real servers so you can have a host-to-many-hosts solution. But what I realized, after reading the SCAFFOLD paper, is what we have been trying to accomplish that past +10 years is to create a host-to-service solution with help of these load balancing kludges - e.g. writing this e-mail I don't care on which node the mail gets composed, the most important thing is that the service is available!
> Did Saltzer think of online services, where there online services on the Internet at that time? I believe that mainframes (node) to produce a high available service was mainstream at that time - the online service concept (of the magnitude we see today) has arrived later. The demand for mission-critical services are growing, and here we sit with an outdated stack in our hands, trying to respond to the new requirements.
> 
> Also, if you read RFC 6306 (e.g. Appendix C) you'll notice that the node has a two level "attachment point" - one level describing how the node is connected to the local network and how the local network is connected to the Internet. This model is like the highway where you have several exits to a destination and the nodes can leverage this information to reach another node. The current model that we use, is more or less like the railway - nodes have no choice to make a decision which path is used to reach the destination, instead the railway (network) tries to figure out which path works best for the nodes. If we could use a two-level addressing scheme we could replace multi-homing with multi-pathing and let the transport protocol figure out which path works best for them (like the driver on the highway). Oh, and the transport protocol is pretty good on liveness detection also.
>  
> 2. A given node may be connected to one or more network attachment points, and may need to move from one attachment point to another without losing its identity as a node.[3]
> There is also overloading the hostAddr name with ASAddr and sockID in the example implementation.. This in practice may be challenging but I assume this is just an experimental viewpoint.
> 
> Yes, drop this packet header structure and use the one described in RFC 6306 - it is better suited for the current Internet and there is placeholder for an identifier.
> 
> MPTCP do have a token that can be used as a session identifier - then the application can be moved from one node to another - and I don't care if this e-mail gets composed in Finland or Belgium, as long as the service is available. 
> 
> For these two 
> 
> "IMHO, if you do routing based on identifiers then there is really no decoupling of locators and identifiers - of course this depends on one's definition of Loc/ID split"
> 
> I don't think this has to do with loc/id split, that in fact maybe a false path[2].. But you do routing on a "name/address" relative to the layer. Mux/Demux is just composition/decomposition in the network domain.
> 
> If we take the perspective that devices are multi-homed as the canonical model;  examples being  a device moving from AP to AP, Wi-fi to 3G/4G or DC to DC. Scaffold goes to a lot of effort to make this seamless but the complexity should be an indicator something is wrong.
> 
> I don't think SCAFFOLD address this issue - this is better suited for MPTCP and they have work in progress to solve roaming issues.
>  
> 
> I put together a short example here: http://www.slideshare.net/gaberger/scaffold-8866328 which describes Saltzer's model as well as Scaffold. I also took a stab at what it would actually look like if you incorporated a Node Identifier in the address architecture. I am not saying that this is an easy topic, its been 40 years in the making, the question is where do we concentrate our efforts in properly designing the address architecture and managing the layer bindings.. The latter as a charter in this group "the what", the former as a necessity in understanding "the why".
> 
> I'll have a look on that - I do have a presentation about hIPv4 and uses cases but it need to be updated before releasing it.
> 
> Patrick 
> 
>