Subject: [sami] Welcome to SAMI and something you may like to know.

From: "Yingjie Gu(yingjie)" <[email-masked]>
Date: Thu, 11 Aug 2011 14:04:14 +0800

Hi all,

Thank you very much for your interest in StAte Migration (SAMI). 

In this email, I would like to introduce some background of this mail list,
so that you can get better understanding of what we need and want to do as
the first target of this mail list.

At IETF81 meeting, we had a small size BarBoF. The topic is what you have
seen in mail list description. To let those who haven't attend the BarBoF
know what is state in our context, an example of the state is TCP state on
firewall. For detail information, please go to
https://datatracker.ietf.org/meeting/81/materials.html
->Tsvarea->Yingjie-policies migration, and
http://tools.ietf.org/id/draft-gu-opsawg-policies-migration-00.txt. 

Attendees agree that state migration is useful. What we need to do next is
to narrow down the scope, e.g. state migration within the same
Administration domain or between domains, and collect the use cases in that
scope. Then we need to figure out what kind of state need to be migrated in
the narrow scope, what's the common representation of the states, and, if we
go that far, the potential solutions for state migration.

Please contribute your ideas to the mail list. 

  _____  

Best Regards
Gu Yingjie

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 11 Aug 2011 09:40:35 +0200

> go that far, the potential solutions for state migration.

For me, it is crucial to first define the scope before I can agree
whether state migration is a useful concept or not. Depending on the
timing and the addressing/routing issues, other mechanisms might be
more appropriate.

Bottom line: Be careful with general statements like "Attendees agree
that state migration is useful." before we even agree on the context.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: "Yingjie Gu(yingjie)" <[email-masked]>
Date: Thu, 11 Aug 2011 17:06:53 +0800

Hi Juergen,

I accept your suggestion. I will be more careful with my words. 

We don't need yet more high level discussion on whether it's useful or not. 

Let's first figure out the context: scope, use cases and so on.

Best Regards
Gu Yingjie

-----邮件原件-----
发件人: [email-masked] [mailto:[email-masked]] 代表 Juergen
Schoenwaelder
发送时间: 2011年8月11日 乐乐15:41
收件人: Yingjie Gu(yingjie)
抄送: 'Wesley Eddy'; 'Romascanu, Dan (Dan)'; [email-masked]; 'David
Harrington'; [email-masked]
主题: Re: [sami] Welcome to SAMI and something you may like to know.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>
_______________________________________________
sami mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/sami

---

From: Linda Dunbar <[email-masked]>
Date: Thu, 11 Aug 2011 16:13:44 +0000

I thought at the barBOF that the next step is to identify some use cases. It will be very beneficial to accelerate the progress by identifying some states (or conditions) which today's firewall or security devices can use to continue the proper function. 

Linda

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Yingjie Gu(yingjie)
> Sent: Thursday, August 11, 2011 4:07 AM
> To: 'Juergen Schoenwaelder'
> Cc: 'Wesley Eddy'; 'Romascanu, Dan (Dan)'; [email-masked]; 'David
> Harrington'; [email-masked]
> Subject: Re: [sami] Welcome to SAMI and something you may like to know.
> 
> Hi Juergen,
> 
> > Administration domain or between domains, and collect the use cases
> in
> that
> > scope. Then we need to figure out what kind of state need to be
> migrated
> in
> > the narrow scope, what's the common representation of the states, and,
> if
> we

---

From: "Yingjie Gu(yingjie)" <[email-masked]>
Date: Wed, 17 Aug 2011 16:58:43 +0800

The following is a scope suggested by Dr. Fan.

" According to the current implementation,VM migration is scoped in a layer
2 network with shared storage.Key factors that decide whether hot migration
is successful includes,from the network's side, bandwith and delay.Even the
migration happens between two sites,it may succeed if the bandwidth is wide
enough and the delay is small enough.
 
So,pehhaps we can define the scope as such:A layer 2 subnet with good enough
performance which makes the hot migration successful."

What is your opinion on this scope?

Best Regards
Gu Yingjie

-----邮件原件-----
发件人: Linda Dunbar [mailto:[email-masked]] 
发送时间: 2011年8月12日 乐乐0:14
收件人: Yingjie Gu(yingjie); 'Juergen Schoenwaelder'
抄送: 'Wesley Eddy'; 'Romascanu, Dan (Dan)'; [email-masked]; 'David
Harrington'; [email-masked]
主题: RE: [sami] Welcome to SAMI and something you may like to know.

I thought at the barBOF that the next step is to identify some use cases. It
will be very beneficial to accelerate the progress by identifying some
states (or conditions) which today's firewall or security devices can use to
continue the proper function. 

Linda

---

From: "So, Ning" <[email-masked]>
Date: Wed, 17 Aug 2011 09:59:19 -0400

Yingjie,

You might want to consider limiting the scope to problem definition and provider requirements collection first.  I know a lot of work has gone into documenting what is available today.  However, the provider requirements in this area is still rather thin.   For example, the requirements for optimization-based migration can be quite different from failure restoration-based migration.  The type of services the VMs are supporting can also impact the migration requirements, thus impacting the solutions.  Defining different types of VM migration and the associated requirements should be a higher priority, in my opinion. 

Best regards,
 
Ning So
Verizon Corporate Technology
(office) 972-729-7905
(Cell) 972-955-0914

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Yingjie Gu(yingjie)
Sent: Wednesday, August 17, 2011 3:59 AM
To: 'Linda Dunbar'; 'Juergen Schoenwaelder'
Cc: 'Wesley Eddy'; 'Romascanu, Dan (Dan)'; [email-masked]; 'David Harrington'; [email-masked]
Subject: Re: [sami] Welcome to SAMI and something you may like to know.

" According to the current implementation,VM migration is scoped in a layer
2 network with shared storage.Key factors that decide whether hot migration is successful includes,from the network's side, bandwith and delay.Even the migration happens between two sites,it may succeed if the bandwidth is wide enough and the delay is small enough.
 
So,pehhaps we can define the scope as such:A layer 2 subnet with good enough performance which makes the hot migration successful."

Best Regards
Gu Yingjie

-----邮件原件-----
发件人: Linda Dunbar [mailto:[email-masked]]
发送时间: 2011年8月12日 乐乐0:14
收件人: Yingjie Gu(yingjie); 'Juergen Schoenwaelder'
抄送: 'Wesley Eddy'; 'Romascanu, Dan (Dan)'; [email-masked]; 'David Harrington'; [email-masked]
主题: RE: [sami] Welcome to SAMI and something you may like to know.

Linda

> From: [email-masked] [mailto:[email-masked]] On Behalf 
> Of Yingjie Gu(yingjie)

---

From: Melinda Shore <[email-masked]>
Date: Wed, 17 Aug 2011 16:51:32 -0800

> performance which makes the hot migration successful."

I think it would be worthwhile, should this work go forward, to
describe what happens (or what should happen) if the failover
is unsuccessful - things like at what point state transfer happens.
It seems to me that there's a pretty clear decision point around
whether or not you wait until the failover is complete and
successful (can you know if it's successful before it's complete?)
before transferring middlebox state, and I definitely would
not like to see that dismissed as out-of-scope before any work
is even chartered.

I'd also stay away from "good enough performance" - not really
sure what that means in a technical context.

Scope would be something along the lines of:

    The work will cover transfer of associated middlebox state [and
    i'd probably enumerate examples - probably includes firewall
    pinholes, probably does not include layer 3 routing] needed to
    maintain existing network flows as a network services fail
    over to a hot standby.  Doing the hokey-pokey is out-of-scope.

That needs a lot of work and it would need definition of terms
like "hot standby" and "middlebox," etc., but it describes what
will be included in the work and what will not.

Melinda

---

From: "Yingjie Gu(yingjie)" <[email-masked]>
Date: Thu, 18 Aug 2011 20:03:00 +0800

Ning,

When we talked at Quebec, I remembered that you have impressive understanding on VM migration requirements in/between subnets. Hope you can share your use cases from provider's perspective.
I have talked with two providers in detail, and they both see the strong requirements for state migration as VM migrates. Perhaps we need to document provider's requirements for people's reference. I am working with providers on this document to introduce use cases.

Best Regards
Gu Yingjie

-----邮件原件-----
发件人: So, Ning [mailto:[email-masked]] 
发送时间: 2011年8月17日 乐乐21:59
收件人: Yingjie Gu(yingjie); 'Linda Dunbar'; 'Juergen Schoenwaelder'
抄送: 'Wesley Eddy'; 'Romascanu, Dan (Dan)'; [email-masked]; 'David Harrington'; [email-masked]
主题: RE: [sami] Welcome to SAMI and something you may like to know.

Yingjie,

Best regards,
 
Ning So

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Yingjie Gu(yingjie)
Sent: Wednesday, August 17, 2011 3:59 AM
To: 'Linda Dunbar'; 'Juergen Schoenwaelder'
Cc: 'Wesley Eddy'; 'Romascanu, Dan (Dan)'; [email-masked]; 'David Harrington'; [email-masked]
Subject: Re: [sami] Welcome to SAMI and something you may like to know.

 

Best Regards
Gu Yingjie

-----邮件原件-----
发件人: Linda Dunbar [mailto:[email-masked]]
发送时间: 2011年8月12日 乐乐0:14
收件人: Yingjie Gu(yingjie); 'Juergen Schoenwaelder'
抄送: 'Wesley Eddy'; 'Romascanu, Dan (Dan)'; [email-masked]; 'David Harrington'; [email-masked]
主题: RE: [sami] Welcome to SAMI and something you may like to know.

Linda

---

From: "Yingjie Gu(yingjie)" <[email-masked]>
Date: Thu, 18 Aug 2011 20:57:15 +0800

My understanding of scope, based on Yongbing's scope suggestion, is as follows.

The scope of State migration, i.e. the scope of VM migration, is within a Layer 2 subnet, which guarantees that VM's IP address can keep unchanged. Though some technologies can enable VM migrates between different subnets and keep the IP address unchanged, most requirements for VM migration is within the same L2 subnet for now. 

We need to be aware that, not all L2 subnet has enough bandwidth or shared storage or reasonable migration distance to make VM migration succeed. In our scope, I suggest we only consider the L2 subnet which can support successful VM migration, in which case we has the reason to consider state migration. Let's call it "qualified L2 subnet", which equals to Yongbing's " L2 subnet with good enough performance ". It should be the users of "State Migration mechanism" to judge whether their L2 subnet is suitable for VM Migration and whether they should adopt State Migration mechanism in their subnet.
Of course, even in a "qualified L2 subnet", VM migration can not be successful all the time. So, we also need to consider failure feedback mechanism.

State migration should be agnostic to applications/services running on migrating VM. Some applications/services are sensitive to time to different extents, some are not. However, no matter what kind of applications/services are running on the migrating VM, state migration must be completed or response Sate-Migration-Fail as soon as possible.

States Definition: States on network devices that are essential to service/application continuity. That means, if the state is missing, the running service/applications will be disrupted. Let's name this kind of state as Essential State, e.g. TCP states on Firewall. The other kind of state is states on network devices that may influence the accuracy of billing and accounting, or increase attack risks. At the very beginning, we can focus on Essential State.  We will figure out the Essential State on Firewall and Load Balancer, which I think is a good start. 

Please speak out your opinions on the above scope definition.

Two coming drafts: 
1) Use cases introduction: In this document, we will list use cases from providers, showing the real requirements they see for state migration, in addition to scenarios of VM Migration, services running on the VM and networking. 
2) Essential States Analysis: In this document, we will enumerate essential states on Firewall and Load Balancer and the common representation of the states.

We hope more people can contribute to these two drafts. Please let me know if you are interested in any or both of the drafts.

Thank you very much.

Best Regards
Gu Yingjie

-----邮件原件-----
发件人: Fan Yongbing [mailto:[email-masked]] 
发送时间: 2011年8月18日 乐乐11:30
收件人: Melinda Shore; Yingjie Gu(yingjie)
抄送: [email-masked]
主题: Re: [sami] Welcome to SAMI and something you may like to know.

Thanks for Melinda's suggestion.

A revision version about the scope:

 The work will cover state migration of associated middleboxes(for 
example,firewall,switch and so on)needed to
 maintain existing service as a VM is hot-migrated to another position in 
the same layer 2 subnet.

Fan Yongbing
China Telecom

樊勇兵  博士
--------------------------------------------------
中国电信股份有限公司广州研究院
数据通信研究部
Mobile：13316090609（主用）；13922438897（辅用）
Tel：020-38639121；PHS：020-88338121
Fax：020-38639487
--------------------------------------------------

---

From: "So, Ning" <[email-masked]>
Date: Thu, 18 Aug 2011 09:58:15 -0400

Yingjie,

I can help with one kind of providers' requirements, the traditional telecom service provider who owns the infrastructure end-to-end.  My understanding is not nearly as good when it comes to other type of providers such as GOOGLE, YAHOO, or service broker/whole seller/integrator.  

Best regards,
 
Ning So

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Yingjie Gu(yingjie)
Sent: Thursday, August 18, 2011 7:03 AM
To: So, Ning; 'Linda Dunbar'; 'Juergen Schoenwaelder'
Cc: 'Wesley Eddy'; 'Romascanu, Dan (Dan)'; [email-masked]; 'David Harrington'; [email-masked]
Subject: Re: [sami] Welcome to SAMI and something you may like to know.

Ning,

Best Regards
Gu Yingjie

-----邮件原件-----
发件人: So, Ning [mailto:[email-masked]]
发送时间: 2011年8月17日 乐乐21:59
收件人: Yingjie Gu(yingjie); 'Linda Dunbar'; 'Juergen Schoenwaelder'
抄送: 'Wesley Eddy'; 'Romascanu, Dan (Dan)'; [email-masked]; 'David Harrington'; [email-masked]
主题: RE: [sami] Welcome to SAMI and something you may like to know.

Yingjie,

Best regards,
 
Ning So

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Yingjie Gu(yingjie)
Sent: Wednesday, August 17, 2011 3:59 AM
To: 'Linda Dunbar'; 'Juergen Schoenwaelder'
Cc: 'Wesley Eddy'; 'Romascanu, Dan (Dan)'; [email-masked]; 'David Harrington'; [email-masked]
Subject: Re: [sami] Welcome to SAMI and something you may like to know.

 

Best Regards
Gu Yingjie

-----邮件原件-----
发件人: Linda Dunbar [mailto:[email-masked]]
发送时间: 2011年8月12日 乐乐0:14
收件人: Yingjie Gu(yingjie); 'Juergen Schoenwaelder'
抄送: 'Wesley Eddy'; 'Romascanu, Dan (Dan)'; [email-masked]; 'David Harrington'; [email-masked]
主题: RE: [sami] Welcome to SAMI and something you may like to know.

Linda

---

From: Melinda Shore <[email-masked]>
Date: Thu, 18 Aug 2011 08:02:22 -0800

> Of course, even in a "qualified L2 subnet", VM migration can not be successful all the time. So, we also need to consider failure feedback mechanism.

Absolutely.  It seems to me that if you think that - for the purposes
of state migration - failing because there's insufficient network
resources is different from failing for some other reason, it might be
a good thing to explain why.  I could be convinced otherwise but for
the moment I don't think this is a particularly defensible distinction.

I'm also not clear on how much network bandwidth is needed to support
failover.  Intuitively it seems like the answer is "nearly none."
There may be issues around bandwidth available to the hot spare when
it goes live but if they're on the same layer 2 subnet/network segment,
the failed-from and failed-to servers are going to have pretty much
the same resources available to them.

Melinda

---

From: Melinda Shore <[email-masked]>
Date: Thu, 18 Aug 2011 12:30:52 -0800

I have to admit that I'm becoming more confused rather than less
as the discussions progress.

1) My understanding earlier was that we were talking about failing
    over services between existing VMs rather than copying a VM as
    part of a failover.  Is anybody actually doing that?  Aren't you
    concerned that you'll be copying the failed state?

2) If you're copying a live VM, along with its runtime  state
    (i.e. memory), things like IP addresses, MAC addresses, and
    so on will be moving with it.  If the VM stays on the same
    subnet presumably it will be behind the same middlebox.  The
    only thing I could think of that might become an issue is the
    MAC address being mapped to a particular switch port.  What
    network state do you think will need to move with a VM
    relocation on the same subnet?

It seems to me that there's some interesting work here regarding
transferring middlebox state when the service is being failed over,
not so much when a VM is just being shuttled around a subnet (and
I'm not sure that really happens, anyway).  That said, if what's
at issue is middlebox state moving that probably belongs in the
transport area rather than the ops area.  But first, let's figure
out what's actually under discussion.

Melinda

---

From: "Yingjie Gu(yingjie)" <[email-masked]>
Date: Fri, 19 Aug 2011 09:16:56 +0800

Hi David,

Thank you so much for your attention and such detail suggestion. I really
appreciate it.

I believe there are some misunderstanding. But I appreciate the chance to
clarify our target.

The intention of State Migration is not to standardize VM Migration. The
focus is how to migrate state on network devices when VM is migrating, just
like the name ‘State Migration’ means. How to do virtualization and how to
migrate the VM is out of the scope. 

You can do VM migration either by a standardized way(I am not sure there is
a standard way) or proprietary way. It doesn’t matter. The problem is the
state associated with the VM on old network devices, e.g. Hardware Firewall,
Load Balancer or switches, also need to be correctly presented on new
network devices at the new location. Otherwise the running service will have
to be stopped. 

The state could be TCP states on Firewall or Session states on Load
Balancer. When migrating state, one may find that he has to migrate state
between devices designed by different vendors. That is why we need a
standardized way to do this. 

Currently we are try to narrow down the scope: e.g. do we consider state
migration within a L2 subnet, or also consider state migration between L2
subnets. We will also figure our the state that is essential to service
continuity after VM Migration. 

As conclusion, we are not working on VM Migration or Virtualization. We are
not trying to standardize virtualization, VM migration, TCP/IP stacks,
storage, etc. Neither does virtual host configuration in the scope.

Also I would like to know whether there is confusing text in the mail list
description or in my ever reply to the mail list or in my draft that make
people feel that we are working on VM Migration. If there is, I will
definitely revise them at once.

Again, thank you for your detail suggestion. 

Best Regards

Gu Yingjie

-----邮件原件-----
发件人: David Harrington [mailto:[email-masked]] 
发送时间: 2011年8月18日 乐乐23:25
收件人: 'Yingjie Gu(yingjie)'; [email-masked]
主题: Bringing new work into the IETF

Hi Yingjie,

I trimmed the CC: list to the sami mailing list.

Those who are interested in the sami discussion can join the sami list

and find this discussion in the sami archive.

So far, I think your proposal is not being accepted by the IETF

because it is too general and too abstract.

You are still trying to figure out what problem you want to solve.

To make VM migration standardization a viable problem to work on, you

probably need to get vendors and operators to all agree on complete

standardization for their various technologies - virtualization,

TCP/IP stacks, network management protocols, and so on. 

I think you are very unlikely to ever achieve that because vendors

want to continue to enhance their solutions versus their competitors.

Standardization is not appropriate to solve every problem. 

Most enterprises/providers will choose a specific virtualization

solution, such as VMware.

How VM migration is done depends a great deal on the virtualization

solution in use.

Hypervisors have not been standardized because the companies that

provide virtualization solutions gain competitive advantage from their

hypervisor capabilities. They don't necessarily want to standardize

their hypervisors across vendors.

Most enterprises/providers will choose a specific storage solution,

such as EMC or NetApp.

How VM migration is done depends a great deal on the storage solution

in use.

Storage devices usually use standards, but have not been completely

standardized because the companies that provide storage solutions gain

competitive advantage from their storage solution enhancements. They

don't want to totally standardize storage across vendors.

Most enterprises/providers will choose specific equipment vendors to

deploy in their networks.

Those choices often have to do with the proprietary enhancements

offered by different vendors.

The vendors have no desire to all offer identical products and

identical features.

How VM migration is done depends a great deal on the networking

equipment in use.

IP/TCP stacks have not been totally standardized because the companies

that provide networking equipment gain competitive advantage from

their equipment's enhanced capabilities. They don't want to totally

standardize their equipment across vendors.

Most enterprises/providers will choose specific equipment to deploy in

their networks.

Those choices often have to do with the proprietary solutions to

manage the chosen equipment.

The vendors have no desire to all offer identical management features.

How a migration is done depends a great deal on the networking

management tools in use.

CLIs and proprietary MIB modules and other management interfaces have

not been totally standardized because the companies that provide these

tools gain competitive advantage from their management products'

enhanced capabilities. They don't want to totally standardize their

management interfaces.

To standardize VM migration across all the VM implementations, the

TCP/IP implementations, the storage implementations, the routing

implementations, the security implementations, the application

implementations, the network management implementations, and the

enterprise-specific deployment choices, is like "boiling the ocean".

Not a very achieveable goal.

The IETF does **engineering**.

You would do much better to focus your energy on a **small** problem,

solvable by standardization, that can be turned into an engineering

project appropriate to the IETF. 

For example, you might focus on developing a standard DHCP attribute

or a standard netconf/YANG module for some aspect of virtual host

configuration that could be reasonably standardized across

virtualization platforms and equipment vendors. Each standardization

of a small piece of Internet-related VM configuration could help

standardize VM migration in the long run. And that would be

engineering work appropriate to the IETF. 

David Harrington

Director, IETF Transport Area

[email-masked] (preferred for ietf)

[email-masked]

+1 603 828 1401 (cell)

> From: Yingjie Gu(yingjie) [mailto:[email-masked]] 

> Sent: Thursday, August 18, 2011 8:03 AM

> To: 'So, Ning'; 'Linda Dunbar'; 'Juergen Schoenwaelder'

> Cc: 'Wesley Eddy'; 'Romascanu, Dan (Dan)'; 

> [email-masked]; 'David Harrington'; [email-masked]

> Subject: Re: [sami] Welcome to SAMI and something you may 

> like to know.

> When we talked at Quebec, I remembered that you have 

> impressive understanding on VM migration requirements 

> in/between subnets. Hope you can share your use cases from 

> provider's perspective.

> I have talked with two providers in detail, and they both see 

> the strong requirements for state migration as VM migrates. 

> Perhaps we need to document provider's requirements for 

> people's reference. I am working with providers on this 

> document to introduce use cases.

> 抄送: 'Wesley Eddy'; 'Romascanu, Dan (Dan)'; 

> [email-masked]; 'David Harrington'; [email-masked]

> You might want to consider limiting the scope to problem 

> definition and provider requirements collection first.  I 

> know a lot of work has gone into documenting what is 

> available today.  However, the provider requirements in this 

> area is still rather thin.   For example, the requirements 

> for optimization-based migration can be quite different from 

> failure restoration-based migration.  The type of services 

> the VMs are supporting can also impact the migration 

> requirements, thus impacting the solutions.  Defining 

> different types of VM migration and the associated 

> requirements should be a higher priority, in my opinion. 

> From: [email-masked] [mailto:[email-masked]] On 

> Behalf Of Yingjie Gu(yingjie)

> Cc: 'Wesley Eddy'; 'Romascanu, Dan (Dan)'; 

> [email-masked]; 'David Harrington'; [email-masked]

> Subject: Re: [sami] Welcome to SAMI and something you may 

> like to know.

> " According to the current implementation,VM migration is 

> scoped in a layer

> 2 network with shared storage.Key factors that decide whether 

> hot migration is successful includes,from the network's side, 

> bandwith and delay.Even the migration happens between two 

> sites,it may succeed if the bandwidth is wide enough and the 

> delay is small enough.

> So,pehhaps we can define the scope as such:A layer 2 subnet 

> with good enough performance which makes the hot migration 

> successful."

> 抄送: 'Wesley Eddy'; 'Romascanu, Dan (Dan)'; [email-masked]; 

> 'David Harrington'; [email-masked]

> I thought at the barBOF that the next step is to identify 

> some use cases. It will be very beneficial to accelerate the 

> progress by identifying some states (or conditions) which 

> today's firewall or security devices can use to continue the 

> proper function. 

> > From: [email-masked] [mailto:[email-masked]] 

> On Behalf 

> > Subject: Re: [sami] Welcome to SAMI and something you may 

> like to know.

> > We don't need yet more high level discussion on whether 

> it's useful or 

> > not.

> > 发件人: [email-masked] [mailto:[email-masked]] 代表

> > Juergen

> > 抄送: 'Wesley Eddy'; 'Romascanu, Dan (Dan)'; 

> [email-masked]; 'David 

> > 主题: Re: [sami] Welcome to SAMI and something you may like to

know.

> > On Thu, Aug 11, 2011 at 02:04:14PM +0800, Yingjie Gu(yingjie)

wrote:

> > > Attendees agree that state migration is useful. What we need to

do

> > next is

> > > Administration domain or between domains, and collect the 

> use cases

> > > the narrow scope, what's the common representation of the

states, 

> > > and,

> > whether state migration is a useful concept or not. 

> Depending on the 

> > timing and the addressing/routing issues, other mechanisms might

be 

> > Bottom line: Be careful with general statements like 

> "Attendees agree 

> > that state migration is useful." before we even agree on 

> the context.

> > Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen,

Germany

---

From: "David Harrington" <[email-masked]>
Date: Thu, 18 Aug 2011 11:25:04 -0400

Hi Yingjie,

David Harrington

know.
do
states, 

be 
Germany

---

From: Melinda Shore <[email-masked]>
Date: Thu, 18 Aug 2011 18:05:43 -0800

On 8/18/11 5:16 PM, Yingjie Gu(yingjie) wrote:
> focus is how to migrate state on network devices when VM is migrating, 
> just like the name ‘State Migration’ means. How to do virtualization and 
> how to migrate the VM is out of the scope.

I'm still unclear on what middlebox state needs to be migrated with a VM.
If the IP address is the same, if you're keeping the same ephemeral
ports and
the same MAC address, etc., and you're staying on the same subnet, what
needs to be moved where?

Melinda

---

From: Thomas Narten <[email-masked]>
Date: Thu, 18 Aug 2011 22:55:18 -0400

"Yingjie Gu(yingjie)" <[email-masked]> writes:

> Balancer. When migrating state, one may find that he has to migrate
> state between devices designed by different vendors. That is why we
> need a standardized way to do this.

Well, there are 2 separate problems above. Each involves different
state, and possibly different challenges. It would be useful to ask
specifically for these two scenaros, whether there is a compelling
need to migrate such state. I.e., who needs a solution to this
problem? Which data center operator? Which  customer? Or is this just
a theoretical problem for which if there was a solution, it *might*
get used?

> Currently we are try to narrow down the scope: e.g. do we consider
> state migration within a L2 subnet, or also consider state migration
> between L2 subnets. We will also figure our the state that is
> essential to service continuity after VM Migration.

IMO, this is the less interesting question regarding scope.

The first question is whether there is a compelling case to move state
at all. If so, for which devices? And what do the vendors for those
devices say? If the vendors that own the market for firewalls,
load-balancer, <you name it but they have network state> are not
interested in a solution as is being discussed here, what is the
likelyhood that any solution (if it was developed) would ever be used?

IMO, we should start by first talking about specific scenarios, where
migrating state is necessary, and why the lack of a solution today is
a problem that the IETF needs to develop solutions for.

Thomas

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 19 Aug 2011 15:45:37 +0200

On Thu, Aug 18, 2011 at 08:57:15PM +0800, Yingjie Gu(yingjie) wrote:

> The scope of State migration, i.e. the scope of VM migration, is
> within a Layer 2 subnet, which guarantees that VM's IP address can
> keep unchanged. Though some technologies can enable VM migrates
> between different subnets and keep the IP address unchanged, most
> requirements for VM migration is within the same L2 subnet for now.

Those setups that I have seen (but I have not seen too many and
usually smaller onces) have a single L2 network _behind_ firewalls and
load balancers and hence there is almost no state to migrate when a VM
moves. It appears to me that people design their networks to make
migration feasible (by making it a single L2 network) instead of
trying to migrate across arbitrary network topologies.

For me, the problem becomes interesting for the IETF when someone
wants to migrate a VM from one IP subnet to another IP subnet. In that
case, we have some form of IP mobility (since the IP connectivity
changes) and depending on the nature of the applications running on
the VMs, there might be some period in which some tunneling is needed
(because the state associated with existing transport connections in
middleboxes is likely harder to migrate correctly than doing some
mobile IP like tunneling). Some people said in Quebec that we can
ignore the IP addressing issues and just focus on the state migration
but I feel that is wrong because the way you deal with the mobility at
the IP level will have impact on how much state needs migration or
whether any operational state needs to be migrated at all (and my
personal preference would be to design the network to avoid the need
for state migration - that is to produce guidelines or best practices
to follow rather than producing more special purpose standards).

To what extend this is a real-world problem and to what extend vendors
of middle-boxes are interested to implement a standard in this area, I
do not know. But a good indication of interest is usually that some of
those vendors actively participate in the specification of a solution.

As stated in Quebec, to me this still sounds more like a research
effort than a standardization effort but I am happy to get convinced
otherwise.

/js

---

From: "So, Ning" <[email-masked]>
Date: Fri, 19 Aug 2011 12:54:37 -0400

David,

I agree with you that the problems are not clearly defined, the requirements are not clear, and the scope is too large.  However, I disagree that providers do not want standardized solution. At least that is not the case for the traditional telecom service providers who are also trying to become the cloud/data center service providers. 

Telecom service provider such as Verizon have significant business with enterprises and government agencies.  One significant cloud service business for that market in the foreseeable future is to host overflow/expansion capacity for the enterprise/government data centers.  That means the provider data centers will have to be multi-tenant in nature with seamless inter-working with the customer data centers.  There are quite a few (I cannot number them) hypervisors out there in the market today, and the customers I know have all of them (although usually each customer has one or two hypervisors only).   Without standardization means the provider data center has to have all of the hypervisors in each and every data center in order to provide the services, and it also means the server/network capacity in the provider data centers are dedicated per Hypervisor without capacity sharing.  

Of course I am only stating a large problem for some providers, not the ones like GOOGLE and YAHOO who have total control of their data centers so they can deploy the same hypervisor everywhere.  And I also understand vendors love the proprietary environment because wasted provider capacity means more business for them.  However, providers are not all dummies.  We will eventually see what is happening, and we will use the well-established equipment selection process (RFP) to drive standardization in order to become more efficient.  

Verizon has always been big on networking technology standardization, and I don't see any difference when it comes to data center and cloud.   From my perspective, standardization is good for IETF, and good for the industry as a whole.  

Best regards,
 
Ning So

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of David Harrington
Sent: Thursday, August 18, 2011 10:25 AM
To: 'Yingjie Gu(yingjie)'; [email-masked]
Subject: [sami] Bringing new work into the IETF

Hi Yingjie,

I trimmed the CC: list to the sami mailing list.
Those who are interested in the sami discussion can join the sami list and find this discussion in the sami archive.

So far, I think your proposal is not being accepted by the IETF because it is too general and too abstract.
You are still trying to figure out what problem you want to solve.

To make VM migration standardization a viable problem to work on, you probably need to get vendors and operators to all agree on complete standardization for their various technologies - virtualization, TCP/IP stacks, network management protocols, and so on. 
I think you are very unlikely to ever achieve that because vendors want to continue to enhance their solutions versus their competitors.
Standardization is not appropriate to solve every problem. 

Most enterprises/providers will choose a specific virtualization solution, such as VMware.
How VM migration is done depends a great deal on the virtualization solution in use.
Hypervisors have not been standardized because the companies that provide virtualization solutions gain competitive advantage from their hypervisor capabilities. They don't necessarily want to standardize their hypervisors across vendors.

Most enterprises/providers will choose a specific storage solution, such as EMC or NetApp.
How VM migration is done depends a great deal on the storage solution in use.
Storage devices usually use standards, but have not been completely standardized because the companies that provide storage solutions gain competitive advantage from their storage solution enhancements. They don't want to totally standardize storage across vendors.

Most enterprises/providers will choose specific equipment vendors to deploy in their networks.
Those choices often have to do with the proprietary enhancements offered by different vendors.
The vendors have no desire to all offer identical products and identical features.
How VM migration is done depends a great deal on the networking equipment in use.
IP/TCP stacks have not been totally standardized because the companies that provide networking equipment gain competitive advantage from their equipment's enhanced capabilities. They don't want to totally standardize their equipment across vendors.

Most enterprises/providers will choose specific equipment to deploy in their networks.
Those choices often have to do with the proprietary solutions to manage the chosen equipment.
The vendors have no desire to all offer identical management features.
How a migration is done depends a great deal on the networking management tools in use.
CLIs and proprietary MIB modules and other management interfaces have not been totally standardized because the companies that provide these tools gain competitive advantage from their management products'
enhanced capabilities. They don't want to totally standardize their management interfaces.

To standardize VM migration across all the VM implementations, the TCP/IP implementations, the storage implementations, the routing implementations, the security implementations, the application implementations, the network management implementations, and the enterprise-specific deployment choices, is like "boiling the ocean".
Not a very achieveable goal.

The IETF does **engineering**.
You would do much better to focus your energy on a **small** problem, solvable by standardization, that can be turned into an engineering project appropriate to the IETF. 

For example, you might focus on developing a standard DHCP attribute or a standard netconf/YANG module for some aspect of virtual host configuration that could be reasonably standardized across virtualization platforms and equipment vendors. Each standardization of a small piece of Internet-related VM configuration could help standardize VM migration in the long run. And that would be engineering work appropriate to the IETF. 

David Harrington
Director, IETF Transport Area
[email-masked] (preferred for ietf) [email-masked]
+1 603 828 1401 (cell)

> When we talked at Quebec, I remembered that you have impressive 
> understanding on VM migration requirements in/between subnets. Hope 
> you can share your use cases from provider's perspective.
> I have talked with two providers in detail, and they both see the 
> strong requirements for state migration as VM migrates.
> Perhaps we need to document provider's requirements for people's 
> reference. I am working with providers on this document to introduce 
> use cases.
> 
> 
> 
> Best Regards
> Gu Yingjie
> 
> 
> -----邮件原件-----
> You might want to consider limiting the scope to problem definition 
> and provider requirements collection first.  I know a lot of work has 
> gone into documenting what is available today.  However, the provider 
> requirements in this
> for optimization-based migration can be quite different from failure 
> restoration-based migration.  The type of services the VMs are 
> supporting can also impact the migration requirements, thus impacting 
> the solutions.  Defining different types of VM migration and the 
> associated requirements should be a higher priority, in my opinion.
> 
>  
> Best regards,
>  
> Ning So
> 2 network with shared storage.Key factors that decide whether hot 
> migration is successful includes,from the network's side, bandwith and 
> delay.Even the migration happens between two sites,it may succeed if 
> the bandwidth is wide enough and the delay is small enough.
>  
> So,pehhaps we can define the scope as such:A layer 2 subnet with good 
> enough performance which makes the hot migration successful."
> 
> I thought at the barBOF that the next step is to identify some use 
> cases. It will be very beneficial to accelerate the progress by 
> identifying some states (or conditions) which today's firewall or 
> security devices can use to continue the proper function.
> 
> 
> Linda
> 
know.
do
states, 

be 
Germany

---

From: Melinda Shore <[email-masked]>
Date: Fri, 19 Aug 2011 11:14:11 -0800

On 08/19/2011 08:54 AM, So, Ning wrote:
 > [ ... ]
> Telecom service provider such as Verizon have significant business
 > with enterprises and government agencies.  One significant cloud
 > service business for that market in the foreseeable future is to
 > host overflow/expansion capacity for the enterprise/government
 > data centers.  That means the provider data centers will have to
 > be multi-tenant in nature with seamless inter-working with the
 > customer data centers.  There are quite a few (I cannot number them)
 > hypervisors out there in the market today, and the customers I know
 > have all of them (although usually each customer has one or two
 > hypervisors only).   Without standardization means the provider
 > data center has to have all of the hypervisors in each and every
 > data center in order to provide the services, and it also means
 > the server/network capacity in the provider data centers are
 > dedicated per Hypervisor without capacity sharing.

You guys are being asked for a technical discussion of what's
being proposed.  This is marketing.

I've mostly been appalled by how stuff (I initially typed "work,"
realized that that was not correct, and "stuff" is the only thing
I could come up with to describe what's under discussion) related to
clouds/data centers has been brought to the IETF.  There's never
been a clear statement of need for a specific piece of work, no
requirements, just fishing for something that someone might possibly
be able to get a publication out of.

The service migration stuff struck me as different, because for
better or worse there's a lot of flow-coupled state inside the
network, often related to security but sometimes related to
transport optimization, policy routing, etc.  How to move
that when live network sessions move is a real problem, and one
that's hard to solve.  Unfortunately you guys totally blew that
one out of the water when you reframed this as 1) moving VMs,
and 2) moving them within the same subnet.  If this is really,
truly, honestly what you want to do, you're going to start to
identify specific use cases and pieces of state that you think
will need to be transferred.  The only thing I can come up with
is address/port mappings in switches, and honestly that's not
very compelling.  This piece of email from Ning continues the
vagueness and really doesn't provide any information that might
help evaluate the work on its technical merit.

Melinda Shore

---

From: Thomas Narten <[email-masked]>
Date: Mon, 22 Aug 2011 08:25:08 -0400

Just so I understand...

"So, Ning" <[email-masked]> writes:

> host overflow/expansion capacity for the enterprise/government data
> centers.  That means the provider data centers will have to be
> multi-tenant in nature with seamless inter-working with the customer
> data centers.  There are quite a few (I cannot number them)
> hypervisors only).  Without standardization means the provider data
> center has to have all of the hypervisors in each and every data
> center in order to provide the services, and it also means the
> server/network capacity in the provider data centers are dedicated
> per Hypervisor without capacity sharing.

Are you saying you want interoperabilty between hypervisors? That is,
You want to be able to move a VM from hypervisor type A (i.e, from
vendor A) to a hypervisor of type B (i.e,. from Vendor B), and you'd
like the IETF to develop standards that allow this?

If so, that seems like trying to bite off a pretty big task, well
beyound what most others have been calling for.

It would be good to get clarity as to whether this is a goal and
whether there any of the key players (i.e., hypervisor vendors) agree.

Thomas

---

From: "So, Ning" <[email-masked]>
Date: Mon, 22 Aug 2011 09:52:18 -0400

Thomas,

Please see my reply inline.  

Best regards,
 
Ning So

-----Original Message-----
From: Thomas Narten [mailto:[email-masked]] 
Sent: Monday, August 22, 2011 7:25 AM
To: So, Ning
Cc: David Harrington; 'Yingjie Gu(yingjie)'; [email-masked]
Subject: Re: [sami] Bringing new work into the IETF

Just so I understand...

"So, Ning" <[email-masked]> writes:

> service business for that market in the foreseeable future is to host 
> overflow/expansion capacity for the enterprise/government data 
> server/network capacity in the provider data centers are dedicated per 
> Hypervisor without capacity sharing.

Are you saying you want interoperabilty between hypervisors? That is, You want to be able to move a VM from hypervisor type A (i.e, from vendor A) to a hypervisor of type B (i.e,. from Vendor B), and you'd like the IETF to develop standards that allow this?

[Ning]:  Yes. 

If so, that seems like trying to bite off a pretty big task, well beyound what most others have been calling for.

It would be good to get clarity as to whether this is a goal and whether there any of the key players (i.e., hypervisor vendors) agree.

[Ning]:  It is important to have a common agreement on requirements.  I stated my opinion and use case.  I will reach out to the carriers I have been work with in other areas to see if they also have the similar requirements.  If it is true, then IETF has to play a key role in this.  Other SDO such as DMTF will also play a key role.  I hope the use cases and requirements can drive player agreement/participation, especially secondary players.          

Thomas

---

From: <[email-masked]>
Date: Mon, 22 Aug 2011 10:16:25 -0400

Hi Melinda,

> be able to get a publication out of.

I concur with that viewpoint.  The former "clouds" mailing list was
a time-sink that I had to pay attention to.

> that's hard to solve.

I also agree with this - there is an actual state migration problem
in here struggling to get out.

> Unfortunately you guys totally blew that

Ah, that I can help with ... ;-).

If one restricts a subnet to a physical L2 subnet, I think your
conclusion has a lot of merit (I reserve the right to quibble
later, but this message is about something else ...).  OTOH,
L2 connectivity is being provided over L3 infrastructure where
one may not expect it - examples of technologies that can do
that include GRE, L2TP and OTV. For OTV, see draft-hasmit-otv-03,
http://datatracker.ietf.org/doc/draft-hasmit-otv/ - that's
a good place to start as it contains a lot of worked-out details
for stretching L2 connectivity over L3 carrier infrastructure.

Live migration of a virtual machine across this sort of stretched
L2 connectivity is an important use case, because it's interesting
(yes, people really do want to do this), and the VM's L2 and L3
addresses can't change during the migration.  To a first
approximation, changing any of the VM's addresses makes the VM
migration non- transparent and disruptive courtesy of things
like ARP caches in other VMs.

One of the first things that is often done when stretching L2
subnets in this fashion is to distribute the L3 default gateway.
If one moves  a VM from San Jose, CA to San Francisco, CA (about 50mi),
one would prefer to hand off that VM's traffic to/from the (public)
Internet to an ISP in San Francisco, as opposed to hauling it back
to San Jose (the latter is colloquially referred to as "hairpinning").
If it helps to understand the scenario, assume that the AS that
encompasses the San Francisco and San Jose facilities (and the
connection between them) is multi-homed to different ISPs in
San Francisco and San Jose.

The good news is that routing protocols don't like hairpinning, and
will try to move the traffic to use the ISP connection in San Francisco.
The bad news is that there's a stateful inspection firewall on the
ISP connection in San Jose that contains things like TCP state.  Hence
moving the VM's TCP connections to San Francisco results in trying to
use the corresponding stateful inspection firewall on the ISP connection
in San Francisco, which has no state for those TCP connections, and
hence bit buckets all of their packets.  That would be bad, and the
result is that the middlebox (firewall) requires that the TCP connections
be hairpinned back to San Jose even though the routing protocols are
perfectly capable of optimizing that hairpinning away via the ISP in
San Francisco.

So, I'm confident that there's a real problem in here, but I also
strongly encourage re-reading Dave Harrington's message that started
this "Bringing new work ..." thread, because it outlines some important
practical considerations about whether the IETF can usefully solve this
problem.  To put those concerns bluntly:

	If we built "it", would anyone "come"?

Thanks,
--David

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Melinda Shore
> Sent: Friday, August 19, 2011 3:14 PM
> To: [email-masked]

---

From: Melinda Shore <[email-masked]>
Date: Mon, 22 Aug 2011 07:27:37 -0800

On 8/22/11 5:52 AM, So, Ning wrote:
> It is important to have a common agreement on requirements.  I
 > stated my opinion and use case.  I will reach out to the carriers
 > I have been work with in other areas to see if they also have the
 > similar requirements.  If it is true, then IETF has to
 > play a key role in this.

I tend to think that this is basically of a piece with wishing for a
pony. That said, as important it is to reach an agreement on
requirements, it's even more important to reach an agreement on
what piece of work is to be done.

That said, the IETF is the Internet Engineering Task
Force.  Just because something moves across a network doesn't mean
that it's necessarily a network problem.  There's also the question
of past being prologue - in an odd feat of chartering rserpool
tackled moving live servers but not moving any of the state associated
with them, and the question of moving middlebox state didn't even
come up.  I tend to think that it's because people involved in the
chartering decision looked at the problem and decided that if the
working group took it on they'd never finish.  Anyway, to
circle back around it seems to me that it's probably a good idea to
stay focused on the network aspects of, erm, whatever it is you
think the IETF should do.  This is not just because of the
question of organizational scope, but also because of the
expertise within the group.

I've also thought about the question of running layer 2 subnets
over the transport layer but that's really not what has been
proposed so far.  I think that a good way to get a handle on
the answer to David's question, "If we build it, will they come?"
would be for one of the proponents of this work to try to
convince Huge Virtualization Vendor <x> to participate in
developing a mechanism that would allow their images to run on
Freeware Hypervisor <y>.

Melinda

---

From: Benson Schliesser <[email-masked]>
Date: Mon, 22 Aug 2011 10:35:44 -0500

Hi, Ning.

On 8/22/11 8:52 AM, "So, Ning" <[email-masked]> wrote:
>> Thomas Narten said:
> Are you saying you want interoperabilty between hypervisors? That is, You want
> to be able to move a VM from hypervisor type A (i.e, from vendor A) to a
> hypervisor of type B (i.e,. from Vendor B), and you'd like the IETF to develop
> standards that allow this?
> 
> [Ning]:  Yes. 
> ...
> [Ning]:  It is important to have a common agreement on requirements.  I stated
> my opinion and use case.  I will reach out to the carriers I have been work
> with in other areas to see if they also have the similar requirements.  If it
> is true, then IETF has to play a key role in this.  Other SDO such as DMTF
> will also play a key role.  I hope the use cases and requirements can drive
> player agreement/participation, especially secondary players.

It's good that you mention the DMTF.  You should also investigate other
groups such as ODCA (http://www.opendatacenteralliance.org/), which has a
working group on this topic.

Assuming that hypervisor compatibility, control APIs, etc, are worked out in
a different forum - what role do you think the IETF has in this space?  I
have my own thoughts on this topic, but I don't want to put words in your
mouth. Can you be specific?

Thanks,
-Benson

---

From: "So, Ning" <[email-masked]>
Date: Mon, 22 Aug 2011 14:49:46 -0400

Benson,

Thank you for bringing ODCA to my attention.  I have had very limited exposure to DMTF, and no exposure to ODCA at all.  After reading a bit more on their work, those SDOs appear to be better places for the type of work I have in mind.  I cannot cover those SDOs, so I will alert internally to have other people involved.  

Best regards,
 
Ning So

-----Original Message-----
From: Benson Schliesser [mailto:[email-masked]] 
Sent: Monday, August 22, 2011 10:36 AM
To: So, Ning; Thomas Narten
Cc: [email-masked]
Subject: Re: [sami] Bringing new work into the IETF

Hi, Ning.

On 8/22/11 8:52 AM, "So, Ning" <[email-masked]> wrote:
> [Ning]:  It is important to have a common agreement on requirements.  
> I stated my opinion and use case.  I will reach out to the carriers I 
> have been work with in other areas to see if they also have the 
> similar requirements.  If it is true, then IETF has to play a key role 
> in this.  Other SDO such as DMTF will also play a key role.  I hope 
> the use cases and requirements can drive player agreement/participation, especially secondary players.

It's good that you mention the DMTF.  You should also investigate other groups such as ODCA (http://www.opendatacenteralliance.org/), which has a working group on this topic.

Assuming that hypervisor compatibility, control APIs, etc, are worked out in a different forum - what role do you think the IETF has in this space?  I have my own thoughts on this topic, but I don't want to put words in your mouth. Can you be specific?

Thanks,
-Benson

---

From: Warren Kumari <[email-masked]>
Date: Mon, 22 Aug 2011 20:12:53 -0400

> trying to migrate across arbitrary network topologies.

I was unable to find a better place in this thread to put this, so I'm somewhat arbitrarily putting it here…

I have a draft (that seriously needs some revisioning) that outlines, at a VERY high level how VM Mobility can be implemented over a L3 network (tl;dr summary -- you resolver the link-layer address via a directory server and then encapsulate the traffic between the hypervisors)

This isn't the venue to discuss this (it's vaguely being discussed over on ARMD, although it's not really intended as input to ARMD), I just wanted to bring it to y'alls attention (and muddy the water somewhat):

> A new version of I-D, draft-wkumari-dcops-l3-vmmobility-00.txt has been successfully submitted by Warren Kumari and posted to the IETF repository.
> 
> Filename:	 draft-wkumari-dcops-l3-vmmobility
> Revision:	 00
> Title:		 Virtual Machine mobility in L3 Networks.
> Creation date:	 2011-08-11
> WG ID:		 Individual Submission
> Number of pages: 8
> 
> Abstract:
>  This document outlines how Virtual Machine mobility can be
>  accomplished in datacenter networks that are based on L3
>  technologies.  It is not really intended to solve (or fully define)
>  the problem, but rather to outline it at a very high level to
>  determine if standardization within the IETF makes sense.
> 
> 
> 
> 
> The IETF Secretariat

This will probably not help the scope discussions :-P

W

---

From: Melinda Shore <[email-masked]>
Date: Tue, 23 Aug 2011 13:43:03 -0800

As the discussion has progressed it's become more-or-less clear
that there might possibly perhaps be one or two interesting
problems here, maybe, although the people pushing for the
chartering of something-or-other (anything!) to do with
data centers and virtualization seem to be working very hard
indeed to obscure anything that might turn out to have some
value.

First, there appears to be a substantial problem related to
moving flow-associated state in middleboxes when a network
connection (deliberately keeping "connection" vague for the
moment) fails over.  This has certainly come up in the context
of multihomed connections in sctp, and to be honest I
haven't followed that as closely as I should.  Still, even
if sctp has something that works brilliantly there would be
questions about applying their mechanism in a different
context.

Second, there appears to be another substantial problem,
this one related to how to handle routing and network
state when a VM is migrated from one hypervisor to a
network-topographically-remote hypervisor when both are
on the same layer 2 subnet being tunneled over a layer
3 transport (or when they're not on the same subnet
at all, which I gather is considerably less common).

The problem here is that it's not at all clear that there's
a constituency for the work.  I'm not talking about people
to write and review internet drafts, but rather companies
standing up and saying "We want this problem solved and we
think it should be solved through an open standards process,"
and data centers/service providers standing up and saying
"We would deploy this."  Saying that you're absolutely
certain that somebody else would say one of these things
does not count.

It would be unfortunate if the IETF were to sink valuable
resources into another effort that nobody cares about
other than people looking for stuff to put on their CV.
Is there an audience for this work?

Melinda

---

From: "Yingjie Gu(yingjie)" <[email-masked]>
Date: Wed, 24 Aug 2011 13:39:45 +0800

I think we are still on the right way to move forward. 
We need to hear different voice so that we can know what go to people's mind
when they see "State Migration".

For now, we have heard different opinions, e.g. 
'state migration for service failover' 
'need differentiate requirements for Optimization-based migration and
restoration-based migration', 
'layer 2 connectivity over L3 infrastructure', 
'think about IP mobility instead of state migration'
'services running on VM will impact the way of state migration' and many
others. 
Sorry that I don't list all of the input. 
Though not all of them will finally be in scope, or we finally fail to
charter a workable scope in IETF, they are highly valuable input for scope
discussion. 
Thank you very much.

Meanwhile, I also try to ask DC provider to share their use cases. China
Telecom and China Mobile, who are also looking forward to become DC
provider, see the concrete requirements and prepare to write down a draft to
introduce the use cases in real scenarios. But they may introduce some use
cases to mail list before the draft is completed.

Best Regards
Gu Yingjie

-----邮件原件-----
发件人: [email-masked] [mailto:[email-masked]] 代表 Melinda
Shore
发送时间: 2011年8月24日 乐乐5:43
收件人: [email-masked]
主题: [sami] Trying to figure out where we are

Melinda

---

From: "zhangyunfei" <[email-masked]>
Date: Wed, 24 Aug 2011 14:03:54 +0800

I can share a use case here from our point of view in data center operation(will detail this in the upcoming draft as Yingjie said):
We are considering running different services (e.g., VoIP and streaming services) by VM in the same cluster withinn data center. We know different services show different user visiting behaviors. For example, in the midnight, there are few VoIP usage while there are still large amount of streaming usage( this is just an example, maybe not so accurate to fit with the reality). In such case, we'd like merge the existing VoIP and streaming services into fewer machines and cut down the machines running only few VoIP usage to reduce the power consumption.
In this senario, we need to consider how to migrate the existing VM states.

Looking forward to discussing more use cases in the mailing list.

BR
Yunfei

zhangyunfei
2011-08-24

发件人： Yingjie Gu(yingjie)
发送时间： 2011-08-24 13:41:04
收件人： 'Melinda Shore'; [email-masked]
抄送： 
主题： Re: [sami] Trying to figure out where we are

Best  Regards
Gu  Yingjie

-----邮件原件-----

Melinda

---

From: Thomas Narten <[email-masked]>
Date: Wed, 24 Aug 2011 08:23:05 -0400

"zhangyunfei" <[email-masked]> writes:

> I can share a use case here from our point of view in data center
> operation(will detail this in the upcoming draft as Yingjie said): We
> are considering running different services (e.g., VoIP and streaming
> services) by VM in the same cluster withinn data center. We know
> different services show different user visiting behaviors. For
> example, in the midnight, there are few VoIP usage while there are
> still large amount of streaming usage( this is just an example, maybe
> not so accurate to fit with the reality). In such case, we'd like
> merge the existing VoIP and streaming services into fewer machines and
> cut down the machines running only few VoIP usage to reduce the power
> consumption.  In this senario, we need to consider how to migrate the
> existing VM states.

How is this any different than standard VM migration as practiced
today? What special requirements result from the above scenario?

The topic of this effort is "state migration". In order to make
progress, we need to be very specific about what kind of state and in
what devices needs to be transferred. And why existing approaches as
practiced today are insufficient.

If folk cannot (in one or two paragraphs) answer the above questions,
I do not think writing an Internet Draft about the above would be a
good use of anyone's time.

Thomas

---

From: Jamal Hadi Salim <[email-masked]>
Date: Wed, 24 Aug 2011 08:30:04 -0400

> good use of anyone's time.

+1.

cheers,
jamal

---

From: "Yingjie Gu(yingjie)" <[email-masked]>
Date: Thu, 25 Aug 2011 20:51:52 +0800

In previous mail discussion, there are questions on why should VM be
migrated. 
And also, in the side meeting at Quebec, people want to learn the use case
from real scenario. 
I guess Yunfei's use case could be an answer to that. 

Here are some of the states that I can see on the devices that need to be
migrated.

1. States on switches:
1.1  DHCP snooping table on ports;
1.2  IGMP snooping table on ports;
1.3  Dynamic ACL which is created by traffic or authentication;

2. States on FW
2.1  TCP Connect States
2.2  Dynamic ACL

3. States on LB
3.1  Connect States.
3.2  Session States.

4. States on IPS/IDS.
4.1 Cumulative data

We can think about two scenarios:
1. VM migrate within the subnet under the same FW/LB/IPS. In this case,
states also migrate under the same FW/LB/IPS etc.
States need to be migrated include 1.1, 1.2, 1.3.

2. VM migrate between two geographic data center sites, but the two sites
are in the same subnet (refer to David Black's email). In this case, states
also migrate between FW/LB/IPS etc
All the above states need to be migrated.

Best Regards
Gu Yingjie

-----邮件原件-----
发件人: [email-masked] [mailto:[email-masked]] 代表 Thomas
Narten
发送时间: 2011年8月24日 乐乐20:23
收件人: zhangyunfei
抄送: Yingjie Gu(yingjie); 'Melinda Shore'; [email-masked]
主题: Re: [sami] Trying to figure out where we are

Thomas

---

From: Thomas Narten <[email-masked]>
Date: Thu, 25 Aug 2011 09:11:08 -0400

> In previous mail discussion, there are questions on why should VM be

I think we should take it as a given that VMs move around. People do
that today, we all see the value.

> And also, in the side meeting at Quebec, people want to learn the use case

Yes. looking at specific scenarios allows us to ask the question of
whether the scenario is realistic and corresponds to what operators
actual do today (or want to do), or whether it is just a theoretical
problem (in which case it's very questionable whether the IETF should
do work.)

> I guess Yunfei's use case could be an answer to that.

See my previous posting, I don't understand what use case this is yet.

> 1.1  DHCP snooping table on ports;

How often does DHCP snooping take place in data centers? In other
words, is this a real problem, or a theoretical one?

> 1.2  IGMP snooping table on ports;

I think we need to outline some specific scenarios that describe what
the problem actually is here. Who is doing snooping, why, and why a VM
move leaves something not working.

> 1.3  Dynamic ACL which is created by traffic or authentication;

A specific scenario  would be helpful here. Both so I can understand
the details of the problem,  and so we can discuss whether the
scenario is a real problem in existing datacenters today, or just a
theoretical problem.

> 2.2  Dynamic ACL

Same as above.

> 3.2  Session States.

Same as above.

> 4.1 Cumulative data

Same as above.

> States need to be migrated include 1.1, 1.2, 1.3.

Before we spend a lot of time on this, we need to establish that there
are actual real scenarios that correspond to real problems happening
in data centers today. If a problem is just theoretical, it becomes
very unclear whether the IETF should do any work (at this time).

I am not interested in discussing solutions, or even the problem in
much detail, if there is no compelling scenario motivating the
problems for which solutions are being considred.

Thomas

---

From: Warren Kumari <[email-masked]>
Date: Mon, 29 Aug 2011 13:20:20 -0400

> that today, we all see the value.

Yup.

> very unclear whether the IETF should do any work (at this time).

Yes.

> problems for which solutions are being considred.

My message here is basically content free, I just wanted to support / agree with what Thomas said -- this is fairly unusual, so worth noting...

W

---

From: "Yingjie Gu(yingjie)" <[email-masked]>
Date: Tue, 30 Aug 2011 09:30:58 +0800

Hi Thomas and all,

We are making effort on a use case draft now, and will present it at early
Sep.

Best Regards
Gu Yingjie

-----邮件原件-----
发件人: Thomas Narten [mailto:[email-masked]] 
发送时间: 2011年8月25日 乐乐21:11
收件人: Yingjie Gu(yingjie)
抄送: 'zhangyunfei'; 'Melinda Shore'; [email-masked]
主题: Re: [sami] Trying to figure out where we are

Same as above.

Same as above.

Same as above.

Thomas