Subject: [sami] A new draft on state migration use cases is submitted.

From: A tao <[email-masked]>
Date: Mon, 3 Oct 2011 22:49:31 +0800

Hi all,
We have submited a draft on state migration use cases.
You can find it at
http://www.ietf.org/id/draft-yang-opsawg-policies-migration-use-case-00.txt

We talked with persons from the DC providers (China Mobile, China Telecom,
Tencent), Firewall & Switch vendors (Huawei Symantec, Ruijie, and etc.) to
make sure the states and use cases in the draft are reasonable.

We appreciate your comments and suggestions.

BTW, it's a public holiday in China from 1st to 7th Oct. So I may not
respond to your comments in time.

Best Regards.

---

From: 刘茗(研六 福州) <[email-masked]>
Date: Sun, 9 Oct 2011 13:41:24 +0000

One of our customers, the leader of online shopping provider in china, have the same requirement.  They run VMs on the power x86 machine with KVM hypervisor. For some security reasons, they applied the ACLs through the Linux’s IPtable running on the Hypervisor. But when the VM floating , the IPtable profile can not be migrated to the other machine. So they hope the switch can replace the IPTable  and can migrates the ACL profiles for the VM when floating .

From: A tao [mailto:[email-masked]]
Sent: Monday, October 03, 2011 10:50 PM
To: [email-masked]
Subject: [sami] A new draft on state migration use cases is submitted.

Hi all,

We talked with persons from the DC providers (China Mobile, China Telecom, Tencent), Firewall & Switch vendors (Huawei Symantec, Ruijie, and etc.) to make sure the states and use cases in the draft are reasonable.

BTW, it's a public holiday in China from 1st to 7th Oct. So I may not respond to your comments in time.

Best Regards.

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Sun, 9 Oct 2011 18:01:38 +0200

> One of our customers, the leader of online shopping provider in china, have the same requirement.  They run VMs on the power x86 machine with KVM hypervisor. For some security reasons, they applied…

The switches really have nothing to do with ACLs sitting in the
hypervisor. Making the switches responsible for migrating the ACLs
seems broken to me.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: "Yingjie Gu(yingjie)" <[email-masked]>
Date: Mon, 10 Oct 2011 09:25:30 +0800

Ming, you'd better introduce yourself :)

My understanding of these words is that, instead of deploying ACLs on Hypervisor and try to migrate ACLs between Hypervisors, the customer would like the ACLs be deployed on switches and migrate ACLs between switches. 

Is this what you mean, Ming?

Best Regards
Gu Yingjie

-----邮件原件-----
发件人: [email-masked] [mailto:[email-masked]] 代表 Juergen Schoenwaelder
发送时间: 2011年10月10日 乐乐0:02
收件人: 刘茗(研六 福州)
抄送: A tao; [email-masked]
主题: Re: [sami] A new draft on state migration use cases is submitted.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>
_______________________________________________
sami mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/sami

---

From: 刘茗(研六 福州) <[email-masked]>
Date: Mon, 10 Oct 2011 14:24:35 +0000

Dear  Yingjie,

Yes, you got my point. Our customers deploy the virtualization in order to improve the utility of hardware resources, especially the CPU . But the security policy executed by the hypervisor will consume the CPU resource without money back. So if the switches can migrate the security policy across the physical machine, it will make more money back. 

Oh, I forgot introducing myself. My name is Ming Liu. I'm  a product manager from a network product vendor in China mainland and in charge of solutions and products for Data Center. And our customers include government, universities, ICP and so on .

-----Original Message-----
From: Yingjie Gu(yingjie) [mailto:[email-masked]] 
Sent: Monday, October 10, 2011 9:25 AM
To: 'Juergen Schoenwaelder'; 刘茗(研六 福州)
Cc: 'A tao'; [email-masked]
Subject: Re: [sami] A new draft on state migration use cases is submitted.

Best Regards
Gu Yingjie

-----邮件原件-----
发件人: [email-masked] [mailto:[email-masked]] 代表 Juergen Schoenwaelder
发送时间: 2011年10月10日 乐乐0:02
收件人: 刘茗(研六 福州)
抄送: A tao; [email-masked]
主题: Re: [sami] A new draft on state migration use cases is submitted.

> One of our customers, the leader of online shopping provider in china, have the same requirement.  They run VMs on the power x86 machine with KVM hypervisor. For some security reasons, they applied…

The switches really have nothing to do with ACLs sitting in the hypervisor. Making the switches responsible for migrating the ACLs seems broken to me.

/js

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 10 Oct 2011 19:49:52 +0200

On Mon, Oct 10, 2011 at 02:24:35PM +0000, 刘茗(研六 福州) wrote:
> Dear  Yingjie,
> 
> Yes, you got my point. Our customers deploy the virtualization in
> order to improve the utility of hardware resources, especially the
> CPU . But the security policy executed by the hypervisor will
> consume the CPU resource without money back. So if the switches can
> migrate the security policy across the physical machine, it will
> make more money back.

Do you have any data to share with us how much of your hypervisor CPU
time is spent on ACL processing? I hear you saying that you are short
on hypervisor CPU cycles but have plenty of CPU cycles on the switches
(or that it is cheaper to buy CPU cycles on switches than on
hypervisor platforms). Since it is the first time I hear this
argument, I am wondering whether you have some numbers to share.

/js

---

From: Linda Dunbar <[email-masked]>
Date: Mon, 10 Oct 2011 20:45:39 +0000

Tao, 

That is an interesting description. Can you elaborate a little bit on pros and cons of hypervisor CPU taken by the security vs. the extra processing on switches? 

Linda 

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> 刘茗(研六 福州)
> To: Yingjie Gu(yingjie); 'Juergen Schoenwaelder'
> Yes, you got my point. Our customers deploy the virtualization in order
> to improve the utility of hardware resources, especially the CPU . But
> the security policy executed by the hypervisor will consume the CPU
> resource without money back. So if the switches can migrate the
> security policy across the physical machine, it will make more money
> back.
> 
> Oh, I forgot introducing myself. My name is Ming Liu. I'm  a product
> manager from a network product vendor in China mainland and in charge
> of solutions and products for Data Center. And our customers include
> government, universities, ICP and so on .
> 
> My understanding of these words is that, instead of deploying ACLs on
> Hypervisor and try to migrate ACLs between Hypervisors, the customer
> would like the ACLs be deployed on switches and migrate ACLs between
> switches.
> 
> > One of our customers, the leader of online shopping provider in china,
> have the same requirement.  They run VMs on the power x86 machine with
> KVM hypervisor. For some security reasons, they applied the ACLs
> through the Linux’s IPtable running on the Hypervisor. But when the VM
> floating , the IPtable profile can not be migrated to the other machine.
> So they hope the switch can replace the IPTable  and can migrates the
> ACL profiles for the VM when floating .
>

---

From: 卓志强(研七 福州) <[email-masked]>
Date: Tue, 11 Oct 2011 06:06:34 +0000

I have the following documents to get some reference data,
http://people.netfilter.org/kadlec/nftest.pdf

The "filtering rules" is like ACLs. As the numbers increase, the performance of different software have declined.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Linda Dunbar
Sent: Tuesday, October 11, 2011 4:46 AM
To: 刘茗(研六 福州); Yingjie Gu(yingjie); 'Juergen Schoenwaelder'
Cc: 'A tao'; [email-masked]
Subject: Re: [sami] A new draft on state migration use cases is submitted.

Tao, 

Linda

---

From: Melinda Shore <[email-masked]>
Date: Mon, 10 Oct 2011 22:30:02 -0800

> The "filtering rules" is like ACLs. As the numbers increase, the performance of different software have declined.

I read that paper.  If you go to section 4.3 you'll see that
performance varies with the underlying data structures, and
while ipfilter uses a really witless linear search the others
mentioned do not, and don't show the same performance impacts
from large numbers of installed filter rules:

"The lines for iptables on both figures show clearly the non-scaling 
behaviour of iptables. However both nf- hipac and ipset performed almost 
indifferently with regard of the number of rules. ipset was a tiny bit 
better than nf-hipac, but ipset is more lighter and simpler than 
nf-hipac. The last figure shows the required time to add the given 
number of rules to the kernel. Again, iptables suffers from its linear 
algorithms (which produces exponential behaviour in rule-addition due to 
the cumulating effect) while nf-hipac and ipset are immune from such 
problems."

Melinda

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 11 Oct 2011 09:51:43 +0200

On Tue, Oct 11, 2011 at 06:06:34AM +0000, 卓志强(研七 福州) wrote:
> The "filtering rules" is like ACLs. As the numbers increase, the
> performance of different software have declined.

Your proposition is that moving filtering rules from the endpoint (the
hypervisor) into the network is cheaper and easier to scale. To make
any sense of the above quoted document, I would need to know what the
performance of middleboxes is going to be and I would need to know how
many hypervisors are going to be behind a middlebox. If its lets say
1:10 (one switch to ten hypervisors), then the middleboxes would need
to outperform a kernel software filter by a factor significant larger
than 10. I guess you have done this calculation so it should be easy
to provide a concrete data point.

Once the scalability trade-off is clear, one can look at the costs of
the various options (essentially more hypervisors vs. fancier switches
that peak into L4 and perhaps even beyond). I would love if problem
statement documents would go into exactly this level of detail.

/js

---

From: Melinda Shore <[email-masked]>
Date: Tue, 11 Oct 2011 06:46:02 -0800

On 10/10/11 11:51 PM, Juergen Schoenwaelder wrote:
> many hypervisors are going to be behind a middlebox.

Well, I'm not sure.  It seems to me that "easier to scale" is an issue
only if what's being done fails to scale with the number of rules.  The
cited paper doesn't do that - in fact, it says that if you use freely
available open source software, the marginal cost of adding more rules
is 0.

I agree with your point more generally, however.  It's counterintuitive
to think that it's cheaper to aggregate rulesets in network devices than
it is to push the platform-specific ones out to the edges.  But, unless
there's a real problem here I think the discussion is irrelevant, at
least in the context of trying to figure out whether or not to charter
some IETF work.

Melinda

---

From: A tao <[email-masked]>
Date: Wed, 12 Oct 2011 00:01:51 +0800

No doubt that a performance test results can help people to understand, to
some extent, why there are requirements and implementations to move security
functions, such as firewall/DHCP snooping/ACLs, out of Hypervisor or
servers.

If Ming can provide this results, that will be best.

We don't have the data but we also talked with DC providers, and they gave
me an example to explain why they want the firewall being executed by
network side. They deployed a VM to run Firewall functions on a x86 server.
And by test, they find the firewall consumes more than large amount of CPU
capacity which otherwise can be used to support 2-3 VMs. I don't get the
real data and the test situation, this is only a information.

2011/10/11 Juergen Schoenwaelder <[email-masked]>

---

From: "Yingjie Gu(yingjie)" <[email-masked]>
Date: Wed, 12 Oct 2011 10:57:04 +0800

I would say that the goal of this test should not be regarded as a proof to
show which model is better, security functions deployed inside of
server/Hypervisor or outside. People need to choose a way based on their
real scenario.

I think there are some knowledge that should be the base of SAMI, they are:

1. IT staff in DCs or in enterprise networks hope to see clear edge of
Network and Computing. That is let the Network do the network things,e.g.
forwarding and security, and Servers do the computing things. Argument
exists in this topic, but at least some providers we know of show the
requirements on clear edge. 

2. Some large ISPs we talked with, who are also DC service providers,
require to move additional network functions to network side, so that they
can save CPU processing capability to create more VMs, and selling VMs is
the way they can earn money. 

(The performance test results, if we have, can be used as a side proof to
support DC providers' requirements.)

(I know some IDC/DC providers register this mail list, and I really look
forward to hear their opinions)

3. There are many existing DCs where ACLs and firewalls are deployed on
network side. Hypervisor can run some part of security functions, but not
all of the security functions are deployed on Hypervisor. 

4. Even for the security functions deployed on Hyperviser, not all of the
Hypervisor vendors can support state live migration. This is another reason,
mentioned by Ming also, why people want to move network functions to
network.

Of course, I am willing to hear people's opinions on these knowledge.

If we agree on the above knowledge, we can move forward, to discuss whether
the states and scenarios make sense to you, 

  _____  

Best Regards
Gu Yingjie

发件人: A tao [mailto:[email-masked]] 
发送时间: 2011年10月12日 乐乐0:02
收件人: Juergen Schoenwaelder
抄送: 卓志强(研七 福州); Linda Dunbar; 刘茗(研六 福州); Yingjie Gu(yingjie);
[email-masked]
主题: Re: [sami] A new draft on state migration use cases is submitted.

/js

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 12 Oct 2011 08:17:38 +0200

> the way they can earn money. 

I assume security functions embedded in network equipment do not come
for free either. So these ISPs seem to believe that security functions
in network devices are easier to manage and cheaper to scale.
 
> mentioned by Ming also, why people want to move network functions to

My understanding is that today neither hypervisors nor 'switches'
support the state live migration you want. Hence, these ISPs seem to
believe that 'switches' are easier to get updated with state migration
functions than hypervisors (even though there is quite some software
out there to manage VM infrastructures).

Since I am not a big ISP, I can't really judge any of this. But those
two arguments at least seem somewhat counter intuitive to me. Anyway,
I like to point your attention to the "Software Driven Networks" BOF
(scroll down on http://trac.tools.ietf.org/bof/trac/wiki/WikiStart).
This might actually be the solution to your problem.

/js

---

From: Melinda Shore <[email-masked]>
Date: Tue, 11 Oct 2011 22:41:17 -0800

On 10/11/11 10:17 PM, Juergen Schoenwaelder wrote:
> support the state live migration you want.

There's been some work done around a more static scenario - SCTP and
NAT - although I'm not sure what state it's in.  Probably worth
checking out whether anything's been done with regard to this in
mobile IP.

I will say, based on the midcom architectural work and the follow-on
stuff with route-coupled signaling protocols that the topology-related
problems with moving flow-coupled state around the network is
exceptionally difficult - sufficiently difficult, I think, that
it really is a good idea to avoid it when possible.

I still think there's a core of an interesting idea here and I'm
pretty frustrated that on the one hand its advocates have not been
able to make a stronger case and that on the other hand they won't
let it stew quietly at the back of the stove for a little while.
In particular, in a crunchy-on-the-outside network security model
there's state associated with firewalling and with NAT *in* *network*
*devices* that will probably need to move, but the authors haven't
presented a case for that, the service providers are apparently not
asking for that, and it seems to have drifted out of the picture.
Instead, we're getting some fairly confusing (and unsupported)
assertions about performance and DHCP sniffing and all sorts of
things that mostly leave me doubting what I'm reading (they're
not helped much by the early history of the cloud/datacenter
advocacy at the IETF).

If you can forgive a badly-mixed metaphor I think this particular
stewpot has run out of track.

Melinda

---

From: "Yingjie Gu(yingjie)" <[email-masked]>
Date: Wed, 12 Oct 2011 14:51:39 +0800

Well, I won't try to answer why ISPs and DC providers raise these
requirments. I am sure not all of ISPs and DC providers ask for the same
thing, but we do get the requirements from our customers, at least they can
represent part of the operation views. 

As for your point of "neither Hypervisors nor switches support state live
migration", that is not big deal. Because we make all these discussions here
to figure out whether this is a problem and whether we can resolve it,
instead of whether there is existing solutions for this. 

I think Software Driven Network has no clear relationship to State Migration
after I read the drafts. I may be wrong, could you give some hints based on
your knowledge?

Thanks.

Best Regards
Gu Yingjie

-----邮件原件-----
发件人: [email-masked] [mailto:[email-masked]] 代表 Juergen
Schoenwaelder
发送时间: 2011年10月12日 乐乐14:18
收件人: Yingjie Gu(yingjie)
抄送: 'A tao'; '卓志强(研七 福州)'; [email-masked]; '刘茗(研六 福州)'; 'Linda
Dunbar'
主题: Re: [sami] 答复: A new draft on state migration use cases is
submitted.

 

/js

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 12 Oct 2011 09:42:11 +0200

> after I read the drafts. I may be wrong, could you give some hints based on

Software Driven Network seems to be a different name for what is known
as OpenFlow networks <http://www.openflow.org>, where the control
logic of a network is separated from the hardware taking care of
packet forwarding. If you have OpenFlow switches in front of your
hypervisors, than you control the flow processing behaviour of the
switches from a logically centralized external controller. Moving a VM
then just requires to inform the controller and the controller in turn
will "reprogram" the OpenFlow switches to move the data where it needs
to be.

/js

---

From: "Yingjie Gu(yingjie)" <[email-masked]>
Date: Wed, 12 Oct 2011 17:22:39 +0800

Thank you Melinda. You have mentioned a very interesting area, that is Mobile IP.
At last IETF meeting, I prepared a picture of flow of state migration mime UMTS Serving RNC Handoff. I believe we can learn from UMTS Serving RNC Handoff when we try to resolve this problem.

About NAT, we removed this issue from the latest use case draft, that's a decision based on our investigating. The draft-gu-opsawg-policies-migration is an old version of problem description, in which some of the use cases are not applicable. I plan to ask authors to merge problem statement and use case drafts to make a consistent description of the problem.

Sorry for my bad English, by " they won't let it stew quietly at the back of the stove for a little while." do you mean the advocates should stay a little quiet on the mail list? If that's what you mean, I feel confused. We do a lot of work to investigate and present it to persons, and reply to questions on the problem. When someone raise a question on the mail list, isn't he looking forward to a response? Well, I am not very familiar of how to make new work proposal. My understanding of a advocate of a proposal in IETF should try his best to investigate problems and invite experts to join discussion. I would hear more suggestions on how to manage the mail list, you can send private mails to me. 

Thanks.

Best Regards
Gu Yingjie

-----邮件原件-----
发件人: Melinda Shore [mailto:[email-masked]] 
发送时间: 2011年10月12日 乐乐14:41
收件人: Juergen Schoenwaelder
抄送: Yingjie Gu(yingjie); 'A tao'; "'卓志强(研七 福州)'"; [email-masked]; "'刘茗(研六 福州)'"; 'Linda Dunbar'
主题: Re: [sami] 答复: A new draft on state migration use cases is submitted.

Melinda

---

From: <[email-masked]>
Date: Wed, 12 Oct 2011 10:40:22 -0400

Hi Melinda,

> not helped much by the early history of the cloud/datacenter

I'll go further here - having taken another detailed look, I don't see a
a DHCP snooping/sniffing problem that needs a protocol solution.  The DHCP
snooping/sniffing and related ARP filtering technology is intended to prevent
unwanted changes to MAC/IP address binding (e.g., ARP cache poisoning).  When
a VM moves, MAC/IP bindings are unaffected; what changes is the interface
that the VM's MAC(s) now reside(s) at.  It should be sufficient to ensure
that the gratuitous ARP issued after a VM moves updates not only the MAC
learning tables in the switch, but also any interface information in the
DHCP snooping/sniffing info retained by the switches.

That feels like an implementation problem:
	- Make sure the info obtained from DHCP snooping/sniffing is shared
		across the L2 network domain that supports live VM migration.
		As part of this, the entire L2 network domain should uniformly
		support DHCP snooping/sniffing, not just part of the domain.
	- Make sure that the gratuitous ARP issued after a live VM migration
		updates the interface in the DHCP snooping/sniffing info in
		addition to updating MAC learning information.

I still think there's a firewall problem, but the really compelling case may be
VM live migration across distance (different data centers), and that's still an
emerging use case.  That requires stretching an L2 domain across the data centers
- there are a number of ways to do this, including L2VPNs and OTV (see
draft-hasmit-otv for the latter).

I'm not sure about NAT state, as neither the IP nor the MAC changes when the
VM moves - for this reason, moving a VM across sides of a NAT doesn't seem like
something that's likely to work well.

Thanks,
--David

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Melinda Shore
> Sent: Wednesday, October 12, 2011 2:41 AM
> To: Juergen Schoenwaelder
> Cc: Yingjie Gu(yingjie); "'卓志强(研七 福州)'"; 'Linda Dunbar'; 'A tao'; "'刘茗(研六 福州)'";
> sami@i etf.org
> Subject: Re: [sami] 答复: A new draft on state migration use cases is submitted.
>

---

From: Melinda Shore <[email-masked]>
Date: Wed, 12 Oct 2011 11:00:19 -0800

On 10/12/2011 01:22 AM, Yingjie Gu(yingjie) wrote:
> Sorry for my bad English, by " they won't let it stew quietly at
 > the back of the stove for a little while." do you mean the
 > advocates should stay a little quiet on the mail list? If
 > that's what you mean, I feel confused. We do a lot of work
 > to investigate and present it to persons, and reply to questions
 > on the problem. When someone raise a question on the mail
 > list, isn't he looking forward to a response? Well, I am
 > not very familiar of how to make new work proposal.
 > My understanding of a advocate of a proposal in IETF should
 > try his best to investigate problems and invite experts to
 > join discussion.

That's the gist of it, and I apologize for using such idiomatic
language.  Basically my sense is that we're just not making any
progress.  You guys are doing a lot of work and my feeling is that
it's not fair to ask you to do more unless there's a reasonable
expectation of progress, and at this point I don't think there is.
I think there's been some hope that at some point someone would
put together a clear, compact problem statement with a defined,
manageable scope and instead what's been happening is that you
put something out, questions are raised, and rather than using them
to refine the problem statement you've been pivoting - saying
"Okay, how about this instead?"  To a certain extent I feel like
you're being asked to a bunch of probably pointless exercises and
that it's not fair to you if there's not, as I said, a reasonable
expectation of progress.

I think 1) the problem space is a good one, 2) the problem
statements so far have been just awful, and 3) I don't know where
to go from here.  I think backing off a little bit and resuming
the discussions later might not be the worst idea.

Melinda