Subject: [nvo3] comments on draft-drake-nvo3-evpn-control-plane

From: Sunny Rajagopalan <[email-masked]>
Date: Tue, 25 Sep 2012 14:57:00 -0700

I have some comments/suggestions on draft-drake-nvo3-evpn-control-plane 
(some of these suggestions have been made privately to the 
draft-marques-l3vpn-end-system authors as well):

1) I would suggest *not* altering the semantics of the MPLS label in the 
BGP route. Instead, use the route distinguisher to carry the 24-bit VNID 
(this is arguably better since the semantics of the RD align better with 
the semantics of the VNID). I would suggest encoding this as a type 0 RD, 
with the VNID going into the Assigned number sub-field. In addition, call 
out that an MPLS label value of 0 in the BGP route is a valid value, and 
will be used by PEs which do not support MPLS encap. 

2) I've separately mentioned that the draft should call out that the PE 
control plane need not be co-located with the PE forwarding plane, and 
that XMPP could be used as the messaging format between the two. Each 
endpoint update should look like this:{endpoint_mac, {endpoint_ips}, {NVE 
IPs}, RD, label}. This allows for true decoupling of the control plane 
from the frame format.

3) Even though it's not called out explicitly, the model used in both 
draft-drake-nvo3-evpn-control-plane and draft-marques-l3vpn-end-system 
assumes that the PE forwarding plane is interested in having all of the 
endpoint routes in their participating VNIDs. In my view, this puts an 
unnecessary load on NVEs. Instead, we can modify XMPP so that the NVE can 
request end point resolution of specific addresses within a VNID, so it 
only ever needs to cache information about flows that are transiting it. 

The issue here is how to push endpoint updates to the NVE when endpoints 
move (since having the PE keep track of which update to push to which NVE 
is unreasonable), My proposal here is to place the onus of requesting 
updates on the NVE, possibly triggered by the receipt of an ICMP error 
message. In other words, mandate that when an NVE receives a packet that 
after decapsulation is found to belong to an end-host that is no longer 
present in the attached customer network, it generates an ICMP error (rate 
controlled) back to source, taking care to include in the ICMP error 
packet enough of the payload so that the remote PE can figure out the 
customer endpoints that were attempting to communicate. On receipt of such 
an ICMP error, the NVE can extract the endpoint information from the 
payload and request the control plane for an endpoint update.

4) I would also suggest not having the NVE keep track of the encapsulation 
used by the remote endpoint. (this means that the tunnel encapsulation 
attribute in the draft would be unnecessary). Instead, the onus of 
translating between encapsulation methods should be on gateways. If you 
define the XMPP format well, you should be able to communicate end point 
information in a way that is agnostic of the encap method used by the NVE, 
allowing it to do the one encap it does best. A gateway can do this 
translation without BGP control plane intervention, because it would be 
configured to have interfaces that are (for eg) NVGRE on one arm and VXLAN 
on the other, and it would be obvious as to what encap to put on a packet 
going from one arm to the other. Applying an MPLS label would involve the 
gateway participating in BGP.

5) The multi-homing discussed in the draft only covers the case where the 
CE devices are physically separated from the PE devices by physical links. 
In the case where the PE forwarding plane is implemented in the 
hypervisor, the more important multi-homing question is what to do if the 
NVE is connected to two or more upstream devices, (basically, the NVE has 
two or more IP addresses). What I would like to happen is have a mac 
address route be associated with multiple NVE addresses. I believe this is 
possible using the framework established in the 
draft-raggarwa-sajassi-l2vpn-evpn, but it might be worthwhile to call out 
this case in the draft-drake-nvo3-evpn-control-plane draft. This is useful 
because draft-raggarwa-sajassi-l2vpn-evpn treats multiple PE IP addresses 
associated with the same mac address as belonging to separate MESs, and 
assumes that the CE-PE links will be labelled with an ethernet segment, 
which is not the case for the hypervisor NVE/PE.

Long email, thanks for reading!
--
Sunny
p.s. John, your email address in the draft is incorrect.

---

From: Kireeti Kompella <[email-masked]>
Date: Wed, 26 Sep 2012 10:27:01 -0700

Hi Sunny,

On Sep 25, 2012, at 14:57, Sunny Rajagopalan <[email-masked]> wrote:

> 3) Even though it's not called out explicitly, the model used in both draft-drake-nvo3-evpn-control-plane and draft-marques-l3vpn-end-system assumes that the PE forwarding plane is interested in having all of the endpoint routes in their participating VNIDs. In my view, this puts an unnecessary load on NVEs.

I'm glad you brought this up. Actually, this conversation has happened several times, to my knowledge, without a firm conclusion. I doubt we can close it, but at least, let's air it. 

Push: send route updates to everyone (first see Aldrin's comment about RT Constraint) as soon as you (the AUTHORITY/ORACLE) get them. 

Pull: sit on updates you get until someone asks for them. 

I could try to convince you what a terrible idea Pull is. I could refer to the Internet, which is all Push, and scales reasonably well. I could tell you about companies that staked all on Pull, and aren't here any more to implore us not to go there. I could ask you what happens to packets while the Pull is being responded to, or a bunch of related questions. I won't. 

> In my view, this puts an unnecessary load on NVEs.

Let's talk instead about the "unnecessary load". Can someone quantify this?  Is it CPU? memory? messaging? What's the bottleneck or pain point?

Here's my back-of-the-envelop calculation for memory, normalized to a VM. Let's say a VM has 10,000 friends in the DC that it might possibly want to talk to, but only one that it really wants to talk to. Let's say that a FIB route entry takes 100 bytes. That adds up to a possible total of 1MB vs. an actual of 100 bytes. Is 1MB really something one should optimize, especially considering that the VM has probably been allocated 4GB?

Maybe there is a dimension to this that really is an issue. I would love to know, especially with numbers backing it up. But let's first convince ourselves that this is a problem worth solving before spending cycles solving it. 

Thanks,
Kireeti

---

From: Thomas Narten <[email-masked]>
Date: Wed, 26 Sep 2012 14:04:10 -0400

Hi Kireeti.

Kireeti Kompella <[email-masked]> writes:

> I'm glad you brought this up. Actually, this conversation has
>  happened several times, to my knowledge, without a firm
>  conclusion. I doubt we can close it, but at least, let's air it.

> Push: send route updates to everyone (first see Aldrin's comment about
> RT Constraint) as soon as you (the AUTHORITY/ORACLE) get them.

> I could try to convince you what a terrible idea Pull is. I could
> refer to the Internet, which is all Push, and scales reasonably
> well.

You mean like DNS or ARP?

I do not think we should say "push is good, pull is bad". That is just
too categorical a statement.

> I could ask you what happens to packets while the Pull is being
> responded to, or a bunch of related questions. I won't.

They get queued. Or dropped. Or possibly something else. Yes, there
are implications to that. But not necessarily a show stopper either.

> Let's talk instead about the "unnecessary load". Can someone quantify
> this?  Is it CPU? memory? messaging? What's the bottleneck or pain
> point?

Some or all of the above.

If typical VNs are smallish, I agree that an NVE can preload full
tables with no problem. But what about for very large VNs? Should the
architecture *force* such preloading of full tables, even if the
working set of routes is actually very small?

And what about for very large VNs where there is a lot of VM mobility?
Should all NVEs be required to get update info even for destinations
they don't care about?

> Here's my back-of-the-envelop calculation for memory, normalized to a
> VM. Let's say a VM has 10,000 friends in the DC that it might possibly
> want to talk to, but only one that it really wants to talk to. Let's
> say that a FIB route entry takes 100 bytes. That adds up to a possible
> total of 1MB vs. an actual of 100 bytes. Is 1MB really something one
> should optimize, especially considering that the VM has probably been
> allocated 4GB?

Are you really arguing that the difference between 1MB and 100 bytes
is just noise?	 And who says this is in conventional memory on a host?
I could see this being done in the ASIC...

> Maybe there is a dimension to this that really is an issue. I would
> love to know, especially with numbers backing it up. But let's first
> convince ourselves that this is a problem worth solving before
> spending cycles solving it.

I do not think we should today require that the NVO3 architecture (in
a MUST sense) support only push. I think we should allow for either
push or pull, or some combination. I can see benefits with both
approaches.

Note also that we may be looking at the problem from different
perspectives. For example, in a single data center, I can imagine a
centralized directory service holding the complete address mapping
information for all the VNs in the DC. An NVE in such cases can query
such a mapping system with very very low latency.

Thomas

---

From: Melinda Shore <[email-masked]>
Date: Wed, 26 Sep 2012 10:21:19 -0800

> push or pull, or some combination. I can see benefits with both

Right, me too.  I do think that specifying push or pull before
identifying what problem you're trying to solve and what
characteristics the solution should have is getting ahead of
things, a bit.

Melinda

---

From: Sunny Rajagopalan <[email-masked]>
Date: Wed, 26 Sep 2012 12:09:05 -0700

Thomas has made some very valid points, and I would like to add the 
following observations:
1) 16,000: that's the number of remote endpoint entries (across all 
tenants) supported in today's top-of-the-line ASICs which can do 
VXLAN/NVGRE.You don't have room in there to store all the endpoints from 
all tenants. The asic vendors could make room, but that'd be a more 
expensive product.

2) The provider doesn't make any money from the networking software 
running on the server. To the extent that the networking software consumes 
resources, you have that much fewer resources available for the VMs (which 
is where the provider makes money). Further, unlike the VMs which can be 
moved around when they consume too many resources, the switch software is 
a permanent resident. 

In addition, switches implemented in the hypervisor kernel have to contend 
with a limited stack size (typically 4 or 8kb), and virtual memory 
constraints. You don't want the standard to force switch implementations 
to be in a VM, away from the direct data path in the kernel.

Again, I'm not advocating doing away with the push model. I'm just saying 
that you need to build in support for pull. 
--
Sunny

From:   [email-masked]
To:     Kireeti Kompella <[email-masked]>, 
Cc:     "[email-masked]" <[email-masked]>
Date:   09/26/2012 11:05 AM
Subject:        Re: [nvo3] Push or pull?
Sent by:        [email-masked]

Hi Kireeti.

Are you really arguing that the difference between 1MB and 100 bytes
is just noise?            And who says this is in conventional memory on a 
host?
I could see this being done in the ASIC...

Thomas

_______________________________________________
nvo3 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nvo3

---

From: "Fedyk, Donald (Don)" <[email-masked]>
Date: Wed, 26 Sep 2012 19:45:15 +0000

Hi Kieta

I think what you are trying to say is.

Push from an edge device what you know to a server/set of servers.  Include your community of interests(route targets or whatever).
Push from the server(s) to each edge device constrained by community of interests of that device.

The second point is Community of interest granularity.  That has to be at least as large as the set of devices that want to communicate.

This is one push model that scales well.

Don

From: [email-masked] [mailto:[email-masked]] On Behalf Of Kireeti Kompella
Sent: Wednesday, September 26, 2012 1:27 PM
To: Sunny Rajagopalan
Cc: [email-masked]
Subject: [nvo3] Push or pull?

Hi Sunny,

On Sep 25, 2012, at 14:57, Sunny Rajagopalan <[email-masked]<mailto:[email-masked]>> wrote:
3) Even though it's not called out explicitly, the model used in both draft-drake-nvo3-evpn-control-plane and draft-marques-l3vpn-end-system assumes that the PE forwarding plane is interested in having all of the endpoint routes in their participating VNIDs. In my view, this puts an unnecessary load on NVEs.

Thanks,
Kireeti

---

From: "Shah, Himanshu" <[email-masked]>
Date: Wed, 26 Sep 2012 15:45:36 -0400

I kind of agree with Thomas.

Cisco gave LISP (pull based) presentation which is a working model, during NVO3 interim.
I believe there are several ways to skin a cat and we should not limit our options.
Besides, I also got an impression from the chairs that discussing preference of one solution over other is
rather premature based on where the NVO3 is.

Regards,
himanshu

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Narten
Sent: Wednesday, September 26, 2012 2:04 PM
To: Kireeti Kompella
Cc: [email-masked]
Subject: Re: [nvo3] Push or pull?

Hi Kireeti.

Kireeti Kompella <[email-masked]> writes:

> happened several times, to my knowledge, without a firm  conclusion. I 
> doubt we can close it, but at least, let's air it.

> refer to the Internet, which is all Push, and scales reasonably well.

I do not think we should say "push is good, pull is bad". That is just too categorical a statement.

> responded to, or a bunch of related questions. I won't.

They get queued. Or dropped. Or possibly something else. Yes, there are implications to that. But not necessarily a show stopper either.

If typical VNs are smallish, I agree that an NVE can preload full tables with no problem. But what about for very large VNs? Should the architecture *force* such preloading of full tables, even if the working set of routes is actually very small?

And what about for very large VNs where there is a lot of VM mobility?
Should all NVEs be required to get update info even for destinations they don't care about?

> spending cycles solving it.

I do not think we should today require that the NVO3 architecture (in a MUST sense) support only push. I think we should allow for either push or pull, or some combination. I can see benefits with both approaches.

Note also that we may be looking at the problem from different perspectives. For example, in a single data center, I can imagine a centralized directory service holding the complete address mapping information for all the VNs in the DC. An NVE in such cases can query such a mapping system with very very low latency.

Thomas

---

From: "Fedyk, Donald (Don)" <[email-masked]>
Date: Wed, 26 Sep 2012 19:47:27 +0000

Sorry Kireeti  my spell checker hacked your name.

Don

From: Fedyk, Donald (Don)
Sent: Wednesday, September 26, 2012 3:45 PM
To: 'Kireeti Kompella'; Sunny Rajagopalan
Cc: [email-masked]
Subject: RE: [nvo3] Push or pull?

Hi Kireeti

Don

Hi Sunny,

Thanks,
Kireeti

---

From: Lucy yong <[email-masked]>
Date: Wed, 26 Sep 2012 20:02:30 +0000

I agree with Thomas. Both "push" and "pull" models have their application space. To add on two points, in push model, BGP peer group or ORF may be used to avoid every NVE to have all endpoint routes; in the pull model, an NVE will have temporary caching to reduce the number of queries.

Lucy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Shah, Himanshu
Sent: Wednesday, September 26, 2012 2:46 PM
To: Thomas Narten; Kireeti Kompella
Cc: [email-masked]
Subject: Re: [nvo3] Push or pull?

Regards,
himanshu

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Narten
Sent: Wednesday, September 26, 2012 2:04 PM
To: Kireeti Kompella
Cc: [email-masked]
Subject: Re: [nvo3] Push or pull?

Hi Kireeti.

Thomas

---

From: Lucy yong <[email-masked]>
Date: Wed, 26 Sep 2012 20:07:34 +0000

In addition, we should allow a solution to use both push and pull model.
Lucy

-----Original Message-----
From: Lucy yong 
Sent: Wednesday, September 26, 2012 3:03 PM
To: 'Shah, Himanshu'; Thomas Narten; Kireeti Kompella
Cc: [email-masked]
Subject: RE: [nvo3] Push or pull?

Lucy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Shah, Himanshu
Sent: Wednesday, September 26, 2012 2:46 PM
To: Thomas Narten; Kireeti Kompella
Cc: [email-masked]
Subject: Re: [nvo3] Push or pull?

Regards,
himanshu

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Narten
Sent: Wednesday, September 26, 2012 2:04 PM
To: Kireeti Kompella
Cc: [email-masked]
Subject: Re: [nvo3] Push or pull?

Hi Kireeti.

Thomas

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Wed, 26 Sep 2012 20:10:23 +0000

> in push model, BGP peer group or ORF may be used to avoid every NVE to have all endpoint routes;

In BGP VPN case, it is most efficient to use RT Constraint [RFC 6484] for selective route distribution - only send the VPN routes to the peer who has the relevant VPNs.
Luyuan

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Lucy yong
> Sent: Wednesday, September 26, 2012 4:03 PM
> To: Shah, Himanshu; Thomas Narten; Kireeti Kompella
> Cc: [email-masked]
> I agree with Thomas. Both "push" and "pull" models have their
> application space. To add on two points, in push model, BGP peer group
> or ORF may be used to avoid every NVE to have all endpoint routes; in
> the pull model, an NVE will have temporary caching to reduce the number
> of queries.
> 
> Lucy
> 
> They get queued. Or dropped. Or possibly something else. Yes, there are
> implications to that. But not necessarily a show stopper either.
> 
> I do not think we should today require that the NVO3 architecture (in a
> MUST sense) support only push. I think we should allow for either push
> or pull, or some combination. I can see benefits with both approaches.
>

---

From: Lucy yong <[email-masked]>
Date: Wed, 26 Sep 2012 20:13:28 +0000

RT is easy to define different topology, but not flexible to changes on fly. Peer group or ORF is more about filtering setting, it may better fit here.
Lucy

-----Original Message-----
From: Luyuan Fang (lufang) [mailto:[email-masked]] 
Sent: Wednesday, September 26, 2012 3:10 PM
To: Lucy yong; Shah, Himanshu; Thomas Narten; Kireeti Kompella
Cc: [email-masked]
Subject: RE: [nvo3] Push or pull?

> Shah, Himanshu
> Thomas Narten

---

From: John E Drake <[email-masked]>
Date: Wed, 26 Sep 2012 13:16:37 -0700

Right, a given PE only receives information for those VPNs of which it is a member.  This is a basic scaling concept common to all L2/L3 VPN technologies. 

Yours irrespectively,

John

> Luyuan Fang (lufang)
> Sent: Wednesday, September 26, 2012 1:10 PM
> > in push model, BGP peer group or ORF may be used to avoid every NVE
> to
> > have all endpoint routes;
> 
> In BGP VPN case, it is most efficient to use RT Constraint [RFC 6484]
> for selective route distribution - only send the VPN routes to the peer
> who has the relevant VPNs.
> Luyuan
> 
> 
> > From: [email-masked] [mailto:[email-masked]] On Behalf
> > Of Lucy yong
> > the pull model, an NVE will have temporary caching to reduce the
> > number of queries.
> >
> > Lucy
> >
> > I believe there are several ways to skin a cat and we should not
> limit
> > our options.
> > Besides, I also got an impression from the chairs that discussing
> > preference of one solution over other is rather premature based on
> > where the NVO3 is.
> >
> > Regards,
> > himanshu
> >
> conclusion.
> > I
> > > Let's talk instead about the "unnecessary load". Can someone
> > > quantify this?  Is it CPU? memory? messaging? What's the bottleneck
> > > or pain point?
> >
> > > Here's my back-of-the-envelop calculation for memory, normalized to
> > > a VM. Let's say a VM has 10,000 friends in the DC that it might
> > possibly
> > > should optimize, especially considering that the VM has probably
> > > been allocated 4GB?
> >

---

From: Kireeti Kompella <[email-masked]>
Date: Wed, 26 Sep 2012 13:29:29 -0700

Hi Thomas,

> You mean like DNS or ARP?

DNS is not in the data path.  ARP is, so good example.  Another is Ethernet flooding to find MACs.  Both are examples of Pull in the data plane, and I rest my case. :-)

> too categorical a statement.

Okay.  How about: Push is simple and known to work; but it costs.  Pull is harder, but may be cheaper.  Still a very broad statement, but it allows the following question:

Is the savings (in whatever dimension) that Pull offers over Push worth the effort?

If the answer (even theoretical) is Yes, sure, go for it.

> are implications to that. But not necessarily a show stopper either.

It was a show-stopper for Ipsilon.

> working set of routes is actually very small?

I thought I picked a fairly large VN, with 10,000 members.  Even if you take a larger one, say 100,000 members, the memory cost is (imo) small.

<snipped, see next email>

> push or pull, or some combination. I can see benefits with both

No question.

> such a mapping system with very very low latency.

Perhaps so (I would love to get more details.)  But what did you gain?

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Wed, 26 Sep 2012 20:45:19 +0000

RT constraint filters automatically based on RTs without manual intervention, it was aimed to scales by design. The initial motivation was Inter-AS VPN routes filtering when first started... It applies to RRs, ASBRs, and PEs within single AS or among diff ASes. It has been commonly used by many SPs for years. 

Peer group and ORF exist long before BGP L3VPN, they are still widely used for different purposes. The operator need to config the peer group and ORF filtering criteria for each one... RT constraint was particularly created for VPN auto filtering.

Why do you want to change RT on the fly? Would not that create security issues? (And peer group or ORF don't change RT on the fly anyway).

RT-rewrite can be done when inter-connecting the VPN across two ASes which are (or were) administrated by different administrative domains, but they are carefully controlled/designed and configured by the operators on the ASBRs or RRs. No RT change on the fly as I know of.

Luyuan

> From: Lucy yong [mailto:[email-masked]]
> Sent: Wednesday, September 26, 2012 4:13 PM
> To: Luyuan Fang (lufang); Shah, Himanshu; Thomas Narten; Kireeti
> Kompella
> Cc: [email-masked]
> RT is easy to define different topology, but not flexible to changes on
> fly. Peer group or ORF is more about filtering setting, it may better
> fit here.
> Lucy
> 
> to have all endpoint routes;
> 
> Of
> number
> Of
> > preference of one solution over other is
> Of
> are
> quantify
> a
> been
> a

---

From: Lucy yong <[email-masked]>
Date: Wed, 26 Sep 2012 20:48:09 +0000

Why do you want to change RT on the fly? Would not that create security issues? (And peer group or ORF don't change RT on the fly anyway).
[[LY]] to support VM mobility. What is an interested endpoint moving from one NVE to another?
lucy

Luyuan

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Wed, 26 Sep 2012 21:00:04 +0000

Why does the VM need to change RT when it moves? Is not the VM supposed to stay in the same VPN and only changing location?
The VM should keep the same RT in order to maintain the membership of that VPN it belongs to.

Luyuan

> Sent: Wednesday, September 26, 2012 4:48 PM
> Why do you want to change RT on the fly? Would not that create security
> issues? (And peer group or ORF don't change RT on the fly anyway).
> RT-rewrite can be done when inter-connecting the VPN across two ASes
> which are (or were) administrated by different administrative domains,
> but they are carefully controlled/designed and configured by the
> operators on the ASBRs or RRs. No RT change on the fly as I know of.
> 
> Luyuan
> 
>

---

From: John E Drake <[email-masked]>
Date: Wed, 26 Sep 2012 14:03:54 -0700

Right, the whole point is that a given VM is a member of a set of EVIs, where each EVI is itself defined in terms of a set of RTs.  The set of EVIs is invariant under VM Mobility. 

Yours irrespectively,

John

> Sent: Wednesday, September 26, 2012 2:00 PM
> Why does the VM need to change RT when it moves? Is not the VM supposed
> to stay in the same VPN and only changing location?
> > Why do you want to change RT on the fly? Would not that create
> > security issues? (And peer group or ORF don't change RT on the fly
> anyway).
> > > RT is easy to define different topology, but not flexible to
> changes
> > on
> > > fly. Peer group or ORF is more about filtering setting, it may
> > > better fit here.
> > > Lucy
> > >
> > > In BGP VPN case, it is most efficient to use RT Constraint [RFC
> > > 6484] for selective route distribution - only send the VPN routes
> to
> > > the
> > peer
> > > > or ORF may be used to avoid every NVE to have all endpoint
> routes;
> > in
> > > > > Push: send route updates to everyone (first see Aldrin's
> comment
> > > > about
> > > > > I could try to convince you what a terrible idea Pull is. I
> > > > > could refer to the Internet, which is all Push, and scales
> > > > > reasonably
> > > well.
> > > >
> > > > > Here's my back-of-the-envelop calculation for memory,
> normalized
> > to
> > > a
> > > > > total of 1MB vs. an actual of 100 bytes. Is 1MB really
> something
> > > one
> > > > perspectives. For example, in a single data center, I can imagine
> > > > a centralized directory service holding the complete address
> > > > mapping information for all the VNs in the DC. An NVE in such
> > > > cases can
> > query

---

From: Lucy yong <[email-masked]>
Date: Wed, 26 Sep 2012 21:21:47 +0000

OK, I see we have different interpretations on NVE to have all the endpoint route. My interpretation is that although an EVI provides the communication among the VMs in a closed use group, it is not necessary for a VM to communicate to all other VMs in the group at a time or all time, therefore having each NVE to maintain all the endpoint routes in an EVI is not necessary. This is the concern I got from Sunny's mail.

Seem that you have different interpretation.
Lucy 

-----Original Message-----
From: Luyuan Fang (lufang) [mailto:[email-masked]] 
Sent: Wednesday, September 26, 2012 4:00 PM
To: Lucy yong; Shah, Himanshu; Thomas Narten; Kireeti Kompella
Cc: [email-masked]
Subject: RE: [nvo3] Push or pull?

Luyuan

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Wed, 26 Sep 2012 21:33:14 +0000

Not clear what your interpretation has to do with change RT when VM moves...
Anyway, if a NVE does not even know what RTs/VPNs he has, maybe he should be fired? :-)
Luyuan

> Sent: Wednesday, September 26, 2012 5:22 PM
> OK, I see we have different interpretations on NVE to have all the
> endpoint route. My interpretation is that although an EVI provides the
> communication among the VMs in a closed use group, it is not necessary
> for a VM to communicate to all other VMs in the group at a time or all
> time, therefore having each NVE to maintain all the endpoint routes in
> an EVI is not necessary. This is the concern I got from Sunny's mail.
> 
> security
> better
> 6484]
> > > > Cisco gave LISP (pull based) presentation which is a working
> model,
> > > > during NVO3 interim.
> could

---

From: Sunny Rajagopalan <[email-masked]>
Date: Wed, 26 Sep 2012 14:35:18 -0700

There's two different topics being talked about here:
Push, which can be optimized using RTs. RTs can be used to ensure that an 
NVE only receives routes for the VPNs it participates in. The RT could 
change when the VM moves, in case the VM moves to an NVE which wasn't 
previously subscribed to the VPN that the VM belongs to. Due to the state 
this creates in the route server, you would want to use it at a VPN-level 
granularity.

Pull, which works by the NVE requesting the route server to "please give 
me endpoint information about address X in VPN Y". This doesn't result in 
additional state at the route server, and the route server doesn't push 
locators for any other end points in that VPN.

I think we're reaching consensus that both are needed, right?

--
Sunny

From:   Lucy yong <[email-masked]>
To:     "Luyuan Fang (lufang)" <[email-masked]>, "Shah, Himanshu" 
<[email-masked]>, [email-masked], Kireeti Kompella 
<[email-masked]>, 
Cc:     "[email-masked]" <[email-masked]>
Date:   09/26/2012 02:22 PM
Subject:        Re: [nvo3] Push or pull?
Sent by:        [email-masked]

OK, I see we have different interpretations on NVE to have all the 
endpoint route. My interpretation is that although an EVI provides the 
communication among the VMs in a closed use group, it is not necessary for 
a VM to communicate to all other VMs in the group at a time or all time, 
therefore having each NVE to maintain all the endpoint routes in an EVI is 
not necessary. This is the concern I got from Sunny's mail.

-----Original Message-----
From: Luyuan Fang (lufang) [mailto:[email-masked]] 
Sent: Wednesday, September 26, 2012 4:00 PM
To: Lucy yong; Shah, Himanshu; Thomas Narten; Kireeti Kompella
Cc: [email-masked]
Subject: RE: [nvo3] Push or pull?

Why does the VM need to change RT when it moves? Is not the VM supposed to 
stay in the same VPN and only changing location?
The VM should keep the same RT in order to maintain the membership of that 
VPN it belongs to.

Luyuan

> > > is just noise?              And who says this is in conventional 
memory on

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Wed, 26 Sep 2012 21:51:24 +0000

Ø  The RT could change when the VM moves, in case the VM moves to an NVE which wasn't previously subscribed to the VPN that the VM belongs to.

Not correct. The RT does not change when VM moves, or it breaks VM mobility.
When you move the VM to a new NVE which does not have the VPN the VM belongs to, the VPN/VRF/RT... need to be configured on the new NVE. The RT of the VM does not change.

Don't we need to config  the relevant VN/VPN on the new NVE which does not have it yet, regardless the model?

Luyuan

From: [email-masked] [mailto:[email-masked]] On Behalf Of Sunny Rajagopalan
Sent: Wednesday, September 26, 2012 5:35 PM
To: Lucy yong
Cc: Luyuan Fang (lufang); Kireeti Kompella; [email-masked]
Subject: Re: [nvo3] Push or pull?

There's two different topics being talked about here:
Push, which can be optimized using RTs. RTs can be used to ensure that an NVE only receives routes for the VPNs it participates in. The RT could change when the VM moves, in case the VM moves to an NVE which wasn't previously subscribed to the VPN that the VM belongs to. Due to the state this creates in the route server, you would want to use it at a VPN-level granularity.

Pull, which works by the NVE requesting the route server to "please give me endpoint information about address X in VPN Y". This doesn't result in additional state at the route server, and the route server doesn't push locators for any other end points in that VPN.

--
Sunny

From:        Lucy yong <[email-masked]>
To:        "Luyuan Fang (lufang)" <[email-masked]>, "Shah, Himanshu" <[email-masked]>, [email-masked], Kireeti Kompella <[email-masked]>,
Cc:        "[email-masked]" <[email-masked]>
Date:        09/26/2012 02:22 PM
Subject:        Re: [nvo3] Push or pull?
Sent by:        [email-masked]
________________________________

-----Original Message-----
From: Luyuan Fang (lufang) [mailto:[email-masked]]
Sent: Wednesday, September 26, 2012 4:00 PM
To: Lucy yong; Shah, Himanshu; Thomas Narten; Kireeti Kompella
Cc: [email-masked]
Subject: RE: [nvo3] Push or pull?

Luyuan

---

From: Kireeti Kompella <[email-masked]>
Date: Wed, 26 Sep 2012 15:00:48 -0700

Hi Thomas,

I'd like to focus on the following couple of points.  Note that the goal is not whether push is better than pull; the goal is to find a metric where doing the work to achieve pull is worthwhile.

On Sep 26, 2012, at 11:04 , Thomas Narten <[email-masked]> wrote:

> is just noise?	 

Yes.  But see below.

> And who says this is in conventional memory on a host?

I helped build some of those ASICs: eminently doable, fairly cost effective, simpler than implementing a new encap.  But in any case, I thought the point of overlays was to shield VM scale from ASICs (whole separate conversation).

So, here's my attempt to quantify the cost of push by normalizing the cost per VM.  This is all very naive and preliminary (for now), but the numbers, such as they are, may speak for themselves.

Let's go back to the case of a VN with 10,000 VMs (each with 4GB RAM); as the worst case, assume each VM is on a different hypervisor.

1) assuming 100 bytes per route, the memory cost per VM (as above) is 1MB for FIB space.  For a 4GB VM, that's a "tax" of 0.025%

2) For VM motion, each VM moves needs an update to 10,000 NVEs.  Assume each update is ~1000 bytes; that's a total of 10MB sent.  Moving the VM is 4GB (or more) sent.  The "tax" is 0.25%.

3) Processing an update message is a very small number of cycles.  I don't know enough to compute the "tax" for this, but I'd guess very small.

Clearly, there are other dimensions to this problem; folks might have better numbers or estimates.  But so far, push is looking pretty good.

So, is there a metric whereby push really hurts, and considering pull becomes a necessity?  Or is the suggestion to explore pull models as due diligence, irrespective of the need?  Or …?

Thanks,
Kireeti.

---

From: Kireeti Kompella <[email-masked]>
Date: Wed, 26 Sep 2012 15:06:44 -0700

Hi Sunny,

> Pull, which works by the NVE requesting the route server to "please give me endpoint information about address X in VPN Y". This doesn't result in additional state at the route server, and the rout…

Agreed.

> I think we're reaching consensus that both are needed, right? 

I'm convinced that RT constraint is needed.  Some folks think both are needed.

I'd like to understand whether a pull model solves a significant problem, in which case I'll join the gang.

Kireeti.

---

From: "Shah, Himanshu" <[email-masked]>
Date: Wed, 26 Sep 2012 18:11:42 -0400

Attachment Circuit to that EVPN instance may not exist either, when VM moves to different host.
The need to sprout AC was noted in the NVO3 interim meeting as part of gap analysis on L2VPN based solution.

/himanshu

From: [email-masked] [mailto:[email-masked]] On Behalf Of Luyuan Fang (lufang)
Sent: Wednesday, September 26, 2012 5:51 PM
To: Sunny Rajagopalan; Lucy yong
Cc: Kireeti Kompella; [email-masked]
Subject: Re: [nvo3] Push or pull?

Luyuan

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]<mailto:[mailto:[email-masked]]> On Behalf Of Sunny Rajagopalan
Sent: Wednesday, September 26, 2012 5:35 PM
To: Lucy yong
Cc: Luyuan Fang (lufang); Kireeti Kompella; [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] Push or pull?

--
Sunny

From:        Lucy yong <[email-masked]<mailto:[email-masked]>>
To:        "Luyuan Fang (lufang)" <[email-masked]<mailto:[email-masked]>>, "Shah, Himanshu" <[email-masked]<mailto:[email-masked]>>, [email-masked]<mailto:[email-masked]>, Kireeti Kompella <[email-masked]<mailto:[email-masked]>>,
Cc:        "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Date:        09/26/2012 02:22 PM
Subject:        Re: [nvo3] Push or pull?
Sent by:        [email-masked]<mailto:[email-masked]>
________________________________

-----Original Message-----
From: Luyuan Fang (lufang) [mailto:[email-masked]]
Sent: Wednesday, September 26, 2012 4:00 PM
To: Lucy yong; Shah, Himanshu; Thomas Narten; Kireeti Kompella
Cc: [email-masked]<mailto:[email-masked]>
Subject: RE: [nvo3] Push or pull?

Luyuan

> From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of
> Lucy yong
> Cc: [email-masked]<mailto:[email-masked]>
> > Cc: [email-masked]<mailto:[email-masked]>
> > Cc: [email-masked]<mailto:[email-masked]>
> > > From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On
> Behalf
> > Of
> > > Lucy yong
> > > Cc: [email-masked]<mailto:[email-masked]>
> > > From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On
> Behalf
> > Of
> > > Shah, Himanshu
> > > Cc: [email-masked]<mailto:[email-masked]>
> > > From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On
> Behalf
> > Of
> > > Thomas Narten
> > > Cc: [email-masked]<mailto:[email-masked]>
> > > Kireeti Kompella <[email-masked]<mailto:[email-masked]>> writes:
> > >
> > > [email-masked]<mailto:[email-masked]>
> > > [email-masked]<mailto:[email-masked]>
> > > [email-masked]<mailto:[email-masked]>
> [email-masked]<mailto:[email-masked]>
_______________________________________________
nvo3 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nvo3

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Wed, 26 Sep 2012 22:21:34 +0000

Agreed.
But I'm not sure that is a gap for nvo3, you have to do that in normal VPN case anyway, right? If you consider dealing with VMs is special than dealing with typical CE, may be...
Luyuan

From: Shah, Himanshu [mailto:[email-masked]]
Sent: Wednesday, September 26, 2012 6:12 PM
To: Luyuan Fang (lufang); Sunny Rajagopalan; Lucy yong
Cc: Kireeti Kompella; [email-masked]
Subject: RE: [nvo3] Push or pull?

/himanshu

Luyuan

--
Sunny

-----Original Message-----
From: Luyuan Fang (lufang) [mailto:[email-masked]]
Sent: Wednesday, September 26, 2012 4:00 PM
To: Lucy yong; Shah, Himanshu; Thomas Narten; Kireeti Kompella
Cc: [email-masked]<mailto:[email-masked]>
Subject: RE: [nvo3] Push or pull?

Luyuan

---

From: "Shah, Himanshu" <[email-masked]>
Date: Wed, 26 Sep 2012 18:30:59 -0400

In case of L2VPN, service provisioning is more static-ish.

With VMs moving around like gremlins at speed of light, first arrival and last left
EVPN members has different characteristics than traditional L2VPN service provisioning.

Not saying it can not be done, just another work item (like you pointed out about RT).
/himanshu

From: Luyuan Fang (lufang) [mailto:[email-masked]]
Sent: Wednesday, September 26, 2012 6:22 PM
To: Shah, Himanshu; Sunny Rajagopalan; Lucy yong
Cc: Kireeti Kompella; [email-masked]
Subject: RE: [nvo3] Push or pull?

Agreed.

From: Shah, Himanshu [mailto:[email-masked]]<mailto:[mailto:[email-masked]]>
Sent: Wednesday, September 26, 2012 6:12 PM
To: Luyuan Fang (lufang); Sunny Rajagopalan; Lucy yong
Cc: Kireeti Kompella; [email-masked]<mailto:[email-masked]>
Subject: RE: [nvo3] Push or pull?

/himanshu

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]<mailto:[mailto:[email-masked]]> On Behalf Of Luyuan Fang (lufang)
Sent: Wednesday, September 26, 2012 5:51 PM
To: Sunny Rajagopalan; Lucy yong
Cc: Kireeti Kompella; [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] Push or pull?

Luyuan

--
Sunny

-----Original Message-----
From: Luyuan Fang (lufang) [mailto:[email-masked]]
Sent: Wednesday, September 26, 2012 4:00 PM
To: Lucy yong; Shah, Himanshu; Thomas Narten; Kireeti Kompella
Cc: [email-masked]<mailto:[email-masked]>
Subject: RE: [nvo3] Push or pull?

Luyuan

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Wed, 26 Sep 2012 22:46:29 +0000

Yep, I thought about it ... that you might be talking it's far more dynamic in cloud/dc environment than typical vpn, very true.
And all said apply to l3vpn as well.
Luyuan

Sent: Wednesday, September 26, 2012 6:31 PM

Agreed.

/himanshu

Luyuan

--
Sunny

-----Original Message-----
From: Luyuan Fang (lufang) [mailto:[email-masked]]
Sent: Wednesday, September 26, 2012 4:00 PM
To: Lucy yong; Shah, Himanshu; Thomas Narten; Kireeti Kompella
Cc: [email-masked]<mailto:[email-masked]>
Subject: RE: [nvo3] Push or pull?

Luyuan

---

From: John E Drake <[email-masked]>
Date: Wed, 26 Sep 2012 15:59:08 -0700

Himanshu,

E-VPN already has optimized VM Mobility mechanisms such that all PEs in a given EVI  know behind which PE a given VM currently resides.

Yours irrespectively,

John

Sent: Wednesday, September 26, 2012 3:31 PM

Agreed.

/himanshu

Luyuan

--
Sunny

-----Original Message-----
From: Luyuan Fang (lufang) [mailto:[email-masked]]
Sent: Wednesday, September 26, 2012 4:00 PM
To: Lucy yong; Shah, Himanshu; Thomas Narten; Kireeti Kompella
Cc: [email-masked]<mailto:[email-masked]>
Subject: RE: [nvo3] Push or pull?

Luyuan

---

From: Aldrin Isaac <[email-masked]>
Date: Wed, 26 Sep 2012 19:14:36 -0400

For VM there is no need to send AC (ESI) route since the notion of a
VNIC LAG doesn't make sense.  ESI route to indicate AC is used when
there is a need for multi-homing.  So specifically for a single VM
there is only MAC route.

On Wed, Sep 26, 2012 at 6:11 PM, Shah, Himanshu <[email-masked]> wrote:
> Ø  The RT could change when the VM moves, in case the VM moves to an NVE
> which wasn't previously subscribed to the VPN that the VM belongs to.
>
>
>
> When you move the VM to a new NVE which does not have the VPN the VM belongs
> to, the VPN/VRF/RT… need to be configured on the new NVE. The RT of the VM
> does not change.
>
>
>
> Don’t we need to config  the relevant VN/VPN on the new NVE which does not
> have it yet, regardless the model?
>
>
>
> Luyuan
>
>
>
> Sunny Rajagopalan
> Pull, which works by the NVE requesting the route server to "please give me
> endpoint information about address X in VPN Y". This doesn't result in
> OK, I see we have different interpretations on NVE to have all the endpoint
> route. My interpretation is that although an EVI provides the communication
> among the VMs in a closed use group, it is not necessary for a VM to
> communicate to all other VMs in the group at a time or all time, therefore
> having each NVE to maintain all the endpoint routes in an EVI is not
> necessary. This is the concern I got from Sunny's mail.
>

---

From: Thomas Nadeau <[email-masked]>
Date: Wed, 26 Sep 2012 20:17:30 -0400

E-VPN already works for mobility cases. The speed at which the changes happen is not the issue.

	--Tom

> But I’m not sure that is a gap for nvo3, you have to do that in normal VPN case anyway, right? If you consider dealing with VMs is special than dealing with typical CE, may be…
> Luyuan
>  
> When you move the VM to a new NVE which does not have the VPN the VM belongs to, the VPN/VRF/RT… need to be configured on the new NVE. The RT of the VM does not change.
>  
> Don’t we need to config  the relevant VN/VPN on the new NVE which does not have it yet, regardless the model?
>  
> Luyuan
>

---

From: John E Drake <[email-masked]>
Date: Thu, 27 Sep 2012 06:51:10 -0700

Hi,

Upon reflection, I thought a VM Mobility reference might be useful:  http://tools.ietf.org/html/draft-ietf-l2vpn-evpn-01#section-16

Yours irrespectively,

John

From: John E Drake
Sent: Wednesday, September 26, 2012 3:59 PM
To: 'Shah, Himanshu'
Cc: [email-masked]
Subject: RE: [nvo3] Push or pull?

Himanshu,

Yours irrespectively,

John

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Shah, Himanshu

Agreed.

/himanshu

Luyuan

--
Sunny

-----Original Message-----
From: Luyuan Fang (lufang) [mailto:[email-masked]]
Sent: Wednesday, September 26, 2012 4:00 PM
To: Lucy yong; Shah, Himanshu; Thomas Narten; Kireeti Kompella
Cc: [email-masked]<mailto:[email-masked]>
Subject: RE: [nvo3] Push or pull?

Luyuan

---

From: Lucy yong <[email-masked]>
Date: Thu, 27 Sep 2012 14:27:18 +0000

Let me try more. RT lets SP operator to control route or traffic in a VPN, it is very useful for VPN services because CE are typical network site or LANs. As you said, RT provides constraint capability.

However, in Data Center, a closed used group have many VMs or servers. Although an NVO instance provides a communication for the VMs in the group, some VMs may not need to communicate with some other VMs. Who need to communicate with whom depends on the application running on VMs. DC operator may not have any clue ahead. Thus, how can they use RT to control or constraint the endpoint route?

Stand in host shoes for a moment, why ARP is designed for host protocol long time ago? It is because a host connecting to a network does not need to know all other hosts on the same network. the protocol lets a host query which one a host want to know at any time. This is different between host and a switch. 

When NVE is on a server, there will be just a small set of VMs attaching to it although it is a switch. When these VMs only needs to communicate with some small set of other VMs in the closed user group, why NVE needs to have all the VM locations in the group. Furthermore, depending on the application on VMs, VM communication pattern may dynamically change, how can DC operator use RT accomplish this? 

I have no doubt that SP VPN is very mature technology and have plenty features for various purposes. However, this is not necessary mean that all these apply to DC virtual overlay well. NVO3 application is quite different from SP VPN application. This is why we write vpn-gap-analysis draft to show what are in common and where is the gap. Draft-hy-nvo3-vpn-gap-analysis. 

To make clear, this expression does not mean I suggest we should develop a new solution for NV03. In fact, I think we can extend and/or simplify VPN solution for NVO3.

Hope this helps.

Lucy

---

From: Lucy yong <[email-masked]>
Date: Thu, 27 Sep 2012 14:40:49 +0000

Hi Kireeti,

I guess another dimension is that a VM in NV does not need to communicate with all other VMs in CUG, why NVE need to have all the endpoint locations even hardware is afford. They are difference between host and switch. Will NVE on server behave more like a group of hosts or a switch?

Lucy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Kireeti Kompella
Sent: Wednesday, September 26, 2012 5:01 PM
To: Thomas Narten
Cc: Kireeti Kompella; [email-masked]
Subject: Re: [nvo3] Push or pull?

Hi Thomas,

Yes.  But see below.

So, is there a metric whereby push really hurts, and considering pull becomes a necessity?  Or is the suggestion to explore pull models as due diligence, irrespective of the need?  Or ...?

Thanks,
Kireeti.

---

From: Jon Hudson <[email-masked]>
Date: Thu, 27 Sep 2012 10:39:06 -0700

A few comments inline...

On Sep 27, 2012, at 7:27 AM, Lucy yong <[email-masked]> wrote:

> However, in Data Center, a closed used group have many VMs or servers. Although an NVO instance provides a communication for the VMs in the group, some VMs may not need to communicate with some other VMs.

True, and desirable. If VM_A_01 only needs to talk to VM_A_02-07, then I don't want it to be able to touch VM_A_08 or VM_B_x.

> Who need to communicate with whom depends on the application running on VMs. DC operator may not have any clue ahead.

Not likely. All the policies should have be setup when the application was first setup and all traffic is deliberate and known down to src_ip_port<->dst_ip_port. 

In the case of a customer in a large public cloud those policies may be more open ended as given from customer -> public_cloud_operator: there will still be well define enough policies to know what communication flows need to be open if you move VM_A_01 from DC01:POD02:RM05:RW02:RK08:pSERV23 ---> DC01:POD02:RM02:RW07:RK11:pSERV42

> Thus, how can they use RT to control or constraint the endpoint route?
> 

You _may_ see a policy as open ended as Cust_BA has VLANs 11-42, and any<->any is permitted between those VLANs and so as new VMs are     
instantiated a new learning sequence is needed. However firewall, access, backup etc policies will likely need to be setup for the VM with a fix ipaddr and so VMs are not typically just appearing willy-nilly like Tribbles.

> 
> When NVE is on a server, there will be just a small set of VMs attaching to it although it is a switch.

We can't assume this. I can easily buy systems today with 32 cores and up to 1TB of memory. For many folks that's 16 VMs minimum and 128 possible.

> When these VMs only needs to communicate with some small set of other VMs in the closed user group, why NVE needs to have all the VM locations in the group. Furthermore, depending on the application on VMs, VM communication pattern may dynamically change, how can DC operator use RT accomplish this? 

Perhaps with a new unknown application. But if VM_A_01 has for weeks been talking to VM_A_[04,7,11,15] for weeks or months and suddenly starts talking to VM_A_22, your VM may be possessed. 

In a Datacenter, surprise is a bad thing. Boring is your goal.

---

From: Kireeti Kompella <[email-masked]>
Date: Thu, 27 Sep 2012 10:51:21 -0700

Hi Jon,

> True, and desirable. If VM_A_01 only needs to talk to VM_A_02-07, then I don't want it to be able to touch VM_A_08 or VM_B_x.

Two very different things:

Lucy said: "some VMs may not need to communicate" … but that could change, and is not dictated by policy, but rather by current need.

IMO, what you are asking calls for VM_A_01 and VM_A_02-07 to be in (say) VN_A, and VM_A_08 and VM_B_x in different VN(s) (not sure what the numbering means, but I'm just going with the flow.)

If one were to put VM_A_01 and VM_A_08 in the same VN, one would have to hand-craft the intra-VN policies.  Doable, but not scalable (administratively).  For example, what happens if a new VM, VM_A_22 shows up (elasticity) … who can it talk to?

KIreeti.

---

From: Jon Hudson <[email-masked]>
Date: Thu, 27 Sep 2012 13:01:24 -0700

Greetings Kireeti =)

My notation was to indicated separate vlans A and B. I should have been
more clear. I was describing what I see today not what I imagine things
will be like with something like NVo3 in place. Sorry for the lack of
clarity.

But to your question I would want any system that can elastically 1.) Clone
a VM from template or source 2.) Give it a new unique ipaddr and name 3.)
assign it relevant gateway/network/netmask/dns info would also 4.)
change/add whatever firewall, logging, backup, access_control, and intra_VN
policy needed.

Otherwise it's not very elastic in my opinion.

Now I suppose you could have a system that takes a pool of pre-instantiated
VMs that already have all relevant configuration information setup, and the
elasticity is to suspend and resume VMs based on need. But even in that
case, the needed policies could be set up the same way and taken from say a
"quarantine" level and promoted to "full production" as part of the process
of unsuspending the VM.

J

On Thu, Sep 27, 2012 at 10:51 AM, Kireeti Kompella <
[email-masked]> wrote:

> >> However, in Data Center, a closed used group have many VMs or servers.
> Although an NVO instance provides a communication for the VMs in the group,
> some VMs may not need to communicate with some other VMs.
> >
> > True, and desirable. If VM_A_01 only needs to talk to VM_A_02-07, then I
> don't want it to be able to touch VM_A_08 or VM_B_x.
>
> Lucy said: "some VMs may not need to communicate" … but that could change,
> and is not dictated by policy, but rather by current need.
>
> IMO, what you are asking calls for VM_A_01 and VM_A_02-07 to be in (say)
> VN_A, and VM_A_08 and VM_B_x in different VN(s) (not sure what the
> numbering means, but I'm just going with the flow.)
>
> If one were to put VM_A_01 and VM_A_08 in the same VN, one would have to
> hand-craft the intra-VN policies.  Doable, but not scalable
> (administratively).  For example, what happens if a new VM, VM_A_22 shows
> up (elasticity) … who can it talk to?
>
> KIreeti.
>
>

-- 
"Do not lie. And do not do what you hate."

---

From: "Fedyk, Donald (Don)" <[email-masked]>
Date: Thu, 27 Sep 2012 20:33:28 +0000

Hi Lucy

Can you point to more information your CUG model somewhere as a reference?  
Historically, I'm familiar with closed user groups and those CUGs on L2 networks (usually for providing Router connectivity). 
L2 CUGs gave way to L3 VPNs (and L2 VPNs) because those L2 CUGs were implemented as access connection parameters. 
One negative as I recall was the CUGs/L2 addresses had to be assigned and configured and that was a both a burden initially and in migration scenarios. 
Also in clients with L2 CUGs address scope was unrestricted and they could do denial of service type of attacks (accidentally or on purpose) because they could see all L2 addresses but only connect to ones with the same CUGs.  CUG identifiers were only applied at the end.  
So for me CUG has a negative connotation. But perhaps you means something else by CUG?

In the L2/L3 VPN using RT no member of one VPN can see addresses or connect to another VPN. But the RT is dynamic in that once you have a site that announces the RT it is included in the VPN. The clients (in this case VMs) do not have access to the RT identifier directly.  RT association can be by port and configured VLAN for example. In this case VMs could show up dynamically in a such a VLAN with new MAC and IP addresses and RT association need not change.

Thanks,
Don 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Lucy yong
Sent: Thursday, September 27, 2012 10:27 AM
To: Luyuan Fang (lufang); Shah, Himanshu; Thomas Narten; Kireeti Kompella
Cc: [email-masked]
Subject: Re: [nvo3] Push or pull?

Hope this helps.

Lucy

---

From: Thomas Narten <[email-masked]>
Date: Thu, 27 Sep 2012 16:51:26 -0400

> Okay.  How about: Push is simple and known to work; but it costs.
>  Pull is harder, but may be cheaper.  Still a very broad statement,
>  but it allows the following question:

I think it's not particularly useful to have a high-level discussion
about whether something as general as "push" vs. "pull" will work or
not. Push vs. pull are just two approaches. They both have plusses and
minuses. To decide which makes sense (and in what context), you have
to look at the overall problem, requirements, and a whole bunch of
factors. And different folk may weigh various factors differently.

I'm not saying push can't work. In fact, I'm sure it can. But just
because it can work, doesn't mean other approaches should be ruled
out.

> Is the savings (in whatever dimension) that Pull offers over Push

I think so. At least, I don't want to see it ruled out up front.

> It was a show-stopper for Ipsilon.

Now there is a soundbite. But I'm not sure exactly how that applies
here...

> I thought I picked a fairly large VN, with 10,000 members.  Even if
> you take a larger one, say 100,000 members, the memory cost is (imo)
> small.

It's not just memory, there is also the signalling cost.  And if you
have 10K VMs, no doubt at any one time some (more?) are moving around,
with the corresponding signaling (using push), going out all the way
to *all* NVEs.

> <snipped, see next email>

OK.

> Perhaps so (I would love to get more details.)  But what did you
  gain?

Um, some large data centers already have centralized components. Like
the entire VM orchestration system.  Adding more to that
centralization doesn't necessarily change the single point of
failure/risk factor in a substantial way. E.g., in the ARMD
discussions, one large operator said they'd like to centralize more of
that info, and reduce the signaling (which was effectively
redundent). There was little benefit in dynamic use of ARP, for
instance, when all addresses and their locations were already known
centrally.

Thomas

---

From: Thomas Narten <[email-masked]>
Date: Thu, 27 Sep 2012 16:54:41 -0400

Hi Kireeti.

Kireeti Kompella <[email-masked]> writes:

> So, here's my attempt to quantify the cost of push by normalizing
>  the cost per VM.  This is all very naive and preliminary (for now),
>  but the numbers, such as they are, may speak for themselves.

> Let's go back to the case of a VN with 10,000 VMs (each with 4GB
>  RAM); as the worst case, assume each VM is on a different
>  hypervisor.

> 1) assuming 100 bytes per route, the memory cost per VM (as above)
>  is 1MB for FIB space.  For a 4GB VM, that's a "tax" of 0.025%

> 2) For VM motion, each VM moves needs an update to 10,000 NVEs.
>  Assume each update is ~1000 bytes; that's a total of 10MB sent.
>  Moving the VM is 4GB (or more) sent.  The "tax" is 0.25%.

10MB worth of updates for a single VM move doesn't seem insignificant
to me. A few MB here, a few MB there, pretty soon we're talking about
real bandwith. :)

The .25% tax figures is apples to oranges. The VM that is moving pays
the cost of doing the 4MB move. But that tax you are talking about is
paid by the NVEs, not the VM doing the move. If we assume 10 VMs per
NV, thats 1000 NVEs paying that tax.

> 3) Processing an update message is a very small number of cycles.  I
>  don't know enough to compute the "tax" for this, but I'd guess very
>  small.

A small tax here, a small tax there, ...

> Clearly, there are other dimensions to this problem; folks might
>  have better numbers or estimates.  But so far, push is looking
>  pretty good.

> So, is there a metric whereby push really hurts, and considering
>  pull becomes a necessity?  Or is the suggestion to explore pull
>  models as due diligence, irrespective of the need?  Or …?

I doubt that it's worthwile trying to come up with a metric in
advance. Most likely, we'll just start arguing about the assumptions
being fed into the mode. I don't see the point in restricting the
architecture to a "push" model at this early juncture. I also don't
think its necessary.

Thomas

---

From: Thomas Narten <[email-masked]>
Date: Thu, 27 Sep 2012 17:31:01 -0400

One of the things I've been meaning to clarify here is what is the
defintion of a CUG?

In offlist discussions I've had, I've come to the conclusion that a
CUG is the same thing as a VN. That is, it's a set of machine that are
administratively placed into a group and are allowed to communicate
with each other, but not with others outside of that CUG.

Correct?

And,  RFC 4364 says:

>    Suppose it is desired to create a fully meshed closed user group,
>    i.e., a set of sites where each can send traffic directly to the
>    other, but traffic cannot be sent to or received from other sites.
>    Then each site is associated with a VRF, a single Route Target
>    attribute is chosen, that Route Target is assigned to each VRF as
>    both the Import Target and the Export Target, and that Route Target
>    is not assigned to any other VRFs as either the Import Target or the
>    Export Target.

Is there another (different?) definition of CUG?

Thomas

---

From: Yakov Rekhter <[email-masked]>
Date: Thu, 27 Sep 2012 14:55:06 -0700

Thomas,

> with the corresponding signaling (using push), going out all the way

Perhaps you should learn how RT Constrain could be used to
avoid sending the state for 10K VM "to *all* NVEs"

Yakov.

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Thu, 27 Sep 2012 22:04:04 +0000

Lucy,

I have trouble to see where you are driving, no need for VN? 

  > Who need to communicate with whom depends on the application running on VMs. DC operator may not have any clue ahead. 

Then the operator cannot provision the VM in the correct VN. Even that VM never talk to anyone, it needs to belong to a particular VN. I thought we are talking VN in nvo3, talking multi-tenancy isolation. If you don't need VN, that is something else. 

Being in a member of VN, you are allowed to talk/listen to another member of VM, but do so only when you need to and the policy permits. 
In VPN case, you can also use RT import/export for policy control... etc.  

Extend/Modify VPN to fit into DC environment is a good idea, remove RT is a bad idea.

Luyuan

> Fedyk, Donald (Don)
> Sent: Thursday, September 27, 2012 4:33 PM
> To: Lucy yong; [email-masked]
> Subject: [nvo3] What is CUG model was RE: Push or pull?
> 
> Hi Lucy
> 
> Historically, I'm familiar with closed user groups and those CUGs on L2
> networks (usually for providing Router connectivity).
> L2 CUGs gave way to L3 VPNs (and L2 VPNs) because those L2 CUGs were
> implemented as access connection parameters.
> One negative as I recall was the CUGs/L2 addresses had to be assigned
> and configured and that was a both a burden initially and in migration
> scenarios.
> Also in clients with L2 CUGs address scope was unrestricted and they
> could do denial of service type of attacks (accidentally or on purpose)
> because they could see all L2 addresses but only connect to ones with
> the same CUGs.  CUG identifiers were only applied at the end.
> In the L2/L3 VPN using RT no member of one VPN can see addresses or
> connect to another VPN. But the RT is dynamic in that once you have a
> site that announces the RT it is included in the VPN. The clients (in
> this case VMs) do not have access to the RT identifier directly.  RT
> association can be by port and configured VLAN for example. In this
> case VMs could show up dynamically in a such a VLAN with new MAC and IP
> addresses and RT association need not change.
> 
> 
> Thanks,
> Don
> 
> 
> Let me try more. RT lets SP operator to control route or traffic in a
> VPN, it is very useful for VPN services because CE are typical network
> site or LANs. As you said, RT provides constraint capability.
> 
> Although an NVO instance provides a communication for the VMs in the
> group, some VMs may not need to communicate with some other VMs. Who
> need to communicate with whom depends on the application running on
> VMs. DC operator may not have any clue ahead. Thus, how can they use RT
> to control or constraint the endpoint route?
> 
> Stand in host shoes for a moment, why ARP is designed for host protocol
> long time ago? It is because a host connecting to a network does not
> need to know all other hosts on the same network. the protocol lets a
> host query which one a host want to know at any time. This is different
> between host and a switch.
> 
> When NVE is on a server, there will be just a small set of VMs
> attaching to it although it is a switch. When these VMs only needs to
> communicate with some small set of other VMs in the closed user group,
> why NVE needs to have all the VM locations in the group. Furthermore,
> depending on the application on VMs, VM communication pattern may
> dynamically change, how can DC operator use RT accomplish this?
> 
> I have no doubt that SP VPN is very mature technology and have plenty
> features for various purposes. However, this is not necessary mean that
> all these apply to DC virtual overlay well. NVO3 application is quite
> different from SP VPN application. This is why we write vpn-gap-
> analysis draft to show what are in common and where is the gap. Draft-
> hy-nvo3-vpn-gap-analysis.
> 
> To make clear, this expression does not mean I suggest we should
> develop a new solution for NV03. In fact, I think we can extend and/or
> simplify VPN solution for NVO3.
> 
> Hope this helps.
> 
> Lucy
> 
> > > [[LY]] to support VM mobility. What is an interested endpoint
> moving
> > > from one NVE to another?
> > > lucy
> > >
> > > > for selective route distribution - only send the VPN routes to
> the
> > > peer
> > > > > application space. To add on two points, in push model, BGP
> peer
> > > > group
> > > > > I do not think we should say "push is good, pull is bad". That
> is
> > > > just
> > > > > tables with no problem. But what about for very large VNs?
> Should
> > > the
> > > > > > want to talk to, but only one that it really wants to talk
> to.
> > > > Let's
> > > > > > love to know, especially with numbers backing it up. But
> let's
> > > > first
> > > > > I do not think we should today require that the NVO3
> architecture
> > > (in
> > > > a
> > > > > MUST sense) support only push. I think we should allow for
> either
> > > > push

---

From: Yakov Rekhter <[email-masked]>
Date: Thu, 27 Sep 2012 15:03:47 -0700

Thomas,

> Is there another (different?) definition of CUG?

There is a definition of L2-based CUG in draft-rekhter-nvo3-vm-mobility-issues

Yakov.

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Thu, 27 Sep 2012 22:26:41 +0000

Thomas,

> with each other, but not with others outside of that CUG.

I would rephrase it as: A virtual network (VN) behaves as a "Closed User Group" (CUG) of machines that are allowed to exchange
traffic freely, while traffic between virtual networks is subject to access controls.

Maria

---

From: Lucy yong <[email-masked]>
Date: Thu, 27 Sep 2012 22:28:53 +0000

Gee, I thought we had used CUG term a lot in nv03 mailing discussions and don't know when it disappeared.
Thank you, Tom, for the rescue. That is what in my mind about CUG meaning.

Lucy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Narten
Sent: Thursday, September 27, 2012 4:31 PM
To: Fedyk, Donald (Don)
Cc: [email-masked]; Lucy yong
Subject: Re: [nvo3] What is CUG model was RE: Push or pull?

Correct?

Thomas

---

From: Lucy yong <[email-masked]>
Date: Thu, 27 Sep 2012 22:35:03 +0000

Luyuan,

I am ready to drive to my home soon. My statement does not say no need VN, don't know where you get that impression.

All what want to point out is that a tenant virtual network may have a lot of TESes attached, it is not necessary that all TESes need to communicated each other at a time although VN provides that capability. DC operator may not know what application program that TESes run and which other TESes the program need to communicate ahead. For example, if DC provider provides a VN with a 1000 VMs to an enterprise customer, the enterprise decides what application run on each VM, not DC provider. In this case, DC provider may not know which VMs will communicate with which other VMs ahead, so they may not able to use RT to prevent what each NVE need to know correctly. Of course, DC operator has the control where VM can be placed and moved.

I do not have any intention to remove RT in VPN. RT is very useful for DC operator to control VPN traffic. 

Lucy

-----Original Message-----
From: Luyuan Fang (lufang) [mailto:[email-masked]] 
Sent: Thursday, September 27, 2012 5:04 PM
To: Fedyk, Donald (Don); Lucy yong; [email-masked]
Subject: RE: [nvo3] What is CUG model was RE: Push or pull?

Lucy,

Luyuan

---

From: Lucy yong <[email-masked]>
Date: Thu, 27 Sep 2012 22:43:31 +0000

Don,

snip..

In the L2/L3 VPN using RT no member of one VPN can see addresses or connect to another VPN. But the RT is dynamic in that once you have a site that announces the RT it is included in the VPN. The clients (in this case VMs) do not have access to the RT identifier directly.  RT association can be by port and configured VLAN for example. In this case VMs could show up dynamically in a such a VLAN with new MAC and IP addresses and RT association need not change.
[[LY]] I am still not clear if RT in "push" model can replace or equivalent to query function in pull in a VN. Please advice more. Note that I am not saying that "pull" is better than "push" here. Just want know. 

Lucy

Thanks,
Don 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Lucy yong
Sent: Thursday, September 27, 2012 10:27 AM
To: Luyuan Fang (lufang); Shah, Himanshu; Thomas Narten; Kireeti Kompella
Cc: [email-masked]
Subject: Re: [nvo3] Push or pull?

Hope this helps.

Lucy

---

From: Kireeti Kompella <[email-masked]>
Date: Thu, 27 Sep 2012 16:25:45 -0700

Hi Thomas,

> with each other, but not with others outside of that CUG.

Almost. Entities inside a CUG may communicate to entities outside through a policy specified by the appropriate administrator. the default policy is as you say: no communication. 

> And,  RFC 4364 says:

Essentially, RFC 4364 defined an L3 CUG, and the EVPN draft an L2 CUG. 

Kireeti

---

From: Kireeti Kompella <[email-masked]>
Date: Thu, 27 Sep 2012 16:38:40 -0700

Hi Thomas,

On Sep 27, 2012, at 13:54, Thomas Narten <[email-masked]> wrote:

> architecture to a "push" model at this early juncture.

Fair enough. 

My guess is that, once you have both (push and pull technologies), it would be nice to have a clear understanding of when which would be better. I.e., build a model ...

Kireeti

---

From: Kireeti Kompella <[email-masked]>
Date: Thu, 27 Sep 2012 16:54:16 -0700

Hi Jon,

On Sep 27, 2012, at 13:01, Jon Hudson <[email-masked]> wrote:

> Greetings Kireeti =)
> 
> My notation was to indicated separate vlans A and B. I should have been more clear. I was describing what I see today not what I imagine things will be like with something like NVo3 in place. Sorry for the lack of clarity. 

Kind of what I guessed. When you say "I don't want VM_A_01 to touch VM_A_08", wouldn't you do that by putting them in different vlans, rather than both in vlan A?

> But to your question I would want any system that can elastically 1.) Clone a VM from template or source 2.) Give it a new unique ipaddr and name 3.) assign it relevant gateway/network/netmask/dns info would also 4.) change/add whatever firewall, logging, backup, access_control, and intra_VN policy needed. 

I can see inter-VN policy. Can you help me see how an intra-VN policy can be made elastic?

> Otherwise it's not very elastic in my opinion. 

Totally agree!

> Now I suppose you could have a system that takes a pool of pre-instantiated VMs that already have all relevant configuration information setup, and the elasticity is to suspend and resume VMs based on need. But even in that case, the needed policies could be set up the same way and taken from say a "quarantine" level and promoted to "full production" as part of the process of unsuspending the VM. 

I suppose. Sounds like "poor man's elasticity" to me. :-)

Kireeti

> J
> 
> On Thu, Sep 27, 2012 at 10:51 AM, Kireeti Kompella <[email-masked]> wrote:
> Hi Jon,
>

---

From: Aldrin Isaac <[email-masked]>
Date: Thu, 27 Sep 2012 20:14:25 -0400

Outbound Route Filtering (of which RT Constrain is a type) is really both push and pull depending on how you look at it.

The purpose of ORF is to deliver to a PE only the set of routes associated to community that locally connected CE are members of.  When a new CE is configured on a PE that is a member of a community which existing CE were not members of, than the PE requests (pulls) the routes for that new community from it's RR/peers.  In the case of NVE and hypervisors, when a VM belonging to a community moves to an NVE which does not have other members of that community then the NVE will request (pull) from it's RR the routes for that community.  

So, in regards to ORF, the question is not really whether ORF is pull or push but about what the trigger for a pull request should be (VM move, conversational level, etc) and what the pull should be for (community, destination, community+destination).  I think the good folk at the IETF can certainly define new types such that ORF based pulls meet real-world requirements.  But let's first get proof that the requirements are real-world.

Best regards -- aldrin

---

From: Jon Hudson <[email-masked]>
Date: Thu, 27 Sep 2012 17:53:19 -0700

Hey...

> Kind of what I guessed. When you say "I don't want VM_A_01 to touch VM_A_08", wouldn't you do that by putting them in different vlans, rather than both in vlan A?

Yes ideally! However since today you can't move a live VM from one VLAN to another, more and more interpret "flat network" to mean one VLAN. And will then use per interface ACLs or other more silly things like playing with netmasks to create larger spheres of mobility. 

If you want every VM to have the option of moving to every possible hypervisor then you are either doing one VLAN, VLAN tagging, or putting hypervisors on multiple networks. 

To be very honest, it's all this madness that I am describing that makes NVo3 coming out right so very important.

> I can see inter-VN policy. Can you help me see how an intra-VN policy can be made elastic?

Well perhaps my image of intra-VN policies is off. But essentially what I was thinking was to have per VM profiles that trigger changes to Intra-VN (and inter-VN) policies based on the the addition, removal, or modification of VN membership or requirements. 

For example having quarantined VMs that you may want to have in the VN, pulling live data, but not responding to requests or being added to resource queues or even viewable until they are blessed and promoted to production. 

But perhaps I am thinking of intra-VN policies incorrectly?

> I suppose. Sounds like "poor man's elasticity" to me. :-)

Except some Datacenters don't allow dhcp and want pre-instantiated pre-configured VMs with static ipaddrs pre- allocated.

It also doesn't help that the ability to clone based on load also allows for a very interesting denial of service attack =)

But yes, I personally totally agree, dynamic instantiation and deletion based on metrics is the only fun way to do this.

> 
> Kireeti
> 
>> J
>> 
>> On , Sep 27, 2012 at 10:51 AM, Kireeti Kompella <[email-masked]> wrote:
>>> Hi Jon,
>>>

---

From: Kireeti Kompella <[email-masked]>
Date: Thu, 27 Sep 2012 18:10:01 -0700

Hi Jon,

On Sep 27, 2012, at 17:53, Jon Hudson <[email-masked]> wrote:

<snipped>

> Yes ideally!

Cool.

> However since today you can't move a live VM from one VLAN to another, more and more interpret "flat network" to mean one VLAN.

Got it. 

> And will then use per interface ACLs or other more silly things like playing with netmasks to create larger spheres of mobility. 

<shudder>

> To be very honest, it's all this madness that I am describing that makes NVo3 coming out right so very important.

With you. 

<snipped>

> For example having quarantined VMs that you may want to have in the VN, pulling live data, but not responding to requests or being added to resource queues or even viewable until they are blessed a…

*Very* interesting example!  Gotta mull over this. 

> But perhaps I am thinking of intra-VN policies incorrectly?

I think rather, you're struggling with two orthogonal notions: a VN as a mobility domain and a VN as a CUG (or ACL domain: talk freely inside, talk via firewall outside).  That's something we absolutely need to fix in NVO3, to your point above.

Kireeti

---

From: "Fedyk, Donald (Don)" <[email-masked]>
Date: Fri, 28 Sep 2012 01:14:21 +0000

So what I heard/read is this:
draft-rekhter-nvo3-vm-mobility-issues-02.txt:
An L2-Closed User Group has some defining attribute (VLAN-ID for Ethernet) that allows free communication between members.
- IP address assignment is based on L2-CUG. 

- A Virtual Machine(VM) may belong to one or more L2-based CUGs.(One VID per L2-CUG) (One IP Prefixes per CUG)

- And Policy can allow a VM in one L2-Based CUG to communicate with other VMs in other CUGs through IP.

- So the mechanism enabling a L2-CUG is a VLAN. 

And I believe a Route Target in a EVPN is a mechanism that can extend the VLAN. (extending the L2-CUG over IP WAN). 
It would follow that a Route Target in an L3 VPN can extend and interconnect an IP subnet(s) (over the IP WAN) and this is (I think) independent of the L2-CUG although it could also be aligned with the L2-CUG.

So getting back to Push or PULL 

An L2-CUG uses a VLAN as a Push Mechanism for Learning (Frames are pushed). 
An IP-subnet on a L2-CUG can use ARP (Pull) (ARP responds with IP address). 
The IP-Subnet could also use routing and go between L2-CUGs. 
Between IP-Subnets we use routing (IGP or BGP + Policy between subnets)
I'd say EVPN RT is primarily push (RT + VLAN) = L2-CUG. 
L3 VPN is Push RT + Routing. 

This aligns with those that say a VM can talk to other VMs in other Virtual Networks only at IP. 
Push and Pull is not so clear cut.  

I can sort this scheme out in my head. But I don't know other alternative schemes. 
Perhaps delete EVPN and L3VPN and use something else? 

At is level it makes some sense to me.
 
Thanks,
Don 

-----Original Message-----
From: Kireeti Kompella [mailto:[email-masked]] 
Sent: Thursday, September 27, 2012 7:26 PM
To: Thomas Narten
Cc: Fedyk, Donald (Don); [email-masked]; Lucy yong
Subject: Re: [nvo3] What is CUG model was RE: Push or pull?

Hi Thomas,

Kireeti

---

From: Xuxiaohu <[email-masked]>
Date: Fri, 28 Sep 2012 02:24:32 +0000

Besides push and pull, please don’t forget the data-plane MAC learning mechanism is also a proven choice for Layer2 overlay approaches.

Best regards,
Xiaohu

> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代表 Lucy
> yong
> 发送时间: 2012年9月27日 22:41
> 收件人: Kireeti Kompella; Thomas Narten
> 抄送: [email-masked]
> 主题: Re: [nvo3] Push or pull?
> 
> Hi Kireeti,
> 
> I guess another dimension is that a VM in NV does not need to communicate
> with all other VMs in CUG, why NVE need to have all the endpoint locations
> even hardware is afford. They are difference between host and switch. Will
> NVE on server behave more like a group of hosts or a switch?
> 
> Lucy
> 
> Kireeti Kompella
> I'd like to focus on the following couple of points.  Note that the goal is not
> whether push is better than pull; the goal is to find a metric where doing the
> work to achieve pull is worthwhile.
> 
> I helped build some of those ASICs: eminently doable, fairly cost effective,
> simpler than implementing a new encap.  But in any case, I thought the point
> of overlays was to shield VM scale from ASICs (whole separate conversation).
> 
> So, here's my attempt to quantify the cost of push by normalizing the cost per
> VM.  This is all very naive and preliminary (for now), but the numbers, such as
> they are, may speak for themselves.
> 
> Let's go back to the case of a VN with 10,000 VMs (each with 4GB RAM); as the
> worst case, assume each VM is on a different hypervisor.
> 
> 1) assuming 100 bytes per route, the memory cost per VM (as above) is 1MB
> for FIB space.  For a 4GB VM, that's a "tax" of 0.025%
> 
> 2) For VM motion, each VM moves needs an update to 10,000 NVEs.  Assume
> each update is ~1000 bytes; that's a total of 10MB sent.  Moving the VM is
> 4GB (or more) sent.  The "tax" is 0.25%.
> 
> 3) Processing an update message is a very small number of cycles.  I don't
> know enough to compute the "tax" for this, but I'd guess very small.
> 
> Clearly, there are other dimensions to this problem; folks might have better
> numbers or estimates.  But so far, push is looking pretty good.
> 
> So, is there a metric whereby push really hurts, and considering pull becomes a
> necessity?  Or is the suggestion to explore pull models as due diligence,
> irrespective of the need?  Or ...?
> 
> Thanks,
> Kireeti.
>

---

From: Xuxiaohu <[email-masked]>
Date: Fri, 28 Sep 2012 03:54:38 +0000

> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代表
> Xuxiaohu
> 发送时间: 2012年9月28日 10:25
> 收件人: Lucy yong; Kireeti Kompella; Thomas Narten
> 抄送: [email-masked]
> Besides push and pull, please don’t forget the data-plane MAC learning
> mechanism is also a proven choice for Layer2 overlay approaches.

Most importantly, the data-plane MAC learning mechanism can work well in both the push mode and the pull mode. For instance, in a push mode, if local MAC addresses are learnt by NVEs with the data-plane MAC learning mechanism, once some local MAC addresses are aged out, their corresponding MAC routes would be withdrawn accordingly. In this way, the NVEs in the push mode also don't need to store all MAC addresses within each attached VN instance. Meanwhile, in the pull mode, if the ingress NVE receives an Ethernet frame and finds no matching forwarding entry for that frame, the frame could be flooded by the ingress NVE as a normal bridge does before received a mapping response from the mapping system.

Best regards,
Xiaohu

> Assume

---

From: Thomas Narten <[email-masked]>
Date: Fri, 28 Sep 2012 09:09:26 -0400

This discussion is interesting, but I still have not seen an answer to
the question: what is (or is there) a difference between a VN and an
CUG?

If there is no difference, that would be great, because then we can
use the terms interchangably. If there are differences, we need to
understand what those differences are, or there will be confusion in
our discussions.

That definition appears to be:

>    Consider a set of VMs that (as a matter of policy) are allowed to
>    communicate with each other, and a collection of devices that
>    interconnect these VMs. If communication among any VMs in that set
>    could be accomplished in such a way as to preserve MAC source and
>    destination addresses in the Ethernet header of the packets exchanged
>    among these VMs (as these packets traverse from their sources to
>    their destinations), we will refer to such set of VMs as an Layer 2
>    based Closed User Group (L2-based CUG).

This defines an L2 CUG, but not a CUG generally. That said, my read of
he above is that a CUG is nothing more than "a set of VMs that (as a
matter of poliocy) are allowed to communicate with each other". That
sure sounds like what I thought a VN is.

Interesting, the draft-ietf-nvo3-framework-00.txt.p7s doesn't seem to
even define VN (oops).

draft-ietf-nvo3-overlay-problem-statement-00.txt says:

>   To end stations, a virtual network looks like a normal network
>   (e.g., providing an ethernet or L3 service), except that the only
>   end stations connected to the virtual network are those belonging
>   to a tenant's specific virtual network.

My assumption has always been that VN is the set of machines that (by
policy) are allowed to talk to each other. Communication with machins
outside of a VN takes place outside of NVO3, i.e., by (at least
conceptually) going outside of the VN, having policy applied (as
appropriate) and then (if policy allows) forwarding to another
VN. I.e., analogous to forwarding between VLANs where one has to go
through L3.

> Almost. Entities inside a CUG may communicate to entities outside
> through a policy specified by the appropriate administrator. the
> default policy is as you say: no communication.

I think we are in complete agreement here. Although I say that to go
from one VN to another one must (conceptually) go out of the VN, one
can certainly optimize that case and (if policy allows) have direct
tunneling from a VM in one VN to a VM in another.

But the fundamental notion of a VN (and from what I can tell, a CUG)
is that it is a set of machines that are part of a single
administrative group/network and that are allowed to talk to each
other, and that normally do not talk to machines in other VNs.

> Essentially, RFC 4364 defined an L3 CUG,

> and the EVPN draft an L2 CUG.

What draft would that be? Neither draft-ietf-l2vpn-evpn-req-00.txt nor
draft-ietf-l2vpn-evpn-01.txt use the term "closed" or "CUG".

Thomas

---

From: Kireeti Kompella <[email-masked]>
Date: Fri, 28 Sep 2012 06:35:56 -0700

Hi Thomas,

> the question: what is (or is there) a difference between a VN and an

No difference. 

> understand what those differences are, or there will be confusion in

For clarity, we should define VN in some draft (framework?), state that VN and CUG are the same, but stick to using one term, preferably VN. 

Note that Wikipedia defines CUG as follows:

Closed User Groups are groups of GSM mobile telephone subscribers who can only make calls and receive calls from members within the group. Any other calls would be rejected.

The definition is pretty close, but it goes to show that CUG comes with overtones that may confuse people. 

One refinement might be to define L2 and L3 VNs off the base definition of VN. 

>> draft-rekhter-nvo3-vm-mobility-issues

Thomas, the reference I gave below about where L2-CUG is defined is wrong -- not the EVPN draft, but the one above. 

<rest snipped>

Kireeti

---

From: "Toy, Mehmet" <[email-masked]>
Date: Fri, 28 Sep 2012 14:17:45 +0000

CUG term has been used since the days of X.25.
To me CUG only represents  entities/users that are on the network (i.e. VN), not the network or a set of connections making the virtual network.
On the other hand, VN represents the network or a set of connections making the virtual network, but not the users on the network.

Mehmet

From: [email-masked] [mailto:[email-masked]] On Behalf Of Kireeti Kompella
Sent: Friday, September 28, 2012 9:36 AM
To: Thomas Narten
Cc: [email-masked]
Subject: Re: [nvo3] What is CUG model was RE: Push or pull?

Hi Thomas,

On Sep 28, 2012, at 6:09, Thomas Narten <[email-masked]<mailto:[email-masked]>> wrote:
This discussion is interesting, but I still have not seen an answer to
the question: what is (or is there) a difference between a VN and an
CUG?

No difference.

Closed User Groups are groups of GSM<http://en.wikipedia.org/wiki/GSM> mobile telephone<http://en.wikipedia.org/wiki/Mobile_telephone> subscribers who can only make calls and receive calls from members within the group. Any other calls would be rejected.

Yakov Rekhter <[email-masked]<mailto:[email-masked]>> writes:

There is a definition of L2-based CUG in
draft-rekhter-nvo3-vm-mobility-issues

<rest snipped>

Kireeti

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Fri, 28 Sep 2012 14:29:44 +0000

What's wrong with (which I sent before):

A virtual network (VN) behaves as a "Closed User Group" (CUG) of machines that are allowed to exchange
traffic freely, while traffic between virtual networks is subject to access controls.

Maria

> Sent: Friday, September 28, 2012 9:09 AM
> To: [email-masked]

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Fri, 28 Sep 2012 15:39:15 +0000

I think Mehmet has a subtle, but valid point about the term CUG.  A CUG is a group of users, while a VN provides connectivity between those users.

There is one other aspect of a VNs that we had previously discussed, which is that TES connected to different VNs can use overlapping addresses.  This is an aspect that goes beyond what users can communicate with each other to what addresses they can use.  Presumably, if one were to allow two members of different CUGs to communicate (by policy), then the addresses used by the members of the two CUGs better not overlap.  However, if a gateway is used to interconnect two VNs, then that gateway could also provide a NAT function to allow communication between TES on the two VNs even if they overlap.  So, perhaps address overlap is another distinction between VNs and CUGs?

This leads me to a question about using "policy" (without a gateway) to allow two CUGs to communicate.  Can this "policy" also define NAT to allow overlapping addresses between two CUGs?

Thanks, Larry

From: <Toy>, Mehmet <[email-masked]<mailto:[email-masked]>>
Date: Friday, September 28, 2012 7:17 AM
To: Kireeti Kompella <[email-masked]<mailto:[email-masked]>>, Thomas Narten <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [nvo3] What is CUG model was RE: Push or pull?

Mehmet

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Kireeti Kompella
Sent: Friday, September 28, 2012 9:36 AM
To: Thomas Narten
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] What is CUG model was RE: Push or pull?

Hi Thomas,

No difference.

<rest snipped>

Kireeti

---

From: "Fedyk, Donald (Don)" <[email-masked]>
Date: Fri, 28 Sep 2012 16:11:59 +0000

Mehmet is pointing  to what I remember.

I think we should break down the definitions in terms of
Policy
Mechanism
Behavior.

For example:

A mechanism for a L2 CUG is a VLAN-ID
The policy is that a Ethernet network logically separates traffic from between VLANs on the same network.
A closed user group is a behavior where there is free communication between a group.  For example MAC based services on a Ethernet VLAN.

Don

From: [email-masked] [mailto:[email-masked]] On Behalf Of Larry Kreeger (kreeger)
Sent: Friday, September 28, 2012 11:39 AM
To: [email-masked]
Subject: Re: [nvo3] What is CUG model was RE: Push or pull?

Thanks, Larry

Mehmet

Hi Thomas,

No difference.

<rest snipped>

Kireeti

---

From: Kireeti Kompella <[email-masked]>
Date: Fri, 28 Sep 2012 09:21:38 -0700

Hi Larry,

> I think Mehmet has a subtle, but valid point about the term CUG.  A CUG is a group of users, while a VN provides connectivity between those users.

More than that, a VN may also provide connectivity to other entities (storage, appliances).

So, good: the notions of CUG and VN are close in many aspects; however, VNs are what we want, and don't carry baggage.  Let's define VNs, then use them consistently across NVO3 documents.

> There is one other aspect of a VNs that we had previously discussed, which is that TES connected to different VNs can use overlapping addresses.  

Certainly, VNs belonging to different "customers" must support overlapping addresses.  VNs belonging to the same customer may or may not have overlapping addresses, depending on how the customer chooses to allocate addresses.

> This is an aspect that goes beyond what users can communicate with each other to what addresses they can use.  Presumably, if one were to allow two members of different CUGs to communicate (by policy), then the addresses used by the members of the two CUGs better not overlap.  However, if a gateway is used to interconnect two VNs, then that gateway could also provide a NAT function to allow communication between TES on the two VNs even if they overlap.  So, perhaps address overlap is another distinction between VNs and CUGs?
> 

Policy, whether via a gateway or otherwise, includes NAT.  Whether one has gateways or not is up to implementation and deployment.  I now pass the mike to Aldrin :-)

Kireeti.

> Thanks, Larry
> 
> From: <Toy>, Mehmet <[email-masked]>
> To: Kireeti Kompella <[email-masked]>, Thomas Narten <[email-masked]>

---

From: "Fedyk, Donald (Don)" <[email-masked]>
Date: Fri, 28 Sep 2012 16:29:42 +0000

Hi Lucy

Inline.

-----Original Message-----
From: Lucy yong [mailto:[email-masked]] 
Sent: Thursday, September 27, 2012 6:44 PM
To: Fedyk, Donald (Don); [email-masked]
Subject: RE: What is CUG model was RE: [nvo3] Push or pull?

Don,

snip..

[DF] I think we should state policy, mechanism and behavior (see my previous messages on this.) 
A BGP-AD based RT is a mechanism, a group attribute. 
The policy is that all members that advertise the same route target receive the information associated with that Route target. 
The behavior is that this enable information exchange between members a closed user group behavior. 

RT is a push mechanism because the information is pushed to members that announce the same membership. You could argue RT is a bit of a pull too because until you announce RT you don't get the information about the RT.  But once you are a member you receive all updates as a push. I'm thinking that push and pull categorization is less important than administrative control. I believe you are asking for a mechanism that does not have the same administrative control on connectivity as an l3 VPN, or EVPN that uses a RT. The RT starts where the BGP-AD policy is enabled. 

Regards,
Don  

Lucy

Thanks,
Don 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Lucy yong
Sent: Thursday, September 27, 2012 10:27 AM
To: Luyuan Fang (lufang); Shah, Himanshu; Thomas Narten; Kireeti Kompella
Cc: [email-masked]
Subject: Re: [nvo3] Push or pull?

Hope this helps.

Lucy

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Fri, 28 Sep 2012 17:34:23 +0000

Hi Kireeti,

A couple of responses below with LK>.

 - Larry

From: Kireeti Kompella <[email-masked]<mailto:[email-masked]>>
Date: Friday, September 28, 2012 9:21 AM
To: Cisco Employee <[email-masked]<mailto:[email-masked]>>
Cc: Kireeti Kompella <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [nvo3] What is CUG model was RE: Push or pull?

Hi Larry,

On Sep 28, 2012, at 08:39 , "Larry Kreeger (kreeger)" <[email-masked]<mailto:[email-masked]>> wrote:

LK> OK, so you are differentiating "users" of a CUG (presumably TES) from shared "services" provided by the cloud service provider.  Do you think we need some terminology for this differentiation or not?

LK> I agree!

Policy, whether via a gateway or otherwise, includes NAT.  Whether one has gateways or not is up to implementation and deployment.

LK> OK, just saying policy is general enough to include NAT…so let me ask my real question.  Does the notion of Policy as used by many in this group who work on BGP based Lx-VPN (using RT import/export), include the notion of NAT?

Thanks, Larry

 I now pass the mike to Aldrin :-)

Kireeti.

Thanks, Larry

Mehmet

From:[email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Kireeti Kompella
Sent: Friday, September 28, 2012 9:36 AM
To: Thomas Narten
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] What is CUG model was RE: Push or pull?

Hi Thomas,

No difference.

<rest snipped>

Kireeti

---

From: Lucy yong <[email-masked]>
Date: Fri, 28 Sep 2012 19:10:47 +0000

Don and all,

[[LY]] I like that phrases. It is more clear. RT is designed for network operator to implement policy on VPN routing. Although RT impose can be seen as "pull" like, it is more policy configuration that have be done by net operator (or say initiated). The pull function I refer is more about NVE initiates the request on behalf of VM request. 

To make my early case more straight forward, all the VMs are in a single VN which provides a communication among VMs. However, when an IT application runs on these VMs, DC operator may not know ahead what communication pattern that the application will have among these VMs. It may result to a case that some VM may only need to communicate some other, not all. I did not intend to make any complex case which user or operator want to constrain the communication between some VMs for security reason or other reason.

BTW: I am so glad to be in this community and to be forced offline for a while. Thank you so much for folks (Don, Jon, Kireeti, Aldrin, Yakov, Mehmet, Tom, more) to contribute cases and help to clarify the scenarios. (so save my time to reply and address every response, and enjoy reading other's views first) I fully agree with Aldrin, NVO3 WG needs to have a consent first where and when and what the pull request should be in the push model; and whether and when and what the push is necessary in the pull model. We need to know real world requirements and hope DC operator contribute more on this.

Thanks a lot.
Lucy 

Regards,
Don  

Lucy

Thanks,
Don 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Lucy yong
Sent: Thursday, September 27, 2012 10:27 AM
To: Luyuan Fang (lufang); Shah, Himanshu; Thomas Narten; Kireeti Kompella
Cc: [email-masked]
Subject: Re: [nvo3] Push or pull?

Hope this helps.

Lucy

---

From: Yakov Rekhter <[email-masked]>
Date: Fri, 28 Sep 2012 14:58:53 -0700

Kireeti,

> No difference.=20

At least in the context of L2-based CUG, as defined in
draft-rekhter-nvo3-vm-mobility-issues, it is defined in terms
of the service that it provides:

Given that, would you say that there is no difference between
an L2-based CUG and a VN ?

Yakov.

> For clarity, we should define VN in some draft (framework?), state that VN a=
> nd CUG are the same, but stick to using one term, preferably VN.=20
> 
> Closed User Groups are groups of GSM mobile telephone subscribers who can on=
> ly make calls and receive calls from members within the group. Any other cal=
> ls would be rejected.
> 
> The definition is pretty close, but it goes to show that CUG comes with over=
> tones that may confuse people.=20
> 
> One refinement might be to define L2 and L3 VNs off the base definition of V=
> N.=20
> 
> >=20
> Thomas, the reference I gave below about where L2-CUG is defined is wrong --=
>  not the EVPN draft, but the one above.=20
> 
> <rest snipped>
> 
> Kireeti
> 
> 
> --Apple-Mail-F4CA13A0-392E-49A4-8B68-13EF56C87F2F
> Content-Transfer-Encoding: quoted-printable
> Content-Type: text/html;
> 	charset=utf-8
> 
> <html><head></head><body bgcolor=3D"#FFFFFF"><div>Hi Thomas,<br><br>On Sep 2=
> 8, 2012, at 6:09, Thomas Narten &lt;<a href=3D"mailto:[email-masked]">nar=
> [email-masked]</a>&gt; wrote:<br><br></div><div></div><blockquote type=3D"ci=
> te"><div><span>This discussion is interesting, but I still have not seen an a
=
> nswer to</span><br><span>the question: what is (or is there) a difference be=
> tween a VN and an</span><br><span>CUG?</span><br></div></blockquote><div><br=
> ></div>No difference.&nbsp;<div><br><blockquote type=3D"cite"><div><span>If t
=
> here is no difference, that would be great, because then we can</span><br><s=
> pan>use the terms interchangably. If there are differences, we need to</span=
> ><br><span>understand what those differences are, or there will be confusion=
>  in</span><br><span>our discussions.</span><br></div></blockquote><div><br><=
> /div>For clarity, we should define VN in some draft (framework?), state that=
>  VN and CUG are the same, but stick to using one term, preferably VN.&nbsp;<=
> /div><div><br></div><div>Note that Wikipedia defines CUG as follows:</div><d=
> iv><br></div><p style=3D"margin-top: 0.4em; margin-right: 0px; margin-bottom=
> : 0.5em; margin-left: 0px; "><b>Closed User Groups</b>&nbsp;are groups of&nb=
> sp;<a href=3D"http://en.wikipedia.org/wiki/GSM" title=3D"GSM" style=3D"text-=
> decoration: none; background-image: none; background-attachment: initial; ba=
> ckground-origin: initial; background-clip: initial; background-color: initia=
> l; ">GSM</a>&nbsp;<a href=3D"http://en.wikipedia.org/wiki/Mobile_telephone" t
=
> itle=3D"Mobile telephone" class=3D"mw-redirect" style=3D"text-decoration: no=
> ne; background-image: none; background-attachment: initial; background-origi=
> n: initial; background-clip: initial; background-color: initial; ">mobile te=
> lephone</a>&nbsp;subscribers who can only make calls and receive calls from m
=
> embers within the group. Any other calls would be rejected.</p><div><br></di=
> v><div>The definition is pretty close, but it goes to show that CUG comes wi=
> th overtones that may confuse people.&nbsp;</div><div><br></div><div>One ref=
> inement might be to define L2 and L3 VNs off the base definition of VN.&nbsp=
> ;</div><div><br><blockquote type=3D"cite"><div><span>Yakov Rekhter &lt;<a hr=
> ef=3D"mailto:[email-masked]">[email-masked]</a>&gt; writes:</span><br>=
> <span></span><br><blockquote type=3D"cite"><span>There is a definition of L2=
> -based CUG in</span><br></blockquote><blockquote type=3D"cite"><span> draft-=
> rekhter-nvo3-vm-mobility-issues</span></blockquote></div></blockquote><div><=
> br></div><div>Thomas, the reference I gave below about where L2-CUG is defin=
> ed is wrong -- not the EVPN draft, but the one above.&nbsp;</div><br></div><=
> div>&lt;rest snipped&gt;</div><div><br></div><div>Kireeti</div><div><br></di=
> v></body></html>=
> 
> --Apple-Mail-F4CA13A0-392E-49A4-8B68-13EF56C87F2F--
> 
> --===============6522178728508190782==
> Content-Type: text/plain; charset="us-ascii"
> MIME-Version: 1.0
> Content-Transfer-Encoding: 7bit
> Content-Disposition: inline
> 
> --===============6522178728508190782==--