Subject: Re: Solicit reviews and comments on draft-hao-l3vpn-inter-nvo3-vpn-00

From: Robert Raszuk <[email-masked]>
Date: Tue, 1 Jul 2014 19:21:33 +0200

Hi Weiguo,

I have read your document with interest expecting to see something new ..
well I have not seen anything which other L3VPN documents would not have
already covered.

Section 4 lists 16 million number as issue of option A. Can you explain
where in option A architecture such number is stated ?

As to the number of sessions issue between ASBRs I would recommend lecture
of draft-mapathak-interas-ab-00.txt

Bottom line I am not finding anything new in this document which would not
be already well known or even shipping in open source or commercial
implementations.

Best regards,
R.

On Tue, Jul 1, 2014 at 3:29 AM, Haoweiguo <[email-masked]> wrote:

> Hi all,
> We submit a new draft of "Inter-AS Option B between NVO3 and BGP/MPLS IP
> VPN network", please review it and warmly appreciate your comments and
> suggestions.
> Thanks
> weiguo
>

---

From: Haoweiguo <[email-masked]>
Date: Wed, 2 Jul 2014 03:52:46 +0000

Hi Robert,

Currently there is no heterogeneous option-B inter-as solution between NVO3 and MPLS L3VPN network, only option-A inter-as solution is provided in mainstream NVO3 solution. This draft mainly discribes the procedures of NVO3 distributed gateway integrated with inter-as option-B solution, it can provide directions for open source or commercial implementations.

Pls see my detail reply inline [weiguo].

Thanks

weiguo

________________________________
发件人: [email-masked] [[email-masked]] 代表 Robert Raszuk [[email-masked]]
发送时间: 2014年7月2日 1:21
收件人: Haoweiguo
抄送: [email-masked]
主题: Re: Solicit reviews and comments on draft-hao-l3vpn-inter-nvo3-vpn-00

Hi Weiguo,

I have read your document with interest expecting to see something new .. well I have not seen anything which other L3VPN documents would not have already covered.

Section 4 lists 16 million number as issue of option A. Can you explain where in option A architecture such number is stated ?
As to the number of sessions issue between ASBRs I would recommend lecture of draft-mapathak-interas-ab-00.txt
[weiguo]:Because theoretically 16M VN are supported in a NVO3 network, if centralized layer 3 gateway and inter-as option-A solution is used, then 16M sub-interfaces should be supported on each ASBR. VPN traffic separation still relies on VLAN.
In draft-mapathak-interas-ab-00,EBGP session can be greatly reduced, but I don't know how can you separate different VPN's traffic? Does it still relies on VLAN or sub-interface?

Bottom line I am not finding anything new in this document which would not be already well known or even shipping in open source or commercial implementations.
[weiguo]:
The difference from traditional RFC 4364 inter-as option-B are as follows:

Internal DC to external DC direction routing distribution procedures:
ASBR1 allocates MPLS VPN Label per tenant (VN ID) per NVE, the incoming forwarding table on ASBR1 is as following:
   +--------------------+------------------+
   |  MPLS VPN Label    |  NVE  + VN ID    |
   +--------------------+------------------+
   |       1000         |  NVE1 + 10       |
   +--------------------+------------------+
   |       2000         |  NVE1 + 20       |
   +--------------------+------------------+

External DC to internal DC direction routing distribution procedures:
ASBR1 allocates VN ID for each VPN Label receiving from ASBR2, The outgoing forwarding table on ASBR1 is as follows:
   +------------------+--------------------+
   |       VN ID      |   Out VPN Label    |
   +------------------+--------------------+
   |      10000       |        3000        |
   +------------------+--------------------+
   |      10001       |        4000        |
   +------------------+--------------------+

In this scenario, VN ID has local significance and is similar to MPLS Label, it's different from VN usage in NVO3 network, in NVO3 network VN has network wide globally significance.

Also, NVO3 support NVE-NVA architecture, in this architecture, NVA should support inter-as option-B function,  the forwarding table on ASBR and NVEs are downloaded through NVA.

Best regards,
R.

On Tue, Jul 1, 2014 at 3:29 AM, Haoweiguo <[email-masked]<mailto:[email-masked]>> wrote:
Hi all,
We submit a new draft of "Inter-AS Option B between NVO3 and BGP/MPLS IP VPN network", please review it and warmly appreciate your comments and suggestions.
Thanks
weiguo

---

From: Haoweiguo <[email-masked]>
Date: Mon, 7 Jul 2014 01:23:13 +0000

Hi Robert,

In inter-as option-B between NVO3 and MPLS/IP VPN network case,  the DC Gateway router(ASBR in NVO3 network) should perform translation between VN-IDs and IP VPN labels while forwarding packets between the DC and WAN interfaces without performing an IP lookup. This draft describes how to set up VN-ID and MPLS Label mapping relationship on NVO3 network gateway(ASBR). This is the new point of this draft, and has some difference from traditional RFC4364 based inter-as option-B solution.

Thanks

weiguo

________________________________
发件人: Haoweiguo
发送时间: 2014年7月2日 11:52
收件人: Robert Raszuk
抄送: [email-masked]
主题: 答复: Solicit reviews and comments on draft-hao-l3vpn-inter-nvo3-vpn-00

Hi Robert,

Thanks

weiguo

Hi Weiguo,

Best regards,
R.

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 8 Jul 2014 01:13:33 +0200

Hi,

> between the DC and WAN interfaces without performing an IP lookup.

What's wrong with IP lookup per VPN on ASBR ?

Are you sure you want to allocate as many VN_IDs as you have VPN
prefixes in each VPN and inject all of those into end systems in the
DC ?

Moreover do you think there are advantages to continue to churn your
Vn_IDs intra-dc advertisements at the same rate as your WAN L3VPN IP
reachability changes say between any of the two PEs for multihomed VPN
sites ?

I would consider that topology hiding is an advantage not a drawback.

Note that DC facing ASBR may demux to a VRF on VN_ID basis per VPN
(not per VPN prefix), perform again per VPN scoped IP lookup then
follow with encapsulating in correct VPN label towards WAN. Pretty
simple setup :)

Thx,
R.

On Mon, Jul 7, 2014 at 3:23 AM, Haoweiguo <[email-masked]> wrote:
> Hi Robert,
>
> In inter-as option-B between NVO3 and MPLS/IP VPN network case,  the DC
> Gateway router(ASBR in NVO3 network) should perform translation between
> VN-IDs and IP VPN labels while forwarding packets between the DC and WAN
> interfaces without performing an IP lookup. This draft describes how to set
> up VN-ID and MPLS Label mapping relationship on NVO3 network gateway(ASBR).
> This is the new point of this draft, and has some difference from
> traditional RFC4364 based inter-as option-B solution.
>
> Thanks
>
> weiguo
>
>
>
> Currently there is no heterogeneous option-B inter-as solution between NVO3
> and MPLS L3VPN network, only option-A inter-as solution is provided in
> mainstream NVO3 solution. This draft mainly discribes the procedures of NVO3
> distributed gateway integrated with inter-as option-B solution, it can
> provide directions for open source or commercial implementations.
>
> [weiguo]:Because theoretically 16M VN are supported in a NVO3 network, if
> centralized layer 3 gateway and inter-as option-A solution is used, then 16M
> sub-interfaces should be supported on each ASBR. VPN traffic separation
> still relies on VLAN.
> In draft-mapathak-interas-ab-00,EBGP session can be greatly reduced, but I
> don't know how can you separate different VPN's traffic? Does it still
> relies on VLAN or sub-interface?
>
> ASBR1 allocates MPLS VPN Label per tenant (VN ID) per NVE, the incoming
> forwarding table on ASBR1 is as following:
>
> In this scenario, VN ID has local significance and is similar to MPLS Label,
> it's different from VN usage in NVO3 network, in NVO3 network VN has network
> wide globally significance.
>
>
>
> Also, NVO3 support NVE-NVA architecture, in this architecture, NVA should
> support inter-as option-B function,  the forwarding table on ASBR and NVEs
> are downloaded through NVA.
>
>
> Best regards,
> R.
>
>

---

From: Haoweiguo <[email-masked]>
Date: Tue, 8 Jul 2014 02:24:45 +0000

Hi,
Thanks for your deep level comments. Pls see my reply inline with [weiguo].
Thanks
weiguo

________________________________________
发件人: [email-masked] [[email-masked]] 代表 Robert Raszuk [[email-masked]]
发送时间: 2014年7月8日 7:13
收件人: Haoweiguo
抄送: [email-masked]
主题: Re: 答复: Solicit reviews and comments on draft-hao-l3vpn-inter-nvo3-vpn-00

Hi,

[weiguo]:  This is a inter-as option between NVO3 and MPLS VPN network. When traffic from internal NVO3 network is forwarded to external MPLS VPN network, the ASBR(NVO3 Gateway) terminates into a VRF and performs IP lookup, MPLS encapsulation and then forwards the traffic to MPLS VPN network. This mechanism is called Option-A in the draft. Currently most of  the  commercial used NVO3 gateways support this solution.
In extreme case, 16M VN/Tenants and several million end systems need to be supported in a data center,  inter-as option-A solution  has following drawbacks:
1.Up to 16 million (16M) gateway interfaces (virtual/physical) and 16M EBGP session(Option-AB can optimize this point to reduce EBGP session) need to exist between the ASBRs. 
2.UP to 16M VRFs need to be supported on border routers.  
3.Several million routing entries need to be supported on border routers.

[weiguo]: Assuming the ASBR in NVO3 network is ASBR1, the ASBR in MPLS VPN network is ASBR2.
 1. In external to internal DC direction, the route distribution procedures are as follows:
On ASBR1, VN ID is allocated per MPLS VPN Label received from peer ASBR2 in MPLS VPN network. If remote PE in MPLS VPN network allocates MPLS VPN Label per VPN, there won't be so many MPLS VPN Label on ASBR2 and so many VN ID on ASBR1. NVEs need to know ASBR1's allocated VN ID to forward  traffic data  from internal to external DC.
2. In internal to external DC direction, the route distribution procedures are as follows:
ASBR1 allocates MPLS VPN Label  per NVE per VN(tenant), and then advertise the MPLS VPN Label to peer ASBR2 in MPLS VPN network. When ASBR1 receives traffic data from external to internal DC,ASBR1 looks up NVO3 encapsulation by MPLS VPN Label, then perform NVO3 encapsulation and sends the traffic to destination NVE. 

[weiguo]: Yes, frequent VPN Label and VN ID advertisements is a issue. To ensure no network fluctuations due to VPN IP prefix changes, MPLS VPN Label had better be allocated per VPN on each PE.
In internal to external DC direction, MPLS VPN Label is allocated per VN per NVE, TS creation,deletion and motion won't cause MPLS VPN Label frequent allocation and revotion, or at least to mitigate this issue.

[weiguo]: Topology hiding is good, the drawback is the scalability of ASBR. Option-A is suitable to small or medium data center, option-B is more suitable to large scale data center.

[weiguo]: This solution has already been described in the draft as Option-A solution.

Thx,
R.

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 8 Jul 2014 11:06:38 +0200

Hi Weiguo,

> [weiguo]:  This is a inter-as option between NVO3 and MPLS VPN
> network. When traffic from internal NVO3 network is forwarded to
> external MPLS VPN network, the ASBR(NVO3 Gateway)
> terminates into a VRF and performs IP lookup, MPLS encapsulation
> and then forwards the traffic to MPLS VPN network. This
> mechanism is called Option-A in the draft.

Then the draft is wrong as Option-A does not use any MPLS encapsulation.

You are confusing on which side of ASBR VRFs sit. I am describing the
model where ASBR is an option-B ASBR with in the same time integrated
PE towards the DC side. There is no option A.

Note also for clarity that packets are arriving at such PE from DC
side not traditionally based on bounding to interface to a VRF but
based on VN_ID bounding to a VRF. Some vendors call this "vrf select"
mode.

Sorry if this model/architecture was not clear in my former mail.

> In extreme case, 16M VN/Tenants and several million end systems
> need to be supported in a data center,  inter-as option-A solution
> has following drawbacks:

I am not talking about option A so the following list does not apply.
Moreover each solution needs to scale and I would not put any hard
limit regardless what it is.

So if particular box reaches its capacity you add another box to act
as a gateway. That's it.

> On ASBR1, VN ID is allocated per MPLS VPN Label received from peer
> ASBR2 in MPLS VPN network.

That is very inefficient and I would argue that architecturally wrong.
This puts all the WAN churn and load towards all compute nodes which
just does not belong there and brings no value.

> If remote PE in MPLS VPN network allocates MPLS VPN Label per VPN,

There is no such a thing. At best you can allocate a VPN label per PE
per VRF, but this would require a lot of reconfiguration of the entire
WAN L3VPN network which may not be welcome by number of providers.

Best regards,
R.

---

From: Haoweiguo <[email-masked]>
Date: Tue, 8 Jul 2014 10:11:51 +0000

Hi Robert,
Thanks for your clear description.

In this email, your model are clearly described, it's different from the model in our draft. In your model, NVO3 gateway and PE are integrated in a single device. In our model, NVO3 gateway and PE are two independent devices.
If data center network and MPLS VPN WAN network belongs to two different companies (for example, data center network belongs to Microsoft, MPLS VPN network belongs to AT&T), NVO3 gateway and PE function normally are splitted into two devices instead of integrating into single device. In this case, inter-as solution between NVO3 gateway and PE should be used, there are two solutions of Option-A and Option-B. In this draft, Option-B solution is described in detail.

Pls see my other replies inline with [weiguo].

Then the draft is wrong as Option-A does not use any MPLS encapsulation.
[weiguo]. Sorry, it's a slip of the pen. Yes, Option-A has no MPLS encapsulation, native IP forwarding is used between ASBRs. Every VPN people should know the procedures.
The correct description is:
This is a inter-as option between NVO3 and MPLS VPN network. When traffic from internal NVO3 network is forwarded to external MPLS VPN network, the ASBR(NVO3 Gateway)
terminates into a VRF and performs IP lookup,  then forwards the traffic to peer ASBR. This mechanism is called Option-A in the draft.

You are confusing on which side of ASBR VRFs sit. I am describing the
model where ASBR is an option-B ASBR with in the same time integrated
PE towards the DC side. There is no option A.
[weiguo]: I understand what's your meaning now, we are not discussing the same scenario. In your description, PE and NVO3 gateway are integrated in a single device, so no Option-A here. If MPLS VPN network and data center network belongs to same company(such as AT&T, china telecom, and etc), your model are suitable, PE and NVO3 gateway are integrated on same device, VN ID is bounded to a VRF on the device.
In our draft,  PE and NVO3 gateway are splitted into two devices, PE belongs to carrier network(such as china telecom, AT&T,and etc), NVO3 gateway belongs to data center(such as Facebook, Microsoft, Tencent, and etc), in this case inter-as Option-A or Option-B solution should be used. In this case, NVO3 gateway in data center is ASBR1, PE in carrier network is ASBR2. If sub-interface is used to separate VPN traffic between ASBR1 and ASBR2, the solution is Option-A. If MPLS VPN Label is used to separate VPN traffic between ASBR1 and ASBR2, it is Option-B solution, the procedures is described in detail in this draft.

Note also for clarity that packets are arriving at such PE from DC
side not traditionally based on bounding to interface to a VRF but
based on VN_ID bounding to a VRF. Some vendors call this "vrf select"
mode.
[weiguo]: Yes, in your model, bounding interface to VRF isn't needed, only VN ID bounding to VRF exists.

Sorry if this model/architecture was not clear in my former mail.
[weiguo]: I understand your model now, your model are different from the model described in our draft.

Thanks
weiguo
________________________________________
发件人: [email-masked] [[email-masked]] 代表 Robert Raszuk [[email-masked]]
发送时间: 2014年7月8日 17:06
收件人: Haoweiguo
抄送: [email-masked]
主题: Re: 答复: 答复: Solicit reviews and comments on draft-hao-l3vpn-inter-nvo3-vpn-00

Hi Weiguo,

Best regards,
R.

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 8 Jul 2014 14:56:19 +0200

Hi Weiguo,

> In this email, your model are clearly described

I don't think so based on your reply. Your interpretation is
completely unrelated to my model :) Sorry.

Let me try to illustrate:

                              Data Center
                        WAN

 VMs/namespaces
hypervisor + kernel (PE1)
   compute node
                |
                |
                 --------------------- PE2/ASBR --- option B --- ASBR
------------- PE3

I am describing PE2/ASBR network element where you combine plain
option B with the VRF select based on VN_ID PE.

DC can be operated by any provider so can WAN. Those are completely
not related to each other except agreement to establish L3VPN option B
peering session between corresponding ASBRs.

PE2 is acting as WAN side aggregator on a per VRF basis. There is no
option A in this model anywhere.

L3 VPN architecture does not provide such model hence perhaps you were
trying to map it to something which does not really exist in any RFC
document and this was the src of your confusion.

I suggest if you want to build scalable interconnect between DC based
virtualized resources and arbitrary WAN you use this model. Of course
for completenss such mode must also provide independent access from
Internet either plain or over IPSec, but I assume this is obvious and
here we are only focusing on hybrid L3 VPN interconnect case.

PE1 may be just a data plane or it could be also control plane and
data plane. The choice there is flexible.

Hopefully with the picture this is more clear now.

Regards,
R.

---

From: Haoweiguo <[email-masked]>
Date: Wed, 9 Jul 2014 07:27:46 +0000

Hi Robert,
Your illustration is very clear. Your model is centralized NVO3 gateway + Inter-as Option-B, our model in the draft is distributed NVO3 gateway + Inter-as Option-B.
Your model is very good and easy to be deployed, it will be a great complementary to this solution.
In your model PE2 have all VRFs of the data center. For the traffic data  from NVO3 side to MPLS VPN side, VRF is selected based on VN ID. For the traffic data from MPLS VPN side to NVO3 side, VRF is selected based on MPLS VPN Label instead of sub-interface between ASBRs.
In our model , gateway is located on each NVE instead of on the PE2/ASBR, VRF is located on each NVE, only VN ID and MPLS VPN Label switching table exists on the ASBR, VRF don't need to be setup on the ASBR.It is more similar to vanilla inter-as Option-B.
Thanks
weiguo
________________________________________
发件人: [email-masked] [[email-masked]] 代表 Robert Raszuk [[email-masked]]
发送时间: 2014年7月8日 20:56
收件人: Haoweiguo
抄送: [email-masked]
主题: Re: 答复: 答复: 答复: Solicit reviews and comments on draft-hao-l3vpn-inter-nvo3-vpn-00

Hi Weiguo,

                              Data Center
                        WAN

 VMs/namespaces

Regards,
R.

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 9 Jul 2014 10:33:08 +0200

Hi Weiguo,

You still missing it .. sorry. There is nothing centralized in my model.

As I said my model does not exist today - it is new proposal - so any
attempts to map it to existing RFC will not be successful.

I said on the picture that NVE is still there on the hosts (compute
nodes) in the role of PE1.

VRFs on PE2/ASBR are only for aggregation purposes.

Moreover I thought this was obvious but solution I proposed is a
superset .. it means ASBR for some VPNs can be just pure vanilla
option B and for those which have a lot of prefixes can be enhanced
with per VRF aggregation function based on the vrf select mode per
VN_ID.

Effectively what I am proposing in this long thread is to define few
modes of VN_ID allocations (just like we have few modes of VPN label
allocation on a classic PE).

For some VPNs you may have VN-ID directly mapped to VPN label and for
some where aggregation of VPN routes makes sense you map VN_ID to a
local VRF.

Thx,
R.

---

From: Haoweiguo <[email-masked]>
Date: Wed, 9 Jul 2014 10:07:12 +0000

Hi Robert,
Thanks for your good comments and detail explaination.
Yes, for some VPNs ASBR can be pure vanilla option-B. For some VPN which have a lot of prefixes can be enhanced
with your suggested solution, per VRF aggregation function based on the vrf select mode per VN_ID. It is useful and necessary in some cases.
And it is indeed necessary to define the modes of VN_ID allocations, we should add the related description in future version.
Thanks
weiguo
________________________________________
发件人: [email-masked] [[email-masked]] 代表 Robert Raszuk [[email-masked]]
发送时间: 2014年7月9日 16:33
收件人: Haoweiguo
抄送: [email-masked]
主题: Re: 答复: 答复: 答复: 答复: Solicit reviews and comments on draft-hao-l3vpn-inter-nvo3-vpn-00

Hi Weiguo,

Thx,
R.