Subject: Re: [bess] draft-hao-bess-inter-nvo3-vpn-optionc

From: Diego Garcia del Rio <[email-masked]>
Date: Tue, 17 Nov 2015 22:25:22 -0800

Some comments from my side,

I think the draft makes quite a few assumptions on specific
implementation details that are way too general to be considered
widely available.

Most of the TOR devices today already pay a double-pass penalty when
doing routing of traffic in/out of vxlan-type tunnels. Only the newest
generation can route into tunnels without additional passes. And are
definitively limited in being able to arbitrary select UDP ports on a
per tunnel basis. In fact, most are even limited at using more than
one VNID per "service" of sorts.

The IP-addressed based implementation which would, I assume, imply
assigning one or more CIDRs to a loopback interface on the ASBR-d is
also quite arbitrary and does not seem like a technically "clean"
solution. (besides burning tons of IPs). As a side-note, most PE-grade
routers i've worked with were quite limited in terms of IP addresses
used for tunnel termination and it wasn't that just a simple pool can
be used.

Wim's mentions on cases where the Application itself, hosted in a
datacenter, would be part of the option-C interconnect, was dismissed
without much discussion so far, while, if we look in detail at the
type of users which will even consider a complex topology like model-C
its most likely users and operators very familiar with MPLS VPNs in
the WAN. Those type of operators will most likely be very interested
in deploying MPLS or WAN-grade applications (i.e., virtual-routers or
other VNFs) in the DC and thus its highly likely that the interconnect
would not terminate at the NVE itself but rather the TS (the virtual
machine).

Also, the use of UDP ports at random would imply quite complex logic
on the ASBR-d IMHO. Im not saying its impossible, but since the
received packet now not only has to be received on a random (though
locally chosen) UDP port and this information preserved in the
pipeline to be able to do the double-tunnel-stitching, it sounds quite
complex. I am sure someone in the list will mention this has already
been implemented somewhere, and I won't argue with that. And let's not
even bring into account what this would do to any DC middlebox that
now has to look at vxlan over almost any random port. We have to go
back to the "is it a 4 or is it a 6 in byte x" heuristics to try to
guess whether the packet is vxlan or just something entirely different
running over IP.

In general I feel the proposed solution seems to be fitting of a
specific use-case which is not really detailed in the draft and does
not describe   a solution that would "elegantly" solve the issues at
hand. It just feels like we're using any available bit-space to stuff
data into places that do not necesarily belong.

Yes, MPLS encapsulations on virtual switches are not yet fully
available, and there can be some performance penalty on the TORs, but
the solutions are much cleaner from a control and data plane point of
view. Maybe I'm too utopic.

Best regards,

Diego

---------------------------------------------------------------------------------

Hi,

The problem we are trying to solve is to reduce data center
GW/ASBR-d's forwarding table size, the motivation is same as
traditional MPLS VPN option-C. Currently, the most common practise on
ASBR-d is to terminate VXLAN encapsulation, look up local routing
table, and then perform MPLS encapsulation to the WAN network. ASBR-d
needs to maintain all VM's MAC/IP. In Option-C method, only transport
layer information needed to be maintained at GW/ASBR-d, the
scalability will be greatly enhanced. Traditonal Option-C is only for
MPLS VPN network interworking, because VXLAN is becoming pervasive in
data center,  the solution in this draft was proposed for the
heterogeneous network interworking.

The advantage of this solution is that only VXLAN encapsulation is
required for OVS/TOR. Unlike Wim's solution, east-west bound traffic
uses VXLAN encap, while north-south bound traffic uses MPLSoGRE/UDP
encap.

There are two solutions in this draft:

1. Using VXLAN tunnel destination IP for stitching at ASBR-d.

No data plane modification requirements on OVS or TOR switches, only
hardware changes on ASBR-d. ASBR-d normally is router, it has
capability to realize the hardware changes. It will consume many IP
addresses and the IP pool for allocation needs to be configured on
ASBR-d beforehand.

2. Using VXLAN destination UDP port for stitching at ASBR-d.

Compared with solution 1, less IP address will be consumed for
allocation. If UDP port range is too large, we can combine with
solution 1 and 2.

In this solution, both data plane modification changes are needed at
OVS/TOR and ASBR-d. ASBR-d also has capability to realize the hardware
changes. For OVS, it also can realize the data plane changes. For TOR
switch, it normally can't realize this function.  This solution mainly
focuses on pure software based overlay network, it has more
scalability. In public cloud data center, software based overlay
network is the majority case.

Whether using solution 1 or 2 depends on the operators real envionment.

So I think our solution has no flaws, it works fine.

Thanks,

weiguo

________________________________
From: BESS [[email-masked]] on behalf of John E Drake
[[email-masked]]
Sent: Wednesday, November 18, 2015 2:49
To: Henderickx, Wim (Wim); EXT - [email-masked]; BESS
Subject: Re: [bess] draft-hao-bess-inter-nvo3-vpn-optionc

Hi,

I think Wim has conclusively demonstrated that this draft has fatal
flaws and I don’t support it.  I also agree with his suggestion that
we first figure out what problem we are trying to solve before solving
it.

Yours Irrespectively,

John

From: BESS [mailto:[email-masked]] On Behalf Of Henderickx, Wim (Wim)
Sent: Tuesday, November 17, 2015 12:49 PM
To: EXT - [email-masked]; BESS
Subject: Re: [bess] draft-hao-bess-inter-nvo3-vpn-optionc

— Snip —

No, the spec as it is can be implemented in its VXLAN variant with
existing vswitches (e.g. OVS allows to choose the VXLAN destination
port, ditto for the linux kernel stack).

(ToR is certainly another story, most of them not having a flexible
enough VXLAN dataplane nor support for any MPLS-over-IP.)

WH> and how many ports simultaneously would they support? For this to
work every tenant needs a different VXLAN UDP destination port/receive
port.
There might be SW elements that could do some of this, but IETF
defines solutions which should be implemented across the board
HW/SW/etc. Even if some SW switches can do this, the proposal will
impose so many issues in HW/data-plane engines that I cannot be behind
this solution.

To make this work generically we will have to make changes anyhow.
Given this, we better do it in the right way and guide the industry to
a solution which does not imply those complexities. Otherwise we will
stick with these specials forever with all consequences (bugs, etc).

- snip -

From: "[email-masked]<mailto:[email-masked]>"
<[email-masked]<mailto:[email-masked]>>
Organization: Orange
Date: Tuesday 17 November 2015 at 01:37
To: Wim Henderickx
<[email-masked]<mailto:[email-masked]>>,
BESS <[email-masked]<mailto:[email-masked]>>
Subject: Re: [bess] draft-hao-bess-inter-nvo3-vpn-optionc

Hi Wim, WG,

2015-11-16, Henderickx, Wim (Wim):

2015-11-13, Henderickx, Wim (Wim):
Thomas, we can discuss forever and someone need to describe
requirements, but the current proposal I cannot agree to for the
reasons explained.

TM> Well, although discussing forever is certainly not the goal, the
reasons for rejecting a proposal need to be thoroughly understood.
WH> my point is what is the real driver for supporting a plain VXLAN
data-plane here, the use cases I have seen in this txt is always where
an application behind a NVE/TOR is demanding option c, but none of the
NVE/TOR elements.

My understanding is that the applications  are contexts where overlays
are present is when workloads (VMs or baremetal) need to be
interconnected with VPNs. In these contexts, there can be reasons to
want Option C to reduce the state on ASBRs.

In these context, its not the workload (VM or baremetal) that would
typically handle VRFs, but really the vswitch or ToR.

WH2> can it not be all cases: TOR/vswitch/Application. I would make
the solution flexible to support all of these not?

2015-11-13, Henderickx, Wim (Wim):

TM> The right trade-off to make may in fact depend on whether you prefer:
(a) a new dataplane stitching behavior on DC ASBRs (the behavior
specified in this draft)
or (b) an evolution of the encaps on the vswitches and ToRs to support
MPLS/MPLS/(UDP or GRE)

WH> b depends on the use case

I don't get what you mean by "b depends on the use case".
WH> see my above comment. If the real use case is an application
behind NVE/TOR requiring model C, than all the discussion on impact on
NVE/TOR is void. As such I want to have a discussion on the real
driver/requirement for option c interworking with an IP based Fabric.

Although I can agree than detailing requirements can always help, I
don't think one can assume a certain application to dismiss the
proposal.

WH> for me the proposal is not acceptable for the reasons explained:
too much impact on the data-planes

I wrote the above based on the idea that the encap used in
MPLS/MPLS/(UDP or GRE), which hence has to be supported on the ToRs
and vswitches.
Another possibility would be service-label/middle-label/Ethernet
assuming an L2 adjacency between vswitches/ToRs and ASBRs, but this
certainly does not match your typical DC architecture. Or perhaps had
you something else in mind ?

WH> see above. The draft right now also requires changes in existing
TOR/NVE so for me all this discussion/debate is void.

No, the spec as it is can be implemented in its VXLAN variant with
existing vswitches (e.g. OVS allows to choose the VXLAN destination
port, ditto for the linux kernel stack).

(ToR is certainly another story, most of them not having a flexible
enough VXLAN dataplane nor support for any MPLS-over-IP.)

WH> and how many ports simultaneously would they support?

WH> and depending on implementation you don’t need to change any of
the TOR/vswitches.

Does this mean that for some implementations you may not need to
change any of the TOR/vswitches, but that for some others you may ?

WH> any proposal on the table requires changes, so for me this is not
a valid discussion

See above, the proposal in the draft does not necessarily need changes
in vswitches.

Let me take a practical example : while I can quite easily see how to
implement the procedures in draft-hao-bess-inter-nvo3-vpn-optionc
based on current vswitch implementations of VXLAN, the lack of
MPLS/MPLS/(UDP, GRE) support in commonplace vswitches seems to me as
making that alternate solution you suggest harder to implement.

WH> I would disagree to this. Tell me which switch/TOR handles
multiple UDP ports for VXLAN ?

I mentioned _v_switches, and many do support a variable destination
port for VXLAN, which is sufficient to implement what the draft
proposes.

-Thomas

From: Thomas Morin <[email-masked]<mailto:[email-masked]>>
Organization: Orange
Date: Friday 13 November 2015 at 09:57
To: Wim Henderickx
<[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [bess] draft-hao-bess-inter-nvo3-vpn-optionc

Hi Wim,

I agree on the analysis that this proposal is restricted to
implementations that supports the chosen encap with non-IANA ports
(which may be hard to achieve for instance on hardware
implementations, as you suggest), or to context where managing
multiple IPs would be operationally viable.

However, it does not seem obvious to me how the alternative you
propose [relying on 3-label option C with an MPLS/MPLS/(UDP|GRE)
encap] addresses the issue of whether the encap behavior is supported
or not (e.g. your typical ToR chipset possibly may not support this
kind of encap,  and even software-based switches may not be ready to
support that today).

My take is that having different options to adapt to various
implementations constraints we may have would have value.

(+ one question below on VXLAN...)

-Thomas

2015-11-12, Henderickx, Wim (Wim):
On VXLAN/NVGRE, do you challenge the fact that they would be used with
a dummy MAC address that would be replaced by the right MAC by a
sender based on an ARP request when needed ?

Is the above the issue you had in mind about VXLAN and NVGRE ?

WH> yes

I you don't mind me asking : why do you challenge that ?

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations
confidentielles ou privilegiees et ne doivent donc

pas etre diffuses, exploites ou copies sans autorisation. Si vous avez
recu ce message par erreur, veuillez le signaler

a l'expediteur et le detruire ainsi que les pieces jointes. Les
messages electroniques etant susceptibles d'alteration,

France Telecom - Orange decline toute responsabilite si ce message a
ete altere, deforme ou falsifie. Merci

This message and its attachments may contain confidential or
privileged information that may be protected by law;

they should not be distributed, used or copied without authorization.

If you have received this email in error, please notify the sender and
delete this message and its attachments.

As emails may be altered, France Telecom - Orange shall not be liable
if this message was modified, changed or falsified.

Thank you.

---

From: Zhuangshunwan <[email-masked]>
Date: Wed, 18 Nov 2015 08:25:33 +0000

Hi Diego,

Thanks for your comments. Pls see inline with [Vincent].

Vincent

发件人: BESS [mailto:[email-masked]] 代表 Diego Garcia del Rio
发送时间: 2015年11月18日 14:25
收件人: [email-masked]
主题: Re: [bess] draft-hao-bess-inter-nvo3-vpn-optionc

I think the draft makes quite a few assumptions on specific implementation details that are way too general to be considered widely available.

Most of the TOR devices today already pay a double-pass penalty when doing routing of traffic in/out of vxlan-type tunnels. Only the newest generation can route into tunnels without additional passes. And are definitively limited in being able to arbitrary select UDP ports on a per tunnel basis. In fact, most are even limited at using more than one VNID per "service" of sorts.

[Vincent]: Yes, the new generation BCM chipset has already supported VXLAN routing without additional passes. For OVS/TOR, VXLAN encapsulation is more popular than MPLSoGRE/UDP, and has better performance.

The IP-addressed based implementation which would, I assume, imply assigning one or more CIDRs to a loopback interface on the ASBR-d is also quite arbitrary and does not seem like a technically "clean" solution. (besides burning tons of IPs). As a side-note, most PE-grade routers i've worked with were quite limited in terms of IP addresses used for tunnel termination and it wasn't that just a simple pool can be used.

[Vincent]: I think the larger VTEP IP address range on ASBR-d has no limitations. For the hardware on ASBR-d, it has capability to terminate multiple VXLAN tunnels with arbitrary destination VTEP IP addresses.

Wim's mentions on cases where the Application itself, hosted in a datacenter, would be part of the option-C interconnect, was dismissed without much discussion so far, while, if we look in detail at the type of users which will even consider a complex topology like model-C its most likely users and operators very familiar with MPLS VPNs in the WAN. Those type of operators will most likely be very interested in deploying MPLS or WAN-grade applications (i.e., virtual-routers or other VNFs) in the DC and thus its highly likely that the interconnect would not terminate at the NVE itself but rather the TS (the virtual machine).

Also, the use of UDP ports at random would imply quite complex logic on the ASBR-d IMHO. Im not saying its impossible, but since the received packet now not only has to be received on a random (though locally chosen) UDP port and this information preserved in the pipeline to be able to do the double-tunnel-stitching, it sounds quite complex. I am sure someone in the list will mention this has already been implemented somewhere, and I won't argue with that. And let's not even bring into account what this would do to any DC middlebox that now has to look at vxlan over almost any random port. We have to go back to the "is it a 4 or is it a 6 in byte x" heuristics to try to guess whether the packet is vxlan or just something entirely different running over IP.

[Vincent]: Using NP or multi-core CPU hardware technology, it can be implemented although deeper packet inspection is needed to perform UDP port and MPLS stitching.

In general I feel the proposed solution seems to be fitting of a specific use-case which is not really detailed in the draft and does not describe   a solution that would "elegantly" solve the issues at hand. It just feels like we're using any available bit-space to stuff data into places that do not necesarily belong.

Yes, MPLS encapsulations on virtual switches are not yet fully available, and there can be some performance penalty on the TORs, but the solutions are much cleaner from a control and data plane point of view. Maybe I'm too utopic.

[Vincent]: I think pure VXLAN solution has its scenario, it's general rather than specific. We can't require all OVS/NVEs support VXLAN + MPLSoGRE at the same time.

Best regards,

Diego

Hi,

The problem we are trying to solve is to reduce data center GW/ASBR-d's forwarding table size, the motivation is same as traditional MPLS VPN option-C. Currently, the most common practise on ASBR-d is to terminate VXLAN encapsulation, look up local routing table, and then perform MPLS encapsulation to the WAN network. ASBR-d needs to maintain all VM's MAC/IP. In Option-C method, only transport layer information needed to be maintained at GW/ASBR-d, the scalability will be greatly enhanced. Traditonal Option-C is only for MPLS VPN network interworking, because VXLAN is becoming pervasive in data center,  the solution in this draft was proposed for the heterogeneous network interworking.

The advantage of this solution is that only VXLAN encapsulation is required for OVS/TOR. Unlike Wim's solution, east-west bound traffic uses VXLAN encap, while north-south bound traffic uses MPLSoGRE/UDP encap.

No data plane modification requirements on OVS or TOR switches, only hardware changes on ASBR-d. ASBR-d normally is router, it has capability to realize the hardware changes. It will consume many IP addresses and the IP pool for allocation needs to be configured on ASBR-d beforehand.

Compared with solution 1, less IP address will be consumed for allocation. If UDP port range is too large, we can combine with solution 1 and 2.

In this solution, both data plane modification changes are needed at OVS/TOR and ASBR-d. ASBR-d also has capability to realize the hardware changes. For OVS, it also can realize the data plane changes. For TOR switch, it normally can't realize this function.  This solution mainly focuses on pure software based overlay network, it has more scalability. In public cloud data center, software based overlay network is the majority case.

Thanks,

weiguo

From: BESS [[email-masked]<mailto:[email-masked]>] on behalf of John E Drake [[email-masked]<mailto:[email-masked]>]

To: Henderickx, Wim (Wim); EXT - [email-masked]<mailto:[email-masked]>; BESS

Hi,

I think Wim has conclusively demonstrated that this draft has fatal flaws and I don’t support it.  I also agree with his suggestion that we first figure out what problem we are trying to solve before solving it.

Yours Irrespectively,

John

From: BESS [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Henderickx, Wim (Wim)

To: EXT - [email-masked]<mailto:[email-masked]>; BESS

— Snip —

No, the spec as it is can be implemented in its VXLAN variant with existing vswitches (e.g. OVS allows to choose the VXLAN destination port, ditto for the linux kernel stack).

(ToR is certainly another story, most of them not having a flexible enough VXLAN dataplane nor support for any MPLS-over-IP.)

WH> and how many ports simultaneously would they support? For this to work every tenant needs a different VXLAN UDP destination port/receive port.

There might be SW elements that could do some of this, but IETF defines solutions which should be implemented across the board HW/SW/etc. Even if some SW switches can do this, the proposal will impose so many issues in HW/data-plane engines that I cannot be behind this solution.

To make this work generically we will have to make changes anyhow. Given this, we better do it in the right way and guide the industry to a solution which does not imply those complexities. Otherwise we will stick with these specials forever with all consequences (bugs, etc).

- snip -

From: "[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>" <[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>>

Organization: Orange

To: Wim Henderickx <[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>>, BESS <[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>>

Hi Wim, WG,

Thomas, we can discuss forever and someone need to describe requirements, but the current proposal I cannot agree to for the reasons explained.

TM> Well, although discussing forever is certainly not the goal, the reasons for rejecting a proposal need to be thoroughly understood.

WH> my point is what is the real driver for supporting a plain VXLAN data-plane here, the use cases I have seen in this txt is always where an application behind a NVE/TOR is demanding option c, but none of the NVE/TOR elements.

My understanding is that the applications  are contexts where overlays are present is when workloads (VMs or baremetal) need to be interconnected with VPNs. In these contexts, there can be reasons to want Option C to reduce the state on ASBRs.

In these context, its not the workload (VM or baremetal) that would typically handle VRFs, but really the vswitch or ToR.

WH2> can it not be all cases: TOR/vswitch/Application. I would make the solution flexible to support all of these not?

(a) a new dataplane stitching behavior on DC ASBRs (the behavior specified in this draft)

or (b) an evolution of the encaps on the vswitches and ToRs to support MPLS/MPLS/(UDP or GRE)

WH> see my above comment. If the real use case is an application behind NVE/TOR requiring model C, than all the discussion on impact on NVE/TOR is void. As such I want to have a discussion on the real driver/requirement for option c interworking with an IP based Fabric.

Although I can agree than detailing requirements can always help, I don't think one can assume a certain application to dismiss the proposal.

WH> for me the proposal is not acceptable for the reasons explained: too much impact on the data-planes

I wrote the above based on the idea that the encap used in MPLS/MPLS/(UDP or GRE), which hence has to be supported on the ToRs and vswitches.

Another possibility would be service-label/middle-label/Ethernet assuming an L2 adjacency between vswitches/ToRs and ASBRs, but this certainly does not match your typical DC architecture. Or perhaps had you something else in mind ?

WH> see above. The draft right now also requires changes in existing TOR/NVE so for me all this discussion/debate is void.

No, the spec as it is can be implemented in its VXLAN variant with existing vswitches (e.g. OVS allows to choose the VXLAN destination port, ditto for the linux kernel stack).

(ToR is certainly another story, most of them not having a flexible enough VXLAN dataplane nor support for any MPLS-over-IP.)

WH> and depending on implementation you don’t need to change any of the TOR/vswitches.

Does this mean that for some implementations you may not need to change any of the TOR/vswitches, but that for some others you may ?

WH> any proposal on the table requires changes, so for me this is not a valid discussion

See above, the proposal in the draft does not necessarily need changes in vswitches.

Let me take a practical example : while I can quite easily see how to implement the procedures in draft-hao-bess-inter-nvo3-vpn-optionc based on current vswitch implementations of VXLAN, the lack of MPLS/MPLS/(UDP, GRE) support in commonplace vswitches seems to me as making that alternate solution you suggest harder to implement.

WH> I would disagree to this. Tell me which switch/TOR handles multiple UDP ports for VXLAN ?

I mentioned _v_switches, and many do support a variable destination port for VXLAN, which is sufficient to implement what the draft proposes.

-Thomas

From: Thomas Morin <[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>>

Organization: Orange

To: Wim Henderickx <[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>>

Cc: "[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>" <[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>>

Hi Wim,

I agree on the analysis that this proposal is restricted to implementations that supports the chosen encap with non-IANA ports (which may be hard to achieve for instance on hardware implementations, as you suggest), or to context where managing multiple IPs would be operationally viable.

However, it does not seem obvious to me how the alternative you propose [relying on 3-label option C with an MPLS/MPLS/(UDP|GRE) encap] addresses the issue of whether the encap behavior is supported or not (e.g. your typical ToR chipset possibly may not support this kind of encap,  and even software-based switches may not be ready to support that today).

My take is that having different options to adapt to various implementations constraints we may have would have value.

-Thomas

On VXLAN/NVGRE, do you challenge the fact that they would be used with a dummy MAC address that would be replaced by the right MAC by a sender based on an ARP request when needed ?

WH> yes

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc

pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler

a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,

France Telecom - Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci

This message and its attachments may contain confidential or privileged information that may be protected by law;

If you have received this email in error, please notify the sender and delete this message and its attachments.

As emails may be altered, France Telecom - Orange shall not be liable if this message was modified, changed or falsified.

Thank you.

---

From: Thomas Morin <[email-masked]>
Date: Thu, 19 Nov 2015 11:52:59 +0100

I don't believe the discussion can be usefully summarized in terms of 
"fatal flaws".
Let me try to summarize my understanding of the thing discussed in this 
thread.

The solution in the draft has an impact on ASBR hardware due to the new 
kind of stitching required (as well summarized by Diego). Whether this 
is a big issue or not depends on the vendor. The solution in the draft, 
for the NVE part, would be implementable in software quite easily. But, 
because of the limitations in the widespread ToR chipsets for VXLAN (and 
their lack of support for MPLS/(GRE or UDP)), its multiple-UDP-port 
variant would be harder to implement in ToRs or at least not without a 
performance hit (people who know that better, feel free to correct).  
The multiple-loopback approach has operational drawbacks, but whether or 
not these are killer issues may depend on the targeted scale (in terms 
of number of NVEs) and may depend on other factors (operator practices, 
ability to automate ASBR configs).

The alternative approach, put forward by Wim, consist in using existing 
Option-C with a variant of the 3-label variant relying on an 
MPLS/MPLS/UDP-or-GRE encap (instead of the 3-label MPLS stack).  This 
approach would not necessitate new standardized procedures, would 
require less changes on ASBRs. However it is not supported today by 
vswitches or ToRs (unless the bottom MPLS label is passed to the VM, 
which I think would restrict the application to a subset of the 
scenarios for which Option C is interesting). Having it supported in 
vswitches may be a simple matter of writing the software, but ToR 
support seems to require evolution in chipsets ; whether such a support 
is likely to appear hasn't been discussed in the thread.

All in all, it seems there is no solution to cover an Option C scenario 
with today's generation ToRs, and the question for tomorrow's generation 
ToRs hasn't been really discussed.

Based on the above, I would think the question boils down to whether it 
is desirable to specify an Option C variant usable with vswitches as 
they are today and possibly usable with a performance hit with today's 
ToRs chipsets, at the expense of a required evolution on ASBRs.  The 
alternative requiring some evolution on vswitches and waiting for future 
ToRs chipsets.

-Thomas

Zhuangshunwan  :
> *发件人:*BESS [mailto:[email-masked]] *代表 *Diego Garcia del Rio
> *发送时间:*2015年11月18日14:25
> *收件人:[email-masked]
> *主题:*Re: [bess] draft-hao-bess-inter-nvo3-vpn-optionc
>
> [Vincent]: Yes, the new generation BCM chipset has already supported 
> VXLAN routing without additional passes. For OVS/TOR, VXLAN 
> encapsulation is more popular than MPLSoGRE/UDP, and has better 
> performance.
> [Vincent]: I think the larger VTEP IP address range on ASBR-d has no 
> limitations. For the hardware on ASBR-d, it has capability to 
> terminate multiple VXLAN tunnels with arbitrary destination VTEP IP 
> addresses.
> [Vincent]: Using NP or multi-core CPU hardware technology, it can be 
> implemented although deeper packet inspection is needed to perform UDP 
> port and MPLS stitching.
> [Vincent]: I think pure VXLAN solution has its scenario, it's general 
> rather than specific. We can't require all OVS/NVEs support VXLAN + 
> MPLSoGRE at the same time.
> Best regards,
> Diego
> uses VXLAN encap, while north-south bound traffic uses MPLSoGRE/UDP encap.
> From: BESS [[email-masked] <mailto:[email-masked]>] on 
> behalf of John E Drake [[email-masked] <mailto:[email-masked]>]
> we first figure out what problem we are trying to solve before solving it.
> Yours Irrespectively,
> John
> From: BESS [mailto:[email-masked] 
> <mailto:[email-masked]>] On Behalf Of Henderickx, Wim (Wim)
> To: EXT - [email-masked] <mailto:[email-masked]>; BESS
> From: "[email-masked] 
> <mailto:[email-masked]><mailto:[email-masked] 
> <mailto:[email-masked]>>" <[email-masked] 
> <mailto:[email-masked]><mailto:[email-masked] 
> <mailto:[email-masked]>>>
> Organization: Orange
> To: Wim Henderickx <[email-masked] 
> <mailto:[email-masked]><mailto:[email-masked] 
> <mailto:[email-masked]>>>, BESS <[email-masked] 
> <mailto:[email-masked]><mailto:[email-masked] <mailto:[email-masked]>>>
> don't think one can assume a certain application to dismiss the proposal.
> port for VXLAN, which is sufficient to implement what the draft proposes.
> -Thomas
> From: Thomas Morin <[email-masked] 
> <mailto:[email-masked]><mailto:[email-masked] 
> <mailto:[email-masked]>>>
> Organization: Orange
> To: Wim Henderickx <[email-masked] 
> <mailto:[email-masked]><mailto:[email-masked] 
> <mailto:[email-masked]>>>
> Cc: "[email-masked] <mailto:[email-masked]><mailto:[email-masked] 
> <mailto:[email-masked]>>" <[email-masked] 
> <mailto:[email-masked]><mailto:[email-masked] <mailto:[email-masked]>>>

---

From: "Henderickx, Wim (Wim)" <[email-masked]>
Date: Thu, 19 Nov 2015 15:02:40 +0000

Thomas, thx for the time summarising this and great summary.

>performance hit (people who know that better, feel free to correct).  

WH> some HW cannot do this at all, so lets dismiss this idea.

>ability to automate ASBR configs).

WH> ok now we have not discussed the constraints some HW vendors have with respect to global VNIDs. To make this work all VNID/Labels need to be globally unique. Hmmmmm 

>alternative requiring some evolution on vswitches and waiting for future 

WH> I vote for a an evolution of switches/TORs that have proper support for this.
I hope some HW vendors of TOR chips shime in, but I am told the MPLS solution is possible in the next generation chips they are working on.

The other options requires baggage forever on the ASBR elements that does not bring value and it is better to avoid this and build an architecture which is future proof and more cost effective. My 2 cents
>_______________________________________________
>BESS mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/bess

---

From: Diego Garcia del Rio <[email-masked]>
Date: Thu, 19 Nov 2015 10:00:56 -0800

Hi Weiguo,

If we're really talking about a public cloud operator where MOST of the
endpoints would be software, interconnecting DC and WAN environments seems
pretty blunt to be honest. You're going to be leaking every single /32 IP
address of the virtual machines into the wide area and most carriers tend
to avoid that kind of thing since its quite disrutptive in terms of
scalability.

Furthermore, wim's proposal does NOT imply changes on the east-west traffic
patterns. Those would remain on vxlan or whatever new encapsulation we
manage to come up with at NVO3.

Model C interconnect presuposes a high trust relationship between both
parties, and if the cloud and WAN are operated by the same company, that is
valid.

Finally, as wim also mentions, the issue of the globally allocated VNIDs is
quite an issue. This affects the existing TORs since now all the WAN PEs
would have to organize themselves not to ever allocate overlapping labels
(then converted to VNIDs) since the DC gear today cannot handle using the
same VNID on different "services".

Again, you dismiss the top of racks, but I don't think we can propose a
standard that just simply ignores a good chunk of the DC space.

And at the same time, all the DC edge routers where we propose to implement
these complex mapping have larger and larger scaling tables with route
capacities in the order of millions of prefixes these days. And while model
A has the burden of provisioning the DC edge with the VRF instance, the
fact it can do prefix lookups means it doesnt need to leak every single /32
to the wide area and instead expose the aggregates.

Best Regards,

On Thu, Nov 19, 2015 at 7:02 AM, Henderickx, Wim (Wim) <
[email-masked]> wrote:

> On 19/11/15 02:52, "BESS on behalf of Thomas Morin" <[email-masked]
> on behalf of [email-masked]> wrote:
>
> WH> ok now we have not discussed the constraints some HW vendors have with
> respect to global VNIDs. To make this work all VNID/Labels need to be
> globally unique. Hmmmmm
>
> >
> I hope some HW vendors of TOR chips shime in, but I am told the MPLS
> solution is possible in the next generation chips they are working on.
>
> The other options requires baggage forever on the ASBR elements that does
> not bring value and it is better to avoid this and build an architecture
> which is future proof and more cost effective. My 2 cents
> >
> >-Thomas
> >
> >
> >Zhuangshunwan  :
> >>
> >> Hi Diego,
> >>

---

From: Haoweiguo <[email-masked]>
Date: Fri, 20 Nov 2015 06:37:35 +0000

Hi Wim,
Pls see inline with [weiguo2].
weiguo
________________________________________
From: BESS [[email-masked]] on behalf of Henderickx, Wim (Wim) [[email-masked]]
Sent: Thursday, November 19, 2015 23:02
To: Thomas Morin; [email-masked]
Subject: Re: [bess] draft-hao-bess-inter-nvo3-vpn-optionc
Thomas, thx for the time summarising this and great summary.

WH> some HW cannot do this at all, so lets dismiss this idea.
[weiguo2]:  Currently TOR switches can't realize multiple-UDP-port variant VXLAN encapsulation, but we can't give conclusion that hardware can't realize this function forever.
The TOR only need to do the a little more complicated encap work for the traffic from DC to WAN direction. For the traffic from WAN to DC direction, the encapsulation is normal VXLAN encapsulation, it has no extra decap requirements for the TOR switches. Normally encapsulation process is simpler than decapsulation. The more complicated decap work is performed at ASBR-d which is router device and has more flexibility.
Currently MPLS+MPLS+GRE encap and decap process may be possible to be realized relying on internal loop in TOR switch, but just as i have mentioned before, many switch doesn't use MPLS over GRE for north-south bound traffic forwarding, they just want to use VXLAN encapsulation for both north-south and east-west bound traffic. I think variant UDP port solution is reasonable, it can't be dismissed.

WH> ok now we have not discussed the constraints some HW vendors have with respect to global VNIDs. To make this work all VNID/Labels need to be globally unique. Hmmmmm
[weiguo2]:  In SDN scenario, a virtual network normally is represented by a global VN ID or MPLS VPN Label to simplify network management. I think it's not strange to allocate global unique MPLS VPN Label or VN ID for a virtual network now.
WH> I vote for a an evolution of switches/TORs that have proper support for this.
I hope some HW vendors of TOR chips shime in, but I am told the MPLS solution is possible in the next generation chips they are working on.
[weiguo2]:  I think it's better not to change current TOR hardware to realize option-C interconnection. No TOR hardware modification solution must be provided. But for future case, we can propose extra hardware requirements for TORs.
The other options requires baggage forever on the ASBR elements that does not bring value and it is better to avoid this and build an architecture which is future proof and more cost effective. My 2 cents

>> ― Snip ―

---

From: Haoweiguo <[email-masked]>
Date: Fri, 20 Nov 2015 08:59:33 +0000

Hi Diego,

Thanks for your comments. Pls see inline with  [weiguo3].

Thanks,

weiguo

________________________________
From: BESS [[email-masked]] on behalf of Diego Garcia del Rio [[email-masked]]
Sent: Friday, November 20, 2015 2:00
To: Henderickx, Wim (Wim)
Cc: Thomas Morin; [email-masked]
Subject: Re: [bess] draft-hao-bess-inter-nvo3-vpn-optionc

Hi Weiguo,

If we're really talking about a public cloud operator where MOST of the endpoints would be software, interconnecting DC and WAN environments seems pretty blunt to be honest. You're going to be leaking every single /32 IP address of the virtual machines into the wide area and most carriers tend to avoid that kind of thing since its quite disrutptive in terms of scalability.
[weiguo3]: I just give an example for pure software based overlay network, i don't mean the option-c solution should be used in public cloud envionment or public cloud is the focus of the option-C. My meaning is that in pure software envionment, currently OVS can be modified for the UDP-port-variant VXLAN encapsulation. For hardware TOR, it relies on next generation chipset.

Furthermore, wim's proposal does NOT imply changes on the east-west traffic patterns. Those would remain on vxlan or whatever new encapsulation we manage to come up with at NVO3.
[weiguo3]: Yes, i understood wim's proposal that is east-west bound encap uses VXLAN and north-south bound encap uses MPLSoGRE. However, some DC operators don't use this pattern, they use VXLAN encapsulation for both north-south bound and east-west bound traffic.

Model C interconnect presuposes a high trust relationship between both parties, and if the cloud and WAN are operated by the same company, that is valid.
[weiguo3]: Yes, i agree.
Finally, as wim also mentions, the issue of the globally allocated VNIDs is quite an issue. This affects the existing TORs since now all the WAN PEs would have to organize themselves not to ever allocate overlapping labels (then converted to VNIDs) since the DC gear today cannot handle using the same VNID on different "services".
[weiguo3]: In data center, normally globally unique VN ID mechanism is used for management simplicity. In the option-c solution, similarly, MPLS VPN Label and VN ID also is recommended to be globally managed through SDN solution.

Again, you dismiss the top of racks, but I don't think we can propose a standard that just simply ignores a good chunk of the DC space.
[weiguo3]: Currently if the UDP-port-variant VXLAN encap solution is deployed, the top of racks need to be dismissed, the hardware capability status quo on the TOR switch  is limited. In the future, if there are some chipsets support the encapsulation, it can also be used on TORs. In summay, TOR dismissing is only for current deployment, that's not for the future.

And at the same time, all the DC edge routers where we propose to implement these complex mapping have larger and larger scaling tables with route capacities in the order of millions of prefixes these days. And while model A has the burden of provisioning the DC edge with the VRF instance, the fact it can do prefix lookups means it doesnt need to leak every single /32 to the wide area and instead expose the aggregates.
[weiguo3]: This is the common issue for Option-C. Just as you said, it is very suitable for a high trustworthy and security envionment, such as cloud data center and WAN PE belongs to same company.

Best Regards,

On Thu, Nov 19, 2015 at 7:02 AM, Henderickx, Wim (Wim) <[email-masked]<mailto:[email-masked]>> wrote:
Thomas, thx for the time summarising this and great summary.

>> *发件人:*BESS [mailto:[email-masked]<mailto:[email-masked]>] *代表 *Diego Garcia del Rio
>> *发送时间:*2015年11月18日14:25
>> *收件人:[email-masked]<mailto:[email-masked]>
>> From: BESS [[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>] on
>> behalf of John E Drake [[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>]
>> To: Henderickx, Wim (Wim); EXT - [email-masked]<mailto:[email-masked]>
>> <mailto:[email-masked]<mailto:[email-masked]>>; BESS
>> From: BESS [mailto:[email-masked]<mailto:[email-masked]>
>> <mailto:[email-masked]<mailto:[email-masked]>>] On Behalf Of Henderickx, Wim (Wim)
>> To: EXT - [email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>; BESS
>> From: "[email-masked]<mailto:[email-masked]>
>> <mailto:[email-masked]<mailto:[email-masked]>><mailto:[email-masked]<mailto:[email-masked]>
>> <mailto:[email-masked]<mailto:[email-masked]>>>" <[email-masked]<mailto:[email-masked]>
>> <mailto:[email-masked]<mailto:[email-masked]>><mailto:[email-masked]<mailto:[email-masked]>
>> <mailto:[email-masked]<mailto:[email-masked]>>>>
>> Organization: Orange
>> To: Wim Henderickx <[email-masked]<mailto:[email-masked]>
>> <mailto:[email-masked]<mailto:[email-masked]>><mailto:[email-masked]<mailto:[email-masked]>
>> <mailto:[email-masked]<mailto:[email-masked]>>>>, BESS <[email-masked]<mailto:[email-masked]>
>> <mailto:[email-masked]<mailto:[email-masked]>><mailto:[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>>>
>> From: Thomas Morin <[email-masked]<mailto:[email-masked]>
>> <mailto:[email-masked]<mailto:[email-masked]>><mailto:[email-masked]<mailto:[email-masked]>
>> <mailto:[email-masked]<mailto:[email-masked]>>>>
>> Organization: Orange
>> To: Wim Henderickx <[email-masked]<mailto:[email-masked]>
>> <mailto:[email-masked]<mailto:[email-masked]>><mailto:[email-masked]<mailto:[email-masked]>
>> <mailto:[email-masked]<mailto:[email-masked]>>>>
>> Cc: "[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>><mailto:[email-masked]<mailto:[email-masked]>
>> <mailto:[email-masked]<mailto:[email-masked]>>>" <[email-masked]<mailto:[email-masked]>
>> <mailto:[email-masked]<mailto:[email-masked]>><mailto:[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>>>
>[email-masked]<mailto:[email-masked]>
_______________________________________________
BESS mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/bess

---

From: <[email-masked]>
Date: Fri, 20 Nov 2015 10:01:35 +0100

2015-11-19, Henderickx, Wim (Wim):
> WH> I vote for a an evolution of switches/TORs that have proper
> support for this. I hope some HW vendors of TOR chips shime in, but I
> am told the MPLS solution is possible in the next generation chips
> they are working on.

Well, it looks like the key questions are:
- when would ToR chips support MPLS/MPLS/UDP ?  (the generation that has 
been released recently but not present in most shipping ToRs yet, the 
next one ?)
- do we want an interim VXLAN-based solution ? (that will involve at 
best a performance penalty with existing chips, and at worse impossible 
to implement -- we haven't seen clear information in this thread)

-Thomas

>>> *发件人:*BESS [mailto:[email-masked]] *代表 *Diego Garcia del
>>> Rio *发送时间:*2015年11月18日14:25 *收件人:[email-masked] *主题:*Re: [bess]
>>> draft-hao-bess-inter-nvo3-vpn-optionc
>>>
>>> Some comments from my side, I think the draft makes quite a few
>>> assumptions on specific implementation details that are way too
>>> general to be considered widely available. Most of the TOR
>>> devices today already pay a double-pass penalty when doing
>>> routing of traffic in/out of vxlan-type tunnels. Only the newest
>>> generation can route into tunnels without additional passes. And
>>> are definitively limited in being able to arbitrary select UDP
>>> ports on a per tunnel basis. In fact, most are even limited at
>>> using more than one VNID per "service" of sorts. [Vincent]: Yes,
>>> the new generation BCM chipset has already supported VXLAN
>>> routing without additional passes. For OVS/TOR, VXLAN
>>> performance. The IP-addressed based implementation which would, I
>>> assume, imply assigning one or more CIDRs to a loopback interface
>>> on the ASBR-d is also quite arbitrary and does not seem like a
>>> technically "clean" solution. (besides burning tons of IPs). As a
>>> side-note, most PE-grade routers i've worked with were quite
>>> limited in terms of IP addresses used for tunnel termination and
>>> it wasn't that just a simple pool can be used. [Vincent]: I think
>>> the larger VTEP IP address range on ASBR-d has no limitations.
>>> For the hardware on ASBR-d, it has capability to terminate
>>> multiple VXLAN tunnels with arbitrary destination VTEP IP
>>> addresses. Wim's mentions on cases where the Application itself,
>>> hosted in a datacenter, would be part of the option-C
>>> interconnect, was dismissed without much discussion so far,
>>> while, if we look in detail at the type of users which will even
>>> consider a complex topology like model-C its most likely users
>>> and operators very familiar with MPLS VPNs in the WAN. Those type
>>> of operators will most likely be very interested in deploying
>>> MPLS or WAN-grade applications (i.e., virtual-routers or other
>>> VNFs) in the DC and thus its highly likely that the interconnect
>>> would not terminate at the NVE itself but rather the TS (the
>>> virtual machine). Also, the use of UDP ports at random would
>>> imply quite complex logic on the ASBR-d IMHO. Im not saying its
>>> impossible, but since the received packet now not only has to be
>>> received on a random (though locally chosen) UDP port and this
>>> information preserved in the pipeline to be able to do the
>>> double-tunnel-stitching, it sounds quite complex. I am sure
>>> someone in the list will mention this has already been
>>> implemented somewhere, and I won't argue with that. And let's
>>> not even bring into account what this would do to any DC
>>> middlebox that now has to look at vxlan over almost any random
>>> port. We have to go back to the "is it a 4 or is it a 6 in byte
>>> x" heuristics to try to guess whether the packet is vxlan or just
>>> something entirely different running over IP. [Vincent]: Using NP
>>> or multi-core CPU hardware technology, it can be implemented
>>> although deeper packet inspection is needed to perform UDP port
>>> and MPLS stitching. In general I feel the proposed solution seems
>>> to be fitting of a specific use-case which is not really detailed
>>> in the draft and does not describe   a solution that would
>>> "elegantly" solve the issues at hand. It just feels like we're
>>> using any available bit-space to stuff data into places that do
>>> not necesarily belong. Yes, MPLS encapsulations on virtual
>>> switches are not yet fully available, and there can be some
>>> performance penalty on the TORs, but the solutions are much
>>> cleaner from a control and data plane point of view. Maybe I'm
>>> too utopic. [Vincent]: I think pure VXLAN solution has its
>>> scenario, it's general rather than specific. We can't require all
>>> OVS/NVEs support VXLAN + MPLSoGRE at the same time. Best
>>> regards, Diego
Hi,
>>> traditional MPLS VPN option-C. Currently, the most common
>>> practise on ASBR-d is to terminate VXLAN encapsulation, look up
>>> local routing table, and then perform MPLS encapsulation to the
>>> WAN network. ASBR-d needs to maintain all VM's MAC/IP. In
>>> Option-C method, only transport layer information needed to be
>>> maintained at GW/ASBR-d, the scalability will be greatly
>>> enhanced. Traditonal Option-C is only for MPLS VPN network
>>> interworking, because VXLAN is becoming pervasive in data center,
>>> the solution in this draft was proposed for the heterogeneous
>>> network interworking. The advantage of this solution is that only
>>> VXLAN encapsulation is required for OVS/TOR. Unlike Wim's
>>> solution, east-west bound traffic uses VXLAN encap, while
>>> north-south bound traffic uses MPLSoGRE/UDP encap. There are two
>>> solutions in this draft: 1. Using VXLAN tunnel destination IP for
>>> stitching at ASBR-d. No data plane modification requirements on
>>> OVS or TOR switches, only hardware changes on ASBR-d. ASBR-d
>>> normally is router, it has capability to realize the hardware
>>> changes. It will consume many IP addresses and the IP pool for
>>> allocation needs to be configured on ASBR-d beforehand. 2. Using
>>> VXLAN destination UDP port for stitching at ASBR-d. Compared with
>>> solution 1, less IP address will be consumed for allocation. If
>>> UDP port range is too large, we can combine with solution 1 and
>>> 2. In this solution, both data plane modification changes are
>>> needed at OVS/TOR and ASBR-d. ASBR-d also has capability to
>>> realize the hardware changes. For OVS, it also can realize the
>>> data plane changes. For TOR switch, it normally can't realize
>>> this function.  This solution mainly focuses on pure software
>>> based overlay network, it has more scalability. In public cloud
>>> data center, software based overlay network is the majority
>>> case. Whether using solution 1 or 2 depends on the operators real
>>> envionment. So I think our solution has no flaws, it works fine.
>>> Thanks, weiguo ________________________________ From: BESS
>>> [[email-masked] <mailto:[email-masked]>] on behalf
>>> of John E Drake [[email-masked] <mailto:[email-masked]>]
>>> Sent: Wednesday, November 18, 2015 2:49 To: Henderickx, Wim
>>> (Wim); EXT - [email-masked]
>>> <mailto:[email-masked]>; BESS Subject: Re: [bess]
>>> draft-hao-bess-inter-nvo3-vpn-optionc Hi, I think Wim has
>>> conclusively demonstrated that this draft has fatal flaws and I
>>> don’t support it.  I also agree with his suggestion that we first
>>> figure out what problem we are trying to solve before solving
>>> it. Yours Irrespectively, John From: BESS
>>> [mailto:[email-masked] <mailto:[email-masked]>] On
>>> Behalf Of Henderickx, Wim (Wim) Sent: Tuesday, November 17, 2015
>>> 12:49 PM To: EXT - [email-masked]
>>> <mailto:[email-masked]>; BESS Subject: Re: [bess]
>>> draft-hao-bess-inter-nvo3-vpn-optionc — Snip — No, the spec as it
>>> is can be implemented in its VXLAN variant with existing
>>> vswitches (e.g. OVS allows to choose the VXLAN destination port,
>>> ditto for the linux kernel stack). (ToR is certainly another
>>> story, most of them not having a flexible enough VXLAN dataplane
>>> nor support for any MPLS-over-IP.) WH> and how many ports
>>> simultaneously would they support? For this to work every tenant
>>> needs a different VXLAN UDP destination port/receive port. There
>>> might be SW elements that could do some of this, but IETF defines
>>> solutions which should be implemented across the board HW/SW/etc.
>>> Even if some SW switches can do this, the proposal will impose so
>>> many issues in HW/data-plane engines that I cannot be behind this
>>> solution. To make this work generically we will have to make
>>> changes anyhow. Given this, we better do it in the right way and
>>> guide the industry to a solution which does not imply those
>>> complexities. Otherwise we will stick with these specials forever
>>> with all consequences (bugs, etc). - snip - From:
>>> "[email-masked]
>>> <mailto:[email-masked]>>> Organization: Orange Date:
>>> Tuesday 17 November 2015 at 01:37 To: Wim Henderickx
>>> <[email-masked]
>>> <mailto:[email-masked]><mailto:[email-masked]
>>> <mailto:[email-masked]>>> Subject: Re: [bess]
>>> draft-hao-bess-inter-nvo3-vpn-optionc Hi Wim, WG, 2015-11-16,
>>> Henderickx, Wim (Wim): 2015-11-13, Henderickx, Wim (Wim): Thomas,
>>> we can discuss forever and someone need to describe requirements,
>>> but the current proposal I cannot agree to for the reasons
>>> explained. TM> Well, although discussing forever is certainly not
>>> the goal, the reasons for rejecting a proposal need to be
>>> thoroughly understood. WH> my point is what is the real driver
>>> for supporting a plain VXLAN data-plane here, the use cases I
>>> have seen in this txt is always where an application behind a
>>> NVE/TOR is demanding option c, but none of the NVE/TOR elements.
>>> My understanding is that the applications  are contexts where
>>> overlays are present is when workloads (VMs or baremetal) need to
>>> be interconnected with VPNs. In these contexts, there can be
>>> reasons to want Option C to reduce the state on ASBRs. In these
>>> context, its not the workload (VM or baremetal) that would
>>> typically handle VRFs, but really the vswitch or ToR. WH2> can it
>>> not be all cases: TOR/vswitch/Application. I would make the
>>> solution flexible to support all of these not? 2015-11-13,
>>> Henderickx, Wim (Wim): TM> The right trade-off to make may in
>>> fact depend on whether you prefer: (a) a new dataplane stitching
>>> behavior on DC ASBRs (the behavior specified in this draft) or
>>> (b) an evolution of the encaps on the vswitches and ToRs to
>>> support MPLS/MPLS/(UDP or GRE) WH> b depends on the use case I
>>> don't get what you mean by "b depends on the use case". WH> see
>>> my above comment. If the real use case is an application behind
>>> NVE/TOR requiring model C, than all the discussion on impact on
>>> driver/requirement for option c interworking with an IP based
>>> Fabric. Although I can agree than detailing requirements can
>>> always help, I don't think one can assume a certain application
>>> to dismiss the proposal. WH> for me the proposal is not
>>> acceptable for the reasons explained: too much impact on the
>>> data-planes I wrote the above based on the idea that the encap
>>> used in MPLS/MPLS/(UDP or GRE), which hence has to be supported
>>> on the ToRs and vswitches. Another possibility would be
>>> service-label/middle-label/Ethernet assuming an L2 adjacency
>>> between vswitches/ToRs and ASBRs, but this certainly does not
>>> match your typical DC architecture. Or perhaps had you something
>>> else in mind ? WH> see above. The draft right now also requires
>>> changes in existing TOR/NVE so for me all this discussion/debate
>>> is void. No, the spec as it is can be implemented in its VXLAN
>>> variant with existing vswitches (e.g. OVS allows to choose the
>>> VXLAN destination port, ditto for the linux kernel stack). (ToR
>>> is certainly another story, most of them not having a flexible
>>> enough VXLAN dataplane nor support for any MPLS-over-IP.) WH> and
>>> how many ports simultaneously would they support? WH> and
>>> depending on implementation you don’t need to change any of the
>>> TOR/vswitches. Does this mean that for some implementations you
>>> may not need to change any of the TOR/vswitches, but that for
>>> some others you may ? WH> any proposal on the table requires
>>> changes, so for me this is not a valid discussion See above, the
>>> proposal in the draft does not necessarily need changes in
>>> vswitches. Let me take a practical example : while I can quite
>>> easily see how to implement the procedures in
>>> draft-hao-bess-inter-nvo3-vpn-optionc based on current vswitch
>>> implementations of VXLAN, the lack of MPLS/MPLS/(UDP, GRE)
>>> support in commonplace vswitches seems to me as making that
>>> alternate solution you suggest harder to implement. WH> I would
>>> disagree to this. Tell me which switch/TOR handles multiple UDP
>>> ports for VXLAN ? I mentioned _v_switches, and many do support a
>>> variable destination port for VXLAN, which is sufficient to
>>> implement what the draft proposes. -Thomas From: Thomas Morin
>>> <[email-masked]
>>> <mailto:[email-masked]>>> Organization: Orange Date:
>>> Friday 13 November 2015 at 09:57 To: Wim Henderickx
>>> <[email-masked]
>>> <mailto:[email-masked]><mailto:[email-masked]
>>> <mailto:[email-masked]>>> Subject: Re: [bess]
>>> draft-hao-bess-inter-nvo3-vpn-optionc Hi Wim, I agree on the
>>> analysis that this proposal is restricted to implementations that
>>> supports the chosen encap with non-IANA ports (which may be hard
>>> to achieve for instance on hardware implementations, as you
>>> suggest), or to context where managing multiple IPs would be
>>> operationally viable. However, it does not seem obvious to me how
>>> the alternative you propose [relying on 3-label option C with an
>>> MPLS/MPLS/(UDP|GRE) encap] addresses the issue of whether the
>>> encap behavior is supported or not (e.g. your typical ToR chipset
>>> possibly may not support this kind of encap,  and even
>>> software-based switches may not be ready to support that today).
>>> implementations constraints we may have would have value. (+ one
>>> question below on VXLAN...) -Thomas 2015-11-12, Henderickx, Wim
>>> (Wim): On VXLAN/NVGRE, do you challenge the fact that they would
>>> be used with a dummy MAC address that would be replaced by the
>>> right MAC by a sender based on an ARP request when needed ? Is
>>> the above the issue you had in mind about VXLAN and NVGRE ? WH>
>>> yes I you don't mind me asking : why do you challenge that ?
>>>

---

From: <[email-masked]>
Date: Fri, 20 Nov 2015 10:05:30 +0100

2015-11-20, Haoweiguo:
> WH> ok now we have not discussed the constraints some HW vendors have
> with respect to global VNIDs. To make this work all VNID/Labels need
> to be globally unique. Hmmmmm
 > [weiguo2]:  In SDN scenario, a virtual
> network normally is represented by a global VN ID or MPLS VPN Label
> to simplify network management. I think it's not strange to allocate
> global unique MPLS VPN Label or VN ID for a virtual network now.

In an option C scenario you have to take into account what is done in 
the WAN. Globally-assigned labels are not an option there at all.

-Thomas

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: "Henderickx, Wim (Wim)" <[email-masked]>
Date: Fri, 20 Nov 2015 13:27:18 +0000

In-line with WH>

>been released recently but not present in most shipping ToRs yet, the 
WH> the next generation TOR chips seem to support it, at least this is the info I got from 3 vendors
>to implement -- we haven't seen clear information in this thread)
WH> in my view no, since we also have the global VNID issue and a continuous tax and ASBR boxes for this.

---

From: "Henderickx, Wim (Wim)" <[email-masked]>
Date: Fri, 20 Nov 2015 13:35:24 +0000

>the WAN. Globally-assigned labels are not an option there at all.

WH> Agreed. As I said before the proposal in the draft has too many implications/restrictions in existing network deployments + requires a dedicated data plane in ASBR devices, which we cannot reuse and will add a continuous tax in SW/HW (add cost for nothing).
This is why I am against this draft and we should drive the industry to a proper solution that does not have these implications.

---

From: "Henderickx, Wim (Wim)" <[email-masked]>
Date: Fri, 20 Nov 2015 13:49:51 +0000

In-line with WH2>

>Pls see inline with [weiguo2].

WH2> this is where we disagree since there are to many implication: global VNID, HW tax, etc
>[weiguo2]:  In SDN scenario, a virtual network normally is represented by a global VN ID or MPLS VPN Label to simplify network management. I think it's not strange to allocate global unique MPLS VPN…

WH2> I also disagree on this -> see other email/thread
>[weiguo2]:  I think it's better not to change current TOR hardware to realize option-C interconnection. No TOR hardware modification solution must be provided.

WH2> our mission in IETF to help drive the industry in the right direction, hence I am against this draft
> But for future case, we can propose extra hardware requirements for TORs.

---

From: Lucy yong <[email-masked]>
Date: Fri, 20 Nov 2015 16:48:53 +0000

Share my 2 cent.

Cloud providers want to tunnel its customer traffic through DC (AS)BR. Option C is a way to realize it. Both solutions summarized by Thomas have no change on WAN VPN side and seamlessly work with WAN VPN option C. However, to support either solution, DC has to do some enhancement on DC BR or ToR switch, etc, which dictates to different implementations within a DC. Option C pro and con remains regardless what implementation is used in a DC. 

Two solutions should not coexist in one DC (does not make a sense), but it does not matter if one DC operator prefers to use one solution and another DC prefers to use another solution. Since there are many cloud providers today, it is worth for the WG to document both solutions and point out the implementation requirements on impacted components. Then, up to vendors and operators to select a solution for their DC.

It does not make a sense for us to vote which solution is better here because a better solution for a DC depends on many factors.

Lucy

-----Original Message-----
From: BESS [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Friday, November 20, 2015 3:02 AM
To: Henderickx, Wim (Wim); [email-masked]
Subject: Re: [bess] draft-hao-bess-inter-nvo3-vpn-optionc

Well, it looks like the key questions are:
- when would ToR chips support MPLS/MPLS/UDP ?  (the generation that has been released recently but not present in most shipping ToRs yet, the next one ?)
- do we want an interim VXLAN-based solution ? (that will involve at best a performance penalty with existing chips, and at worse impossible to implement -- we haven't seen clear information in this thread)

-Thomas

>>> *发送时间:*2015年11月18日14:25 *收件人:[email-masked] *主题:*Re: [bess] 
>>> general to be considered widely available. Most of the TOR devices 
>>> today already pay a double-pass penalty when doing routing of 
>>> traffic in/out of vxlan-type tunnels. Only the newest generation can 
>>> route into tunnels without additional passes. And are definitively 
>>> limited in being able to arbitrary select UDP ports on a per tunnel 
>>> basis. In fact, most are even limited at using more than one VNID 
>>> per "service" of sorts. [Vincent]: Yes, the new generation BCM 
>>> chipset has already supported VXLAN routing without additional 
>>> passes. For OVS/TOR, VXLAN encapsulation is more popular than 
>>> MPLSoGRE/UDP, and has better performance. The IP-addressed based 
>>> implementation which would, I assume, imply assigning one or more 
>>> CIDRs to a loopback interface on the ASBR-d is also quite arbitrary 
>>> and does not seem like a technically "clean" solution. (besides 
>>> burning tons of IPs). As a side-note, most PE-grade routers i've 
>>> worked with were quite limited in terms of IP addresses used for 
>>> tunnel termination and it wasn't that just a simple pool can be 
>>> used. [Vincent]: I think the larger VTEP IP address range on ASBR-d 
>>> has no limitations.
>>> For the hardware on ASBR-d, it has capability to terminate multiple 
>>> VXLAN tunnels with arbitrary destination VTEP IP addresses. Wim's 
>>> mentions on cases where the Application itself, hosted in a 
>>> datacenter, would be part of the option-C interconnect, was 
>>> dismissed without much discussion so far, while, if we look in 
>>> detail at the type of users which will even consider a complex 
>>> topology like model-C its most likely users and operators very 
>>> familiar with MPLS VPNs in the WAN. Those type of operators will 
>>> most likely be very interested in deploying MPLS or WAN-grade 
>>> applications (i.e., virtual-routers or other
>>> machine). Also, the use of UDP ports at random would imply quite 
>>> complex logic on the ASBR-d IMHO. Im not saying its impossible, but 
>>> since the received packet now not only has to be received on a 
>>> random (though locally chosen) UDP port and this information 
>>> preserved in the pipeline to be able to do the 
>>> double-tunnel-stitching, it sounds quite complex. I am sure someone 
>>> in the list will mention this has already been implemented 
>>> somewhere, and I won't argue with that. And let's not even bring 
>>> into account what this would do to any DC middlebox that now has to 
>>> look at vxlan over almost any random port. We have to go back to the 
>>> "is it a 4 or is it a 6 in byte x" heuristics to try to guess 
>>> whether the packet is vxlan or just something entirely different 
>>> running over IP. [Vincent]: Using NP or multi-core CPU hardware 
>>> technology, it can be implemented although deeper packet inspection 
>>> is needed to perform UDP port and MPLS stitching. In general I feel 
>>> the proposed solution seems to be fitting of a specific use-case 
>>> which is not really detailed
>>> "elegantly" solve the issues at hand. It just feels like we're using 
>>> any available bit-space to stuff data into places that do not 
>>> necesarily belong. Yes, MPLS encapsulations on virtual switches are 
>>> not yet fully available, and there can be some performance penalty 
>>> on the TORs, but the solutions are much cleaner from a control and 
>>> data plane point of view. Maybe I'm too utopic. [Vincent]: I think 
>>> pure VXLAN solution has its scenario, it's general rather than 
>>> specific. We can't require all OVS/NVEs support VXLAN + MPLSoGRE at 
>>> the same time. Best regards, Diego
Hi,
>>> traditional MPLS VPN option-C. Currently, the most common practise 
>>> on ASBR-d is to terminate VXLAN encapsulation, look up local routing 
>>> table, and then perform MPLS encapsulation to the WAN network. 
>>> ASBR-d needs to maintain all VM's MAC/IP. In Option-C method, only 
>>> transport layer information needed to be maintained at GW/ASBR-d, 
>>> the scalability will be greatly enhanced. Traditonal Option-C is 
>>> only for MPLS VPN network interworking, because VXLAN is becoming 
>>> pervasive in data center, the solution in this draft was proposed 
>>> for the heterogeneous network interworking. The advantage of this 
>>> solution is that only VXLAN encapsulation is required for OVS/TOR. 
>>> Unlike Wim's solution, east-west bound traffic uses VXLAN encap, 
>>> while north-south bound traffic uses MPLSoGRE/UDP encap. There are 
>>> two solutions in this draft: 1. Using VXLAN tunnel destination IP 
>>> for stitching at ASBR-d. No data plane modification requirements on 
>>> solution 1, less IP address will be consumed for allocation. If UDP 
>>> port range is too large, we can combine with solution 1 and 2. In 
>>> this solution, both data plane modification changes are needed at 
>>> OVS/TOR and ASBR-d. ASBR-d also has capability to realize the 
>>> hardware changes. For OVS, it also can realize the data plane 
>>> changes. For TOR switch, it normally can't realize this function.  
>>> This solution mainly focuses on pure software based overlay network, 
>>> it has more scalability. In public cloud data center, software based 
>>> overlay network is the majority case. Whether using solution 1 or 2 
>>> depends on the operators real envionment. So I think our solution 
>>> has no flaws, it works fine.
>>> [[email-masked] <mailto:[email-masked]>] on behalf of 
>>> John E Drake [[email-masked] <mailto:[email-masked]>]
>>> Sent: Wednesday, November 18, 2015 2:49 To: Henderickx, Wim (Wim); 
>>> EXT - [email-masked] <mailto:[email-masked]>; BESS 
>>> Subject: Re: [bess] draft-hao-bess-inter-nvo3-vpn-optionc Hi, I 
>>> think Wim has conclusively demonstrated that this draft has fatal 
>>> we first figure out what problem we are trying to solve before 
>>> solving it. Yours Irrespectively, John From: BESS 
>>> draft-hao-bess-inter-nvo3-vpn-optionc — Snip — No, the spec as it is 
>>> can be implemented in its VXLAN variant with existing vswitches 
>>> (e.g. OVS allows to choose the VXLAN destination port, ditto for the 
>>> linux kernel stack). (ToR is certainly another story, most of them 
>>> not having a flexible enough VXLAN dataplane nor support for any 
>>> MPLS-over-IP.) WH> and how many ports simultaneously would they 
>>> support? For this to work every tenant needs a different VXLAN UDP 
>>> destination port/receive port. There might be SW elements that could 
>>> do some of this, but IETF defines solutions which should be 
>>> implemented across the board HW/SW/etc.
>>> solution. To make this work generically we will have to make changes 
>>> anyhow. Given this, we better do it in the right way and guide the 
>>> industry to a solution which does not imply those complexities. 
>>> Otherwise we will stick with these specials forever with all 
>>> consequences (bugs, etc). - snip - From:
>>> "[email-masked]
>>> Henderickx, Wim (Wim): 2015-11-13, Henderickx, Wim (Wim): Thomas, we 
>>> can discuss forever and someone need to describe requirements, but 
>>> the current proposal I cannot agree to for the reasons explained. 
>>> data-plane here, the use cases I have seen in this txt is always 
>>> where an application behind a NVE/TOR is demanding option c, but 
>>> none of the NVE/TOR elements.
>>> overlays are present is when workloads (VMs or baremetal) need to be 
>>> want Option C to reduce the state on ASBRs. In these context, its 
>>> not the workload (VM or baremetal) that would typically handle VRFs, 
>>> but really the vswitch or ToR. WH2> can it not be all cases: 
>>> TOR/vswitch/Application. I would make the solution flexible to 
>>> support all of these not? 2015-11-13, Henderickx, Wim (Wim): TM> The 
>>> right trade-off to make may in fact depend on whether you prefer: 
>>> specified in this draft) or
>>> (b) an evolution of the encaps on the vswitches and ToRs to support 
>>> MPLS/MPLS/(UDP or GRE) WH> b depends on the use case I don't get 
>>> what you mean by "b depends on the use case". WH> see my above 
>>> comment. If the real use case is an application behind NVE/TOR 
>>> requiring model C, than all the discussion on impact on NVE/TOR is 
>>> void. As such I want to have a discussion on the real 
>>> Fabric. Although I can agree than detailing requirements can always 
>>> help, I don't think one can assume a certain application to dismiss 
>>> the proposal. WH> for me the proposal is not acceptable for the 
>>> reasons explained: too much impact on the data-planes I wrote the 
>>> above based on the idea that the encap used in MPLS/MPLS/(UDP or 
>>> GRE), which hence has to be supported on the ToRs and vswitches. 
>>> certainly does not match your typical DC architecture. Or perhaps 
>>> had you something else in mind ? WH> see above. The draft right now 
>>> also requires changes in existing TOR/NVE so for me all this 
>>> discussion/debate is void. No, the spec as it is can be implemented 
>>> in its VXLAN variant with existing vswitches (e.g. OVS allows to 
>>> choose the VXLAN destination port, ditto for the linux kernel 
>>> stack). (ToR is certainly another story, most of them not having a 
>>> flexible enough VXLAN dataplane nor support for any MPLS-over-IP.) 
>>> WH> and how many ports simultaneously would they support? WH> and 
>>> TOR/vswitches. Does this mean that for some implementations you may 
>>> not need to change any of the TOR/vswitches, but that for some 
>>> others you may ? WH> any proposal on the table requires changes, so 
>>> for me this is not a valid discussion See above, the proposal in the 
>>> draft does not necessarily need changes in vswitches. Let me take a 
>>> practical example : while I can quite easily see how to implement 
>>> the procedures in draft-hao-bess-inter-nvo3-vpn-optionc based on 
>>> current vswitch implementations of VXLAN, the lack of 
>>> making that alternate solution you suggest harder to implement. WH> 
>>> I would disagree to this. Tell me which switch/TOR handles multiple 
>>> UDP ports for VXLAN ? I mentioned _v_switches, and many do support a 
>>> supports the chosen encap with non-IANA ports (which may be hard to 
>>> achieve for instance on hardware implementations, as you suggest), 
>>> or to context where managing multiple IPs would be operationally 
>>> viable. However, it does not seem obvious to me how the alternative 
>>> you propose [relying on 3-label option C with an
>>> MPLS/MPLS/(UDP|GRE) encap] addresses the issue of whether the encap 
>>> behavior is supported or not (e.g. your typical ToR chipset possibly 
>>> may not support this kind of encap,  and even software-based 
>>> switches may not be ready to support that today).
>>> (Wim): On VXLAN/NVGRE, do you challenge the fact that they would be 
>>> used with a dummy MAC address that would be replaced by the right 
>>> MAC by a sender based on an ARP request when needed ? Is the above 
>>> the issue you had in mind about VXLAN and NVGRE ? WH> yes I you 
>>> don't mind me asking : why do you challenge that ?
>>>

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, France Telecom - Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci

This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorization.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, France Telecom - Orange shall not be liable if this message was modified, changed or falsified.
Thank you.

---

From: John E Drake <[email-masked]>
Date: Fri, 20 Nov 2015 16:53:34 +0000

Lucy,

That presupposes that the group likes either of the two proposed solutions in your draft.  

Yours Irrespectively,

John

> From: BESS [mailto:[email-masked]] On Behalf Of Lucy yong
> Sent: Friday, November 20, 2015 11:49 AM
> To: EXT - [email-masked]; Henderickx, Wim (Wim);
> [email-masked]
> Cloud providers want to tunnel its customer traffic through DC (AS)BR.
> Option C is a way to realize it. Both solutions summarized by Thomas have no
> change on WAN VPN side and seamlessly work with WAN VPN option C.
> However, to support either solution, DC has to do some enhancement on DC
> BR or ToR switch, etc, which dictates to different implementations within a
> DC. Option C pro and con remains regardless what implementation is used in
> a DC.
> 
> Two solutions should not coexist in one DC (does not make a sense), but it
> does not matter if one DC operator prefers to use one solution and another
> DC prefers to use another solution. Since there are many cloud providers
> today, it is worth for the WG to document both solutions and point out the
> implementation requirements on impacted components. Then, up to
> vendors and operators to select a solution for their DC.
> 
> It does not make a sense for us to vote which solution is better here because
> a better solution for a DC depends on many factors.
> 
> 
> Lucy
> 
> 
> 
> 
> 
> been released recently but not present in most shipping ToRs yet, the next
> one ?)
> - do we want an interim VXLAN-based solution ? (that will involve at best a
> performance penalty with existing chips, and at worse impossible to
> implement -- we haven't seen clear information in this thread)
> 
> -Thomas
> 
> 
> >> Zhuangshunwan  :
> >>>
> >>> Hi Diego,
> >>>
> >>> <mailto:wim.henderickx@alcatel-
> lucent.com><mailto:wim.henderickx@alc
> >>> atel-lucent.com
> >>>
> >>>
> we
> >>> <mailto:wim.henderickx@alcatel-
> lucent.com><mailto:wim.henderickx@alc
> >>> atel-lucent.com
> >>>
> >>>
> encap
> __________________________________________________________
> __________________________________________________________
> _____
> 
> confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites
> ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez
> le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les
> messages electroniques etant susceptibles d'alteration, France Telecom -
> Orange decline toute responsabilite si ce message a ete altere, deforme ou
> falsifie. Merci
> 
> This message and its attachments may contain confidential or privileged
> information that may be protected by law; they should not be distributed,
> used or copied without authorization.
> As emails may be altered, France Telecom - Orange shall not be liable if this
> message was modified, changed or falsified.
> Thank you.
>

---

From: Lucy yong <[email-masked]>
Date: Fri, 20 Nov 2015 16:58:45 +0000

That is my 2 cents. Anyone can share his/her opinion. :)
Lucy

-----Original Message-----
From: John E Drake [mailto:[email-masked]] 
Sent: Friday, November 20, 2015 10:54 AM
To: Lucy yong; EXT - [email-masked]; Henderickx, Wim (Wim); [email-masked]
Subject: RE: [bess] draft-hao-bess-inter-nvo3-vpn-optionc

Lucy,

Yours Irrespectively,

John

> Option C is a way to realize it. Both solutions summarized by Thomas 
> have no change on WAN VPN side and seamlessly work with WAN VPN option C.
> However, to support either solution, DC has to do some enhancement on 
> DC BR or ToR switch, etc, which dictates to different implementations 
> within a DC. Option C pro and con remains regardless what 
> implementation is used in a DC.
> 
> Two solutions should not coexist in one DC (does not make a sense), 
> but it does not matter if one DC operator prefers to use one solution 
> and another DC prefers to use another solution. Since there are many 
> cloud providers today, it is worth for the WG to document both 
> solutions and point out the implementation requirements on impacted 
> components. Then, up to vendors and operators to select a solution for their DC.
> 
> It does not make a sense for us to vote which solution is better here 
> because a better solution for a DC depends on many factors.
> 
> 
> Lucy
> 
> 
> 
> 
> 
> > support for this. I hope some HW vendors of TOR chips shime in, but 
> > I am told the MPLS solution is possible in the next generation chips 
> > they are working on.
> 
> - when would ToR chips support MPLS/MPLS/UDP ?  (the generation that 
> has been released recently but not present in most shipping ToRs yet, 
> the next one ?)
> best a performance penalty with existing chips, and at worse 
> impossible to implement -- we haven't seen clear information in this 
> thread)
> 
> -Thomas
> 
> 
> >> Zhuangshunwan  :
> >>>
> >>> Hi Diego,
> >>>
> >>> traffic in/out of vxlan-type tunnels. Only the newest generation 
> >>> can route into tunnels without additional passes. And are 
> >>> definitively limited in being able to arbitrary select UDP ports 
> >>> on a per tunnel basis. In fact, most are even limited at using 
> >>> more than one VNID per "service" of sorts. [Vincent]: Yes, the new 
> >>> generation BCM chipset has already supported VXLAN routing without 
> >>> additional passes. For OVS/TOR, VXLAN encapsulation is more 
> >>> popular than MPLSoGRE/UDP, and has better performance. The 
> >>> IP-addressed based implementation which would, I assume, imply 
> >>> assigning one or more CIDRs to a loopback interface on the ASBR-d 
> >>> is also quite arbitrary and does not seem like a technically 
> >>> "clean" solution. (besides burning tons of IPs). As a side-note, 
> >>> most PE-grade routers i've worked with were quite limited in terms 
> >>> of IP addresses used for tunnel termination and it wasn't that 
> >>> just a simple pool can be used. [Vincent]: I think the larger VTEP 
> >>> IP address range on ASBR-d has no limitations.
> >>> interconnect, was dismissed without much discussion so far, while, 
> >>> if we look in detail at the type of users which will even consider 
> >>> a complex topology like model-C its most likely users and 
> >>> operators very familiar with MPLS VPNs in the WAN. Those type of 
> >>> operators will most likely be very interested in deploying MPLS or 
> >>> WAN-grade applications (i.e., virtual-routers or other
> >>> virtual machine). Also, the use of UDP ports at random would imply 
> >>> quite complex logic on the ASBR-d IMHO. Im not saying its 
> >>> someone in the list will mention this has already been implemented 
> >>> into account what this would do to any DC middlebox that now has 
> >>> to look at vxlan over almost any random port. We have to go back 
> >>> to the "is it a 4 or is it a 6 in byte x" heuristics to try to 
> >>> guess whether the packet is vxlan or just something entirely 
> >>> different running over IP. [Vincent]: Using NP or multi-core CPU 
> >>> hardware technology, it can be implemented although deeper packet 
> >>> inspection is needed to perform UDP port and MPLS stitching. In 
> >>> general I feel the proposed solution seems to be fitting of a 
> >>> specific use-case which is not really detailed
> >>> cleaner from a control and data plane point of view. Maybe I'm too 
> >>> utopic. [Vincent]: I think pure VXLAN solution has its scenario, 
> >>> it's general rather than specific. We can't require all OVS/NVEs 
> >>> support VXLAN + MPLSoGRE at the same time. Best regards, Diego
> >>> ------------------------------------------------------------------
> >>> --
> >>> -------------
> >>>
> >>>
> Hi,
> >>> on ASBR-d is to terminate VXLAN encapsulation, look up local 
> >>> routing table, and then perform MPLS encapsulation to the WAN network.
> >>> for stitching at ASBR-d. No data plane modification requirements 
> >>> on OVS or TOR switches, only hardware changes on ASBR-d. ASBR-d 
> >>> UDP port range is too large, we can combine with solution 1 and 2. 
> >>> In this solution, both data plane modification changes are needed 
> >>> at OVS/TOR and ASBR-d. ASBR-d also has capability to realize the 
> >>> This solution mainly focuses on pure software based overlay 
> >>> network, it has more scalability. In public cloud data center, 
> >>> software based overlay network is the majority case. Whether using 
> >>> solution 1 or 2 depends on the operators real envionment. So I 
> >>> think our solution has no flaws, it works fine.
> >>> flaws and I don’t support it.  I also agree with his suggestion 
> >>> that we first figure out what problem we are trying to solve 
> >>> before solving it. Yours Irrespectively, John From: BESS 
> >>> is can be implemented in its VXLAN variant with existing vswitches 
> >>> (e.g. OVS allows to choose the VXLAN destination port, ditto for 
> >>> the linux kernel stack). (ToR is certainly another story, most of 
> >>> them not having a flexible enough VXLAN dataplane nor support for 
> >>> any
> >>> destination port/receive port. There might be SW elements that 
> >>> could do some of this, but IETF defines solutions which should be 
> >>> guide the industry to a solution which does not imply those complexities.
> >>> TM> Well, although discussing forever is certainly not the goal, 
> >>> TM> the
> >>> WH> my point is what is the real driver for supporting a plain 
> >>> WH> VXLAN
> >>> typically handle VRFs, but really the vswitch or ToR. WH2> can it not be all cases:
> >>> support all of these not? 2015-11-13, Henderickx, Wim (Wim): TM> 
> >>> The right trade-off to make may in fact depend on whether you prefer:
> >>> don't get what you mean by "b depends on the use case". WH> see my 
> >>> above comment. If the real use case is an application behind 
> >>> always help, I don't think one can assume a certain application to 
> >>> dismiss the proposal. WH> for me the proposal is not acceptable 
> >>> for the reasons explained: too much impact on the data-planes I 
> >>> wrote the above based on the idea that the encap used in 
> >>> MPLS/MPLS/(UDP or GRE), which hence has to be supported on the ToRs and vswitches.
> >>> assuming an L2 adjacency between vswitches/ToRs and ASBRs, but 
> >>> this certainly does not match your typical DC architecture. Or 
> >>> perhaps had you something else in mind ? WH> see above. The draft 
> >>> right now also requires changes in existing TOR/NVE so for me all 
> >>> this discussion/debate is void. No, the spec as it is can be 
> >>> implemented in its VXLAN variant with existing vswitches (e.g. OVS 
> >>> allows to choose the VXLAN destination port, ditto for the linux 
> >>> kernel stack). (ToR is certainly another story, most of them not 
> >>> having a flexible enough VXLAN dataplane nor support for any 
> >>> MPLS-over-IP.)
> >>> may not need to change any of the TOR/vswitches, but that for some 
> >>> others you may ? WH> any proposal on the table requires changes, 
> >>> so for me this is not a valid discussion See above, the proposal 
> >>> in the draft does not necessarily need changes in vswitches. Let 
> >>> me take a practical example : while I can quite easily see how to 
> >>> multiple UDP ports for VXLAN ? I mentioned _v_switches, and many 
> >>> do support a variable destination port for VXLAN, which is 
> >>> sufficient to implement what the draft proposes. -Thomas From: 
> >>> Thomas Morin <[email-masked]
> >>>
> >>> behavior is supported or not (e.g. your typical ToR chipset 
> >>> right MAC by a sender based on an ARP request when needed ? Is the 
> >>> above the issue you had in mind about VXLAN and NVGRE ? WH> yes I 
> >>> you don't mind me asking : why do you challenge that ?
> >>>
> 
> confidentielles ou privilegiees et ne doivent donc pas etre diffuses, 
> exploites ou copies sans autorisation. Si vous avez recu ce message 
> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi 
> que les pieces jointes. Les messages electroniques etant susceptibles 
> d'alteration, France Telecom - Orange decline toute responsabilite si 
> ce message a ete altere, deforme ou falsifie. Merci
> 
> privileged information that may be protected by law; they should not 
> be distributed, used or copied without authorization.

---

From: <[email-masked]>
Date: Fri, 20 Nov 2015 18:04:03 +0100

> That presupposes that the group likes either of the two proposed solutions in your draft.

John, I think Lucy's "two solutions" was referring to 
draft-hao-bess-inter-nvo3-vpn-optionc solution and the 3-label Optionc 
MPLS/MPLS/UDP solution described by Wim.

-Thomas

---

From: John E Drake <[email-masked]>
Date: Fri, 20 Nov 2015 17:18:54 +0000

Lucy,

My apologies, I misunderstood.

I think we have to accept the fact that we will have to deal with a multiplicity of different encapsulations in the data plane along a packet's e2e path and that we should take a more measured approach to deciding how to deal with this in a general and extensible way before accepting any solutions. 
  
Yours Irrespectively,

John

> From: [email-masked] [mailto:[email-masked]]
> Sent: Friday, November 20, 2015 12:04 PM
> To: John E Drake; Lucy yong; Henderickx, Wim (Wim); [email-masked]
> John, I think Lucy's "two solutions" was referring to draft-hao-bess-inter-
> nvo3-vpn-optionc solution and the 3-label Optionc MPLS/MPLS/UDP solution
> described by Wim.
> 
> -Thomas
> 
> 
> 
> >
> <mailto:[email-masked]>;
> >> BESS
> complexities.
> my
> >> If you have received this email in error, please notify the sender
> >> and delete this message and its attachments.

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Fri, 20 Nov 2015 17:51:08 +0000

+1

-----Original Message-----
From: BESS [mailto:[email-masked]] On Behalf Of John E Drake
Sent: Friday, November 20, 2015 12:19 PM
To: EXT - [email-masked]; Lucy yong; Henderickx, Wim (Wim); [email-masked]
Subject: Re: [bess] draft-hao-bess-inter-nvo3-vpn-optionc

Lucy,

  
Yours Irrespectively,

John

---

From: "Rabadan, Jorge (Jorge)" <[email-masked]>
Date: Fri, 20 Nov 2015 19:31:47 +0000

IMHO if TOR chip vendors can confirm they are seriously looking at MPLS/MPLS/UDP, Wim’s suggestion makes all the sense since we know it works and scales.
My 2 cents.

Jorge

---

From: Lucy yong <[email-masked]>
Date: Fri, 20 Nov 2015 19:57:46 +0000

IMHO: voting on this thread does not make a sense. Both solutions will work and scales.

Lucy 

-----Original Message-----
From: Rabadan, Jorge (Jorge) [mailto:[email-masked]] 
Sent: Friday, November 20, 2015 1:32 PM
To: UTTARO, JAMES; John E Drake; EXT - [email-masked]; Lucy yong; Henderickx, Wim (Wim); [email-masked]
Subject: Re: [bess] draft-hao-bess-inter-nvo3-vpn-optionc

Jorge

On 11/20/15, 9:51 AM, "BESS on behalf of UTTARO, JAMES" <[email-masked] on behalf of [email-masked]> wrote:

>> > That presupposes that the group likes either of the two proposed 
>> > solutions
>> in your draft.
>> 
>> draft-hao-bess-inter- nvo3-vpn-optionc solution and the 3-label 
>> Optionc MPLS/MPLS/UDP solution described by Wim.
>> 
>> -Thomas
>> 
>> 
>> 
>> >
>> >> Option C is a way to realize it. Both solutions summarized by 
>> >> Thomas have no change on WAN VPN side and seamlessly work with WAN 
>> >> VPN
>> option C.
>> >> However, to support either solution, DC has to do some enhancement 
>> >> on DC BR or ToR switch, etc, which dictates to different 
>> >> implementations within a DC. Option C pro and con remains 
>> >> regardless what implementation is used in a DC.
>> >>
>> >> Two solutions should not coexist in one DC (does not make a 
>> >> sense), but it does not matter if one DC operator prefers to use 
>> >> one solution and another DC prefers to use another solution. Since 
>> >> there are many cloud providers today, it is worth for the WG to 
>> >> document both solutions and point out the implementation 
>> >> requirements on impacted components. Then, up to vendors and 
>> >> operators to select a solution for
>> their DC.
>> >>
>> >> It does not make a sense for us to vote which solution is better 
>> >> here because a better solution for a DC depends on many factors.
>> >>
>> >>
>> >> Lucy
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>> support for this. I hope some HW vendors of TOR chips shime in, 
>> >>> but I am told the MPLS solution is possible in the next 
>> >>> generation chips they are working on.
>> >>
>> >> - when would ToR chips support MPLS/MPLS/UDP ?  (the generation 
>> >> that has been released recently but not present in most shipping 
>> >> ToRs yet, the next one ?)
>> >> - do we want an interim VXLAN-based solution ? (that will involve 
>> >> at best a performance penalty with existing chips, and at worse 
>> >>>>> routing of traffic in/out of vxlan-type tunnels. Only the 
>> >>>>> newest generation can route into tunnels without additional 
>> >>>>> passes. And are definitively limited in being able to arbitrary 
>> >>>>> select UDP ports on a per tunnel basis. In fact, most are even 
>> >>>>> limited at using more than one VNID per "service" of sorts. 
>> >>>>> [Vincent]: Yes, the new generation BCM chipset has already 
>> >>>>> supported VXLAN routing
>> without
>> >>>>> assigning one or more CIDRs to a loopback interface on the 
>> >>>>> ASBR-d is also quite arbitrary and does not seem like a 
>> >>>>> technically "clean" solution. (besides burning tons of IPs). As 
>> >>>>> a side-note, most PE-grade routers i've worked with were quite 
>> >>>>> limited in terms of IP addresses used for tunnel termination 
>> >>>>> and it wasn't that just a simple pool can be used. [Vincent]: I 
>> >>>>> think the larger VTEP IP address range on ASBR-d has no limitations.
>> >>>>> addresses. Wim's mentions on cases where the Application 
>> >>>>> itself, hosted in a datacenter, would be part of the option-C 
>> >>>>> while, if we look in detail at the type of users which will 
>> >>>>> even consider a complex topology like model-C its most likely 
>> >>>>> users and operators very familiar with MPLS VPNs in the WAN. 
>> >>>>> Those type of operators will most likely be very interested in 
>> >>>>> deploying MPLS or WAN-grade applications (i.e., virtual-routers 
>> >>>>> or other
>> >>>>> VNFs) in the DC and thus its highly likely that the 
>> >>>>> interconnect would not terminate at the NVE itself but rather 
>> >>>>> the TS (the virtual machine). Also, the use of UDP ports at 
>> >>>>> random would imply quite complex logic on the ASBR-d IMHO. Im 
>> >>>>> not saying its impossible, but since the received packet now 
>> >>>>> not only has to be received on a random (though locally chosen) 
>> >>>>> UDP port and this information preserved in the pipeline to be 
>> >>>>> able to do the double-tunnel-stitching, it sounds quite 
>> >>>>> complex. I am sure someone in the list will mention this has 
>> >>>>> already been implemented somewhere, and I won't argue with 
>> >>>>> that. And let's not even bring into account what this would do 
>> >>>>> to any DC middlebox that now has to look at vxlan over almost 
>> >>>>> any random port. We have to go back to the "is it a 4 or is it 
>> >>>>> a 6 in byte x" heuristics to try to guess whether the packet is 
>> >>>>> vxlan or just something entirely different running over IP. 
>> >>>>> [Vincent]: Using NP or multi-core CPU hardware technology, it 
>> >>>>> can be implemented although deeper packet inspection is needed 
>> >>>>> to perform UDP port and MPLS stitching. In general I feel the 
>> >>>>> proposed solution seems to be fitting of a specific use-case which is not really detailed
>> >>>>> scenario, it's general rather than specific. We can't require 
>> >>>>> all OVS/NVEs support VXLAN + MPLSoGRE at the same time. Best 
>> >>>>> regards, Diego
>> >>>>> WAN
>> network.
>> >>>>> ASBR-d needs to maintain all VM's MAC/IP. In Option-C method, 
>> >>>>> only transport layer information needed to be maintained at 
>> >>>>> GW/ASBR-d, the scalability will be greatly enhanced. Traditonal 
>> >>>>> Option-C is only for MPLS VPN network interworking, because 
>> >>>>> VXLAN is
>> becoming
>> >>>>> pervasive in data center, the solution in this draft was 
>> >>>>> proposed for the heterogeneous network interworking. The 
>> >>>>> advantage of this solution is that only VXLAN encapsulation is required for OVS/TOR.
>> >>>>> Unlike Wim's solution, east-west bound traffic uses VXLAN 
>> >>>>> encap, while north-south bound traffic uses MPLSoGRE/UDP encap. 
>> >>>>> There
>> are
>> >>>>> two solutions in this draft: 1. Using VXLAN tunnel destination 
>> >>>>> IP for stitching at ASBR-d. No data plane modification 
>> >>>>> requirements on OVS or TOR switches, only hardware changes on 
>> >>>>> ASBR-d. ASBR-d normally is router, it has capability to realize 
>> >>>>> the hardware changes. It will consume many IP addresses and the 
>> >>>>> IP pool for allocation needs to be configured on ASBR-d 
>> >>>>> beforehand. 2. Using VXLAN destination UDP port for stitching 
>> >>>>> at ASBR-d. Compared with solution 1, less IP address will be 
>> >>>>> consumed for allocation. If UDP port range is too large, we can combine with solution 1 and 2.
>> >>>>> In this solution, both data plane modification changes are 
>> >>>>> data plane changes. For TOR switch, it normally can't realize this function.
>> >>>>> software based overlay network is the majority case. Whether 
>> >>>>> using solution 1 or 2 depends on the operators real envionment. 
>> >>>>> [[email-masked] <mailto:[email-masked]>] on 
>> >>>>> think Wim has conclusively demonstrated that this draft has 
>> >>>>> fatal flaws and I don’t support it.  I also agree with his 
>> >>>>> suggestion that we first figure out what problem we are trying 
>> >>>>> to solve before solving it. Yours Irrespectively, John From: 
>> >>>>> BESS [mailto:[email-masked] 
>> >>>>> <mailto:[email-masked]>] On Behalf Of Henderickx, Wim 
>> >>>>> (Wim) Sent: Tuesday, November 17, 2015
>> >>>>> draft-hao-bess-inter-nvo3-vpn-optionc — Snip — No, the spec as 
>> >>>>> it is can be implemented in its VXLAN variant with existing 
>> >>>>> vswitches (e.g. OVS allows to choose the VXLAN destination 
>> >>>>> port, ditto for the linux kernel stack). (ToR is certainly 
>> >>>>> another story, most of them not having a flexible enough VXLAN 
>> >>>>> dataplane nor support for any
>> >>>>> support? For this to work every tenant needs a different VXLAN 
>> >>>>> UDP destination port/receive port. There might be SW elements 
>> >>>>> that could do some of this, but IETF defines solutions which 
>> >>>>> should be implemented across the board HW/SW/etc.
>> >>>>> Even if some SW switches can do this, the proposal will impose 
>> >>>>> so many issues in HW/data-plane engines that I cannot be behind 
>> >>>>> this solution. To make this work generically we will have to 
>> >>>>> make changes anyhow. Given this, we better do it in the right 
>> >>>>> way and guide the industry to a solution which does not imply 
>> >>>>> those
>> complexities.
>> >>>>> can discuss forever and someone need to describe requirements, 
>> >>>>> but the current proposal I cannot agree to for the reasons explained.
>> >>>>> TM> Well, although discussing forever is certainly not the 
>> >>>>> TM> goal, the
>> >>>>> data-plane here, the use cases I have seen in this txt is 
>> >>>>> always where an application behind a NVE/TOR is demanding 
>> >>>>> option c, but none of the NVE/TOR elements.
>> >>>>> overlays are present is when workloads (VMs or baremetal) need 
>> >>>>> to be interconnected with VPNs. In these contexts, there can be 
>> >>>>> typically handle VRFs, but really the vswitch or ToR. WH2> can 
>> >>>>> it not
>> be all cases:
>> >>>>> support all of these not? 2015-11-13, Henderickx, Wim (Wim): 
>> >>>>> TM> The right trade-off to make may in fact depend on whether 
>> >>>>> you
>> prefer:
>> >>>>> (a) a new dataplane stitching behavior on DC ASBRs (the 
>> >>>>> behavior specified in this draft) or
>> >>>>> NVE/TOR is void. As such I want to have a discussion on the 
>> >>>>> real driver/requirement for option c interworking with an IP 
>> >>>>> based Fabric. Although I can agree than detailing requirements 
>> >>>>> can always help, I don't think one can assume a certain 
>> >>>>> application to dismiss the proposal. WH> for me the proposal is 
>> >>>>> not acceptable for the reasons explained: too much impact on 
>> >>>>> the data-planes I wrote the above based on the idea that the 
>> >>>>> encap used in MPLS/MPLS/(UDP or GRE), which hence has to be 
>> >>>>> supported on the
>> ToRs and vswitches.
>> >>>>> perhaps had you something else in mind ? WH> see above. The 
>> >>>>> draft right now also requires changes in existing TOR/NVE so 
>> >>>>> for me all this discussion/debate is void. No, the spec as it 
>> >>>>> vswitches (e.g. OVS allows to choose the VXLAN destination 
>> >>>>> port, ditto for the linux kernel stack). (ToR is certainly 
>> >>>>> another story, most of them not having a flexible enough VXLAN 
>> >>>>> dataplane nor support for any
>> >>>>> MPLS-over-IP.)
>> >>>>> changes, so for me this is not a valid discussion See above, 
>> >>>>> the proposal in the draft does not necessarily need changes in 
>> >>>>> multiple UDP ports for VXLAN ? I mentioned _v_switches, and 
>> >>>>> many do support a variable destination port for VXLAN, which is 
>> >>>>> analysis that this proposal is restricted to implementations 
>> >>>>> that supports the chosen encap with non-IANA ports (which may 
>> >>>>> be hard to achieve for instance on hardware implementations, as 
>> >>>>> you suggest), or to context where managing multiple IPs would 
>> >>>>> be operationally viable. However, it does not seem obvious to 
>> >>>>> me how the alternative you propose [relying on 3-label option C 
>> >>>>> with an
>> >>>>> implementations constraints we may have would have value. (+ 
>> >>>>> one question below on VXLAN...) -Thomas 2015-11-12, Henderickx, 
>> >>>>> Wim
>> >>>>> (Wim): On VXLAN/NVGRE, do you challenge the fact that they 
>> >>>>> would be used with a dummy MAC address that would be replaced 
>> >>>>> by the right MAC by a sender based on an ARP request when 
>> >>>>> needed ? Is the above the issue you had in mind about VXLAN and 
>> >>>>> NVGRE ? WH> yes I you don't mind me asking : why do you challenge that ?
>> >>>>>
>> >>
>> >>
>> >> confidentielles ou privilegiees et ne doivent donc pas etre 
>> >> diffuses, exploites ou copies sans autorisation. Si vous avez recu 
>> >> ce message par erreur, veuillez le signaler a l'expediteur et le 
>> >> detruire ainsi que les pieces jointes. Les messages electroniques 
>> >> etant susceptibles d'alteration, France Telecom - Orange decline 
>> >> toute responsabilite si ce message a ete altere, deforme ou 
>> >> falsifie. Merci
>> >>
>> >> privileged information that may be protected by law; they should 
>> >> not be distributed, used or copied without authorization.
>> >> As emails may be altered, France Telecom - Orange shall not be 
>> >> liable if this message was modified, changed or falsified.
>> >> Thank you.
>> >>

---

From: Martin Vigoureux <[email-masked]>
Date: Fri, 20 Nov 2015 21:13:05 +0100

Lucy,

there is no such thing as voting in IETF WGs
And I haven't seen anything like voting as part of this discussion.

Thank you
Martin

---

From: Lucy yong <[email-masked]>
Date: Fri, 20 Nov 2015 20:36:58 +0000

Martin,

Sorry not to express my mind precisely.

I mean to say, debating which solution is a better solution here does not make a sense. There is no need for the two solutions interwork.

Regards,
Lucy

-----Original Message-----
From: BESS [mailto:[email-masked]] On Behalf Of Martin Vigoureux
Sent: Friday, November 20, 2015 2:13 PM
To: [email-masked]
Subject: Re: [bess] draft-hao-bess-inter-nvo3-vpn-optionc

Lucy,

there is no such thing as voting in IETF WGs And I haven't seen anything like voting as part of this discussion.

Thank you
Martin

Le 20/11/2015 20:57, Lucy yong a écrit :
> To: UTTARO, JAMES; John E Drake; EXT - [email-masked]; Lucy 
> yong; Henderickx, Wim (Wim); [email-masked]

---

From: "Rabadan, Jorge (Jorge)" <[email-masked]>
Date: Fri, 20 Nov 2015 23:34:37 +0000

As you said, Lucy, "that is my 2 cents. Anyone can share their opinion” :-)

---

From: Martin Vigoureux <[email-masked]>
Date: Sat, 21 Nov 2015 01:01:33 +0100

Lucy,

Thanks for your clarification.
Yet, the fact that two solutions need not interwork does not render 
useless a debate on the merits of each.

-m

---

From: Haoweiguo <[email-masked]>
Date: Sat, 21 Nov 2015 06:21:11 +0000

Jorge,
We all know that Wim's MPLS/MPLS/UDP solution works, but it's not the only choice. Wim's solution requires MPLSoGRE/UDP encap in data center, but many data centers only use VXLAN/NVGRE encapsulation for both north-south and east-west bound traffic, how to interconnect with outside MPLS VPN network for these data centers? So VXLAN/NVGRE and MPLS VPN network interworking is needed.
And for the interconnection solution, we suggest both no TOR/NVE hardware enhancement solution and future proof solution should be provided.
Thanks,
weiguo
________________________________________
From: BESS [[email-masked]] on behalf of Rabadan, Jorge (Jorge) [[email-masked]]
Sent: Saturday, November 21, 2015 3:31
To: UTTARO, JAMES; John E Drake; EXT - [email-masked]; Lucy yong; Henderickx, Wim (Wim); [email-masked]
Subject: Re: [bess] draft-hao-bess-inter-nvo3-vpn-optionc

Jorge

On 11/20/15, 9:51 AM, "BESS on behalf of UTTARO, JAMES" <[email-masked] on behalf of [email-masked]> wrote:

>> >>>>> draft-hao-bess-inter-nvo3-vpn-optionc ― Snip ― No, the spec as it

---

From: Zhuangshunwan <[email-masked]>
Date: Sat, 21 Nov 2015 07:12:16 +0000

+1

More popular deployments in data center is VXLAN/NVGRE, so the VXLAN/NVGRE and MPLS VPN network interconnecting is necessary. If all TOR/NVEs support MPLSoGRE/oUDP, Wim's solution also can be used, but currently it's not practical to require all TOR/NVEs to support MPLS/MPLS/UDPorGRE.

Vincent
________________________________________
From: Haoweiguo
Sent: Saturday, November 21, 2015 14:21
To: Rabadan, Jorge (Jorge); UTTARO, JAMES; John E Drake; EXT - [email-masked]; Lucy yong; Henderickx, Wim (Wim); [email-masked]
Subject: RE: [bess] draft-hao-bess-inter-nvo3-vpn-optionc

Jorge,

Jorge

---

From: Haoweiguo <[email-masked]>
Date: Sat, 21 Nov 2015 07:17:52 +0000

Hi Jim & John,
Firstly i think option-C interconnection between cloud data center and MPLS VPN network exists. Secondly, i think for different encapsulations, different interconnecting methods are needed. For VXLAN/NVGRE, VTEP IP/UDP Port allocation solutions are needed. Only for MPLSoGRE/oUDP, both standard MPLS+MPLS+GRE/UDP and VTEP IP/UDP Port allocation solution can work, it shoud be measured which one is more suitable or both solutions are proposed to the industry for reference.
Thanks,
weiguo

________________________________________
From: BESS [[email-masked]] on behalf of UTTARO, JAMES [[email-masked]]
Sent: Saturday, November 21, 2015 1:51
To: John E Drake; EXT - [email-masked]; Lucy yong; Henderickx, Wim (Wim); [email-masked]
Subject: Re: [bess] draft-hao-bess-inter-nvo3-vpn-optionc

+1

-----Original Message-----
From: BESS [mailto:[email-masked]] On Behalf Of John E Drake
Sent: Friday, November 20, 2015 12:19 PM
To: EXT - [email-masked]; Lucy yong; Henderickx, Wim (Wim); [email-masked]
Subject: Re: [bess] draft-hao-bess-inter-nvo3-vpn-optionc

Lucy,

Yours Irrespectively,

John

---

From: "Henderickx, Wim (Wim)" <[email-masked]>
Date: Sat, 21 Nov 2015 09:47:00 +0000

It also works with VXLAN/GPE btw. On top the solution I outlined is future proof as well.

The key question is do people want a native VXLAN/NVGRE solution in the DC using existing TOR HW and/or vwitches. This will imply dealing with:
- Global VNID/labels -> driven by current TOR HW (most people will never deploy this)
- We have not spoken about the MAC manipulation in all details but will most likely require a routing lookup in the TOR and will limit performance in some HW TOR(s).

- An implementation on ASBR that only is required for this use case. It will be very expensive to carry forward -> the cost versus the benefit is big and if people want a uniform data plane VXLAN-GPE is an option which is very close to native VXLAN.

---

From: John E Drake <[email-masked]>
Date: Sat, 21 Nov 2015 14:52:29 +0000

Weiguo,

Comment inline.

Yours Irrespectively,

John

> From: BESS [mailto:[email-masked]] On Behalf Of Haoweiguo
> Sent: Saturday, November 21, 2015 2:18 AM
> Firstly i think option-C interconnection between cloud data center and MPLS
> VPN network exists. Secondly, i think for different encapsulations, different
> interconnecting methods are needed. For VXLAN/NVGRE, VTEP IP/UDP Port
> allocation solutions are needed.

[JD]  I don't think this is either the best or the only solution and I don't think we should be in any rush to adopt it. 

> Only for MPLSoGRE/oUDP, both standard
> MPLS+MPLS+GRE/UDP and VTEP IP/UDP Port allocation solution can work, it
> shoud be measured which one is more suitable or both solutions are
> proposed to the industry for reference.
> Thanks,
> weiguo
> 
> I think we have to accept the fact that we will have to deal with a multiplicity
> of different encapsulations in the data plane along a packet's e2e path and
> that we should take a more measured approach to deciding how to deal with
> this in a general and extensible way before accepting any solutions.
> 
> Yours Irrespectively,
> 
> John
> 
> > >> but it does not matter if one DC operator prefers to use one
> > >> solution and another DC prefers to use another solution. Since
> > >>> but I am told the MPLS solution is possible in the next generation
> > >>> chips they are working on.
> > >>
> > >>>>> *发件人:*BESS [mailto:[email-masked]] *代表 *Diego
> Garcia
> > >> del Rio
> > >>>>> *发送时间:*2015年11月18日14:25 *收件人:[email-masked] *主
> 题
> > >> :*Re: [bess]
> > >>>>> routing
> > >>>>> it wasn't that just a simple pool can be used. [Vincent]: I
> > >>>>> and operators very familiar with MPLS VPNs in the WAN. Those
> > >>>>> type of operators will most likely be very interested in
> > >>>>> x" heuristics to try to guess whether the packet is vxlan or
> > >>>>> just something entirely different running over IP. [Vincent]:
> > >>>>> Using NP or multi-core CPU hardware technology, it can be
> > >>>>> implemented although deeper packet inspection is needed to
> > >>>>> perform UDP port and MPLS stitching. In general I feel the
> > >>>>> beforehand. 2. Using VXLAN destination UDP port for stitching at
> > >>>>> ASBR-d. Compared with solution 1, less IP address will be
> behalf
> > >>>>> draft-hao-bess-inter-nvo3-vpn-optionc ― Snip ― No, the spec as
> > >>>>> nor support for any
> explained.
> > >>>>> on the
> > >>>>> draft right now also requires changes in existing TOR/NVE so for
> > >>>>> me all this discussion/debate is void. No, the spec as it is can
> > >>>>> be implemented in its VXLAN variant with existing vswitches
> > >>>>> the linux kernel stack). (ToR is certainly another story, most
> > >>>>> of them not having a flexible enough VXLAN dataplane nor support
> > >>>>> for any
> > >>>>> MPLS-over-IP.)
> > >>>>> MPLS/MPLS/(UDP, GRE) support in commonplace vswitches seems
> to
> > >> me as
> > >>>>> that supports the chosen encap with non-IANA ports (which may be
> > >>>>> hard to achieve for instance on hardware implementations, as you
> > >>>>> operationally viable. However, it does not seem obvious to me
> > >>>>> how the alternative you propose [relying on 3-label option C
> > >>>>> with an
> would