Subject: [nvo3] Comments on http://tools.ietf.org/html/draft-quinn-vxlan-gpe-03

From: David Melman <[email-masked]>
Date: Sun, 27 Jul 2014 15:39:54 +0000

Following this thread, I have a few comments:

1.  To avoid confusion, drop the text about backward compatibility with VXLAN.

2. The VXLAN-GPE draft should focus only on the VXLAN-GPE header and requires the assignment of a new UDP port.   The fact that the VXLAN-GPE header closely resembles VXLAN may be convenient for implementers, but this protocol by definition is not backward compatible with VXLAN.

3. True, the 'P' bit is not needed for backward compatibility, but I'm not against it if there is value to make it consistent with the LISP-GPE header.

David Melman

---

From: Dino Farinacci <[email-masked]>
Date: Sun, 27 Jul 2014 14:21:52 -0700

> 2. The VXLAN-GPE draft should focus only on the VXLAN-GPE header and requires the assignment of a new UDP port.   The fact that the VXLAN-GPE header closely resembles VXLAN may be convenient for im…

If you do that then it will be harder for VXLAN-GPE systems to interoperate with a VXLAN systems. Because a VXLAN-GPE system will need to open and maintain 2 UDP sockets. And an implementaiton will have to be careful to not set the P-bit for the VXLAN socket or clear the P-bit for VXLAN-GPE socket. This is all completely unnecessary and one way or the other should be used.

And if *it was agreed* on to use different UDP port numbers (like the way LISP did it for L2 versus L3 packet encapsulation), we wouldn't need the P-bit at all. But there was push back (by somebody) to not allocate another port for VXLAN, so the demux was forced to be in the VXLAN header. 

And is also the reason this baggage is being carried over the LISP when it really isn't needed.

> 3. True, the ‘P’ bit is not needed for backward compatibility, but I’m not against it if there is value to make it consistent with the LISP-GPE header.

There is no incremental benefit to use the P-bit for LISP. We had a solution but because of the requirement to have no new port for VXLAN, LISP is affected.

Just another example how the working group is putting effort into things that creates more work but no benefit. Don't get me wrong, the cisco guys did this (the VXLAN and LISP same position for P-bit) for consistency, and they should be applauded for that. But if VXLAN could have another port number assigned for "other protocols", maybe the VXLAN-GPE would look so much different.

Something to think about as the working group now has new productivity mentality.

Dino

---

From: Tom Herbert <[email-masked]>
Date: Sun, 27 Jul 2014 17:28:01 -0700

> If you do that then it will be harder for VXLAN-GPE systems to interoperate with a VXLAN systems. Because a VXLAN-GPE system will need to open and maintain 2 UDP sockets. And an implementaiton will…
I am not sure what you are suggesting. AFAICT there is no backwards
compatible means to to add the protocol field to VXLAN which is the
motivation for the new UDP port, which in turn implies a new protocol
(which also implies an opportunity to add a more general set of
protocol features like version number and options extensions which are
also not backwards compatible). Maybe it's possible to break
compatibility within the protocol and assume that out of band
mechanisms could negotiate use of P-bit to compensate, but I assume
there's already quite a bit of VXLAN deployment so that seems pretty
shaky robustness-wise to me.

It's not just adding the protocol field that would be an issue, even
adding the OAM bit to VXLAN would be problematic. Per VXLAN spec
unspecified flag bits are ignored on receive, so if the OAM bit were
subsequently defined in VXLAN it will be ignored by existing
implementations and these packets will be processed normally-- this
seems to be incompatible with the proposed VXLAN-gpe requirement that
"When the O bit is set to 1, the packet is an OAM packet and OAM
processing MUST occur." (btw 'OAM processing' is awfully ambiguous to
be a MUST here IMO).

Allowing the reserved bits in the header to be ignored on receive
limits the usefulness in that new bits that are defined can only be
advisory and not fundamentally change interpretation of the packet.
Had the requirement in VXLAN been that packets with unknown bits set
be dropped, then adding P-bit and O-bit could have been done with
backwards compatibility. This might be a reasonable requirement to
consider if new protocol (i.e. new port number) is undertaken.

Thanks,
Tom

> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3

---

From: Dino Farinacci <[email-masked]>
Date: Sun, 27 Jul 2014 20:39:11 -0400

I am going to let other people explain. I think my email was clear. 

Dino

> On Jul 27, 2014, at 8:28 PM, Tom Herbert <[email-masked]> wrote:
>

---

From: Marc Binderberger <[email-masked]>
Date: Sun, 27 Jul 2014 23:58:48 -0700

Hello Tom,

I'm all for precise wording - but one shouldn't get lost in the wording :-)

The intention of the "compatibility" described in the draft should be clear, 
although the word is confusing: you can re-use your "gpe" send/receive logic 
to send/receive the existing VxLAN encaps. All you need is to send/receive on 
4789/udp instead of the to-be-defined gpe port and set the flags/fields 
accordingly.

The point Dino makes is (I think): is there really a gain to "re-invent" 
VxLAN and LISP data encapsulation? And it might be actually easier to 
implement and test separate encodings instead of the more complex combined 
header in VxLAN-gpe.

The cost of not combining is a potential additional nth header for SFC and 
whatever else may come.

> advisory and not fundamentally change interpretation of the packet.

I agree with your statement in the 2nd half but not your opinion about the 
usefulness. Don't see a problem here, LISP is a good example how ignoring 
unknown flags works well.

Regards, Marc

On Sun, 27 Jul 2014 17:28:01 -0700, Tom Herbert wrote:
>>> 2. The VXLAN-GPE draft should focus only on the VXLAN-GPE header and 
>>> requires the assignment of a new UDP port.   The fact that the VXLAN-GPE 
>>> header closely resembles VXLAN may be convenient for implementers, but 
>>> this protocol by definition is not backward compatible with VXLAN.
>> 
>> If you do that then it will be harder for VXLAN-GPE systems to 
>> interoperate with a VXLAN systems. Because a VXLAN-GPE system will need to 
>> open and maintain 2 UDP sockets. And an implementaiton will have to be 
>> careful to not set the P-bit for the VXLAN socket or clear the P-bit for 
>> VXLAN-GPE socket. This is all completely unnecessary and one way or the 
>> other should be used.
>> 
>> And if *it was agreed* on to use different UDP port numbers (like the way 
>> LISP did it for L2 versus L3 packet encapsulation), we wouldn't need the 
>> P-bit at all. But there was push back (by somebody) to not allocate 
>> another port for VXLAN, so the demux was forced to be in the VXLAN header.
>> 
>>> 3. True, the ‘P’ bit is not needed for backward compatibility, but I’m 
>>> not against it if there is value to make it consistent with the LISP-GPE 
>>> header.
>> 
>> There is no incremental benefit to use the P-bit for LISP. We had a 
>> solution but because of the requirement to have no new port for VXLAN, 
>> LISP is affected.
>> 
>> Just another example how the working group is putting effort into things 
>> that creates more work but no benefit. Don't get me wrong, the cisco guys 
>> did this (the VXLAN and LISP same position for P-bit) for consistency, and 
>> they should be applauded for that. But if VXLAN could have another port 
>> number assigned for "other protocols", maybe the VXLAN-GPE would look so 
>> much different.
>>

---

From: Dino Farinacci <[email-masked]>
Date: Mon, 28 Jul 2014 06:14:02 -0400

> implement and test separate encodings instead of the more complex combined 

Exactly. 

Dino

---

From: Haoweiguo <[email-masked]>
Date: Mon, 28 Jul 2014 12:24:15 +0000

About backward compatibility, i also agree with Dino. VXLAN-GPE should focus on  the VXLAN-GPE header and requires the assignment of a new UDP port, the data format don't need consider backward compatibility with VXLAN header. If NVE1 support VXLAN-GPE, another NVE2 support VXLAN, the two NVE can communicate with each other, no matter inter-subnet layer 3 traffic or intra-subnet layer 2 traffic.
When ingress NVE sends traffic to egress NVE, ingress NVE should use the data format of egress NVE, i.e., when NVE1 sends traffic to NVE2, NVE1 should use VXLAN encapsulation. When NVE2 sends traffic to NVE1, NVE1 should use VXLAN-GPE header.
Although VXLAN currently only support Ethernet in UDP,  layer 3 inter-subnet traffic forwarding between NVEs can also be supported, inner destination MAC can be used to differenciate inter-subnet layer 3 traffic from intra-subnet layer 2 traffic on egress NVE, inner destination MAC should be set gateway interface MAC of egress NVE in inter-subnet communication case.
Thanks
weiguo
________________________________________
发件人: Dino Farinacci [[email-masked]]
发送时间: 2014年7月28日 8:39
收件人: Tom Herbert
抄送: David Melman; [email-masked]; LISP mailing list list
主题: Re: [nvo3] Comments on http://tools.ietf.org/html/draft-quinn-vxlan-gpe-03

Dino

---

From: Dino Farinacci <[email-masked]>
Date: Mon, 28 Jul 2014 08:15:39 -0500

> On Jul 28, 2014, at 7:24 AM, Haoweiguo <[email-masked]> wrote:
> 
> About backward compatibility, i also agree with Dino. VXLAN-GPE should focus on  the VXLAN-GPE header and requires the assignment of a new UDP port, the data format don't need consider backward compatibility with VXLAN header. I

I want to make it clear that supporting backward compatibility is very important since VXLAN-port-4789 is supported in various chips already. 

Dino

---

From: Haoweiguo <[email-masked]>
Date: Mon, 28 Jul 2014 13:55:46 +0000

Hi Dino,
Sorry, i misunderstood you. I think VXLAN-GPE can define a new UDP port and a new data format, P bit in VXLAN-GPE seems to have no any value. As for basic inter-subnet layer 3 communication and intra-subnet layer 2 communication between NVEs, current NVGRE, VXLAN and LISP have already supported, NVGRE,VXLAN,LISP and VXLAN-GPE can be hybrid used to form a NVO3 network if only basic layer 3 and layer 2 forwarding process exists. As for some extra functions of OAM, service chaining,and etc,  only VXLAN-GPE can support, pure VXLAN-GPE network should be used in these cases.
Thanks
weiguo

________________________________________
发件人: Dino Farinacci [[email-masked]]
发送时间: 2014年7月28日 21:15
收件人: Haoweiguo
抄送: Tom Herbert; David Melman; [email-masked]; LISP mailing list list
主题: Re: 答复: [nvo3] Comments on http://tools.ietf.org/html/draft-quinn-vxlan-gpe-03

Dino

---

From: Dino Farinacci <[email-masked]>
Date: Mon, 28 Jul 2014 07:17:53 -0700

> Hi Dino,
> Sorry, i misunderstood you. I think VXLAN-GPE can define a new UDP port and a new data format, P bit 

No worries. 

> in VXLAN-GPE seems to have no any value. As for basic inter-subnet layer 3 communication and intra-subnet layer 2 communication between NVEs, current NVGRE, VXLAN and LISP have already supported, 

VXLAN supports L2 overlays since its goal is to extend subnets. LISP supports L3 overlays so it assumes subnets are local (to the xTR) just like in a routed network. NVGRE can be a combo.

> NVGRE,VXLAN,LISP and VXLAN-GPE can be hybrid used to form a NVO3 network if only basic layer 3 and 

There is motivation to extend an encapsulation header (which is called VXLAN-GPE) so it can support, most importantly NSH. That change also gives VXLAN to support encapsulating layer-2 IPv4 and IPv6, which is duplicate functionality of LISP. But the headers are so similar, it really doens't matter.

However, the P-bit is not needed for anything new in LISP and the OAM-bit is not needed in LISP since LISP has different UDP port number (4342) for control-packets. VXLAN does not have a well defined control protocol so the data-plane has to escape out control-plane pakcets where the first one is this new OAM message.

> layer 2 forwarding process exists. As for some extra functions of OAM, service chaining,and etc,  only VXLAN-GPE can support, pure VXLAN-GPE network should be used in these cases.
> Thanks
> weiguo

Right, agree.

Dino

---

From: Tom Herbert <[email-masked]>
Date: Mon, 28 Jul 2014 08:08:31 -0700

> unknown flags works well.
How so? Adding protocol field to LISP-gpe has the same backwards
compatibility problem as VXLAN-gpe, but is resolving it in a different
way. The addition of the P-bit relies on some (presumably) out of band
mechanism to ensure protocol compatibility. From the LISP-gpe draft:

"A LISP-gpe router MUST not encapsulate non-IP packets to a LISP
router.  A method for determining the capabilities of a LISP router
(gpe or "legacy") is out of the scope of this draft."

Thanks,
Tom

---

From: "Darrel Lewis (darlewis)" <[email-masked]>
Date: Mon, 28 Jul 2014 17:22:01 +0000

>> implement and test separate encodings instead of the more complex combined 

And the authors believe there is value.

-Darrel

---

From: Fabio Maino <[email-masked]>
Date: Mon, 28 Jul 2014 11:09:47 -0700

> (gpe or "legacy") is out of the scope of this draft."

Tom,
with the LISP control plane (as with many other control planes) it's 
trivial to encode the capability of the receiving end of a tunnel (ETR) 
in the mapping itself.  When the sender (ITR) encapsulates the packet it 
can then "choose" the appropriate data path encoding.

The P bit helps taking advantage of similarities between LISP-GPE and 
LISP, and would allow an implementation to send LISP-GPE packets (P=1) 
containing IPv4/v6 encapsulated packets to a legacy LISP router, on the 
UDP port 4341 (LISP). The legacy LISP router, per LISP specification, 
will ignore the P bit and the next protocol field (as long as N,E,V = 
0). This allows to transition LISP fabrics to LISP-GPE without the need 
for a new UDP port. Given we have quite a large installed base of LISP 
devices, this is one of the design goals of the LISP-GPE protocol.

The same applies to VXLAN-GPE, when you use a control plane with the 
characteristics above (e.g. LISP). Note that both LISP and VXLAN define 
reserved bits as set to 0 in tx, ignore in rx.

Since the GPE drafts are about data plane, the method for determining 
the capabilities of the receiving end (GPE or "legacy") is out of the 
scope of those drafts (will certainly be in scope of control plane 
drafts, though).

Thanks,
Fabio

---

From: Dino Farinacci <[email-masked]>
Date: Mon, 28 Jul 2014 11:13:56 -0700

> On Jul 28, 2014, at 11:09 AM, Fabio Maino <[email-masked]> wrote:
> 
> The P bit helps taking advantage of similarities between LISP-GPE and LISP, and would allow an implementation to send LISP-GPE packets (P=1) containing IPv4/v6 encapsulated packets to a legacy LISP router, on the UDP port 4341

Fabio - this could be misleading to make people think without the P-bit that you could not encapsulate IPv4/IPv6/MAC. And we all now we support that in LISP today. 

Dino

---

From: Marc Binderberger <[email-masked]>
Date: Mon, 28 Jul 2014 18:39:40 +0000

Hello Darrel,

> And the authors believe there is value.

otherwise you wouldn't have written it down ;-)

That's what this discussion is (also) about: learning what are the pros
and cons and how to weight them.

Regards, Marc

---

From: Tom Herbert <[email-masked]>
Date: Mon, 28 Jul 2014 12:04:30 -0700

> Tom,
> with the LISP control plane (as with many other control planes) it's trivial
> to encode the capability of the receiving end of a tunnel (ETR) in the
> mapping itself.  When the sender (ITR) encapsulates the packet it can then
> "choose" the appropriate data path encoding.
>
> The P bit helps taking advantage of similarities between LISP-GPE and LISP,
> and would allow an implementation to send LISP-GPE packets (P=1) containing
> IPv4/v6 encapsulated packets to a legacy LISP router, on the UDP port 4341
> (LISP). The legacy LISP router, per LISP specification, will ignore the P
> bit and the next protocol field (as long as N,E,V = 0). This allows to
> transition LISP fabrics to LISP-GPE without the need for a new UDP port.
> Given we have quite a large installed base of LISP devices, this is one of
> the design goals of the LISP-GPE protocol.
>
> Since the GPE drafts are about data plane, the method for determining the
> capabilities of the receiving end (GPE or "legacy") is out of the scope of
> those drafts (will certainly be in scope of control plane drafts, though).

If negotiation is available then the precise set of flags can be
negotiated, so if an unexpected or unknown flag is received it's
obviously invalid and a reasonable recourse is to drop the packet. If
negotiation is essential before communication (which so far seems to
be true for network virtualization), the set of possible flags (or
options) can be constrained a priori so that need to ignore unknown
flags would be moot, by design an end host should never receive an
unknown flag. By this same rationale, I tend to think the "Critical
options" bit is unnecessary in geneve. IMO if a sender took the time
to set a flag or an option it must have done that with the intent that
the receiver (or someone in the network) will consume it.

The caveat to this is that intermediate devices may also need to parse
packets but not necessarily participate in the protocol control plane.
Consider a stateless firewall that parses LISP packets to filter on
encapsulated IP destination address. If the P-bit is implemented in
the end hosts, but we've neglected to update all the firewalls in the
path-- the packet will be misinterpreted if say the firewall sees a
LISP packet with an encapsulated Ethernet frame. In this case,
hopefully the firewall will drop the packet, but there's no guarantee
of that-- the behavior is non-deterministic. Maintaining compatibility
with such devices is a hard problem and might imply constraints on new
options that could fundamental change parsing or interpretation of the
packet (adding protocol type is a good example case).

Thanks,
Tom

>>>>> And is also the reason this baggage is being carried over the LISP when
>>>>> it
>>>>> really isn't needed.
>>>>>

---

From: Dino Farinacci <[email-masked]>
Date: Mon, 28 Jul 2014 12:12:08 -0700

> packet (adding protocol type is a good example case).

I think your point here is (and if I interpreted incorrectly, I'll make a new point then), that if the encapsulated packet is L3 or L2 and described with a port number versus a bit in the payload after the UDP header, it would be easier for firewalls to decide when to filter one versus the other?

Dino

---

From: Tom Herbert <[email-masked]>
Date: Mon, 28 Jul 2014 13:12:22 -0700

> I think your point here is (and if I interpreted incorrectly, I'll make a new point then), that if the encapsulated packet is L3 or L2 and described with a port number versus a bit in the payload a…
My primary concern is not how easy it will be for firewalls to add new
filtering, but rather we can make changes to protocols to ensure
backwards compatibility and robustness. It is much better in the DC
for devices to drop packets for things don't understand rather than to
risk misinterpreting them which leads some non-deterministic behavior.

Generally, I'd say that we'd want an encapsulation protocol that
contains next protocol type field which is always valid as an
invariant-- no P-bit indicating validity of the protocol field and no
need to assign a UDP port for each protocol we might want to
encapsulate (this doesn't scale and we've already gotten push-back
against this model). Processing the protocol type field adds little
overhead especially if we can get that down to 8-bits. GRE/UDP
proposal is a good example of how this model might work, for cost of
one UDP port and four bytes of packet overhead we'll be able to
encapsulate any Ether_type over UDP. Adding support for this in
firewalls for this should be straightforward.

If the choices to extend an existing protocol are between adding P-bit
and assigning a UDP port per each protocol, I still think the P-bit
would be better since it is the more generic solution, but is going to
be more work to ensure backwards compatibility if flags can be ignored
on RX.

Tom

---

From: Dino Farinacci <[email-masked]>
Date: Mon, 28 Jul 2014 14:24:22 -0700

> be more work to ensure backwards compatibility if flags can be ignored

But we are not starting over from scratch. There are implementations that have been around for years. We don't have to luxury you say. And as I said before, every time you add a feature you will need to change the implementation anyways and deal with compatibility. So making a header general with tons of TLV options isn't necessarily the right approach FOR A DATA-PLANE. Versus just changing the header for the new requirements you see in front of you at the time you design the header.

And I mentioned this before at a previous NVO3 meeting. In ~30 years since the IP header was defined, how much have we, in practice, used IP options.

Dino

---

From: Tom Herbert <[email-masked]>
Date: Mon, 28 Jul 2014 14:52:28 -0700

> But we are not starting over from scratch. There are implementations that have been around for years. We don't have to luxury you say. And as I said before, every time you add a feature you will ne…
I agree, at this level we don't need tons of TLV options, but we do
require some basic protocol extensibility-- so to me at least GRE
model is reasonable. As I have said several times now, for network
virtualization we absolutely must have some mechanism to
validate/authenticate the vnid at the receiver. I think the L2TP
cookie is a good starting point at least, and I know that I can do
this simply by appending a 64 bit field in the header like described
in GUE or maybe defining a TLV in geneve-- still not sure how this
could work with VXLAN.

> And I mentioned this before at a previous NVO3 meeting. In ~30 years since the IP header was defined, how much have we, in practice, used IP options.
But, how many TCP options have been defined? The reason that we don't
define new IP options is because nearly all routers assume zero
options for fast path, and so we can never get reasonable forwarding
performance with new options. Fortunately, we've been able to extend
TCP to get what we need for the most part, but we have had to make
concessions because this IP options have been effectively obsoleted (I
imagine something like ECN might have been done differently if IP
options were a realistic option).

Thanks,
Tom

---

From: Dino Farinacci <[email-masked]>
Date: Mon, 28 Jul 2014 15:02:02 -0700

> cookie is a good starting point at least, and I know that I can do

That is why we created the nonce in LISP. It is already there, we don't need an extensible header for it. I was told by many service providers that a changing 24-bit value would be sufficient.

> in GUE or maybe defining a TLV in geneve-- still not sure how this

It can't as defined other than it sets the N-bit in the header (from LISP) and uses the nonce field. But again, we are getting design feature merge conflicts and it is being used as a protocol demux.

And allocating a port per protocol is not as bad as you think. There won't be that many of them. At leaset we have to discpline ourselves to not create so many (unlike what is going on right now).

> But, how many TCP options have been defined? The reason that we don't

Software dude.

> define new IP options is because nearly all routers assume zero

Hardware dude.

> options for fast path, and so we can never get reasonable forwarding

Right, and why do they do that? The more things you put in the more slower and costly hardware gets.

Did you hear Puneet's comment at the mic last week. He is from Broadcom, he has the battle scars on his back. He has tons of experience with this. We NEED TO LISTEN TO HIM!!!

Dino

---

From: "Darrel Lewis (darlewis)" <[email-masked]>
Date: Tue, 29 Jul 2014 00:38:27 +0000

> be more work to ensure backwards compatibility if flags can be ignored

I think this statement summarizes the trade-offs.  

-Darrel

---

From: Marc Binderberger <[email-masked]>
Date: Mon, 28 Jul 2014 20:23:08 -0700

Hello Tom,

my point is your "limits the usefulness". There is some negative vibe and I 
do not agree with that :-)

As mentioned, I agree with the statement that ignore-on-receive flags mean 
you can add new functionality on top of existing ones but not fundamentally 
change things. From practical experience with LISP control plane - which has 
ignore-on-receive reserved fields - this works quite well as it allows to 
add-on features without breaking the fundamental interop with older 
implementations; you have to do it "right" to ensure a system not 
understanding the new feature behaves safe or at least reasonable but it's 
do-able. Thus my reference to LISP, sorry this was probably confusing.

For your LISP-gpe comment Fabio answered already how it could work.

Generally speaking about flags, having both, ignore-when-unknown flags and 
drop-when-unknown flags would be nice IMHO. Problem is our limited space of 
64bit shim header though.

Regards, Marc

---

From: Dino Farinacci <[email-masked]>
Date: Mon, 28 Jul 2014 21:34:15 -0700

> 64bit shim header though.

The flag bits in the LISP header are not used that way. They are enable-bits so you can overload fields and keep the header small. The flags are designed based on "enable-bits" in many hardware designs where the enable-bits indicate which parallel signal lines are in effect (voltage, verify, ECC, parity, etc).

The bits do not mean anything about how the packet is handled but what control data is transmitted with  the data-packet.

Dino

---

From: Marc Binderberger <[email-masked]>
Date: Mon, 28 Jul 2014 21:52:13 -0700

Hello Dino,

thanks for this comment - that's an interesting point of view.

Hmm, how do the O (or RA in another draft) and the P bit fit into this 
picture?  They do mean something about how the packet is handled, don't they?

Regards, Marc

On Mon, 28 Jul 2014 21:34:15 -0700, Dino Farinacci wrote:
> The flag bits in the LISP header are not used that way. They are 
> enable-bits so you can overload fields and keep the header small. The flags 
> are designed based on "enable-bits" in many hardware designs where the 
> enable-bits indicate which parallel signal lines are in effect (voltage, 
> verify, ECC, parity, etc).
> 
> The bits do not mean anything about how the packet is handled but what 
> control data is transmitted with  the data-packet.
> 
> Dino
>

---

From: Dino Farinacci <[email-masked]>
Date: Mon, 28 Jul 2014 22:08:46 -0700

> On Jul 28, 2014, at 9:52 PM, Marc Binderberger <[email-masked]> wrote:
> 
> Hello Dino,
> 

O-bit in LISP is not needed. I told the LUSP-GPE authors this. Because OAM-packets like all other control packets go in UDP port 4342 (where encapsulated packets go in UDP port 4341). 

And P-bit is not needed and the authors indicated the main motivation was to keep consistent with VXLAN so same hardware design could do both. LISP already has port 8473 for L2 frames. And NSH is a service protocol and should run above the transport layer so should have its own port and can address packets anywhere. Just like any other UDP application. If that packet needs to be encapsulated that is a lower level function. Just like IP packets can go in an MPLS based LSP. 

> picture?  They do mean something about how the packet is handled, don't they?

I won't answer that because those bit introductions into the design are indeed design bugs IMO. 

Dino

---

From: Haoweiguo <[email-masked]>
Date: Tue, 29 Jul 2014 10:58:17 +0000

Hi Dino,
Thanks for your detail comments.  As for current VXLAN encapsulation, pls see my comments inline with [weiguo] below.

[weiguo]: Yes, an new encapsulation header should be extended to support NSH. But as for IPv4 and IPv6, i think current VXLAN already supported.  For the layer 3 inter-subnet traffic from NVE1 to NVE2, inner destination MAC is the gateway interface MAC at NVE2.  For the layer 2 intra-subnet traffic from NVE1 to NVE2,  inner destination MAC is the destination TS's MAC. When NVE2 receives VXLAN encapsulated traffic from NVE1, inner destination MAC can be used to differentiate layer 2 traffic from layer 3 traffic. VXLAN distributed layer 3 gateway can be realized through the mechanism, NVE can forward both intra-subnet layer 2 traffic and inter-subnet layer 3 traffic at the same time.

Thanks
weiguo 
________________________________________
发件人: Dino Farinacci [[email-masked]]
发送时间: 2014年7月28日 22:17
收件人: Haoweiguo
抄送: David Melman; [email-masked]; LISP mailing list list; Tom Herbert
主题: Re: [nvo3] Comments on http://tools.ietf.org/html/draft-quinn-vxlan-gpe-03

No worries.

Right, agree.

Dino

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 29 Jul 2014 05:18:08 -0700

I understand what you are saying but it comes down to what the encapsulator is encapsulating. If the packet it is encapsulating is an L2 frame then the encapsulator is supporting a layer-2 overlay service. Otherwise it is a a layer-3 overlay service. 

If the dest MAC Is to the encapsulator that MAC header is stripped before a layer-3 forwarding decision is made. Then, at which time the packet is forwarded natively or encapsulated.

If the dest MAC is not the encapsulator's address, then the MAC header stays with the packet/frame and then encapsulated. 

The former is a layer-3 overlay and the latter is a layer-2 overlay where both can be supported at the same time in one encapsulator device. 

The former is LISP encapsulation and the latter is VXLAN encapsulation. 

With VXLAN-GPE both cases can be supported (and most importantly with a single unified control-plane). 

So I believe VXLAN-GPE could be useful yet redundant but LISP requires no changes (if LISP supports  L2 which is documented in draft-smith-lisp-layer-2 using a different port number). 

Dino

> On Jul 29, 2014, at 3:58 AM, Haoweiguo <[email-masked]> wrote:
> 
> Hi Dino,

---

From: Haoweiguo <[email-masked]>
Date: Wed, 30 Jul 2014 03:15:19 +0000

Hi Dino,
VXLAN can use a unified encapsulation to realize both intra-subnet layer 2 traffic forwarding and inter-subnet layer 3 traffic forwarding at the same time, inner destination MAC is used to differenciate layer 2 traffic from layer 3 traffic. 
LISP uses two different encapsulations for intra-subnet layer 2 traffic forwarding and inter-subnet layer 3 traffic forwarding, UDP port is used to differenciate layer 2 traffic from layer 3 traffic.
From theoretical perspective,  both the two solutions are applicable. From commercial use perspective, the former(unified encap for both layer 2 and layer 3) seems to be more popular currently.
Thanks
weiguo
________________________________________
发件人: Dino Farinacci [[email-masked]]
发送时间: 2014年7月29日 20:18
收件人: Haoweiguo
抄送: David Melman; [email-masked]; LISP mailing list list; Tom Herbert
主题: Re: 答复: [nvo3] Comments on http://tools.ietf.org/html/draft-quinn-vxlan-gpe-03

Dino

---

From: Marc Binderberger <[email-masked]>
Date: Tue, 29 Jul 2014 23:40:19 -0700

Hello Weiguo,

I would say there is a difference if your "layer-3" service is actually a 
layer-2 Ethernet frame with the destination MAC of the encapsulated packet 
being identical with the MAC of the decapsulating system, compared to a 
layer-3 service where your whole processing stays solely in the layer-3, no 
MAC involved.

If I want to define a layer-3 encapsulation (like LISP, like the proposed 
VxLAN-gpe with IPv4/v6 as protocol) then having any layer-2 in the process 
seems a bit "odd". Worse, the involved systems may not have this layer-2 
logic. LISP for example is often used in the WAN, even when some 
encapsulator/decapsulator are DC switches. Some of these WAN systems may be 
pure IPv4/v6 routers.

In other words: having both a layer-2 (VxLAN) and a layer-3 (LISP) 
encapsulation makes good sense. The question is: shall we keep it separated 
as it is today as "LISP" and "VxLAN" or combine the (expensive) multiple data 
planes into one VxLAN-gpe.

Regards, Marc

On Wed, 30 Jul 2014 03:15:19 +0000, Haoweiguo wrote:
> Hi Dino,
> VXLAN can use a unified encapsulation to realize both intra-subnet layer 2 
> traffic forwarding and inter-subnet layer 3 traffic forwarding at the same 
> time, inner destination MAC is used to differenciate layer 2 traffic from 
> layer 3 traffic. 
> LISP uses two different encapsulations for intra-subnet layer 2 traffic 
> forwarding and inter-subnet layer 3 traffic forwarding, UDP port is used to 
> differenciate layer 2 traffic from layer 3 traffic.
> From theoretical perspective,  both the two solutions are applicable. From 
> commercial use perspective, the former(unified encap for both layer 2 and 
> layer 3) seems to be more popular currently.
> Thanks
> weiguo
> 主题: Re: 答复: [nvo3] Comments on 
> http://tools.ietf.org/html/draft-quinn-vxlan-gpe-03
> 
> I understand what you are saying but it comes down to what the encapsulator 
> is encapsulating. If the packet it is encapsulating is an L2 frame then the 
> encapsulator is supporting a layer-2 overlay service. Otherwise it is a a 
> layer-3 overlay service.
> 
> If the dest MAC Is to the encapsulator that MAC header is stripped before a 
> layer-3 forwarding decision is made. Then, at which time the packet is 
> forwarded natively or encapsulated.
> 
> If the dest MAC is not the encapsulator's address, then the MAC header 
> stays with the packet/frame and then encapsulated.
> 
> The former is a layer-3 overlay and the latter is a layer-2 overlay where 
> both can be supported at the same time in one encapsulator device.
> 
> So I believe VXLAN-GPE could be useful yet redundant but LISP requires no 
> changes (if LISP supports  L2 which is documented in 
> draft-smith-lisp-layer-2 using a different port number).
> 
> Dino
> 
>> Thanks for your detail comments.  As for current VXLAN encapsulation, pls 
>> see my comments inline with [weiguo] below.
>> 
>> There is motivation to extend an encapsulation header (which is called 
>> VXLAN-GPE) so it can support, most importantly NSH. That change also gives 
>> VXLAN to support encapsulating layer-2 IPv4 and IPv6, which is duplicate 
>> functionality of LISP. But the headers are so similar, it really doens't 
>> matter.
>> 
>> [weiguo]: Yes, an new encapsulation header should be extended to support 
>> NSH. But as for IPv4 and IPv6, i think current VXLAN already supported.  
>> For the layer 3 inter-subnet traffic from NVE1 to NVE2, inner destination 
>> MAC is the gateway interface MAC at NVE2.  For the layer 2 intra-subnet 
>> traffic from NVE1 to NVE2,  inner destination MAC is the destination TS's 
>> MAC. When NVE2 receives VXLAN encapsulated traffic from NVE1, inner 
>> destination MAC can be used to differentiate layer 2 traffic from layer 3 
>> traffic. VXLAN distributed layer 3 gateway can be realized through the 
>> mechanism, NVE can forward both intra-subnet layer 2 traffic and 
>> inter-subnet layer 3 traffic at the same time.
>> 
>> Thanks
>> weiguo
>> 主题: Re: [nvo3] Comments on 
>> http://tools.ietf.org/html/draft-quinn-vxlan-gpe-03
>> 
>>> Hi Dino,
>>> in VXLAN-GPE seems to have no any value. As for basic inter-subnet layer 
>>> 3 communication and intra-subnet layer 2 communication between NVEs, 
>>> current NVGRE, VXLAN and LISP have already supported,
>> 
>> VXLAN supports L2 overlays since its goal is to extend subnets. LISP 
>> supports L3 overlays so it assumes subnets are local (to the xTR) just 
>> like in a routed network. NVGRE can be a combo.
>> 
>> There is motivation to extend an encapsulation header (which is called 
>> VXLAN-GPE) so it can support, most importantly NSH. That change also gives 
>> VXLAN to support encapsulating layer-2 IPv4 and IPv6, which is duplicate 
>> functionality of LISP. But the headers are so similar, it really doens't 
>> matter.
>> 
>> However, the P-bit is not needed for anything new in LISP and the OAM-bit 
>> is not needed in LISP since LISP has different UDP port number (4342) for 
>> control-packets. VXLAN does not have a well defined control protocol so 
>> the data-plane has to escape out control-plane pakcets where the first one 
>> is this new OAM message.
>> 
>>> layer 2 forwarding process exists. As for some extra functions of OAM, 
>>> service chaining,and etc,  only VXLAN-GPE can support, pure VXLAN-GPE 
>>> network should be used in these cases.
>>> Thanks
>>> weiguo
>> 
>> Right, agree.
>> 
>> Dino
>> 
>>> 
>>> 
>>> 主题: Re: 答复: [nvo3] Comments on 
>>> http://tools.ietf.org/html/draft-quinn-vxlan-gpe-03
>>> 
>>>> About backward compatibility, i also agree with Dino. VXLAN-GPE should 
>>>> focus on  the VXLAN-GPE header and requires the assignment of a new UDP 
>>>> port, the data format don't need consider backward compatibility with 
>>>> VXLAN header. I
>>> 
>>> I want to make it clear that supporting backward compatibility is very 
>>> important since VXLAN-port-4789 is supported in various chips already.
>>> 
>>> Dino

---

From: Haoweiguo <[email-masked]>
Date: Wed, 30 Jul 2014 09:29:41 +0000

Hi Marc,

VXLAN-GPE have next protocol field, so it can combine multiple data planes into one VXLAN-gpe. Currently inner destination MAC can be used to distinguish layer 2 and layer 3 traffic at egress NVE.
Thanks
weiguo
________________________________________
发件人: Marc Binderberger [[email-masked]]
发送时间: 2014年7月30日 14:40
收件人: Haoweiguo
抄送: Dino Farinacci; David Melman; [email-masked]; LISP mailing list list; Tom Herbert
主题: Re: [nvo3] 答复: 答复:  Comments on http://tools.ietf.org/html/draft-quinn-vxlan-gpe-03

Hello Weiguo,

Regards, Marc

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 30 Jul 2014 04:57:26 -0700

A minor correction, LISP does have a unified encapsulation where the packet demux function, that has been defined for a few years, is done with UDP ports. As for VXLAN, the packet demux, has been recently introduced, is done with the P-bit. 

Dino

> On Jul 29, 2014, at 8:15 PM, Haoweiguo <[email-masked]> wrote:
> 
> Hi Dino,

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 30 Jul 2014 05:00:41 -0700

> On Jul 30, 2014, at 2:29 AM, Haoweiguo <[email-masked]> wrote:
> 

You can do what you say today with VXLAN. You don't need GPE for this. The demux is just in the ethertype field if the inner MAC header. 

Dino

---

From: Marc Binderberger <[email-masked]>
Date: Wed, 30 Jul 2014 07:19:55 -0700

Hello Weiguo,

> VXLAN-GPE have next protocol field, so it can combine multiple data planes 
> into one VXLAN-gpe.

so you are fine with the VxLAN-gpe layout then? Okay, I misunderstood this.

> Currently inner destination MAC can be used to 
> distinguish layer 2 and layer 3 traffic at egress NVE.

as I said I see a subtle difference here and would say VxLAN is doing a 
layer-2 operation. Nevertheless, if your encapsulation/decapsulation network 
elements understand this layer-2 aspect, are capable to add Ethernet header, 
VxLAN shim and outer IP header onto the inner IP packet etc., then yes, you 
could transport layer-3 with VxLAN.

Regards, Marc

> 主题: Re: [nvo3] 答复: 答复:  Comments on

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Wed, 30 Jul 2014 21:42:27 +0000

Hi Dino,

On 7/29/14 1:08 AM, "Dino Farinacci" <[email-masked]> wrote:

>O-bit in LISP is not needed. I told the LUSP-GPE authors this. Because
>OAM-packets like all other control packets go in UDP port 4342 (where
>encapsulated packets go in UDP port 4341).

I can see the O-bit being useful when you want to use port 4341 to help
ensure that OAM packets use the same path through the network as the user
data packets.

> 
>
>And P-bit is not needed and the authors indicated the main motivation was
>to keep consistent with VXLAN so same hardware design could do both. LISP
>already has port 8473 for L2 frames.

I assume you meant 8472 (assigned to OTV and used by VXLAN implementation
pre-IANA assignment). I'm not sure you can technically call this a LISP
port.

>And NSH is a service protocol and should run above the transport layer so
>should have its own port and can address packets anywhere.

I think the intent of NSH is to be generic enough to work at different
layers.  The recent addition of the Metadata Type field in the NSH header
allows for it to be used for purposes beyond SFC.  It could theoretically
be used to essentially extend the header of the layer below it (e.g.
VXLAN/LISP).  e.g. I think this could be used for Tom to carry his 64 bit
VNI authentication.

 - Larry

>Just like any other UDP application. If that packet needs to be
>encapsulated that is a lower level function. Just like IP packets can go
>in an MPLS based LSP.
>

---

From: Tom Herbert <[email-masked]>
Date: Wed, 30 Jul 2014 14:46:15 -0700

> VXLAN/LISP).  e.g. I think this could be used for Tom to carry his 64 bit
I'd be interested to see exactly what the headers might look like in
that case. I've tried to extrapolate from the SFC drafts how that
might work but really don't see it...

Thanks,
Tom

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Wed, 30 Jul 2014 22:00:39 +0000

Hi Tom,

First, the VXLAN-GPE Next Protocol field would indicate the value 0x4 for
NSH (as specified in draft-quinn-vxlan-gpe-03).  Then, directly following
the VXLAN-GPE header would be an NSH header.  One would need to define a
new MD Type (not the SFC value of 0x1, specified in draft-quinn-nsh-03).
Then you need to make a decision as to whether you want to possibility of
your authentication value to be processed by hardware or only software.
If you want hardware support, then I would recommend that it not be
encoded as a TLV.  If you only care about software support, then you could
encode the authentication in a TLV using your own organization's TLV
Class, or perhaps an IETF TLV Class if you are standardizing it.  If you
expect hardware to parse and validate the VNI authentication, then I would
encode it somewhere within the 20 bytes following the base NSH header and
not in a TLV.

 - Larry

On 7/30/14 2:46 PM, "Tom Herbert" <[email-masked]> wrote:

>>>> picture?  They do mean something about how the packet is handled,
>>>>don't
>>>>they?
>>>

---

From: Tom Herbert <[email-masked]>
Date: Wed, 30 Jul 2014 17:16:49 -0700

> encode it somewhere within the 20 bytes following the base NSH header and
Any optional data I define which proves useful in the datapath I may
eventually want to implement in HW, and I really wouldn't want to have
to make such a decision up front-- so I'll assume anything we'd want
to define would need to go into NSH headers in order to keep HW
support an option. So then in this model is it correct to say that the
we could arbitrarily extend the protocol by using a chain of NSH
headers each of which provides 20 bytes of data we can use for
optional data and still be "HW friendly"?

Thanks,
Tom

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 30 Jul 2014 17:51:08 -0700

> ensure that OAM packets use the same path through the network as the user

Well there is no reason they wouldn't go the same as the user packets if the destination IP address was the same. Also, the OAM proposal for LISP is to measure paths between RLOCs. The same RLOCs that are used for data packets. You don't need a data-plane header to achieve the same paths. And again, you give more work for the hardware to do.

> pre-IANA assignment). I'm not sure you can technically call this a LISP

Yes, typo.

We will call it a port used by LISP because it is documented in draft-smith-lisp-layer-2. But don't miss the point of referencing it.

> VXLAN/LISP).  e.g. I think this could be used for Tom to carry his 64 bit

No one is questioning the intent of NSH. But it should use a UDP header not a VXLAN header. Why is it any different than DNS or SMTP?

If the NSH UDP packet gets encapsulated downstream by a VXLAN or LISP device, so be it. But why should NSH prepend a VXLAN header. And why would NSH prepend a MAC header before the VXLAN header. I talked to the authors about this, so it is not news to them.

Dino

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Thu, 31 Jul 2014 01:04:45 +0000

On 7/30/14 5:51 PM, "Dino Farinacci" <[email-masked]> wrote:

>Well there is no reason they wouldn't go the same as the user packets if
>the destination IP address was the same. Also, the OAM proposal for LISP
>is to measure paths between RLOCs. The same RLOCs that are used for data
>packets. You don't need a data-plane header to achieve the same paths.
>And again, you give more work for the hardware to do.

I'm assuming that routers and switches will be multipathing based on the
UDP port numbers, so I would expect different destination UDP ports to
take different equal cost paths.

>We will call it a port used by LISP because it is documented in
>draft-smith-lisp-layer-2. But don't miss the point of referencing it.
>
>No one is questioning the intent of NSH. But it should use a UDP header
>not a VXLAN header. Why is it any different than DNS or SMTP?

I agree that for service chaining, NSH can certainly ride within UDP.  I'm
referring to using NSH to extend the VXLAN header (or similar
encapsulation...I'm not sure what layer to call it).  I responded to Tom
on this thread regarding a potential use case for this.

>
>If the NSH UDP packet gets encapsulated downstream by a VXLAN or LISP
>device, so be it. But why should NSH prepend a VXLAN header. And why
>would NSH prepend a MAC header before the VXLAN header. I talked to the
>authors about this, so it is not news to them.
>
>Dino
>
>
>

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Thu, 31 Jul 2014 01:06:31 +0000

>optional data and still be "HW friendly"?

No, I would not necessarily say that.  I think an arbitrary number of NSH
headers makes the expected offset of the payload vary in a similar way
that TLVs would.

>>>>>I won't answer that because those bit introductions into the design
>>>>>are
>>>>>indeed design bugs IMO.
>>>>>
>>>>>Dino

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 30 Jul 2014 18:13:26 -0700

> take different equal cost paths.

Well if OAM is going to be effective, messages need to be sent from any pair of ports that yield 0 through N modulus so multiple paths can be determined. So it doesn't matter with the port number values  you use, those control packets will be ECMPed as well.

If you are also inferring that you want the OAM packets to go through the same data-path of each device on the path, then you will have to put TLVs in the data path, which is traditionally not prudent. See my Puneet reference from previous email.

Dino

---

From: Xuxiaohu <[email-masked]>
Date: Thu, 31 Jul 2014 01:55:15 +0000

> >And NSH is a service protocol and should run above the transport layer
> >so should have its own port and can address packets anywhere.
> 
> I think the intent of NSH is to be generic enough to work at different layers.
> The recent addition of the Metadata Type field in the NSH header allows for it to
> be used for purposes beyond SFC.  It could theoretically be used to essentially
> extend the header of the layer below it (e.g.
> VXLAN/LISP).  e.g. I think this could be used for Tom to carry his 64 bit VNI
> authentication.

Hi Larry,

If the Metadata Type field in the NSH header is going to be used for purposes beyond SFC, it seem much better to make the metadata-specific header and the service-path header as independent as possible from the beginning. 

Best regards,
Xiaohu

---

From: "Paul Quinn (paulq)" <[email-masked]>
Date: Thu, 31 Jul 2014 12:56:14 +0000

<removed LISP list>

> optional data and still be "HW friendly"?

You can’t generalize hardware parsing like that.  The key for easy of parsing is fixed sizes and known offsets.  Stacking headers changes that.  That isn’t to say you couldn’t do what you suggest, but it would be more complex and limits the value of the simple fixed size header.

---

From: Tom Herbert <[email-masked]>
Date: Thu, 31 Jul 2014 08:31:27 -0700

> You can’t generalize hardware parsing like that.  The key for easy of parsing is fixed sizes and known offsets.  Stacking headers changes that.  That isn’t to say you couldn’t do what you suggest,…
I'm not trying generalize hardware parsing, I was trying to
extrapolate how to use NSH to allow different meta data. Unless I
misunderstand, NSH has similar properties to TLVs with the exception
that they are fixed size. I do agree that fixed sizes and known offset
are important for easy parsing, which is precisely why I'm leery about
anything that looks like TLVs in such a low level data path.

The major parsing problem with TLVs (and I presume NSH if more than
one header is allowed) is that the number of possible header
combinations is combinatorial. For example, if there are 5 TLVs this
generates 150 possible combinations. If we use optional flag-fields
like in GUE (based on GRE) there would only be 32 possible
combinations. The latter is much more amenable to HW parsing
techniques like say a TCAM, not to mention the overhead to describe
each TLV potentially increases TCAM width.

We have already verified that both NICs and switches are capable of
parsing GRE with various combinations of the defined variable headers
(keyid, seqnum, csum). I take this as evidence that the is method of
extensibility is HW friendly.

---

From: Pankaj Garg <[email-masked]>
Date: Thu, 31 Jul 2014 15:45:28 +0000

> -----Original Message-----
> From: nvo3 [mailto:[email-masked]] On Behalf Of Larry Kreeger
> (kreeger)
> Sent: Wednesday, July 30, 2014 6:07 PM
> To: Tom Herbert
> Cc: David Melman; Marc Binderberger; LISP mailing list list; Dino Farinacci;
> [email-masked]
> Subject: Re: [nvo3] Comments on http://tools.ietf.org/html/draft-quinn-
> vxlan-gpe-03
> 
> 
> 
> >> First, the VXLAN-GPE Next Protocol field would indicate the value 0x4
> >>for  NSH (as specified in draft-quinn-vxlan-gpe-03).  Then, directly
> >>following  the VXLAN-GPE header would be an NSH header.  One would
> >>need to define a  new MD Type (not the SFC value of 0x1, specified in
> >>draft-quinn-nsh-03).
> >> Then you need to make a decision as to whether you want to
> >>possibility of  your authentication value to be processed by hardware
> >>or only software.
> >>encoded as a TLV.  If you only care about software support, then you
> >>could  encode the authentication in a TLV using your own
> >>organization's TLV  Class, or perhaps an IETF TLV Class if you are
> >>standardizing it.  If you  expect hardware to parse and validate the
> >>VNI authentication, then I would  encode it somewhere within the 20
> >>bytes following the base NSH header and  not in a TLV.
> >>
> >to make such a decision up front-- so I'll assume anything we'd want to
> >define would need to go into NSH headers in order to keep HW support an
> >option. So then in this model is it correct to say that the we could
> >arbitrarily extend the protocol by using a chain of NSH headers each of
> >which provides 20 bytes of data we can use for optional data and still
> >be "HW friendly"?
> 
> headers makes the expected offset of the payload vary in a similar way that
> TLVs would.

[PG] The idea here is that TLV provides a way for the software to experiment and quickly light up new features without requiring changes to NIC offloads. Once the hardware support is needed for those, those TLVs can be converted to standard format metadata and put in the fixed header portion of the NSH header. Using different MDType, one can have a 20-byte fixed header or larger/smaller in future for different MDType. This provides a lot of extensibility in a way that is both hardware and software friendly.

> >>>> I think the intent of NSH is to be generic enough to work at
> >>>>different  layers.  The recent addition of the Metadata Type field
> >>>>in the NSH header  allows for it to be used for purposes beyond SFC.
> >>>>It could theoretically  be used to essentially extend the header of
> >>>>the layer below it (e.g.
> >>>> VXLAN/LISP).  e.g. I think this could be used for Tom to carry his
> >>>>64 bit  VNI authentication.
> >>>>
> >>>>>encapsulated that is a lower level function. Just like IP packets
> >>>>>can go in an MPLS based LSP.
> >>>>>
> >>>>>>don't they?
> >>>>>are indeed design bugs IMO.
> >>>>>
> >>>>>Dino

---

From: Eric Gray <[email-masked]>
Date: Thu, 31 Jul 2014 21:09:09 +0000

Dino,

Would you re-phrase your response?  I am having some trouble parsing it, so 
I must be missing something.

First, I think (when you said "... sent from any pair of ports ...") you meant to
say "... sent with any pair of ports ..."  - but this is a guess.

As for making OAM messages traverse the exact same path as data, this is 
what OAM is expected to do.  In essence, if data follows a path that involves
a non-zero number of gates, while OAM does not, the successful delivery of
OAM is only an approximate indication of the data-path integrity.  Any H/W
that data has to go through, and OAM does not go through, could fail and we
would see an OAM indication of a valid path through which data either would
not go, or would be diverted in some unexpected way.

Ordinarily, this should not be a problem for the hardware, as (ordinarily) the
OAM is indistinguishable from data.  The hardware works no harder to push
OAM than it would to push an equivalent amount of data.

So, what is the problem again?

--
Eric

-----Original Message-----
From: nvo3 [mailto:[email-masked]] On Behalf Of Dino Farinacci
Sent: Wednesday, July 30, 2014 9:13 PM
To: Larry Kreeger
Cc: Tom Herbert; David Melman; Marc Binderberger; LISP mailing list list; [email-masked]
Subject: Re: [nvo3] Comments on http://tools.ietf.org/html/draft-quinn-vxlan-gpe-03

> I'm assuming that routers and switches will be multipathing based on 
> the UDP port numbers, so I would expect different destination UDP 
> ports to take different equal cost paths.

Dino

---

From: Dino Farinacci <[email-masked]>
Date: Thu, 31 Jul 2014 16:16:53 -0700

> say "... sent with any pair of ports ..."  - but this is a guess.

Yes "with" is a better way of stating it.

> what OAM is expected to do.  In essence, if data follows a path that involves

Good luck. I do not how you will be able to control each ECMP path at each path across different vendors as well as the same vendor with different hashing algorithms.

One needs to argue if you really need the granuarlity for the complexity that will needed to get this partially correct.

> not go, or would be diverted in some unexpected way.

Well I think LISP RLOC-probing is good enough, but I am biased.  ;-)

> OAM than it would to push an equivalent amount of data.

If an ITR sends a packet the ETR's address, the middle boxes do not know if it is a control-packet versus a data-packet.

> So, what is the problem again?

I am trying to avoid problems. Seems like things are being over-engineered. Again.

Dino

P.S. Sorry I keep being negative. And if one person says shut up, I'll stop posting.

---

From: Marc Binderberger <[email-masked]>
Date: Thu, 31 Jul 2014 23:08:28 -0700

Hello Dino,

always interesting, different people interpret differently.

You seem to talk about controlling where the OAM data flows. I don't think 
this is what Eric means. As long as a flow is defined in the traditional 3- 
or 5-tuple way from the IP/UDP packet and as long as every router is mapping 
the same flow onto the same ECMP egress interface we are fine. The details - 
the exact algorithm, if the algorithm uses the router-id as an additional 
hash or whatever else - don't matter. The OAM packet would use the same 
IP/UDP header as the data packet it's supervising. And the receiver processes 
OAM packets as "OAM" because of the flag set.

> One needs to argue if you really need the granuarlity for the complexity 
> that will needed to get this partially correct.

> Well I think LISP RLOC-probing is good enough, but I am biased.  ;-)

you are "rich" by having an additional control plane UDP port ;-)
I don't see much complexity although I agree that just knowing if the RLOC is 
alive and ping-able may be enough in many cases. If we can get more like 
in-band OAM - why not?

> If an ITR sends a packet the ETR's address, the middle boxes do not know if 
> it is a control-packet versus a data-packet.

true but e.g. for LISP, while the middle box may have no idea what 4341 and 
4342 as udp ports mean, it could still calculate a different hash bucket for 
ECMP due to the different UDP port.
Hmm, my statement is so trivial - I assume you want to say something 
different with your reply?

> I am trying to avoid problems. Seems like things are being over-engineered. 

I would say the echo nonce in RFC6830 is not fundamentally different from 
OAM. The N+E flags trigger some activity on the receiver, similar to OAM.

> P.S. Sorry I keep being negative. And if one person says shut up, I'll stop 

well, everyone is entitled to his/her opinion and to voice it in his/her 
personal style. You have a point as I was a bit upset myself about this 
draft: there is likely more about it, otherwise why would we discuss 
yet-another-8-byte-header? Fabio offered the simplification the new header 
offers but for a while we will have 2-3 slightly different headers that need 
active support in the code or hardware. Actually in terms of code - be it C 
or Verilog - I'm not sure there is any simplification ever over VxLAN + LISP.

But at least the situation of having both VxLAN and LISP can be simplified by 
having a common umbrella and one common discussion.

Personally I think VxLAN-gpe is how the VxLAN/LISP header could have looked 
like from the start (hindsight is great, I know) and I don't have a technical 
problem with the draft itself. What is missing is enough context to discuss 
it. E.g. I'm still not sure why there is a P flag, if for a hard technical 
reason or for the aesthetics that every field is controlled by a turn-on flag 
;-)

So I encourage and kindly ask the authors to provide more of this context in 
the next draft version.

Regards, Marc

On Thu, 31 Jul 2014 16:16:53 -0700, Dino Farinacci wrote:
> Good luck. I do not how you will be able to control each ECMP path at each 
> path across different vendors as well as the same vendor with different 
> hashing algorithms.
> 
> One needs to argue if you really need the granuarlity for the complexity 
> that will needed to get this partially correct.
> 
> If an ITR sends a packet the ETR's address, the middle boxes do not know if 
> it is a control-packet versus a data-packet.
> 
>> Subject: Re: [nvo3] Comments on 
>> Well if OAM is going to be effective, messages need to be sent from any 
>> pair of ports that yield 0 through N modulus so multiple paths can be 
>> determined. So it doesn't matter with the port number values  you use, 
>> those control packets will be ECMPed as well.
>> 
>> If you are also inferring that you want the OAM packets to go through the 
>> same data-path of each device on the path, then you will have to put TLVs 
>> in the data path, which is traditionally not prudent. See my Puneet 
>> reference from previous email.
>> 
>> Dino
>>