Subject: Re: [nvo3] Comments on http://tools.ietf.org/html/draft-quinn-vxlan-gpe-03

From: Dino Farinacci <[email-masked]>
Date: Fri, 1 Aug 2014 09:24:41 -0700

> Hello Dino,
> 
> always interesting, different people interpret differently.
> 
> You seem to talk about controlling where the OAM data flows. I don't think 
> this is what Eric means. As long as a flow is defined in the traditional 3- 
> or 5-tuple way from the IP/UDP packet and as long as every router is mapping 
> the same flow onto the same ECMP egress interface we are fine. The details - 
> the exact algorithm, if the algorithm uses the router-id as an additional 
> hash or whatever else - don't matter. The OAM packet would use the same 
> IP/UDP header as the data packet it's supervising. And the receiver processes 
> OAM packets as "OAM" because of the flag set.

For overlays, if you want to do RLOC selection at the encapsulator based on the underlay's paths, you will need to know all paths so you can have granular decision control.

I didn't mean to emphasize the algorithm being used at each hop. There is a combinatorial problem by looking at all paths. For example say one hop has ECMP paths (A, B, C) and the next one has (A', B', C'). For different 5-tuples you can have A -> A', A -> B', A -> C',  B -> A', B -> B', B -> C', and C -> A', C -> B', C-> C'. And if the same algorithms are not used, you may have a 1-to-many relationship from hop 1 to hope 2.

If an OAM mechanism is going to measure attributes of a path, it will to do so for all paths, And the number of combinations above was just in 2 boxes with only 3 ECMPs each.

This quickly raises my complexity flag.

>> One needs to argue if you really need the granuarlity for the complexity 
>> that will needed to get this partially correct.
> 
>> Well I think LISP RLOC-probing is good enough, but I am biased.  ;-)
> 
> you are "rich" by having an additional control plane UDP port ;-)
> I don't see much complexity although I agree that just knowing if the RLOC is 
> alive and ping-able may be enough in many cases. If we can get more like 
> in-band OAM - why not?

You can inband OAM for the control-port too.

>> If an ITR sends a packet the ETR's address, the middle boxes do not know if 
>> it is a control-packet versus a data-packet.
> 
> true but e.g. for LISP, while the middle box may have no idea what 4341 and 
> 4342 as udp ports mean, it could still calculate a different hash bucket for 
> ECMP due to the different UDP port.

The hash is modulo the number of ECMP paths. You build the control packets with 5-tuples that generate hashs across all ECMP paths.

> Hmm, my statement is so trivial - I assume you want to say something 
> different with your reply?
> 
> 
>> I am trying to avoid problems. Seems like things are being over-engineered. 
>> Again.
> 
> I would say the echo nonce in RFC6830 is not fundamentally different from 
> OAM. The N+E flags trigger some activity on the receiver, similar to OAM.

It is way different. All we are testing is the forward path from ITR to ETR (and nothing else). OAM proposals also meausre other attributes of the path.

> 
>> P.S. Sorry I keep being negative. And if one person says shut up, I'll stop 
>> posting.
> 
> well, everyone is entitled to his/her opinion and to voice it in his/her 
> personal style. You have a point as I was a bit upset myself about this 
> draft: there is likely more about it, otherwise why would we discuss 
> yet-another-8-byte-header? Fabio offered the simplification the new header 
> offers but for a while we will have 2-3 slightly different headers that need 
> active support in the code or hardware. Actually in terms of code - be it C 
> or Verilog - I'm not sure there is any simplification ever over VxLAN + LISP.

Well if one wants the VXLAN and LISP header to be consistent, it was already that way from the start. And if you wanted to run L3 with a VXLAN header, you use the destination MAC as the xTR's address and you got L3. There is no demux needed because the inner ethernet header has an ethertype that can demux every protocol ever invented in the world.

Do not create a new registry for types authors of GPE. PPP learned this in 1989 and used ethertypes. Why is that different now?

If there is an ethertype allocated for "ethernet" there can be one allocated for NSH. And then if anyone ever wanted to run NSH directly over ethernet, you have your ethertype.

But I AM NOT RECOMMENDING THIS. I believe NSH should run on top of UDP and get its own port number. NSH sends UDP packets, period.

> But at least the situation of having both VxLAN and LISP can be simplified by 
> having a common umbrella and one common discussion.

Agree. 

> Personally I think VxLAN-gpe is how the VxLAN/LISP header could have looked 
> like from the start (hindsight is great, I know) and I don't have a technical 
> problem with the draft itself. What is missing is enough context to discuss 
> it. E.g. I'm still not sure why there is a P flag, if for a hard technical 
> reason or for the aesthetics that every field is controlled by a turn-on flag 
> ;-)

There is a P-flag so you have demux ability in the VXLAN/LISP header. So we will demux at the UDP port level, the P-bit level, and the ethertype level. All these will be demux decisions a forwarder has to deal with. This is quite ridiculous.

> So I encourage and kindly ask the authors to provide more of this context in 
> the next draft version.
> 
> Regards, Marc

Dino

> 
> 
> 
> On Thu, 31 Jul 2014 16:16:53 -0700, Dino Farinacci wrote:
>>> Dino,
>>> 
>>> Would you re-phrase your response?  I am having some trouble parsing it, 
>>> so 
>>> I must be missing something.
>>> 
>>> First, I think (when you said "... sent from any pair of ports ...") you 
>>> meant to
>>> say "... sent with any pair of ports ..."  - but this is a guess.
>> 
>> Yes "with" is a better way of stating it.
>> 
>>> As for making OAM messages traverse the exact same path as data, this is 
>>> what OAM is expected to do.  In essence, if data follows a path that 
>>> involves
>> 
>> Good luck. I do not how you will be able to control each ECMP path at each 
>> path across different vendors as well as the same vendor with different 
>> hashing algorithms.
>> 
>> One needs to argue if you really need the granuarlity for the complexity 
>> that will needed to get this partially correct.
>> 
>>> a non-zero number of gates, while OAM does not, the successful delivery of
>>> OAM is only an approximate indication of the data-path integrity.  Any H/W
>>> that data has to go through, and OAM does not go through, could fail and we
>>> would see an OAM indication of a valid path through which data either would
>>> not go, or would be diverted in some unexpected way.
>> 
>> Well I think LISP RLOC-probing is good enough, but I am biased.  ;-)
>> 
>>> Ordinarily, this should not be a problem for the hardware, as (ordinarily) 
>>> the
>>> OAM is indistinguishable from data.  The hardware works no harder to push
>>> OAM than it would to push an equivalent amount of data.
>> 
>> If an ITR sends a packet the ETR's address, the middle boxes do not know if 
>> it is a control-packet versus a data-packet.
>> 
>>> So, what is the problem again?
>> 
>> I am trying to avoid problems. Seems like things are being over-engineered. 
>> Again.
>> 
>> Dino
>> 
>> P.S. Sorry I keep being negative. And if one person says shut up, I'll stop 
>> posting.
>> 
>>> 
>>> --
>>> Eric
>>> 
>>> -----Original Message-----
>>> From: nvo3 [mailto:[email-masked]] On Behalf Of Dino Farinacci
>>> Sent: Wednesday, July 30, 2014 9:13 PM
>>> To: Larry Kreeger
>>> Cc: Tom Herbert; David Melman; Marc Binderberger; LISP mailing list list; 
>>> [email-masked]
>>> Subject: Re: [nvo3] Comments on 
>>> http://tools.ietf.org/html/draft-quinn-vxlan-gpe-03
>>> 
>>>> I'm assuming that routers and switches will be multipathing based on 
>>>> the UDP port numbers, so I would expect different destination UDP 
>>>> ports to take different equal cost paths.
>>> 
>>> Well if OAM is going to be effective, messages need to be sent from any 
>>> pair of ports that yield 0 through N modulus so multiple paths can be 
>>> determined. So it doesn't matter with the port number values  you use, 
>>> those control packets will be ECMPed as well.
>>> 
>>> If you are also inferring that you want the OAM packets to go through the 
>>> same data-path of each device on the path, then you will have to put TLVs 
>>> in the data path, which is traditionally not prudent. See my Puneet 
>>> reference from previous email.
>>> 
>>> Dino
>>> 
>>> _______________________________________________
>>> nvo3 mailing list
>>> [email-masked]
>>> https://www.ietf.org/mailman/listinfo/nvo3
>>

---

From: Tom Herbert <[email-masked]>
Date: Fri, 1 Aug 2014 10:11:33 -0700

> This quickly raises my complexity flag.
Definitely, the combinatorial problem has made concurrently probing
all paths infeasible. The solution to detecting bad paths has been to
get feedback to the application from the transport (e.g. TCP) for
packet loss or excessive RTT for specific flows. The response to a bad
path is simply reopen the connection and hope for a better path (by
virtue of a different 5-tuple hash for the new connection).

For encapsulation there is good news and bad news. When we're
encapsulating over UDP we can affect the path simply by twiddling the
source port a little, so having application reopen connection might
not be necessary response to bad path. Bad news is that it may be
infeasible to get the feedback from a third party guest about path
quality. If the encapsulation layer wants to perform path selection
somehow inband signaling and per flow might be needed at that layer.

---

From: Dino Farinacci <[email-masked]>
Date: Fri, 1 Aug 2014 10:21:41 -0700

> somehow inband signaling and per flow might be needed at that layer.

This is why in LISP the ITR simply RLOC-probes each RLOC in the locator-set and not concerned with the multiple paths to each RLOC. So if RLOC-probing or echo-noncing determines an RLOC is unreachable, we just switch to another RLOC.

I am not sure if the LISP overlay had Segment Routing underneath it, that it could really stay with an RLOC that has just gone down if it could select another SR path to that RLOC. Maybe we are just splitting hairs too thinly here.

Dino

---

From: Eric Gray <[email-masked]>
Date: Fri, 1 Aug 2014 17:49:33 +0000

Marc/Dino,

	Marc has essentially caught the interpretation I meant.  Sorry if I was not clear.

--
Eric

-----Original Message-----
From: Marc Binderberger [mailto:[email-masked]] 
Sent: Friday, August 01, 2014 2:08 AM
To: Dino Farinacci; Eric Gray
Cc: Larry Kreeger; Tom Herbert; David Melman; LISP mailing list list; [email-masked]
Subject: Re: [nvo3] Comments on http://tools.ietf.org/html/draft-quinn-vxlan-gpe-03
Importance: High

Hello Dino,

You seem to talk about controlling where the OAM data flows. I don't think this is what Eric means. As long as a flow is defined in the traditional 3- or 5-tuple way from the IP/UDP packet and as long as every router is mapping the same flow onto the same ECMP egress interface we are fine. The details - the exact algorithm, if the algorithm uses the router-id as an additional hash or whatever else - don't matter. The OAM packet would use the same IP/UDP header as the data packet it's supervising. And the receiver processes OAM packets as "OAM" because of the flag set.

> One needs to argue if you really need the granuarlity for the 
> complexity that will needed to get this partially correct.

> Well I think LISP RLOC-probing is good enough, but I am biased.  ;-)

you are "rich" by having an additional control plane UDP port ;-) I don't see much complexity although I agree that just knowing if the RLOC is alive and ping-able may be enough in many cases. If we can get more like in-band OAM - why not?

> If an ITR sends a packet the ETR's address, the middle boxes do not 
> know if it is a control-packet versus a data-packet.

true but e.g. for LISP, while the middle box may have no idea what 4341 and
4342 as udp ports mean, it could still calculate a different hash bucket for ECMP due to the different UDP port.
Hmm, my statement is so trivial - I assume you want to say something different with your reply?

> I am trying to avoid problems. Seems like things are being over-engineered. 

I would say the echo nonce in RFC6830 is not fundamentally different from OAM. The N+E flags trigger some activity on the receiver, similar to OAM.

Regards, Marc

---

From: Eric Gray <[email-masked]>
Date: Fri, 1 Aug 2014 18:14:13 +0000

Dino,

	Thanks for the reply.

	I think we are on the same page, but possibly not.

	I don't argue that we should be able to steer an OAM message to follow
a particular path among the multiple equal-cost paths, as long as OAM messages
contain enough entropy to ensure that sending some number of them will result 
in having OAM messages traverse each of the multiple equal-cost paths.  Ideally, 
the mechanisms used - both in OAM and in path selection at each hop - will tend 
to distribute OAM messages and data on a proportional basis (e.g. - it's bad if 20% 
of the OAM messages go along a path that carries ~50% of the data).

	Indeed, it is not clear what "following the same path" would mean on the
basis of any individual data PDU.  In order to do this literally you would first have to 
determine which path the data took and then determine what to put in the OAM
message to make it follow the same path.

	Alternatively, you could clone the parts of a particular data PDU that are 
used by distribution algorithms in each hop, in an OAM PDU that you want to use
the same path.

	Either of these approaches would be very hard, for all of the reasons you 
mentioned, and maybe a few you did not, and it is unclear to me why you would 
need to do this most of the time.  Certainly it will be pointless in proactive OAM.

	If the forwarders on a path are not able to distinguish data from OAM, it
can be relatively easy to ensure that approximately the same distribution of OAM
and data is the case.

	One issue that may come up is if the OAM messages have to be sent like
other data packets, with Source and Destination IP address and transport-layer 
port information of the source and sink of the OAM itself, and these fields are all 
that are included in any mechanism used to distribute traffic across multiple paths, 
then the OAM will all traverse the same path.

	One approach to dealing with this is to add one or more fields used to 
provide the entropy that would otherwise be missing, and require every hop to 
include this field (or these fields) in path selection.

	Where this can cause trouble is if the field(s) are included in path selection
only if the PDU is an OAM message.  In this case, while the entropy is there to allow
OAM message to provide full coverage, there is extra logic required in traversing a
series of hops (where each may have multiple path choices for forwarding) that is 
used to determine if the message is an OAM message.

	In this case, by definition, the forwarding devices will be distinguishing OAM
from data (using the extra logic) and this results in a "different path" (at the micro
level, because of the extra logic - that may itself be subject to failure).

	The issue is less about the fact that the OAM PDU follows a different path 
in the network from that a corresponding data PDU would follow (though it may); 
it is that every OAM PDU will "follow" a different logical path within the node itself.

	Alternatively, the field(s) can be examined in every PDU.  This eliminates 
the risk of "path differentiation" (at the micro-level) because of the use of extra 
logic steps, but introduces a requirement that any PDU source MUST be careful 
about value(s) it puts in these fields (or risk having its flows distributed across 
multiple paths).  This can complicate domain ingress devices (for example, a 
tunnel entry point), and reduce overall entropy for distribution of data PDUs.

--
Eric

-----Original Message-----
From: Dino Farinacci [mailto:[email-masked]] 
Sent: Thursday, July 31, 2014 7:17 PM
To: Eric Gray
Cc: Larry Kreeger; Tom Herbert; David Melman; Marc Binderberger; LISP mailing list list; [email-masked]
Subject: Re: [nvo3] Comments on http://tools.ietf.org/html/draft-quinn-vxlan-gpe-03
Importance: High

> Dino,
> 
> Would you re-phrase your response?  I am having some trouble parsing 
> it, so I must be missing something.
> 
> First, I think (when you said "... sent from any pair of ports ...") 
> you meant to say "... sent with any pair of ports ..."  - but this is a guess.

> As for making OAM messages traverse the exact same path as data, this 
> is what OAM is expected to do.  In essence, if data follows a path 
> that involves

Good luck. I do not how you will be able to control each ECMP path at each path across different vendors as well as the same vendor with different hashing algorithms.

One needs to argue if you really need the granuarlity for the complexity that will needed to get this partially correct.

> a non-zero number of gates, while OAM does not, the successful 
> delivery of OAM is only an approximate indication of the data-path 
> integrity.  Any H/W that data has to go through, and OAM does not go 
> through, could fail and we would see an OAM indication of a valid path 
> through which data either would not go, or would be diverted in some unexpected way.

> Ordinarily, this should not be a problem for the hardware, as 
> (ordinarily) the OAM is indistinguishable from data.  The hardware 
> works no harder to push OAM than it would to push an equivalent amount of data.

If an ITR sends a packet the ETR's address, the middle boxes do not know if it is a control-packet versus a data-packet.

> So, what is the problem again?

I am trying to avoid problems. Seems like things are being over-engineered. Again.

Dino

P.S. Sorry I keep being negative. And if one person says shut up, I'll stop posting.

> Cc: Tom Herbert; David Melman; Marc Binderberger; LISP mailing list 
> list; [email-masked]
> Well if OAM is going to be effective, messages need to be sent from any pair of ports that yield 0 through N modulus so multiple paths can be determined. So it doesn't matter with the port number values  you use, those control packets will be ECMPed as well.
> 
> If you are also inferring that you want the OAM packets to go through the same data-path of each device on the path, then you will have to put TLVs in the data path, which is traditionally not prudent. See my Puneet reference from previous email.
> 
> Dino
>

---

From: Dino Farinacci <[email-masked]>
Date: Fri, 1 Aug 2014 11:26:07 -0700

Yes Eric, you make a lot of good points. Therefore the promise of OAM is very limited and can mislead at best.  ;-)

Dino

---

From: Eric Gray <[email-masked]>
Date: Fri, 1 Aug 2014 20:18:37 +0000

:-)

Well, possibly somewhat limited under unusual circumstances and might mislead at 
worst.

The small possibility of being struck by lightning doesn't really justify standing
outside in a storm for no good reason.

We could argue whether or not there is a good reason, but you might be surprised
at my lack of fervor.

I suspect that where you really stand in the argument depends on whether you 
build forwarding equipment, or use forwarding equipment built by someone else.

I am just passing on arguments I have heard.

--
Eric

-----Original Message-----
From: Dino Farinacci [mailto:[email-masked]] 
Sent: Friday, August 01, 2014 2:26 PM
To: Eric Gray
Cc: LISP mailing list list ([email-masked]); [email-masked]
Subject: Re: [nvo3] Comments on http://tools.ietf.org/html/draft-quinn-vxlan-gpe-03
Importance: High

Dino

--- [SNIP] ---