Subject: Comments on draft-herbert-ipv6-update-dest-ops

From: "C. M. Heard" <[email-masked]>
Date: Sat, 18 Aug 2018 10:16:23 -0700

On Fri, 17 Aug 2018 12:28:45 -0700 , Tom Herbert wrote:

> I wrote up a short draft to try to clarify the requirements of
> processing Destination Options that are before a Routing header. This
> work was motivated by some of the discussions around SRv6. It also
> would one of the arguments that TLVs in SRv6 are needed because
> Destination Options can't be ignored at SRv6 destinations.
>
> The narrative above applies to Section 2, Processing Requirements of
Destination Options
<https://tools.ietf.org/html/draft-herbert-ipv6-update-dest-ops-00#section-2>.
That section would make processing of a Destination Options header that
precedes a routing header optional for all nodes listed in the routing
header, including the final destination node. As justification the draft
states:

   This change has a similar justification for relaxing the requirement
   that all intermediate nodes process Hop-by-Hop options in [RFC8200
<https://tools.ietf.org/html/rfc8200>].

   Intermediate destination nodes may be closer in taxonomy to switches
   and routers than end hosts, so it follows that they may have similar
   processing constraints in efficiently processing extension headers
   and TLVs.

That justification seems questionable to me.

First, the proposal makes processing of a Destination Options header that
precedes a routing header optional for the final destination node, to which
the stated rationale clearly does not apply.

Second, a source-routed packet is explicitly addressed to each intermediate
node that needs to process the routing header. That puts the packet on a
different processing path than the normal router processing path, which
forwards IPv6 packets NOT explicitly addressed to the node itself. Thus the
argument may plausibly be made that the taxonomy of such intermediate nodes
is closer to that of an end host rather than to normal switch or router.

In addition to the justification being weak, there is a technical flaw in
the proposal. Per RFC 8200, Section 4
<https://tools.ietf.org/html/rfc8200#section-4>, extension headers must be
processed strictly in the order in which they appear:

   The contents and semantics of each extension header
   determine whether or not to proceed to the next header.  Therefore,
   extension headers must be processed strictly in the order they appear
   in the packet; a receiver must not, for example, scan through a
   packet looking for a particular kind of extension header and process
   that header prior to processing all preceding ones.

A node that receives a packet addressed to itself cannot determine that
there is an occurrence of a destination options header preceding a routing
header without violating that rule.

Finally, I note that the intended status of the draft is listed as
Informational. Making it optional to process a Destination Options header
that precedes a Routing Header changes a requirement levied by RFC 8200.
Thus, if this proposal is retained, the document needs to be Standards
Track.

> The draft
> also clarifies what it means for a Destination Option to be changeable
> en route.
>
> The narrative above applies to Section 3, Changeable Destination Options
<https://tools.ietf.org/html/draft-herbert-ipv6-update-dest-ops-00#section-3>.
I am in complete agreement with that section, and I think it would be an
excellent clarification to publish in either a Standards Track or
Informational document.

Mike Heard

---

From: Tom Herbert <[email-masked]>
Date: Sat, 18 Aug 2018 11:06:05 -0700

On Sat, Aug 18, 2018 at 10:16 AM, C. M. Heard <[email-masked]> wrote:
> Destination Options. That section would make processing of a Destination
> Options header that precedes a routing header optional for all nodes listed
> in the routing header, including the final destination node. As
> justification the draft states:
>
>    that all intermediate nodes process Hop-by-Hop options in [RFC8200].
>
Hi Mark,

This part of the draft is motivated by reality of implementation. Take
a look at draft-ietf-6man-segment-routing-header-14 section 8.
Implementations of SR are listed that include Linux, Cisco, FD.io,
Barefoot, and Juniper. Of these, the only one that is reported to
support SR TLVs is Linux (the only host stack in the list). Segment
routing does not preclude the use of Destination Options before the
segment routing header, and I am extremely doubtful that any of these
implementations, except again for Linux, properly support processing
of Destination Options as specified in RFC8200. It would be great to
be proven wrong on this, and that all these implementations are indeed
protocol compliant! But, history with Hop-by-Hop options in similar
implementations can't be overlooked, nor the fact that current SR
implementations don't even implement SR TLVs. I think it's much more
likely that they're dropping, skipping over them, or throwing
Destination Options into some some slow path. Of these choices,
skipping over them is the only one of these alternative behaviors that
might not kill our ability to use Destination Options.

> the proposal. Per RFC 8200, Section 4, extension headers must be processed
> strictly in the order in which they appear:
>
The next header field in a Destination Options header can be checked
to determine if a Routing header follows.

Tom

> The narrative above applies to Section 3, Changeable Destination Options. I
> am in complete agreement with that section, and I think it would be an

---

From: "C. M. Heard" <[email-masked]>
Date: Sat, 18 Aug 2018 13:22:00 -0700

Tom,

Some challenges for you.

> > the stated rationale clearly does not apply.

Would you care to address this objection?

> might not kill our ability to use Destination Options.

The implementation issue with HbH, as I have understood it, is that many
hardware- and firmware-based forwarding implementations can process
the fixed part of the IPv6 header in the fast path but cannot process EHs
(of any kind) in the fast path, hence punt to the slow path. It's not just
an
issue of TLVs (though most EHs are a kind of TLV). This applies, of
course, to packets not addressed to the node -- i.e., to normal forwarding.

When a routing header comes into play, the packet is addressed directly
to the node. It seems plausible that for a node with the limitations
described
above, a packet directly addressed to it would need to be punted to the slow
path in any case, as there is  the possibility that the node would to
consume
the packet, and it would definitely need to go to the slow path if node
supports any version of the Routing Header. Do you know how that
all works for the implementations(other than Linux) that are listed in
https://tools.ietf.org/html/draft-ietf-6man-segment-
routing-header-14#section-8?

My point in asking that question is that the burden of proof of showing
that this change is actually useful falls on you, the proponent of the
proposal, and I do not think that you have met that burden.

> to determine if a Routing header follows

That is true if the extension header order is as recommended in
https://tools.ietf.org/html/rfc8200#section-4.1, though doing so would
still violate the strict in-order processing rule. Note, however, that the
header ordering in Section 4.1 is a recommendation, not a requirement.

Incidentally, I note that Section 1.2 of draft-ietf-6man-segment-
routing-header-14
<https://tools.ietf.org/html/draft-ietf-6man-segment-routing-header-14#section-2.1>
 says:

   Type Length Value (TLV) contain OPTIONAL information that may be used
   by the node identified in the Destination Address (DA) of the packet.

   Each TLV has its own length, format and semantic.  The code-point
   allocated (by IANA) to each TLV Type defines both the format and the
   semantic of the information carried in the TLV.  Multiple TLVs may be
   encoded in the same SRH.

That sure sounds like a description of destination options to me, so
perhaps rather
than making a Destination Options header in front of a Routing Header
optional to
process, perhaps it would be better to keep things as they are and define
appropriate
destination options instead of SRH TLVs, as Ron Bonica has suggested (see
https://trac.ietf.org/trac/6man/ticket/38). This comment does not apply to
HMAC,
which is intended to protect the SRH, but it would then not need to be a
TLV.

Mike Heard

---

From: Tom Herbert <[email-masked]>
Date: Sun, 19 Aug 2018 13:30:09 -0700

> Would you care to address this objection?

Mike,

The allowance would have to cover all destination nodes including the
last one because there is no way to to know a node is last in the
routing list without processing the routing header which would violate
RFC8200. I would note that this wouldn't impact DestOpts after a
Routing header which must be processed only by the final destination.

> https://tools.ietf.org/html/draft-ietf-6man-segment-routing-header-14#section-8?

Linux properly implements Destination Options (both before and after
Routing header), including handling of unknown TLV types. It also now
limits the number of TLVs per the recommendations of RFC6434bis (eight
is default limit).

I looked at FD.io VPP code. AFAICT they do NOT implement compliant
support for Destination Options. It appears that the Routing header is
being fished for using the function ip6_ext_header_find_t. That
function will skip over Destination Options before the Routing header.
There are only two inconsequential references to
IP_PROTOCOL_IP6_DESTINATION_OPTIONS (nexthdr constant) in all the VPP
code, and I can't find any evidence that Destination Options TLVs are
ever parsed.

I looked for information on P4 how implements this an couldn't find
much of relevance. As I understand it, P4 does not natively support
loops so parsing an unlimited list of TLVs might not be feasible,
although maybe it's possible to parse a small number of them.

For commercial implementations, like Cisco and Juniper, I'll let it up
to the vendors to report whether or not they have proper support for
Destination Options and how much they have tested that.

> Incidentally, I note that Section 1.2 of
> draft-ietf-6man-segment-routing-header-14 says:
>
> That sure sounds like a description of destination options to me, so perhaps
> rather
Yes, that would be best outcome: SRH TLVs are eliminated, Destination
Options are used to hold the options information that is needed per SR
destinations, and everyone is compliant with RFC8200. If that happens,
then the provision in this draft to allow nodes to ignore DestOpts
would be unnecessary.

Tom

---

From: "C. M. Heard" <[email-masked]>
Date: Sun, 19 Aug 2018 20:10:48 -0700

On Sun, Aug 19, 2018 at 1:30 PM, Tom Herbert <[email-masked]> wrote:
>>>> precedes a routing header optional for the final destination node, to
>>>> which the stated rationale clearly does not apply.
>>

True, but surely a node that can process a Destination Options header that
follows a Routing Header -- which an end node must be able to do -- also has
the capability to process a Destination Options header that precedes a
Routing Header. It should not matter whether or not it is the final
destination of the packet.

[ ... ]

>> Do you know how that

Thanks for providing that information.

Mike Heard

---

From: Tom Herbert <[email-masked]>
Date: Mon, 20 Aug 2018 08:05:39 -0700

Mike,

Yes, that would be true if the final destination is a compliant host
stack (e.g. Linux). In that case Hop-by-Hop options as well as
Destination options before or after a Routing header are properly
parsed. As I mentioned, we can now limit the number of TLVs that are
parsed to reasonably limit processing and prevent some denial of
service attack, but there's no motivation for these stacks to start
completely ignoring options all together-- it's only a MAY in RFC8200
that a stack can ignore HBH and a MAY in this draft to ignore DestOpts
.

The negative converse also seems to be true. If an implementation does
not support parsing of Destantion Options before the Routing header,
then the node probably doesn't support parsing of any Destination
Options. This becomes an issue in router implementations that
terminate tunnels int the network with intent to and forward the
tunneled packets. For instance FD.io VPP supports VXLAN, LISP, SR-IPIP
among other encapsulations, AFAICT if the implementation receives an
encapsulated packet addressed to the node that has a Destination
Option (second type that follows any Routing header), either the
packet is either dropped or the Destination options are ignored.

Tom

---

From: Ole Troan <[email-masked]>
Date: Mon, 20 Aug 2018 17:40:34 +0200

> On 20 Aug 2018, at 17:05, Tom Herbert <[email-masked]> wrote:
> 

[wearing the VPP committer hat]

VPP as a tunnel endpoint does not parse destination options. In fact at least for rfc2473 tunnels I believe it would drop packets containing one. I am not convinced you can use one implementation’s behavior to support your argument (or not). At least from a VPP perspective I would add destination option header support, as soon as there were any options that made sense to support. (And I am not convinced the tunnel encap limit option is one of them.). I certainly wouldn’t build an implementation that blindly processed destination options, if there were any to support.

Cheers 
Ole

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 20 Aug 2018 12:07:51 -0400

I can see two sides to this.  On the one hand, we want tunnel 
termination logic to be fast and effecient.  Requiring that it have only 
the necessary information, and not other random things, is part of our 
tool kit for achieving that.

On the other hand, the outer IP header, in the case of v6, is clearly 
destined for the tunnel terminator.  And the IPv6 spec is very clear 
that certain kinds of processing for IPv6 destination options are 
mandatory at such a destination.  For example, retunring an error if 
ther eia an unknown IPv6 destination option with the bit set to force 
errors.

Clearly, doing the checks for that is more expensive.  Equally, it seems 
to be clearly required by our current specs.

Yours,
Joel

On 8/20/18 11:40 AM, Ole Troan wrote:
> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> [email-masked]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
> --------------------------------------------------------------------
>

---

From: Tom Herbert <[email-masked]>
Date: Mon, 20 Aug 2018 09:10:26 -0700

> VPP as a tunnel endpoint does not parse destination options. In fact at least for rfc2473 tunnels I believe it would drop packets containing one. I am not convinced you can use one implementation’s…
Ole,

What you're describing is the type of deadlock that has led to IPv6
options being ossified. On one side middelboxes are only going to
support them properly once there are "options that make sense" to
support, and on the other hand host developers don't want to work to
define ones that make sense to support since middleboxes don't support
options. In-situ OAM is a good example of this. The original idea was
to put OAM in every encapsulation protocol, however that would be much
better server in either Hop-by-Hop or Destination options and now at
least IPv6 options are being defined for that.

Specifically in the case of VPP, I'm not sure why proper parsing of
Destination Options would be a be major effort. This is just software,
and probably just a few lines of implementation to implement the
parsing loop and then you'd be able to claim the VPP is protocol
compliant in this area. I think this will be a bigger issue for
hardware implementations.

Tom

---

From: Ole Troan <[email-masked]>
Date: Mon, 20 Aug 2018 18:23:07 +0200

Tom,

> hardware implementations.

Absolutely. It’s quick to add. That’s the point. Which is why I’m not adding it until there is something to do with those options.
Note in the example you cited we aren’t talking about a middlebox, but a tunnel endpoint.
I’m not sure I can be bothered adding support for pad options.
I don’t see the deadlock here.

Now, if you are talking about a router/middlebox not passing the options, that’s another thing.

Cheers,
Ole

---

From: Ole Troan <[email-masked]>
Date: Mon, 20 Aug 2018 18:33:18 +0200

Joel,

> I can see two sides to this.  On the one hand, we want tunnel termination logic to be fast and effecient.  Requiring that it have only the necessary information, and not other random things, is part of our tool kit for achieving that.
> 
> On the other hand, the outer IP header, in the case of v6, is clearly destined for the tunnel terminator.  And the IPv6 spec is very clear that certain kinds of processing for IPv6 destination options are mandatory at such a destination.  For example, retunring an error if ther eia an unknown IPv6 destination option with the bit set to force errors.
> 
> Clearly, doing the checks for that is more expensive.  Equally, it seems to be clearly required by our current specs.

Yes, to both.

Best regards,
Ole

---

From: Tom Herbert <[email-masked]>
Date: Mon, 20 Aug 2018 10:16:06 -0700

Ole,

Consider the folling scenario:

Someone is developing a new Destination Option that might be of
interest for use with tunnels and might even become one of those
options "that makes sense to support" . Maybe it's an option for the
aforementioned insitu-OAM, or maybe a general packet CRC, etc. The
developer dutifully uses experimental option type 0x1e for the type
value of their option to test it. In particular, the action taken if
the option is unknown to a receiver is to skip over it. They made that
choice because that allows incremental deployment of the support for
the option in any combination supporting sending side or receivers. In
other words, they don't want a flag day for the option that requires
all nodes to support the new option.

Now they go to test backwards comapatbility of the option by sending
it to a node that hasn't been updated to receive it. If the receiving
node is a Linux system, then the option is ignored and the tunneled
packet is processed as before-- expected behavior per the spec.
However, if it's sent to a VPP node then the packet is dropped-- not
expected. Even if this doesn't lead to incorrectness, it does breaks
interoperability and violates "be liberal in what you receive". This
net effect is that this makes development, test, and pilot deployment
of new options really hard. Just implementing the TLV loop, even if
the implementation doesn't process any of them, would resolve this.

Tom

---

From: Fred Baker <[email-masked]>
Date: Mon, 20 Aug 2018 12:00:06 -0700

From my perspective, a tunnel endpoint is a destination. If we want any given destination, an example of which is a tunnel endpoint, to not process some set of options, I would suggest that we simply don't put those options into the header. The question isn't whether the endpoint should process them; it's whether the option is appropriate and useful.

> On Aug 20, 2018, at 9:07 AM, Joel M. Halpern <[email-masked]> wrote:
> 

--------------------------------------------------------------------------------
The fact that there is a highway to hell and a stairway to heaven is an interesting comment on projected traffic volume...

---

From: Ole Troan <[email-masked]>
Date: Mon, 20 Aug 2018 21:07:07 +0200

Tom,

> the implementation doesn't process any of them, would resolve this.

Absolutely. The set of _optional_ destination options would benefit from this approach.
Of which we have none. I’d much rather just implement new options as they come available (if they ever will).
If I get time, I’ll try to do some performance measurements on what the actual cost is for a software based system.

Best regards,
Ole

---

From: Tom Herbert <[email-masked]>
Date: Mon, 20 Aug 2018 13:17:57 -0700

On Mon, Aug 20, 2018 at 12:07 PM, Ole Troan <[email-masked]> wrote:
> Of which we have none. I’d much rather just implement new options as they come available (if they ever will)

Ole,

I'm not sure what "comes available" mean here. Does that mean VPP
would only support TLVs parsing once one becomes standardized and the
VPP maintainers think is worth supporting? As shown in the scenario
above, with that approach its difficult to develop optional TLVs if
implementations don't properly skip over them.

> If I get time, I’ll try to do some performance measurements on what the actual cost is for a software based system.
Our testing in Linux showed that the performance hit is heavily
dependent on the number of TLVs to be processed. Processing a small
number of TLVs was fairly inconsequential relative the rest of the
header processing. A a packet filled with tiny TLVs so that a single
MTU sized packet has hundreds of them will kill performance on any
software or hardware system and really has no other practical purpose
other than to be a DOS attack. This why we needed to add limits in
RFC6434bis. In Linux, the default  limit of number of options a stack
will process is eight.

Tom

---

From: Tom Herbert <[email-masked]>
Date: Mon, 20 Aug 2018 13:24:50 -0700

> From my perspective, a tunnel endpoint is a destination. If we want any given destination, an example of which is a tunnel endpoint, to not process some set of options, I would suggest that we simp…
Fred,

The problem isn't that the endpoint doesn't handle any unknown options
that would otherwise be skipped, it's that endpoints are dropping
packets simply by virtue that a Destination Options header is present.
That behavior is not consistent with RFC8200 and is detrimental to our
ability to ever define and use useful options.

Tom

---

From: Ole Troan <[email-masked]>
Date: Mon, 20 Aug 2018 23:47:47 +0200

Hi Tom,

> implementations don't properly skip over them.

No, don’t get me wrong.
As in any open source project you are welcome to provide a patch, and I’m pretty sure it would get in.

That said since we’re talking about tunnels, and in those cases you generally know whom you are talking to, I’m not sure I buy the argument that would make deploying new destination options harder. It’s not like a tunnel head end accepts traffic from random sources around the Internet (well, we’ve tried that with stuff like RFC1933, 6to4 and so on, but that didn’t quite pan out).
So you’d end up with a configuration option I’d imagine… 

> RFC6434bis. In Linux, the default  limit of number of options a stack

Right. 

Cheers,
Ole

---

From: Tom Herbert <[email-masked]>
Date: Mon, 20 Aug 2018 18:07:26 -0700

> As in any open source project you are welcome to provide a patch, and I’m pretty sure it would get in.
I would, but for me VPP code is very hard to decipher. For someone who
knows the code this should be relatively easy. The ip6_parse_tlv (in
net-next/net/ipv6/exthdrs.c) function does this in Linux in < 100 LOC.
It's straightforward, supporting padding, unknown options, and limits.

> That said since we’re talking about tunnels, and in those cases you generally know whom you are talking to, I’m not sure I buy the argument that would make deploying new destination options harder.…

I'm not sure this is just about tunnels. It looks like VPP can also
support TCP connections (there's a reference on web to HTTP server for
VPP) as well as UDP sockets, and VPP also can do classification, NAT,
and other firewall functions. Do you know if VPP properly handle
Destination Options in these cases?

I'm afraid this might be an example of an implementation deployed on
middleboxes in the Internet (like firewalls) that arbitrarily drop
packets because they have a DestOpts extension header (RFC7872).
That's exactly a principle reason why Destination Options are unusable
on the Internet, and hence why there's been so much effort to avoid
using them and no one bother's trying to create "useful" options. This
is an example of death of a good protocol feature by undermining it
with a thousand arbitrary decisions on what parts of the spec to
implement or not.

This goes goes back to a major rationale for this draft. If an
implementation really doesn't want to implement the TLV loop for
processing options then so be it, but then at least they should skip
over the options instead of just dropping the packet so as not to kill
the feature for everyone else.

Tom

---

From: Ole Troan <[email-masked]>
Date: Tue, 21 Aug 2018 12:55:18 +0200

Hi Tom,

>>>>> options "that makes sense to support" . Maybe it's an option for thenaforementioned insitu-OAM, or maybe a general packet CRC, etc. The

You have at least 4 arguments going at once here.

1) How to deal with destination options on intermediate devices terminating and forwarding traffic (tunnels, source routing)
2) Dealing with destination options when terminating traffic as a host
3) Dealing with destination options when acting as a layer violating middlebox.
4) Why after 20 years aren’t there any “useful” destination options created?

Best regards,
Ole

---

From: Tom Herbert <[email-masked]>
Date: Tue, 21 Aug 2018 08:09:45 -0700

> 3) Dealing with destination options when acting as a layer violating middlebox.

Note that that RFC8200 doesn't distiguish any of the cases. Per the
spec there is no difference between a tunnel endpoint and a end host
in how they are supposed to process Destination Options. Neither is
there is any difference between how a hardware and software
implemenation of protocol is supposed to work. Middleboxes that aren't
destinations of a packet aren't supposed to be looking at Destination
Options header at all.

> 4) Why after 20 years aren’t there any “useful” destination options created?

One could also argue that IPv6 itself has gotten enough traction until
recently to warrant interest in the creating them.

All these fall under the same root problem, namely how do we undo the
effects of non-compliant implementations for core protocol features
(i.e. Destination Options in this case). It seems like there's three
alternatives: abandon trying to use the feature and try to do
something else to get the functionality, fix implementations to be
compliant, change protocols to adapt to reality of implementation that
best preserves the feature.

VPP is an example of an implementation that should simply be fixed to
be compliant (I have taken that up with the VPP developers). This
draft updates processing of Destination Options before the Routing
header relax the requirements similar to how this was does for
Hop-by-Hop options. It makes the assumption that some intermediate
devices in a routing list won't processing the Destination Options per
the current specification. That assumption seems to be true for VPP,
and it seems like that some other implementations will also not be
processing them ostensibly because of performance hit and processing
cost.

Tom

---

From: Ole Troan <[email-masked]>
Date: Tue, 21 Aug 2018 17:26:12 +0200

> On 21 Aug 2018, at 17:09, Tom Herbert <[email-masked]> wrote:
> 

I don’t think lessening the processing requirements for destination options make any sense at all. 
Then you lose the ability to have mandatory options. 

From an implementation perspective this is largely hypothetical until some real options exist.
I don’t buy your argument that none have been developed because of the existence of middleboxes. The tests I have seen there are a very low drop probability for these through the core of the network. 

Of course any implementation, VPP included will support real options when they exist. And I will likely add a eh parsing node for nothing else than measuring performance , but I am quite unlikely to support enabling that by default just to be able to parse padding and some speculative future optional option (sic). VPP is in any case a framework for building forwarding applications, and people are of course free to do whatever they like with the DOH in their “applications “. 

(Apologies for making this very implementation specific, but these arguments might transfer to other implementations too. )

Cheers,
Ole