Subject: 6man w.g. last call for <draft-ietf-6man-segment-routing-header-11.txt>

From: Bob Hinden <[email-masked]>
Date: Thu, 29 Mar 2018 13:30:45 -0700

This message starts a two week 6MAN Working Group Last Call on advancing:

       Title           : IPv6 Segment Routing Header (SRH)
       Authors         : Stefano Previdi
                         Clarence Filsfils
                         John Leddy
                         Satoru Matsushima
                         Daniel Voyer
	Filename       : draft-ietf-6man-segment-routing-header-11.txt
	Pages          : 34
	Date           : 2018-03-28

     https://tools.ietf.org/html/draft-ietf-6man-segment-routing-header

as a Proposed Standard.  Substantive comments and statements of support
for publishing this document should be directed to the mailing list.
Editorial suggestions can be sent to the author.  This last call will
end on 12 April 2018.

An issue tracker will be setup to track issues raised on this document.

Thanks,
Bob & Ole

---

From: "Joel M. Halpern" <[email-masked]>
Date: Fri, 30 Mar 2018 10:51:24 -0400

I do not think this document is ready to be sent to the IETF and IESG 
for final approval.
There are several kinds of problems.

ECMP1: The document asserts that entropy information is put into the 
flow label.   I wish this worked.  I helped bring this idea forward 
years ago, independent of SRH.  Unfortunately, there are two related 
problems.  First, adoption appears to be low.  Second, and more 
important, there are reports from the field that doing thi actually 
breaks other things and cause packet re-ordering under some 
circumstances.  As a result, vendors are suggesting that people turn it 
off even when it is available.
ECMP2: If, with suitable analysis, we decide that this is actually safe 
to do, then the document needs to include both that analysis and clearly 
spelled out requirements that operators MUST verify that all their 
routers support this behavior and that they have enabled this behavior 
before turning on SRv6 usage.

Structure: The text in section 2.3 says "It is assumed in this document 
that the SRH is added to the packet by its source".  This is at best 
disingenuous.  It is very clear that the value of this behavior lies 
primarily in use by the network, not by the packet sources.  Claiming 
otherwise results in a document with minimal utility.  Even this 
document itself disagrees with this assertion.  Tucked into the very 
next section is text saying that "outer header with an SRH applied to 
the incoming packet".  If this behavior were a clearly spelled out 
requirement, rather than a "typically" and if the text in 2.3 were 
replaced with something realistic, then the document would at least be 
itnernally consistent and match the expected usages.

Edge filtering: The text on edge filtering does not actually state that 
prevention of packets with SRH and a current DA of an internal node is 
mandatory.  Unless it is clearly stated, the security considerations 
text as currently written is significantly weakened.  If it is 
mandatory, then again the deployment section needs to note that an 
operator needs to verify that all of his edge devices support such 
filtering and have it properly enabled in order to use SRv6.

Edge Filtering and hybrids: Other documents have talked about allowing 
external packets with SRv6 entries pointing to internal nodes (which 
means the DA upon arrival at the operator edge will be an internal node 
as I understand it).  It the intention is to permit that with 
appropriate security, then the edge filtering requirements need to be 
clear about the requirements for cryptographic validation at the edge.

Yours,
Joel

On 3/29/18 4:30 PM, Bob Hinden wrote:
> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> [email-masked]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
> --------------------------------------------------------------------
>

---

From: Tom Herbert <[email-masked]>
Date: Fri, 30 Mar 2018 08:59:44 -0700

On Fri, Mar 30, 2018 at 7:51 AM, Joel M. Halpern <[email-masked]> wrote:
> I do not think this document is ready to be sent to the IETF and IESG for
> final approval.
> ECMP1: The document asserts that entropy information is put into the flow
> label.   I wish this worked.  I helped bring this idea forward years ago,
> independent of SRH.  Unfortunately, there are two related problems.  First,
> adoption appears to be low.  Second, and more important, there are reports
> from the field that doing thi actually breaks other things and cause packet
> re-ordering under some circumstances.  As a result, vendors are suggesting
> that people turn it off even when it is available.

Joel,

I assume you're referring to:

"If an array of layer-3 adjacencies is bound to the End.X SID, then
one entry of the array is selected based on a hash of the packet's
header (at least SA, DA, Flow Label)."

The problem with flow label isn't that it can cause OOO packets, it's
interaction with stateful techniques in intermediate nodes like
firewalls and load balancer that require all packets for a flow to
consistently hit the same stateful device. When the flow label changes
in mid flow, packets might be routed to a different device that
doesn't have the flow state and hence are dropped.

I imagine that one could argue that SRH is intended to be used in
closed domains so the operator could ensure there are no stateful
nodes present that would have an issue, but that makes correct
operation of the protocol conditional.

IMO the fix for the flow label/ECMP problem needs to include an update
to RFC6437 specifying 1) the flow label can only be set by the source,
nodes in the path MUST NOT set it, 2) if the flow label is set, it
MUST be an entropy identifier of the encapsulated transport flow so
that (SA, DA, flow label) is a good tuple representation of the flow,
3) the flow label MUST be the same for all packets sent on a flow
including fragments.

Tom

> ECMP2: If, with suitable analysis, we decide that this is actually safe to
> do, then the document needs to include both that analysis and clearly
> spelled out requirements that operators MUST verify that all their routers
> support this behavior and that they have enabled this behavior before
> turning on SRv6 usage.
>
> Structure: The text in section 2.3 says "It is assumed in this document that
> the SRH is added to the packet by its source".  This is at best
> otherwise results in a document with minimal utility.  Even this document
> itself disagrees with this assertion.  Tucked into the very next section is
> text saying that "outer header with an SRH applied to the incoming packet".
> If this behavior were a clearly spelled out requirement, rather than a
> "typically" and if the text in 2.3 were replaced with something realistic,
> then the document would at least be itnernally consistent and match the
> expected usages.
>
> mandatory.  Unless it is clearly stated, the security considerations text as
> currently written is significantly weakened.  If it is mandatory, then again
> the deployment section needs to note that an operator needs to verify that
> all of his edge devices support such filtering and have it properly enabled
> in order to use SRv6.
>
> external packets with SRv6 entries pointing to internal nodes (which means
> the DA upon arrival at the operator edge will be an internal node as I
> understand it).  It the intention is to permit that with appropriate
> security, then the edge filtering requirements need to be clear about the
> requirements for cryptographic validation at the edge.
>
> Yours,
> Joel
>
>

---

From: "Joel M. Halpern" <[email-masked]>
Date: Fri, 30 Mar 2018 12:29:26 -0400

Sorry for being a bit vague.  The issue with packet reordering when 
using flowlabel for ECMP was discussed in a blog post from a senior 
engineer (at a company other than my own) who as far as I know does not 
participate in the IETF.  He went through a number of issues in that.

There are likely ways to work around the problems.  But since the draft 
does not even refer to the issue, there is a serious gap.

Yours,
Joel

---

From: "Joel M. Halpern" <[email-masked]>
Date: Fri, 30 Mar 2018 12:38:13 -0400

I responded a bit too quickly to Tom's note.
I agree with the later parts of his note.  It is the ECMP behavior of 
intermediate routers that cause me concern.
Whether his proposed fix is the right one or not is unclear, as the blog 
post I referred to suggested more issues.  (And I apologize for being 
unable to find a pointer to the blog post.)

Yours,
Joel

On 3/30/18 12:29 PM, Joel M. Halpern wrote:
>>> for
>>> flow
>>> routers
>>> document
>>> text as
>>> means

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 31 Mar 2018 11:50:32 +1300

> unable to find a pointer to the blog post.)

OK, but if this is a real issue with our RFCs, and not an implementation
defect, we need to fix it,independently of the current draft.

More below...

>>> operation of the protocol conditional.

If it's conditional on "source nodes MUST conform to [RFC6437]",
would that be a problem?

>>> nodes in the path MUST NOT set it,

We debated this at great length years ago, and it was explicitly changed
to the rules in RFC6437, because the previous rule was unrealistic. But
it defines clear rules about what forwarding nodes may do.

>>> 2) if the flow label is set, it

Wait, "encapsulated"? The flow label refers to the header it occurs in,
not to an encapsulated header, which it knows nothing about.

Ignoring this, that is the intention of RFC6437, apart from not misusing the
word 'entropy'. Please point to any text in RFC6437 that isn't clear enough.

>>> 3) the flow label MUST be the same for all packets sent on a flow

Yes, there is a problem for fragmented packets if the flow label is set by
a forwarding node, as mentioned at https://tools.ietf.org/html/rfc6437#page-7 
That isn't a problem if the source node sets the flow label, which is of
course the primary recommendation. (Therefore, it isn't a problem
in the RFC 6438 scenario, ECMP in tunnels.)

Should we be adding a BCP to that, requiring this choice:

      *  A forwarding node might use the 2-tuple to define a flow in all
         cases.  In this case, subsequent load distribution would be
         based only on IP addresses.
?
    Brian

---

From: "Joel M. Halpern" <[email-masked]>
Date: Fri, 30 Mar 2018 19:57:29 -0400

As far as I can tell, it is not sufficient for nodes originating SRH to 
be compliant with 6437.  In order for this usage to work for SRH 
packets, all IP routers in the domain have to be using the flow label 
ECMP behavior.  (This is one of the insufficiently specified assumptions 
in the draft.)
Given that all routers are applying this behavior, it follows that this 
will be applied to traffic from all sources.

Thus, for example, hosts which send traffic with varying flow labels 
(which apparently exist, even if I have no clue why they do this) will 
have their traffic damaged.

Yours,
Joel

---

From: Tom Herbert <[email-masked]>
Date: Fri, 30 Mar 2018 17:10:12 -0700

> If it's conditional on "source nodes MUST conform to [RFC6437]",
Hi Brian,

Maybe I'm missing it, but I don't think that RFC6437 plainly says that
the flow label MUST be persistent for the lifetime of the flow.
There's actually some interesting things that can be done by changing
the flow label, for instance we did this in Linux for a failing
connection to reroute the flow through the network. It worked quite
well, but inevitably someone will hit a device that expects the flow
label to be consistent. We ended up turning the turn feature off. The
default behavior in Linux now is that the flow label is consistent for
the life of TCP connections.

> it defines clear rules about what forwarding nodes may do.
Well, sort of clear at best.

A forwarding node may set a zero flow label to something non-zero so
that is a changing flow label. Also, a flow may be routed through
different devices that could set the flow label, however there's no
reason to believe they would set the label to the same value.

Section 6.1 has always puzzled to me. I'm not sure why the flow label
is particularly cited as a dangerous covert channel, I would think
something like the potential slack space in UDP would have been a much
greater concern. In any case, the conditions that the flow label could
be changed for security are not normatively described so I imagine
someone might assume some leeway.

> not to an encapsulated header, which it knows nothing about.
"Encapsulated" being used in it's broadest meaning. If there is a
transport flow in the packet, the flow label respresents that and the
3-tuple is useful for ECMP and such.

> word 'entropy'. Please point to any text in RFC6437 that isn't clear enough.
I think the description is clear, however I'm not sure if the
algorithm in Appendix A is so relevant any more. There are now many
examples of hashes created from packet headers from which the flow
label can be derived (we use Jenkin's hash in Linux, most  NICs
implement Toeplitz, etc.). An even simpler method in TCP is to just
create a random number for each connection that is stored in the
connection context from which the flow label is derived.

>          based only on IP addresses.

Then that doesn't give us flow specific ECMP which is useful for
tunnels. Using ECMP for flows is well deoployed. The opportunity with
the flow labels was to give the same functionality without routers
needing to resort to DPI to find transport headers for the hash.

Tom

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 31 Mar 2018 13:29:22 +1300

> ECMP behavior.  (This is one of the insufficiently specified assumptions 

That may well be, but does it belong in the formal spec of the new header,
or in an operational BCP? After all, out-of-order delivery is part of
standard IP delivery semantics.

> have their traffic damaged.

In the sense of potentially re-ordered, or conceivably delivered to the
wrong load-balanced server, yes. But I'm unclear how varying flow labels
can arise with conformant nodes, except in the fragmentation case. If nodes
are not conformant, that isn't our department. The fragmentation case
seems to need a fix.

   Brian

> .

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 31 Mar 2018 13:45:01 +1300

Hi Tom,
> the flow label MUST be persistent for the lifetime of the flow.

The tricky bit is defining "flow" so essentially we fixed on the idea
that a flow is whatever the source host decides is a flow. By default,
it's a given transport session. But imagine, however, that you open
multiple simultaneous TCP sessions to the same server and want them
all to be load-balanced together. (This isn't hard to imagine.)
Then maybe a better heuristic would be to give them all the same flow
label.

So yes, there is wiggle room in the RFC. But if you do what's suggested
as a default, a stateless hash of the 5-tuple, you get a given flow label
for each entire TCP connection.

> reason to believe they would set the label to the same value.

Ah, right, that could happen. Sounds like a Worst Current Practice.
 
> someone might assume some leeway.

Yes. We were certainly under the impression that some firewalls at the
time simply cleared the flow label to be on the safe side.

> 3-tuple is useful for ECMP and such.

OK.

> algorithm in Appendix A is so relevant any more.

It's not. With all due respect to Von Neumann,it's not even a very
good algorithm. But it's non-normative. 

> There are now many

Sure, if you don't mind adding 20 bits of state.

> needing to resort to DPI to find transport headers for the hash.

Sure, absent fragmentation. But with fragmentation, we will always have
problems (and DPI will fail).

If the flow label is zero and a fragment header is present, what else
can you do but use the 2-tuple?

    Brian