Subject: [v6ops] IPv6 EHs Packet Drops (Fwd: New Version Notification for draft-gont-v6ops-ipv6-ehs-packet-drops-02.txt)

From: Fernando Gont <[email-masked]>
Date: Thu, 4 Feb 2016 23:29:42 -0300

Folks,

We have published a revision of our IETF I-D "Operational Implications
of IPv6 Packets with Extension Headers". The I-D is available at:
<https://tools.ietf.org/html/draft-gont-v6ops-ipv6-ehs-packet-drops-02>.

This revision is mostly meant to:

* Address the detailed comments received from Andrew Yourtchenko

* Address the detailed revision received from Sander Steffann.

* Address Brian Carpenter's comments regarding the Flow Label

* Clearly state the goals of this document, as requested during my
presentation at the Yokohama IETF.

Your feedback will be appreciated.

Thanks!

Best regards,
Fernando

-------- Forwarded Message --------
Subject: New Version Notification for
draft-gont-v6ops-ipv6-ehs-packet-drops-02.txt
Date: Thu, 04 Feb 2016 13:46:39 -0800
From: [email-masked]
To: Nick Hilliard <[email-masked]>, Shucheng LIU (Will)
<[email-masked]>, Gert Doering <[email-masked]>, Will Liu
(Shucheng) <[email-masked]>, Fernando Gont
<[email-masked]>, Warren Kumari <[email-masked]>

A new version of I-D, draft-gont-v6ops-ipv6-ehs-packet-drops-02.txt
has been successfully submitted by Fernando Gont and posted to the
IETF repository.

Name:		draft-gont-v6ops-ipv6-ehs-packet-drops
Revision:	02
Title:		Operational Implications of IPv6 Packets with Extension Headers
Document date:	2016-02-04
Group:		Individual Submission
Pages:		15
URL:
https://www.ietf.org/internet-drafts/draft-gont-v6ops-ipv6-ehs-packet-drops-02.txt
Status:
https://datatracker.ietf.org/doc/draft-gont-v6ops-ipv6-ehs-packet-drops/
Htmlized:
https://tools.ietf.org/html/draft-gont-v6ops-ipv6-ehs-packet-drops-02
Diff:
https://www.ietf.org/rfcdiff?url2=draft-gont-v6ops-ipv6-ehs-packet-drops-02

Abstract:
   This document summarizes the security and operational implications of
   IPv6 extension headers, and attempts to analyze reasons why packets
   with IPv6 extension headers may be dropped in the public Internet.

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

The IETF Secretariat

---

From: Nick Hilliard <[email-masked]>
Date: Fri, 05 Feb 2016 13:07:14 +0000

> * Address Brian Carpenter's comments regarding the Flow Label

the flow label is more nuanced than it looks.

If you're load balancing traffic via LAG or ECMP, you need a consistent
source of flow based entropy which can be determined from an individual
packet so that you can build up a hash key which will allow the router
to send packets belonging to same flow down the same bearer link in a
bundle.  Routers need to do this because if you end up with
out-of-sequence packet delivery, performance will drop considerably.

The problem with the ipv6 flow label spec is that it does not affect how
either the source or the destination handles an individual flow.  This
means that an ipv6 source host can set a flow label to be any value
without affecting how the destination host will handle the value.  The
problem is that the flow label is expected to influence how intermediate
routers process the packet, in particular which bearer links they might
choose in a bundle.

In other words, because the end hosts can control the flow label
arbitrarily, control of LAG/ECMP hashing is substantially put in the
hands of the end hosts, rather than the network operator.  This is the
case because the end hosts can tune the LAG/ECMP hash value without
affecting any other characteristics of a particular data flow.  This
strikes me as being a poor idea.

The relevance of this to draft-gont-v6ops-ipv6-ehs-packet-drops is that
it means that it's a particularly bad idea to depend on the flow label
as the sole source of entropy for LAG / ECMP hashing.  This in turn
means that it would be advisable to depend on other consistent entropy
sources (e.g. the normal 5-tuples).  These can only easily be determined
if the ipv6 header doesn't contain EHs.

Nick

---

From: Joe Touch <[email-masked]>
Date: Fri, 5 Feb 2016 06:56:30 -0800

> if the ipv6 header doesn't contain EHs.

Why do you think that any other endpoint assigned values in these
5-tuples would not have the same problems as the flow label? I.e.,
either they have no entropy (fixed per host) or are assigned within the
host in a way that might not optimize entropy at all.

So there's no reason to worry about EHs for this issue. Either you trust
the endsystem entropy or you don't.

Joe

---

From: Nick Hilliard <[email-masked]>
Date: Fri, 05 Feb 2016 15:34:47 +0000

> the endsystem entropy or you don't.

You're misunderstanding what I'm saying, which is that having a flow
label field doesn't obviate the requirement for core boxes to pull
entropy from the normal src/dst ip/port n-tuples.  If there are EHs in a
packet, fallback to using only the flow label is probably then not a
good idea.

Nick

---

From: Tom Herbert <[email-masked]>
Date: Fri, 5 Feb 2016 07:35:48 -0800

> the endsystem entropy or you don't.
+1

The flow label used for ECMP is critical for us moving forward. Use of
the 5-tuple restricts us to a using a small handful of protocols that
the intermediate network HW arbitrarily decides is worth supporting,
which seems to just be UDP/IP and TCP/IP with no IP options or ext.
hdrs. Using flow label for ECMP is a big step in the battle against
protocol ossification.

With IPv6 flow label devices don't need to parse beyond the IP header
just to switch packets. This means we can deploy any protocol or
protocol feature we want and still get the value of flow based ECMP if
the end host sets the flow label accordingly (which is already the
default in OSes). It has the additional advantage that we can change
the flow label for a flow that is having problems to try to get a
better route through the network.

Tom

> Joe
>
> _______________________________________________
> v6ops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/v6ops

---

From: Joe Touch <[email-masked]>
Date: Fri, 5 Feb 2016 07:39:42 -0800

> packet, fallback to using only the flow label is probably then not a

If you're saying "the 5 tuple adds more entropy than is available in the
flow label", maybe or maybe not. It depends on how the flow label is
assigned.

IMO, it's not worth the risk and effort of digging out the 5-tuple for
this potential increase. (risk = brittleness, e.g., won't work when
payloads are eventually all encrypted).

Joe

---

From: Gert Doering <[email-masked]>
Date: Fri, 5 Feb 2016 16:47:22 +0100

Hi,

> the endsystem entropy or you don't.

There's more than one end system out there.

Nothing can stop them from all using the same flow label value, like "0",
but at least one part of the 5-tuple *will* differ for different senders.

Just sayin'

Gert Doering
        -- NetMaster
-- 
have you enabled IPv6 on something today...?

SpaceNet AG                        Vorstand: Sebastian v. Bomhard
Joseph-Dollinger-Bogen 14          Aufsichtsratsvors.: A. Grundner-Culemann
D-80807 Muenchen                   HRB: 136055 (AG Muenchen)
Tel: +49 (0)89/32356-444           USt-IdNr.: DE813185279

---

From: Joe Touch <[email-masked]>
Date: Fri, 5 Feb 2016 07:52:07 -0800

> but at least one part of the 5-tuple *will* differ for different senders.

The only thing you know will differ is the IP address pair (as a tuple).
The rest might or might not.

That argues for flowlabel + IPaddrs and just that.

Digging deeper to get to transport protocol and corresponding port
numbers seems like a lot of work for unpredictable benefit.

Joe

---

From: Nick Hilliard <[email-masked]>
Date: Fri, 05 Feb 2016 16:22:11 +0000

> payloads are eventually all encrypted).

As Gert suggested, flow labels are often set to zero by default.  It's
not helpful when your entropy source is constant across flows.

Nick

---

From: Nick Hilliard <[email-masked]>
Date: Fri, 05 Feb 2016 16:29:17 +0000

> numbers seems like a lot of work for unpredictable benefit.

Having had to deal extensively with lag/ecmp problems on large public
production networks, I'm going to politely disagree with you.

Nick

---

From: Tom Herbert <[email-masked]>
Date: Fri, 5 Feb 2016 08:34:08 -0800

> not helpful when your entropy source is constant across flows.
The 3-tuple is still usable. This situation is not different than
device trying to switch a packet with a transport protocol it can't
parse where the fallback is just to use 3-tuple. Besides that, default
for IOS and now Linux will be to set the flow label to non-zero based
on a hash of the transport flow (I haven't looked into other OSes).

Tom

---

From: Nick Hilliard <[email-masked]>
Date: Fri, 05 Feb 2016 16:46:56 +0000

Tom Herbert wrote:
> The 3-tuple is still usable.

unless the flow label is set, the 3-tuple will just be a src/dst ip
tuple, and load balancing on src/dst ip is, in my experience, useless.

There is no guarantee that a flow label will be set.

Nick

---

From: Joe Touch <[email-masked]>
Date: Fri, 5 Feb 2016 08:49:54 -0800

> production networks, I'm going to politely disagree with you.

I don't think it's useful to make recommendations even if they're based
on existing practice when we see it sunsetting fairly soon - either
because of network-layer encryption, tunnels, etc.

Joe

---

From: Nick Hilliard <[email-masked]>
Date: Fri, 05 Feb 2016 16:55:04 +0000

> because of network-layer encryption, tunnels, etc.

It's not clear what practices/protocols you're referring to when you say
sunsetting?

Nick

---

From: Joe Touch <[email-masked]>
Date: Fri, 5 Feb 2016 09:00:41 -0800

> It's not clear what practices/protocols you're referring to when you say

DPI to extract transport context for entropy inside the network.

Joe

---

From: Fernando Gont <[email-masked]>
Date: Fri, 5 Feb 2016 15:07:57 -0300

> DPI to extract transport context for entropy inside the network.

I've seen increased deployment of *TLS*... but TLS still allows you to
extract the transport-related information....

Cheers,
-- 
Fernando Gont
SI6 Networks
e-mail: [email-masked]
PGP Fingerprint: 6666 31C6 D484 63B2 8FB1 E3C4 AE25 0D55 1D4E 7492

---

From: Nick Hilliard <[email-masked]>
Date: Fri, 05 Feb 2016 19:01:58 +0000

> DPI to extract transport context for entropy inside the network.

If you're referring to IPsec when you say "network layer encryption", I
haven't noticed any proportional increase as a long term trend.  It's
still rarely more than line noise in almost all situations.

As Fernando mentioned, TLS usage has increased substantially as a
percentage of overall tcp flows, but that doesn't impact load balancing
because it allows extraction of L4 n-tuples in the same way as if the
traffic were unencrypted.

Tunnels other than ipsec tunnel mode exist, but are in the minority in
terms of overall traffic flows on the internet.

I don't see any prospect of sunsetting inspection of layer 4 info on the
basis of your suggestions, even in the long term as a potential possibility.

Nick

---

From: [email-masked]
Date: Fri, 05 Feb 2016 20:03:48 +0100 (CET)

> extract the transport-related information....

I think "fairly soon" needs to be defined. I believe extracting
entropy from address and port numbers will continue to be used at
least as long as we have a significant amount of IPv4 traffic.

Steinar Haug, AS 2116

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 6 Feb 2016 08:41:53 +1300

> tuple, and load balancing on src/dst ip is, in my experience, useless.

Really, with IPv6 addresses? There tends to be a fair amount of entropy
in the bottom 64 bits of the client address.

With the 5-tuple, all you really gain is the entropy in the ephemeral port
number. Adding a well-known port to the hash doesn't help much.

> There is no guarantee that a flow label will be set.

That's true, although we have changed the standard to make it more likely
that future stacks will do so.

But the thing is, what else are you going to do to help load balancing in
the case that the transport header is inaccessible? What do you lose in that
case by adding the flow label to the hash? If it's zero, the hash doesn't get
worse. If it's been set according to RFC6437, the hash gets better.

(I'm assuming that we aren't talking about the RFC6438 scenario.)

    Brian

---

From: Tom Herbert <[email-masked]>
Date: Fri, 5 Feb 2016 12:01:11 -0800

> (I'm assuming that we aren't talking about the RFC6438 scenario.)
RFC6438 is very relevant to this discussion! There is no standard way
to deduce a 5-tuple hash for an encapsulated flow in encapsulation
protocols (GRE, MPLS, IPIP, IPsec, etc.) and in this case the
addresses don't contain any entropy about the flows going over the
tunnel. We really only have two options to get flow based ECMP with
encapsulation: 1) foo-over-UDP encapsulation or 2) Flow labels. UDP
encapsulation "works" but adds yet another layer, complexity, and byte
overhead and doesn't help to solve the EH problem anyway. Flow labels
are really the best solution once you're on IPv6.

Tom

---

From: Nick Hilliard <[email-masked]>
Date: Fri, 05 Feb 2016 20:40:12 +0000

> in the bottom 64 bits of the client address.

that's not the point.  If you depend solely on src/dst ip, all traffic
flows between those two endpoints will be hashed over the same bundle
member.  This does two things: 1. it restricts the maximum bandwidth
between those two hosts to the amount of free bandwidth on the bundle
member in question and 2. it trashes the link for other people,
potentially causing performance problems.

If you add in a flow label as an entropy source, it may change the hash
on a per stream basis, but only if the end-point operating system has
set the flow label to be nonzero.  This is not guaranteed to happen
because the flow label plays no part in the transmission mechanism
between the two end points, only on the intermediate path elements of
the transmission.  I.e. from the point of view of an end point, it's
unnecessary.

> that future stacks will do so.

it's well established that changing the standard doesn't necessarily
change the implementations.  It would be good to have recent data on
this - the most recent formal paper I've seen on this dates from 2005
which is way too old (http://www.maths.tcd.ie/~dwmalone/p/ec2nd05.pdf)

Nick

---

From: Fernando Gont <[email-masked]>
Date: Fri, 5 Feb 2016 17:34:33 -0300

> least as long as we have a significant amount of IPv4 traffic.

just curious: why do you think IPv6 would make a different in this
respect? -- because extracting such information can be difficult and
hence everyone would shift to employing the flow-label, or something else?

Thanks!

Cheers,

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 6 Feb 2016 11:25:11 +1300

On 06/02/2016 09:40, Nick Hilliard wrote:
> member.  

Which doesn't matter in the least if you have thousands of flows
from hundreds of sources on the link. I agree that it matters if you
have multiple flows from one particular bandwidth hog; things like
GridFTP for example, which is designed to take an unfair share by
grabbing multiple ports for one application flow.

> This does two things: 1. it restricts the maximum bandwidth

Yes. It's 4 years since RFC6437 so it might almost be worth looking to
see if things have changed. But this is a long game.

   Brian

---

From: Nick Hilliard <[email-masked]>
Date: Fri, 05 Feb 2016 22:41:16 +0000

Brian E Carpenter wrote:
> from hundreds of sources on the link.

in practice, it matters a great deal.  10G and 40G server connectivity
is now commonplace.  If your egress or core bundle members are of the
same order of magnitude as your edge, this will create visible elephant
flows in usage graphs.

Nick

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 6 Feb 2016 12:47:18 +1300

> same order of magnitude as your edge, this will create visible elephant

I completely understand that for IPv4 traffic, but this is about IPv6,
where there are potentially 64 bits of entropy in the low-order bits
of the address. The high-order address bits don't add much entropy.

I'm not just saying that; it can be seen in the table headed Trace 4
in https://www.cs.auckland.ac.nz/~brian/flowhashRep.pdf, where the rows
for Algorithms 2 and 6 are with and without the top 64 bits of the
addresses.

   Brian

---

From: joel jaeggli <[email-masked]>
Date: Fri, 5 Feb 2016 16:02:50 -0800

> of the address. The high-order address bits don't add much entropy.

today in my world we have polarization of flows between hosts talking to
each other using 4 x 10Gbe nics on either end so oddly I don't see
address entropy as enough ever... maybe source dest and flow label are
enough if the latter is actually an entropy source. but the source/dest
are not.

---

From: Tom Herbert <[email-masked]>
Date: Fri, 5 Feb 2016 16:19:23 -0800

> terms of overall traffic flows on the internet.
Then it's deadlock: we can't widely deploy protocols that are not well
supported on the Internet, and protocols won't get proper support
unless they are widely deployed. This is protocol ossification!

Within the enterprise network this story is very different. Tunnels
are quite common and we will be using EH as well as alternative
transport protocols and flow labels really are the only feasible
solution to get the necessary input entropy for ECMP for encapsulated
flows. IMO support for using flow label for ECMP should now be
considered a requirement for anyone selling networking gear to the
enterprise.

> basis of your suggestions, even in the long term as a potential possibility.
It's not so much a question of sunsetting DPI, but enabling the use of
flow labels in ECMP to cover use cases where DPI fails to find a
5-tuple (e.g. ext. hdrs., alternative transport protocols,
encapsulation protocols).

Tom

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 6 Feb 2016 13:25:17 +1300

> enough if the latter is actually an entropy source. but the source/dest

If you are up against GridFTP or the like, yes, that will happen
and entropy from port numbers is your only hope.

OTOH it should be in the host's own interest to set a useful flow
label on each subflow.

    Brian

---

From: Mark Smith <[email-masked]>
Date: Sun, 7 Feb 2016 07:57:15 +1100

> basis of your suggestions, even in the long term as a potential possibility.

The fundamental problem is that you're trying to solving a layer 2
problem using layer 3 or layer 4 information. If layer 3 or layer 4
information changes or isn't there, your layer 2 solution isn't
anywhere as near as effective or effective at all. LAG is not a
complete solution to more layer 2 link capacity because of this
limitation.

By using layer 3 and layer 4 information to try to solve your layer 2
problem, you're in effect merging the layers, which would mean that
layer 3 or layer 4 changes become limited to occurring when layer 2
changes.

Yet the purpose of layers is to decouple them, so that one layer can
be changed or completely swapped without the others having to be
changed or completely swapped at the same time.

If LAG works for you as a layer 2 lack of capacity solution, then
that's fine, keep using it. However, I don't think changes in other
layers should be prevented just to make LAG work, when it itself
doesn't really fully solve the problem it is intending to - lack of
layer 2 bandwidth.

"LAG is good, but it’s not as good as a fatter pipe"
http://www.ieee802.org/3/hssg/public/apr07/frazier_01_0407.pdf

---

From: Nick Hilliard <[email-masked]>
Date: Sat, 06 Feb 2016 21:42:27 +0000

> doesn't really fully solve the problem it is intending to - lack of

we're talking about both LAG and ECMP - the underlying load-sharing
methodologies are the same for each on most platforms.  In other words,
we're talking about lack of link bandwidth, not lack of specifically
layer 2 bandwidth.

> http://www.ieee802.org/3/hssg/public/apr07/frazier_01_0407.pdf

No doubt well intentioned, that is not a very helpful suggestion.

Nick

---

From: Mark Smith <[email-masked]>
Date: Sun, 7 Feb 2016 09:16:41 +1100

> we're talking about lack of link bandwidth, not lack of specifically

I'm not sure I really see the distinction, isn't link bandwidth layer
2 bandwidth?

I think in either case of ECMP or LAG, the main issues crop up and the
constraints on other layers occur when other layers' information is
used as an input. If only the information within the layer that ECMP
or LAG is operating in is used, then the layer boundaries are
preserved.

> No doubt well intentioned, that is not a very helpful suggestion.

It's an observation that even the people who specified it aren't
recommending it when you have the option of a fatter pipe.

(My recent experience with LAG issues in the last couple of years at a
carrier basically showed that 4 x 1Gbps links in a LAG ended up being
false economy compared to going straight to 1 x 10Gbps for edge
aggregation of layer 2 services - broadcasts, multicasts and any
non-IPv4/TCP/UDP protocols weren't balanced well or at all over the 4
x 1Gbps links. The chips in the devices were pretty smart - if the
layer 4 info wasn't available, they'd still LB on layer 3 etc., but
that still didn't balance the traffic enough that 4 x 1Gbps was really
as good as it should have been. I've subsequently thought that LAG
would be better implemented how Multilink PPP works - split the frames
up, spread the parts across all of the links, then put the parts back
together at the other end. That makes it not depend on any upper layer
protocol values/fields or any other attributes of the frames. I'm
guessing that hasn't been done because it isn't practical to do at
high speed.)

Regards,
Mark.

---

From: Nick Hilliard <[email-masked]>
Date: Sat, 06 Feb 2016 22:40:49 +0000

> or LAG is operating in is used, then the layer boundaries are

preserving layer boundaries is laudable, but operators have networks to run.

> recommending it when you have the option of a fatter pipe.

which is fine, except that the option of a fatter pipe is often not viable.

> x 1Gbps links.

the cost cut-over point for 1G to 10G is usually around 3x1G.  However
the breakeven point for 10G to 100G is - depending on the type of kit
chosen - currently around 8x.  If you need more than 100G - and many
networks do - you need LAGs or ECMP because there is nothing faster at
the moment.

> protocol values/fields or any other attributes of the frames.

one of the design goals of LAG and ECMP is that they can be run over
links of unequal load and/or latency.  This means that packet reassembly
would require the receiving network device to maintain a large amount of
state.  This is an expensive proposition which does not scale well.

Nick

---

From: Mark Smith <[email-masked]>
Date: Sun, 7 Feb 2016 11:20:36 +1100

> preserving layer boundaries is laudable, but operators have networks to run.

Which really just sounds like you're saying that operators should
never think and never have to think about long term possible costs and
consequences of their technical choices.

I've worked on and had to untangle a number of networks that have been
built with that sort of mindset. I don't want to see that short of
short sightedness encoded into the Internet protocols.

> which is fine, except that the option of a fatter pipe is often not viable.

Well then, you have to put up with your LAG and ECMP limitations. That
is the cost of not being able to use a fatter pipe. The compromise of
LAG instead of a fatter pipe is not free, and should not be made
cheaper at the cost of upper layer flexibility in the Internet and in
particular on other operators' networks.

> networks do - you need LAGs or ECMP because there is nothing faster at

I think there is. If you scale out the number of links, the number of
devices they attach to, and the paths into and out of those devices,
you'll get the bandwidth you need, less single points of failure and
lower consequences when there is a failure of a device or link.

> links of unequal load and/or latency.

It is not a specific design goal of LAG, although it isn't precluded.
Even the actual load distribution method is unspecified.

>From 802.1AX,

"Aggregation of links of different data rates is not prohibited nor
required by this standard. Determining how to distribute traffic
across links of different data rates is beyond the scope of this
standard.

NOTE 2—Previous versions of this standard restricted the data rate of
the aggregated links. However, the specified mechanisms did not depend
on whether the links operated at the same rate or not."

>  This means that packet reassembly

> state.  This is an expensive proposition which does not scale well.

"Expensive" is a relative term. If much more effective and
non-protocol field value/existence dependent balancing over 4 x 1 Gbps
links was less cost than a 10Gbps link to carry the same traffic, it
wouldn't be expensive, it would be cheap.

Regards,
Mark.

---

From: Nick Hilliard <[email-masked]>
Date: Sun, 07 Feb 2016 13:12:12 +0000

> consequences of their technical choices.

No, I'm saying that LAGs and ECMP are here to stay, and no amount of
wishful thinking is going to stop them from being a reality that needs
to be dealt with.  If you don't need functional load-balancing on your
network, then good for you, but most network operators do.

Nick

---

From: [email-masked]
Date: Sun, 07 Feb 2016 14:30:39 +0100 (CET)

> network, then good for you, but most network operators do.

As an example of this - we were told by Juniper that when they were
introducing 100G, several years ago, they had very specific requests
from some of their biggest customers that the 100G interfaces needed to
support link aggregation right from the start - because those customers
*already then* had a need for link capacities greater than 100G.

(And yes, I absolutely agree that LAGs and ECMP are here to stay. And I
don't believe peeking at the L3 and L4 info to ensure link distribution
is going to disappear any time soon.)

Steinar Haug, AS2116

---

From: [email-masked]
Date: Sun, 07 Feb 2016 14:38:35 +0100 (CET)

> hence everyone would shift to employing the flow-label, or something else?

I actually *don't* think IPv6 will make a significant difference (at
least not until use of the flow label for entropy is *much* more
widespread). However, IPv4 doesn't have the luxury of the flow label,
and thus link aggregation entropy basically *must* be gained from L3
and L4 info.

Steinar Haug, AS2116

---

From: Philip Homburg <[email-masked]>
Date: Sun, 07 Feb 2016 15:02:05 +0100

>is going to disappear any time soon.)

I could be wrong here, but as far as I know the reason for peeking at L3/L4
is to avoid reordered packets.

So an alternative way to go forward is to fix the higher level protocols
to deal with out of order packets and then switch to statistical multiplexing.

---

From: [email-masked]
Date: Sun, 07 Feb 2016 15:21:51 +0100 (CET)

> is to avoid reordered packets.

You identify one "flow" by peeking at L3/L4 info, and the you ensure
that all packets for that flow are transmitted over the same link, to
avoid reordering. But you *also* gain a better distribution of flows
across links.

> to deal with out of order packets and then switch to statistical multiplexing.

Possibly. Not holding my breath waiting for this to happen, though.

Steinar Haug, AS2116

---

From: Philip Homburg <[email-masked]>
Date: Sun, 07 Feb 2016 15:28:47 +0100

>Possibly. Not holding my breath waiting for this to happen, though.

One option is to do statistical multiplexing for packets protected by an
EH. A small extension header inside EH can easily deal with reordered 
packets. Or people using IPSEC tunnels can try to get TCP implementations
fixed.

---

From: Havard Eidnes <[email-masked]>
Date: Sun, 07 Feb 2016 15:53:06 +0100 (CET)

> I could be wrong here, but as far as I know the reason for
> peeking at L3/L4 is to avoid reordered packets.

I think it's also been pointed out that an important factor is to
get a better distribution for the load balancing.

> So an alternative way to go forward is to fix the higher level
> protocols to deal with out of order packets and then switch to
> statistical multiplexing.

Higher-level protocols "deals" with out-of-order packets just
fine from a correctness perspective.  It's just that the higher-
level protocols are highly optimized for the usual case, i.e. in-
order packet delivery.

Optimizing higher-level protocols (TCP) for out-of-order delivery
doesn't magicially happen just because someone wishes for it...
And ... I'd claim that if it was easy, it would already have been
done.

Regards,

- Håvard

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 8 Feb 2016 08:09:23 +1300

> And ... I'd claim that if it was easy, it would already have been

This is something that network coding mitigates to some degree*, but the
fact that you have to use something as complex as network coding shows that
it is indeed a hard problem. So yes, put it in the "science fiction"
bucket for now.

Getting all sources to set the flow label is physically possible, and seems
to me to be the best hope of making the use of extension headers compatible
with ECMP/LAG. But the operations community would need to push the operating
systems community hard to make it happen.

   Brian

* http://www.hindawi.com/journals/ijdsn/2015/379108/

---

From: Mark Smith <[email-masked]>
Date: Mon, 8 Feb 2016 06:41:17 +1100

On 8 February 2016 at 00:12, Nick Hilliard <[email-masked]> wrote:
> to be dealt with.

So no amount of wishful thinking will stop EHs or some other depth of
encapsulation that invalidates layer 2 inspection of layer 3 and layer
4 fields and values for ECMP and LAG either.

There is a specific field in the IPv6 fixed header to be used for this
and similar purposes - the flow label field. You're allowed to
override it's value if a host sets it to zero (and if you override it
in *your* network when it isn't zero, nobody outside your network is
going to know either).

Use the tool provided for the purpose, and insist that your LAG/ECMP
vendor considers the flow label field in their load distribution
function, as per the RFC.

(and in a network virtualisation context, layer 2 information could be
made visible to layer 3 ECMP without violating layers - see section 5
https://tools.ietf.org/html/draft-smith-enhance-vne-with-ipv6-07)

> If you don't need functional load-balancing on your

---

From: Nick Hilliard <[email-masked]>
Date: Sun, 07 Feb 2016 21:37:29 +0000

Mark Smith wrote:
> You're allowed to override it's value if a host sets it to zero (and
> if you override it in *your* network when it isn't zero, nobody
> outside your network is going to know either).

How are you going to do that in a useful way without inspecting L4 values?

> function, as per the RFC.

this is precisely the problem: it's not my vendor, it's someone else's
vendor.  I'm transporting their packets over my link bundles.

If these packets don't have a flow label set then it comes back to a
choice between poor quality load balancing (which affects me as a
transport provider) or else trying to inspect L4 info to either do
ad-hoc load balancing or to use that info to create a new flow label.

Nick

---

From: joel jaeggli <[email-masked]>
Date: Sun, 7 Feb 2016 18:37:55 -0800

> consequences of their technical choices.

we do a lot of things that involve tradeoffs... some of them are bad.

building networks without L3/L4 ECMP at any signficant scale is not an
option. so  lot of people have to live with that tradeoff.

---

From: Joe Touch <[email-masked]>
Date: Sun, 7 Feb 2016 22:54:49 -0800

Hi, all,

I get the impression the most useful way forward is:

- use the flow field where it is available and nonzero (i.e., follow
available intent where indicated)

- if you have to dig it out from other layers, use it to fill in a flow
label if it's available

- always assume that any method to dig into other layers not only can
fail, but should be avoided in the future because it is more likely to
fail as time goes by

I.e., any document prepared now should not assume continued reliance on
the visibility of upper layers indefinitely. Otherwise, you're
effectively recommending that security that obscures upper layer headers
NOT be deployed, and that seems inappropriate.

Joe

---

From: Mark Smith <[email-masked]>
Date: Mon, 8 Feb 2016 18:20:10 +1100

+1

I'm not against layer 2 and layer 3 devices peaking at other layers' field
values, as long as they are very careful about how they use them*.

What I am against is layer 2 limited  workarounds (limited workarounds
because they don't increase capacity for all upper layer traffic) being
used to justify freezing upper layer development and evolution.

(* I have been burned by a layer 2 device interpreting IP DST of 0.0.0.0 as
a debug switch, causing my testing to fail - I didn't change the testers
default layer 3 test traffic addresses because I was testing layer 2, and
wasted 2 weeks going thorough various support channels before finding out
what was going on.)

---

From: [email-masked]
Date: Mon, 8 Feb 2016 09:59:44 +0100

> and thus link aggregation entropy basically *must* be gained from L3

what's the ratio of flow label = 0 to set flow labels in your network?
anyone studied this over time?

Best regards,
Ole

---

From: [email-masked]
Date: Mon, 08 Feb 2016 14:45:58 +0100 (CET)

> anyone studied this over time?

Highly initial estimate: Flow label = 0 accounts for around 99% of the
IPv6 traffic in my network. YMMV.

Steinar Haug, AS 2116

---

From: Warren Kumari <[email-masked]>
Date: Mon, 08 Feb 2016 14:54:04 +0000

> ad-hoc load balancing or to use that info to create a new flow label.
Another thing to keep in mind with flow labels is that attackers can use
them to steer traffic *in an existing connection*, which makes targeting a
specific link easier.
It you are an attacker, it is *much* simpler to start up a bunch of large
flows and twiddle the flow label until you observe an effect (because
you've managed to hash >N traffic onto an N sized link) than it it so guess
achieve the same thing by fiddling with port numbers, etc - you have more
time and feedback.
Attackers *enjoy* filling individual links (it makes DoS easier), which is
one of the reasons some vendors add a (per-LAG) nonce into the hash, and /
or refuse to discuss the hash function - if attackers know the hash, they
can preplan attack traffic that hashes to a given link....

W

---

From: Nick Hilliard <[email-masked]>
Date: Mon, 08 Feb 2016 15:54:26 +0000

> fail, but should be avoided in the future because it is more likely to

No, always assume that any method to dig into other layers not only can
fail, but is likely to fail if EHs are present in the packet.

Re: https://www.ietf.org/mail-archive/web/v6ops/current/msg21675.html
(for posterity: "Date: Mon, 16 Mar 2015 09:29:35 +0100")

It's too late to fix Steve Deering's intentions to make it hard for
middleboxes to examine extension headers, but the design choice created
a set of problems which are extremely difficult to deal with.

Nick

---

From: Tom Herbert <[email-masked]>
Date: Mon, 8 Feb 2016 16:54:40 +0100

On Mon, Feb 8, 2016 at 3:54 PM, Warren Kumari <[email-masked]> wrote:
> flows and twiddle the flow label until you observe an effect (because you've
> managed to hash >N traffic onto an N sized link) than it it so guess achieve
> the same thing by fiddling with port numbers, etc - you have more time and
> feedback.

Is this a significantly worse attack than: 1) UDP attack with
randomizing source port 2) TCP SYN attack with random source ports 3)
Sending TCP/IPv6 packets with an EH so that devices fallback to doing
3-tuple hash?

> can preplan attack traffic that hashes to a given link....

That's why hashes should be randomly keyed and the keys rotated
periodically (I believe at least the first is commonly done?)

Thanks,
Tom

---

From: Nick Hilliard <[email-masked]>
Date: Mon, 08 Feb 2016 15:59:01 +0000

Tom Herbert wrote:
> randomizing source port 2) TCP SYN attack with random source ports

yes, it is significantly worse.  Randomised src/dst ports for tcp/udp
floods will end up with ~equal load balancing.

> 3) Sending TCP/IPv6 packets with an EH so that devices fallback to
> doing 3-tuple hash?

It is equivalently harmful as this.

Having said that, lots of networks drop packets with extension headers.

Nick

---

From: Tom Herbert <[email-masked]>
Date: Mon, 8 Feb 2016 17:02:06 +0100

> floods will end up with ~equal load balancing.
Randomize was a poor choice of words, I meant using the source port to
do probing like in flow label attack.

---

From: Warren Kumari <[email-masked]>
Date: Mon, 08 Feb 2016 16:27:30 +0000

On Mon, Feb 8, 2016 at 7:54 AM Tom Herbert <[email-masked]> wrote:

> you've
Perhaps, perhaps not.

Managing to get all of your attack traffic onto a single LAG member means
that you only need N bps of attack traffic to cause issues, not N * M bps
(where N is the size of the LAG members, and M is the number) - you steer
all of your traffic onto one link, saturate it and then wait. Usually LACP
will fail, and the link falls out of the bundle - you now steer onto the
next link (keeping in mind that the bundle now has less bandwidth), lather,
rinse, repeat.

If you have a number of long lived TCP flows, and can steer their LAG
assignment by twiddling flow labels (without changing anything else) you
get good telemetry (the stack even does some nice stats collections for
you!), and so tell when you have gotten a bunch of flows to hash onto the
same link.

A UDP / TCP SYN attack doesn't provide you with nearly as much feedback to
know when you are hashing onto the same link. You *can* achieve similar
results by making lots of TCP connections and choosing the source port
yourself, but this is much more time consuming, not nearly as convenient,
requires mote flows, and shows up more in analysis....

> periodically (I believe at least the first is commonly done?)

Vendors are very reluctant to talk about their hashing strategies - some do
include a nonce / key but I'm not sure if anyone rotates this in use - it
would cause some sloshing...
The "hashing: that vendors use is actually often quite complex  - for
example, even if you have lots of small flows, with lots of entropy, you
often get distinctly unbalanced links. Some vendors also *appear* to try
hard to keep traffic on a link it is already hashed to - and so if a new
link is added, it seems to run significantly "cooler"quite a while. This is
one of those things that doesn't really seem to make sense[0], but does
show up in practice.

W
[0]: How can you do this without having to keep state?

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 9 Feb 2016 08:19:27 +1300

> - if you have to dig it out from other layers, use it to fill in a flow

There are some restrictions on middleboxes doing that in RFC 6437.

On 08/02/2016 21:59, [email-masked] wrote:
...
> anyone studied this over time?

We're planning for the long term here. While I agree that this is a very
interesting question, we shouldn't base future practice on it.

On 09/02/2016 05:27, Warren Kumari wrote:
...
> you!), and so tell when you have gotten a bunch of flows to hash onto the

Interesting. This wasn't really in scope for RFC 6438. We covered some related
points in the security considerations of RFC 7098 (Flow Label for Load Balancing
in Server Farms) but maybe this needs writing up.

   Brian

---

From: Joe Touch <[email-masked]>
Date: Mon, 8 Feb 2016 11:49:03 -0800

> fail, but is likely to fail if EHs are present in the packet.

My point is that EHs aren't the only issue, and thus deprecating EHs is
a temporary fix that isn't justified by this issue.

...
> a set of problems which are extremely difficult to deal with.

Well, I could turn it around and say that if it "hurts" when you do that
(i.e., use middleboxes), then perhaps the problem lies in what you're
trying to do, not what gets in its way.

I.e., middleboxes are the design choice that has caused that problem.

Joe

---

From: Nick Hilliard <[email-masked]>
Date: Mon, 08 Feb 2016 21:00:50 +0000

> a temporary fix that isn't justified by this issue.

we need to agree to disagree on this then.  The draft only attempts to
raise awareness of some operational issues.

> I.e., middleboxes are the design choice that has caused that problem.

Could you read the draft and comment on the issues that it raises?

Nick

---

From: Joe Touch <[email-masked]>
Date: Mon, 8 Feb 2016 13:36:47 -0800

...
> Could you read the draft and comment on the issues that it raises?

Sec 4.1 and 4.1.2 has been the context for my views.

Joe

---

From: Nick Hilliard <[email-masked]>
Date: Mon, 08 Feb 2016 21:40:38 +0000

> Sec 4.1 and 4.1.2 has been the context for my views.

Any take on the other subsections in 4.1?

Nick

---

From: Fernando Gont <[email-masked]>
Date: Mon, 8 Feb 2016 17:39:52 -0300

> in Server Farms) but maybe this needs writing up.

Our intent to do something along these lines was:
<https://tools.ietf.org/html/draft-gont-6man-flowlabel-security-03>.
Although the coverage of the topic under discussion is, if anything,
very poor.

I may resubmit some version of this.. if anything, for the sake of
discussion.

Thanks!

Best regards,

---

From: Fernando Gont <[email-masked]>
Date: Mon, 8 Feb 2016 18:41:00 -0300

On 02/08/2016 04:49 PM, Joe Touch wrote:
[....]
> I.e., middleboxes are the design choice that has caused that problem.

Real-world networks run on a variety of middle-boxes. At the end of the
day, it turns out that if what you try to deploy does not work nicely
with the devices you have in the realworld (e.g., a plethora of
middleboxes), the end result is that your protocol is not deployable.

The current situation with EHs is that you cannot really rely on them.
As long as they are not friendly with the deployed world, the end result
is drops rates of around 30%-50% which means that in practice you cannot
really rely on them. So eventually it becomes a question of of whether
you want to "be friendly with the network, and be deployable", or
whether you want your packets to be dropped.

I'm not saying that the situation is "nice"... but it's the current reality.

Cheers,

---

From: Joe Touch <[email-masked]>
Date: Mon, 8 Feb 2016 13:46:03 -0800

> The current situation with EHs is that you cannot really rely on them.

The current situation with EHs<DEL><DEL><DEL>middleboxes is that you
cannot really rely on them.

---

From: Joe Touch <[email-masked]>
Date: Mon, 8 Feb 2016 13:48:13 -0800

> Any take on the other subsections in 4.1?

I disagree with most of them for similar reasons, but I've already
raised these issues repeatedly in v6ops.

Continuing to dumb down the net to support the busted revenue model of
vendors at the expense of the Internet architecture is not something I
intend to support by detailed engagement.

Joe

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 9 Feb 2016 10:52:22 +1300

On 09/02/2016 10:41, Fernando Gont wrote:
> As long as they are not friendly with the deployed world, 

It takes two to tango. The middleboxes need to implement RFC7045. Until
that happens, no amount of accommodation by the hosts will solve the
problem.

    Brian

> the end result

---

From: Nick Hilliard <[email-masked]>
Date: Mon, 08 Feb 2016 21:57:02 +0000

> intend to support by detailed engagement.

Some of us need to deal with these problems though.

Do you have any workable alternatives?

Nick

---

From: Joe Touch <[email-masked]>
Date: Mon, 8 Feb 2016 14:09:28 -0800

> Do you have any workable alternatives?

I gave you some when you implied I was off-topic and hadn't read your doc.

Notably:

	- focus on the difference between core net devices and those
	operating on behalf of an end system at the edge

	- core devices SHOULD limit HBH header use, but MUST allow HBH
	headers; when they present a DOS issue because of underdesign,
	the device MAY preferentially drop those that require increased
	investment, but this drop SHOULD be noted in claims of device
	capacity

	- core devices SHOULD NOT inspect above L3; if they feel the
	need, it SHOULD be only to fill in missing flow labels and not
	on a persistent basis, and MUST NOT assume that such actions
	will always be possible

	- code devices that choose to inspect above L3 MUST NOT drop
	packets simply because of performance limits of that inspection
	(i.e., they MUST forward them instead)

	- edge devices MAY act on behalf of an end system by inspecting
	above L3, but only when they can ensure that such action
	would not err in ways the end system would not

I.e., if it hurts do do what you WANT (loadbalance, reduce reordering,
etc.), then backoff and do what you've contracted (i.e., forward at L3).

These are all workable. These mean that you can do what you want only as
long as you CAN; if you can't, you don't get to move forward by
disabling the Internet. You move forward by disabling the feature you're
trying to support that isn't compatible with the Internet because you
can't keep up.

Beyond that, I'd be writing my own RFC.

Joe

---

From: Fernando Gont <[email-masked]>
Date: Mon, 8 Feb 2016 19:46:51 -0300

> Sec 4.1 and 4.1.2 has been the context for my views.

FWIW, out I-D doesn't propose or suggest that EHs should be deprecated.
We simply raise awareness about the challenge they currently represent,
which eventually leads to packet drops in many network scenarios.

Thanks!

Cheers,

---

From: Nick Hilliard <[email-masked]>
Date: Mon, 08 Feb 2016 23:59:42 +0000

> 	operating on behalf of an end system at the edge

in real-world networks, the line separating core and edge is often not
clear-cut, if it exists at all.

> 	(i.e., they MUST forward them instead)

If you specify something like this, you should expect it to be abused.
I don't like that something like this would be abused, but it would happen.

> etc.), then backoff and do what you've contracted (i.e., forward at L3).

... which is fine until what you've contracted hurts transmission of
other packets because you can't do what you want.

> These are all workable.

Your suggestions are doable.  Silicon could be spun to inspect packet
header extensions to arbitrary chain lengths.  ASICs could be designed
to intercept packets with hbh headers at line rate on all ports and to
handle things appropriately.

Workable is the intersection of doable and practical.  When spinning
silicon, a designer needs to make a choice about what's reasonable and
what's not.  I.e. is it reasonable to look past 64k worth of EHs over
multiple packet fragments or is that just a corner case which is
unlikely ever to be hit?  Or is it reasonable to expect that most ipv6
packets will include a hbh header and that a router should be able to
handle packets with hbh headers at line rate on all ports.

It's not useful to deny the existence of practical limitations of any
particular design.  It's a lot more more useful to acknowledge that
trade-offs occur and to create a set of reasonable implementation
expectations.  For example, that ipv6 implementations should process up
to N bytes of EHs, or that EHs should only be attached to the first
packet and not on subsequent fragments.  If this isn't done, then
vendors will make their own decisions about what's practical for them,
and the results may not be what you want.

Nick

---

From: Warren Kumari <[email-masked]>
Date: Tue, 09 Feb 2016 00:24:37 +0000

> and the results may not be what you want.
If I can add to that:
"If this isn't done, then vendors will make their own decisions about
what's practical for them **and what their customers will actually pay
for**, and the results may not be what you want." ("practical for them"
included this, but I think it is worth spelling out)

Customers (like anyone else) are only willing to pay for things that they
need, at at the moment they don't seem to be needing this.

Telling vendors (and their customers) that they are wrong because the RFCs
say so, or because some corner case doesn't work doesn't really help...

W

---

From: Fernando Gont <[email-masked]>
Date: Mon, 8 Feb 2016 21:22:02 -0300

> packets. Or people using IPSEC tunnels can try to get TCP implementations

Could you elaborate a bit on the EH thing?

Thanks!

Cheers,

---

From: Fernando Gont <[email-masked]>
Date: Mon, 8 Feb 2016 21:32:36 -0300

> that happens, no amount of accommodation by the hosts will solve the

My comment is that any "feature" that is unfriendly to middleboxes by
design is doomed to be blocked.

RFC7045 addresses part of the problem. draft-gont-6man-rfc6564bis is
another part of the problem which, surprisingly to me, we've done
nothing about. Then there's the issue of how long we allow EH chains to
be: so far the only current limit is "up to the PMTUD" (RFC7112). Maybe
if we were to reduce this to something more sensible (even at the
operational level), we could move forward a bit.

Thanks,

---

From: Joe Touch <[email-masked]>
Date: Mon, 8 Feb 2016 17:07:49 -0800

On 2/8/2016 3:59 PM, Nick Hilliard wrote:
> particular design. 

I agree, but when faced with alternatives, the appropriate action is to
support L3 forwarding when you can and give up on "higher services" when
they become your DOS vector.

I.e., it's the support for these higher services that is the issue, not
just (or primarily) the use of EHs.

Let's please not get that backwards.

Joe

---

From: [email-masked]
Date: Tue, 9 Feb 2016 09:20:27 +0100

> interesting question, we shouldn't base future practice on it.

I'm not sure what you imply by that.
if the flow label is set, then router implementations / deployments can be changed  to use the 3-tuple for ECMP instead of the 5-tuple.

if generally flow label = 0, then we could at least identify implementations that needs to be fixed so that we at some point can change.

cheers,
Ole

---

From: [email-masked]
Date: Tue, 9 Feb 2016 09:29:09 +0100

Fernando,

> design is doomed to be blocked.

if we were to design the network according to that "principle", we would at least:

- not do IP fragmentation
- not have asymmetric paths
- ban crypto. no reason why middleboxes need to contain themselves below L7

I would claim your line of argumentation is fundamentally incompatible with rfc1958.

Best regards,
Ole

---

From: Mark Smith <[email-masked]>
Date: Tue, 9 Feb 2016 19:55:38 +1100

> I would claim your line of argumentation is fundamentally incompatible
with rfc1958.

So a few years ago I'd noticed MPTCP, a general rise in crypto and also the
rise in Smartphones, and thought about what that might mean to middleboxes
and the network.

"The Rapid Rise of the Mobile
Multihomed Host,
and what it might mean to the network"
http://www.ausnog.net/sites/default/files/ausnog-2013/presentations/D01%20P06-the%20rapid%20rise%20of%20the%20mobile%20multihomed%20host.pdf

I ended up picking the wrong organisation, it was Apple about a week later
with IOS 7.

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 09 Feb 2016 12:14:24 +0000

> 	(i.e., they MUST forward them instead)

If you can't install layer 3 ACLs to protect either your infrastructure
or other peoples' infrastructure because your kit can't inspect far
enough into an ipv6 packet because the spec allows for 64k of EHs before
you see the L4 header, then from an operational point of view, you need
an option to block by default.

You need this option because some silicon designer or product manager
made a decision that it was more important to spend development
resources building some other feature rather than optimising their
design to handle quantities of EHs that would never been seen unless
someone was attempting to abuse a corner case.

> Let's please not get that backwards.

We have an operational problem on the Internet in that packets with EHs
are routinely dropped:

draft-ietf-v6ops-ipv6-ehs-in-real-world

We need to stop ignoring this problem because ipv6 extension headers are
unusable in production at the moment.  You cannot run a protocol if you
know that it simply won't work on between 10% and 55% of end points.

The first step is to measure the scale of the problem.  The second is to
analyse why this might be happening.  This is what we're trying to do in
this document.

Nick

---

From: Fernando Gont <[email-masked]>
Date: Tue, 9 Feb 2016 09:21:18 -0300

> I would claim your line of argumentation is fundamentally incompatible with rfc1958.

Well, it depends on what you look at in that RFC. e.g., the
aforementioned RFC says:

"Engineering feed-back from real implementations is more important
 than any architectural principles."

I'd say that, if anything, the EH structure is more of an "architectural
principle", and the folks commenting on them being rather
implementation-unfriendly
(<http://www.iepg.org/2015-11-01-ietf94/IEPG-RouterArchitecture-jgs.pdf>) and
the ops folks dropping the packets are the "engineering feedback and
real implementations".

The same RFC says:
" 3.12 Objects should be self decribing (include type and size), within
   reasonable limits."

 -- in this respect, the Net-Header space is not self describing (see
draft-gont-6man-rfc6564bis-01)

That said, I'm not happy with the current state of affairs regarding
EHs. I'm just saying that I'm not surprised (for instance, we finally
got to fully specify the FL not that long ago, and hence there was no
alternative to simply walking the EH chain).

FWIW, I'm all for improving the current state of affairs, rather than
sitting happily with the current situation regarding EHs.

And, in any case, IMHO the only point of looking back at design choices
is for the exercise of trying to learn lessons such that, if one was to
design such a protocol again, one could know which things would make
sense to make different (me, I'm not sure I would have options at the
network-layer.. but if I had, I would opt for something that is limited
in size and that can be easily skipped (more a la IPv4) than for a
daisy-chain structure with no pointer to the upper layer as in IPv6). --
obviously, I realize that it's 2016 and not the early 90s... (so you may
argue that this is like "betting on the horses with Monday's newspaper
in your hands", as we say here).

Thanks!

Cheers,

---

From: [email-masked]
Date: Tue, 9 Feb 2016 13:56:56 +0100

Fernando,

my point was that it was absurd to try to design network protocols in a way that they would accommodate any possible policy decision made in the network or by end-hosts. it isn't the IETFs job to try to bypass whatever policy a network choses to implement.

if we accept that e.g. a transit network needs to look into L4 or deeper, then we also accept we can never make any innovations at this layer. (and we've essentially accepted that the Internet is going to be just an underlay for whatever comes next).

I can't think of many examples where a transit network would require to parse the extension chain.
in the case of ECMP, we are better off with IPv6 than IPv4, although we have to do something to get flow-label based ECMP deployed / implemented.

but even today you can do reasonably well without parsing EHs:

if (flow_label > 0) {
  flow_hash = f4(sa,da,protocol, flow)
} else {
  if (nh == UDP|TCP) {
    flow_hash = f5(sa,da,protocol,sport,dport)
  }
  // Do something with GRE keys
} else {
  flow_hash f3(sa,da,protocol)
}
adj_index = flow_hash & (nadj - 1)

Best regards,
Ole

> On 09 Feb 2016, at 13:21, Fernando Gont <[email-masked]> wrote:
>

---

From: Gert Doering <[email-masked]>
Date: Tue, 9 Feb 2016 15:47:22 +0100

Hi,

> I can't think of many examples where a transit network would require to parse the extension chain.

You might want to read Fernando's draft :-) - there's quite a number 
of examples in there.

Many cases boil down to "protection of <things> which need differenciation
on L4" (like, control plane).

Gert Doering
        -- NetMaster

---

From: Tom Herbert <[email-masked]>
Date: Tue, 9 Feb 2016 16:17:32 +0100

> on L4" (like, control plane).
Most of the examples in the draft refer to network edge device
functionality (e.g. filtering, ACLs, DDOS mitigation, NAT). For
devices that are just switch packets (i.e. fabric switches in our
network) the only reason I see that the would want to parse the header
chain is either hop-by-hop options or for ECMP (the latter case is
what we are addressing using flow label). Are there any other reasons
they need to parse the chain?

Tom

---

From: Philip Homburg <[email-masked]>
Date: Tue, 09 Feb 2016 16:58:51 +0100

In your letter dated Mon, 8 Feb 2016 21:22:02 -0300 you wrote:
>>>> to deal with out of order packets and then switch to statistical multiplexi
>n
>>> g.
>>>

I seem to have skipped a few steps. 

It seems that in practice we have defined 'The Internet' as a network that
does not reorder packets.

Then I was thinking about IPSEC tunnels, and one way to restore that property
for an IPSEC tunnel, if statistical multiplexing is going on some in the
network, is to define a new extension header that just contains a sequence
number. The sending side just increments the number for each packet that goes
out. The receiving side has some heuristics how long to wait to reorder packets.

You could do the same thing in TCP, but it might be easier to write it as a
separate layer to avoid modifiying TCP stacks.

The same way an option could be added to hbh or destination options. (And then
a destination option is really all you need)

I assume streams do not contain fragmented packets.

The main idea is that if a load balancer cannot handle an EH, it should
just use statistical multiplexing and let the hosts deal with the result.

---

From: Joe Touch <[email-masked]>
Date: Tue, 9 Feb 2016 08:41:10 -0800

> someone was attempting to abuse a corner case.

We should not dumb down Internet protocols simply because vendors are
greedy.

And we cannot continue to let vendors simply declare that they support
IPv6 without supporting it's costlier features.

What's missing here is consideration of the following:

	a) v6ops needs to live within the specs written by intarea
		operational issues should describe how to comply
		with requirements, not be an end-run around them

	b) v6ops needs to remember that these issues were discussed
	when IPv6 was designed
		you might think we'd pick different decisions given
		what we know now, but all we'd get are different
		requirements that vendors want to do end-runs around

	c) middleboxes and DPI routers create their own problems
		if it "hurts" when you do something, perhaps it's
		time to reconsider

Operators wanting to do something doesn't make it so. They should direct
their dissatisfaction to vendors who misrepresent their products. E.g.,
VW owners don't go to the EPA asking them to redefine "clean diesel"
simply because it's "really hard" for engine designers to comply.

Joe

---

From: Joe Touch <[email-masked]>
Date: Tue, 9 Feb 2016 08:57:15 -0800

> out. The receiving side has some heuristics how long to wait to reorder packets.

IP (and IP forwarding) doesn't care about order.

The only time order matters is when an intermediate device tries to act
like an end system (inspecting L4) but doesn't want to *do the work* of
being an end system (by reordering or reassembling).

> separate layer to avoid modifiying TCP stacks.

TCP already reorders packets. Modern extensions handle out-of-order
arrivals much better than older versions.

...

+1

---

From: Saku Ytti <[email-masked]>
Date: Tue, 9 Feb 2016 19:14:47 +0200

On 9 February 2016 at 18:41, Joe Touch <[email-masked]> wrote:

Hey,

> simply because it's "really hard" for engine designers to comply.

If end users demand that EH options work, operators will demand from
vendors, and vendors will produce devices where this is true. Right
now operators are just demanding cheaper and faster boxes with
existing feature sets, if device which always drops all EH is cheaper
than competitors device which can somehow process them, operators will
choose the cheaper box without any EH support.

This is classical chicken-egg situation, no one will write application
needing EH, as they do not work, and thus customers won't demand that
they work, as no application needs them.

It shouldn't come as surprise on EH, as we've already experienced it
on IP options. We're starting to experience it on fragmentation,
market does not complain when device is unable to fragment at high
rate, as long as the device is cheaper, it is marketable.

Vast majority of operators do not know if they pass or drop
EH/IP-options, particularly if they pass or drop HBH and just accept
what ever default behaviour device has, and no customer will complain
about it.

I think it's good that EH is better described in RFC, but I don't
think there is any way to press 'reverse' button regardless what
documents and standards are written, I don't believe EH can be made to
work on IPv6 Internet anymore. Heck, only reason ICMP works, is
because machine won't work when it's blocked.

When google tested QUIC, I believe they tried to use new IP protocol
number, as well as riding on top of UDP, and reach of new IP protocol
number was just not sufficient for practical deployment. This really
bothers me, it means that even developing new L4 protocol on the
Internet is going to be huge challenge if possible at all.

-- 
  ++ytti

---

From: Matthew Walster <[email-masked]>
Date: Tue, 9 Feb 2016 17:20:33 +0000

> Internet is going to be huge challenge if possible at all.

​Given enough time, all new protocols seemingly end up served by an HTTP
daemon :S

M​

---

From: "Ackermann, Michael" <[email-masked]>
Date: Tue, 9 Feb 2016 17:48:19 +0000

Regards to your statement ---

", is to define a new extension header that just 
> contains a sequence number."

Please see     https://tools.ietf.org/html/draft-ietf-ippm-6man-pdm-option-01

Which already proposes a sequence number, in a DOH extension header,   for diagnostic and performance monitoring purposes.  

Thanks

Mike

-----Original Message-----
From: v6ops [mailto:[email-masked]] On Behalf Of Joe Touch
Sent: Tuesday, February 09, 2016 11:57 AM
To: Philip Homburg <[email-masked]>; [email-masked]
Cc: Fernando Gont <[email-masked]>
Subject: Re: [v6ops] IPv6 EHs Packet Drops (Fwd: New Version Notification for draft-gont-v6ops-ipv6-ehs-packet-drops-02.txt)

On 2/9/2016 7:58 AM, Philip Homburg wrote:
> It seems that in practice we have defined 'The Internet' as a network 
> that does not reorder packets.
> 
> Then I was thinking about IPSEC tunnels, and one way to restore that 
> property for an IPSEC tunnel, if statistical multiplexing is going on 
> some in the network, is to define a new extension header that just 
> contains a sequence number. The sending side just increments the 
> number for each packet that goes out. The receiving side has some heuristics how long to wait to reorder packets.

The only time order matters is when an intermediate device tries to act like an end system (inspecting L4) but doesn't want to *do the work* of being an end system (by reordering or reassembling).

> You could do the same thing in TCP, but it might be easier to write it 
> as a separate layer to avoid modifiying TCP stacks.

TCP already reorders packets. Modern extensions handle out-of-order arrivals much better than older versions.

...
> The main idea is that if a load balancer cannot handle an EH, it 
> should just use statistical multiplexing and let the hosts deal with the result.

+1

The information contained in this communication is highly confidential and is intended solely for the use of the individual(s) to whom this communication is directed. If you are not the intended recipient, you are hereby notified that any viewing, copying, disclosure or distribution of this information is prohibited. Please notify the sender, by electronic mail or telephone, of any unintended receipt and delete the original message without making any copies.
 
 Blue Cross Blue Shield of Michigan and Blue Care Network of Michigan are nonprofit corporations and independent licensees of the Blue Cross and Blue Shield Association.

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 09 Feb 2016 17:49:42 +0000

> 		with requirements, not be an end-run around them

As far as I can tell, this draft is compatible with v6ops charter
objective #1:

> 1. Solicit input from network operators and users to identify
> operational issues with the IPv6 Internet, and determine solutions
> or workarounds to those issues.

If you feel that there's a problem with the v6ops charter itself, then
that's a different issue.

Nick

---

From: Joe Touch <[email-masked]>
Date: Tue, 9 Feb 2016 10:25:49 -0800

>> or workarounds to those issues.

Solutions or workarounds != deprecating specs.

Joe

---

From: Joe Touch <[email-masked]>
Date: Tue, 9 Feb 2016 10:29:32 -0800

> Solutions or workarounds != deprecating specs.

FWIW, a good example of reasonable workarounds includes limiting the max
length of the chain in typical use *at this time*.

All such recommendations should be limited and should focus on
supporting capabilities, not merely redefining the Internet for
convenience. There are plenty of ways to get there, but the trade-off
needs to recognize that what an operator *wants* to do that isn't
supported by vendors does not necessarily indicate the need to redefine
a protocol spec. Sometimes it's the operator or vendor's problem.

Joe

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 10 Feb 2016 08:07:40 +1300

Ole,

> if the flow label is set, then router implementations / deployments can be changed  to use the 3-tuple for ECMP instead of the 5-tuple.

Because of the chicken/egg problem, we need operators to require ECMP/LAG and server
load balancing products that support the flow label, in order to create an incentive
for users to run host stacks that set the flow label by default. Actually I think
the best thing is to use the 6-tuple (5-tuple plus flow label), and drop back
to 3-tuple if the transport header isn't the first Next Header.

> if generally flow label = 0, then we could at least identify implementations that needs to be fixed so that we at some point can change.

Windows, Linux, OS X, iOS and Android would be a good start.

   Brian

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 09 Feb 2016 19:58:55 +0000

> Solutions or workarounds != deprecating specs.

this ID doesn't deprecate any specs.  It only attempts to identify
problems that people have seen on the Internet when EHs are used, and
why the problems might occur.

Nick

---

From: Tom Herbert <[email-masked]>
Date: Tue, 9 Feb 2016 22:33:23 +0100

> Internet is going to be huge challenge if possible at all.
I think you can take this as once data point that end users are
"demanding" that networks and the Internet can support alternative
transport protocols. The current work being done on extension headers,
such as segment routing, can be taken as evidence extensibility of IP
is required also.

Tom

---

From: Tom Herbert <[email-masked]>
Date: Tue, 9 Feb 2016 22:41:04 +0100

> Windows, Linux, OS X, iOS and Android would be a good start.

AFAIK IOS (and FreeBSD) has been sending non-zero flow labels for some
time now. Support was added in Linux to send non-zero flow labels by
default, Android will pick those up the next time they rebase (I still
don't know what the status is in Windows). In reality, setting the
flow label on the host is near zero cost and as long as network
devices are choking on them (we don't have any evidence of that)
there's no reason why we can't get to widespread usage in a relatively
short period of time.

Tom

---

From: Fernando Gont <[email-masked]>
Date: Tue, 9 Feb 2016 21:20:57 -0300

> such as segment routing, can be taken as evidence extensibility of IP

If that's the case, then, our no action on
<draft-gont-6man-rfc6564bis-01> is really bad.

Thanks,

---

From: Fernando Gont <[email-masked]>
Date: Wed, 10 Feb 2016 05:39:25 -0300

Hi, Ole,

On 02/09/2016 09:56 AM, [email-masked] wrote:
> 
> my point was that it was absurd to try to design network protocols in
> a way that they would accommodate any possible policy decision made
> in the network or by end-hosts. it isn't the IETFs job to try to
> bypass whatever policy a network choses to implement.

My take here is that some design decisions make it hard for networks to
enforce the policies they choos to enforce.

e.g., I recall you quoting S. Deering suggesting that the EH structure
was meant to make the life of middleboxes painful. I regard that as a
rather bad design decision (put another way: would you have EH chains
the way we have them n IPv6 if you were to redesign IPv6?).

> if we accept that e.g. a transit network needs to look into L4 or
> deeper, then we also accept we can never make any innovations at this
> layer. (and we've essentially accepted that the Internet is going to
> be just an underlay for whatever comes next).

Not necessarily. There are essentially two issues here:

1) The Next-Header space is not self describing. It should be possible
to skip past unknown EHs. I presented a proposal
(draft-gont-6man-rfc6564bis-01), but we did nothing about it. (there
could be other alternatives, such as prohibiting the specification of
new EHs... but we did nothing about it).

2) EH chains can be very long. Me, I'd prefer that we get to some
compromise (e.g., "everyone must support EH chains of at least 256 bytes
to be able to claim they are IPv6-ready"), rather than what we have now:
we pretend that nodes support the currently unlimited EH chain lengths,
but in practice it gets hard to pass a 10B chain through.

> to parse the extension chain. 

We have examples in draft-gont-v6ops-ipv6-ehs-packet-drops-02.txt
(supplied by folks running such networks).

> in the case of ECMP, we are better off
> with IPv6 than IPv4, although we have to do something to get
> flow-label based ECMP deployed / implemented.

Agreed.

> if (flow_label > 0) { flow_hash = f4(sa,da,protocol, flow) } else { 
> if (nh == UDP|TCP) { flow_hash = f5(sa,da,protocol,sport,dport) } //
> Do something with GRE keys } else { flow_hash f3(sa,da,protocol) } 

Probably the ECMP case is easier to solve in the near term. OTOH, the
ability to do packet filtering doesn't look easy to solve if we keep the
(rather onerous nowadays) requirement of supporting arbitrarily-long EH
chains.

FWIW, this is just me trying to make things work, and trying to find a
middle-ground between the theoretical requirements we expect
implementations to comply with, and the "I will drop all the EH-enabled
packets" that seems to be rather widespread.

Cheers,

---

From: Sander Steffann <[email-masked]>
Date: Wed, 10 Feb 2016 10:19:53 +0100

Hi,

+1

In theory, there is no difference between theory and practice. But, in practice, there is.

Agreed-upon and expected limitations are much much better than every vendor doing something not well defined.

Cheers,
Sander

---

From: [email-masked]
Date: Wed, 10 Feb 2016 10:52:38 +0100

Fernando,

> new EHs... but we did nothing about it).

right, it isn't clear that is solving the problem.
if walking the chain is done by a middlebox for the purpose of "security", then it would be highly unlikely that it would allow an unknown extension header.

for a router doing ECMP we should instead make the flow label usable.

what we have done, is to make it very unlikely that new extension headers will be defined. given that we have the three extensible containers, RH, DestOpt and HBH. and we might be making parsing the HBH optional.

> but in practice it gets hard to pass a 10B chain through.

right, but within the constraints of RFC7112.
for routers that number is and always have been 40. :)
for middleboxes it is dependent on policy, and since that could typically be to parse the whole packet...

> (supplied by folks running such networks).

sorry, I don't see any examples there where a transit network would require to look deeper than 40 bytes.
in the case of DDOS that would be filters that would match a signature, and I'd imagine that include extension headers right.
> packets" that seems to be rather widespread.

right, this is simply a result of the tussle between the different actors on the network.
if anything IETF should try to be neutral in that tussle.

cheers,
Ole

---

From: Fernando Gont <[email-masked]>
Date: Wed, 10 Feb 2016 07:21:59 -0300

On 02/10/2016 06:52 AM, [email-masked] wrote:
>> 
>>> if we accept that e.g. a transit network needs to look into L4
>>> or deeper, then we also accept we can never make any innovations
>>> at this layer. (and we've essentially accepted that the Internet
>>> is going to be just an underlay for whatever comes next).
>> 
>> 1) The Next-Header space is not self describing. It should be
>> possible to skip past unknown EHs. I presented a proposal 
>> (draft-gont-6man-rfc6564bis-01), but we did nothing about it.
>> (there could be other alternatives, such as prohibiting the
>> specification of new EHs... but we did nothing about it).
> 
> right, it isn't clear that is solving the problem. if walking the
> chain is done by a middlebox for the purpose of "security", then it
> would be highly unlikely that it would allow an unknown extension
> header.

That's the point: you cannot tell if what's inside is an EH or an
upper-layer protocol.

If the device is whitelisting stuff, I'd agree with you on the
end-result. OTOH, if the devices means to blacklist stuff, then this may
lead to unnecessarily packet drops.

And, in any case, a bug is a bug.

We're pretending that RFC6564 can be employed to skip past unknown EHs,
when it really can't.

> what we have done, is to make it very unlikely that new extension
> headers will be defined. given that we have the three extensible
> containers, RH, DestOpt and HBH. and we might be making parsing the
> HBH optional.

Then let's say that. Right know, given an unknown Next Header, it's
impossible to tel whether it's an EH or an ULP.

I don't care if we go forward with what we propose in our rfc6564bis
I-D, or we just ban the definition of new EHs. Anything is better than
leaving the bug in.

>> compromise (e.g., "everyone must support EH chains of at least 256
>> bytes to be able to claim they are IPv6-ready"), rather than what
>> we have now: we pretend that nodes support the currently unlimited
>> EH chain lengths, but in practice it gets hard to pass a 10B chain
>> through.
> 
> right, but within the constraints of RFC7112. for routers that number
> is and always have been 40. :)

40B?

If it's an implementation limit, it'd be great that we all agree that
40B is the least common denominator.

Besides, whatever the number, there can be as many instances of HBH as
desired.. so even for routers, you still have to support at least
up-to-MTU long EH chains.

> for middleboxes it is dependent on
> policy, and since that could typically be to parse the whole
> packet...
> 
> sorry, I don't see any examples there where a transit network would
> require to look deeper than 40 bytes. in the case of DDOS that would
> be filters that would match a signature, and I'd imagine that include
> extension headers right.

The examples basically boil down to happing to enforce some sort of
filters that require layer-4 info.

>> Probably the ECMP case is easier to solve in the near term. OTOH,
>> the ability to do packet filtering doesn't look easy to solve if we
>> keep the (rather onerous nowadays) requirement of supporting
>> arbitrarily-long EH chains.
>> 
>> FWIW, this is just me trying to make things work, and trying to
>> find a middle-ground between the theoretical requirements we
>> expect implementations to comply with, and the "I will drop all the
>> EH-enabled packets" that seems to be rather widespread.
> 
> right, this is simply a result of the tussle between the different
> actors on the network. if anything IETF should try to be neutral in
> that tussle.

But when we don't try to be pragmatic and agree on what is supported,
the result is not good: I'd prefer to be "limited" in the EH chains that
I can use, but to know that I can rely on e.g. up to 64B or 256B EH-chains.

Right know, since our requirement to support arbitrarily-long EHs is
generally seen as too onerous, the end result is that folks don't care
to support EHs, and hence you wouldn't even hope to have your 10B EH
chains go through the network.

I'm not even arguing about an actual modification to the spec (i.e., and
enforced upper limit), but e.g. and enforced lower limit that everyone
has to support (which could be increased overtime). -- if we really
won't EHs to be deployable.

Cheers,

---

From: Philip Homburg <[email-masked]>
Date: Wed, 10 Feb 2016 12:29:43 +0100

>And, in any case, a bug is a bug.

Call me paranoid, but I'm quite happy if a security device drops stuff it
doesn't understand.

It shouldn't be that hard to support a white list of unknown extension
headers that can be set by the operator. 

>chains go through the network.

It is always possible to find an excuse for not doing something.

There is a very common pattern of a fragmentation header in front of a UDP
header, which just needs to be supported on todays internet. 

Beyond that, I haven't seen a strong desire for extension headers expressed
outside IETF mailing lists. So there the use cases seem quite remote 
compared to all other new features that need to be supported.

---

From: Fernando Gont <[email-masked]>
Date: Wed, 10 Feb 2016 08:47:13 -0300

> Call me paranoid, but I'm quite happy if a security device drops stuff it

Agreed. What I'm saying is that in scenarios in which you just want to
drop stuff that is known to be evil, but otherwise pass everything else,
you'd need to be able to jump past unknown EHs, and you currently cannot
do that. And I think you should be able to.

(Do I think this is an issue *now*? -- Obviously not... even the
standard EHs don't get to survive the Internet in over 20% of cases).

That aside, we currently assume that it is possible to jump past unknown
EHs when it's not really possible. That's a current bug in our specs.

> headers that can be set by the operator. 

How do you skip past an unknown EH?

> compared to all other new features that need to be supported.

What concerns me is that we pretend that there is stuff (EHs) that is
deployable, when it currently isn't.

So we better try to do something to get some subset of that deployable
(and it becomes clear that you can only rely on such subset to work) and
we declare some sort of humble victory :-), or we forget about it and
assume defeat.

What worries me is when there is a disconnect between the specs and the
real world: a folk that reads the spec assumes that something will work,
but he misses the unwritten wisdom that says that part of what we read
actually fails badly.

---

From: Philip Homburg <[email-masked]>
Date: Wed, 10 Feb 2016 13:18:16 +0100

>EHs when it's not really possible. That's a current bug in our specs.

I think it would be perfectly fine if a vendor supports unknown 
extension headers only if they conform to RFC 6564, which is standards
track.

So the only thing is that operators would have to explictly mark protocol
values as RFC-6564 compatible extension headers that are safe to ignore.

The 'safe to ignore' part has to be manual anyhow.

>but he misses the unwritten wisdom that says that part of what we read

What I'm missing is strong demand for extension headers (other than
fragmentation) outside the IETF.

I.e., if I would design an IPv6 firewall today, I would make sure fragmentation
works, but put all other extension headers as nice to have.

In the end, demand has to come from operators and users.

---

From: [email-masked]
Date: Wed, 10 Feb 2016 13:21:39 +0100

Fernando,

> We're pretending that RFC6564 can be employed to skip past unknown EHs,

no, we're not. we're saying that skipping past unknown headers is not useful/possible in the general case.

> I-D, or we just ban the definition of new EHs. Anything is better than

https://tools.ietf.org/html/draft-ietf-6man-rfc2460bis-03#section-4.8

> 40B is the least common denominator.

IPv6 has had that limit for 20 years.

> up-to-MTU long EH chains.

we're changing that:
https://tools.ietf.org/html/draft-ietf-6man-hbh-header-handling-00

[...]

cheers,
Ole

---

From: [email-masked]
Date: Wed, 10 Feb 2016 13:27:56 +0100

<with absolutely no hat on>

> works, but put all other extension headers as nice to have.

if I would design an IPv6 firewall today I'd drop most fragments. they are nothing but a DOS vector (for middleboxes).
(and I've formed that opinion after actually implementing an RFC7597 BR.)

Best regards,
Ole

---

From: Gert Doering <[email-masked]>
Date: Wed, 10 Feb 2016 13:29:29 +0100

Hi,

> (and I've formed that opinion after actually implementing an RFC7597 BR.)

What about DNS and DNSSEC?  Or are those the exception to "drop *most* 
fragments"?

Gert Doering
        -- NetMaster

---

From: [email-masked]
Date: Wed, 10 Feb 2016 14:13:06 +0100

> What about DNS and DNSSEC?  Or are those the exception to "drop *most*

those would have to fallback to TCP. the exception would be more like in-sequence, back to back fragments with a maximum fragment chain of 2.

this is quickly turning into the "deprecate fragmentation at the network layer thread".
to alleviate all fears, I'm _not_ designing an IPv6 firewall. ;-)

Cheers,
Ole

---

From: Gert Doering <[email-masked]>
Date: Wed, 10 Feb 2016 14:16:29 +0100

Hi,

> those would have to fallback to TCP. the exception would be more like in-sequence, back to back fragments with a maximum fragment chain of 2.

I see.  Which would be fine for the generic DNS+DNSSEC case...

> to alleviate all fears, I'm _not_ designing an IPv6 firewall. ;-)

I didn't want to turn it into the generic all-encompassing fragmentation
thread :-) - just understand what the exception would be.

Gert Doering
        -- NetMaster

---

From: Saku Ytti <[email-masked]>
Date: Wed, 10 Feb 2016 15:50:03 +0200

On 10 February 2016 at 13:29, Philip Homburg <[email-masked]> wrote:

Hey,

> header, which just needs to be supported on todays internet.

I did quick test in NLNOG Ring trying to ping6 with 1501B, meaning src
host will fragment it. Interestingly I also saw:

A=>B NOT_OK
A=>C OK
C=>B OK

Implying sometimes packets with fragment headers are not dropped near the edge?

I had some 130k ping results, based on this sample, about 11% failure
rate with 1501B pings.

This further enforces my belief that fragments will gradually stop
working. Vey common 'high touch' service provider router from Vendor1
handles IPv4 fragments already in linecard CPU, while Vendor2 invested
in silicon and does it very fast in HW. However Vendor2 is not winning
the market, so I can't imagine them supporting IPv4 fragments in HW in
next-gen, why bother? Market does not want it.

-- 
  ++ytti

---

From: Fernando Gont <[email-masked]>
Date: Wed, 10 Feb 2016 10:05:44 -0300

> no, we're not. we're saying that skipping past unknown headers is not useful/possible in the general case.

mmm.. can you explain what RFC6564 buys?

> https://tools.ietf.org/html/draft-ietf-6man-rfc2460bis-03#section-4.8

This still allows the definition of new EHs.

Either ban new EHs, or do something like rfc6564bis. Otherwise we have a
bug in the spec. (what's the point of requiring the format in Section
4.8 of rfc2460bis?)

> IPv6 has had that limit for 20 years.

Sorry, I missed your point. So essentially you're agreeing that Ehs do
not work. IN which scenarios should one implement something as an EH,
since we cannot rely on them?

> https://tools.ietf.org/html/draft-ietf-6man-hbh-header-handling-00

That's good.

---

From: Fernando Gont <[email-masked]>
Date: Wed, 10 Feb 2016 10:40:41 -0300

On 02/10/2016 10:13 AM, [email-masked] wrote:
> those would have to fallback to TCP.

This will be interesting. General DNS serves actually *relying* on TCP
could be an interesting can of worms DoS-wise.

> the exception would be more like in-sequence, back to back fragments with a maximum fragment chain of 2.
> 

I'll ping Ron Bonica to revive his "deprecate IPv6 fragmentation" I-D
and claim you'll support it. :-)

(just kiddin')

Cheers,

---

From: [email-masked]
Date: Wed, 10 Feb 2016 17:01:58 +0100

Fernando,

> since we cannot rely on them?

how did you manage to reach that conclusion?
I'm saying there are very few use cases (ECMP and DDOS) requiring parsing further than the IPv6 header in transit networks.
with the caveats mentioned previously I believe EHs work as well as they can be made to work.

apart from:
 - encourage the deployment of flow-label
 - "fixing" the HBH header
 - strongly discourage new EHs

I think the IETF has done (more than) enough on this topic for a while.

cheers,
Ole

---

From: Nick Hilliard <[email-masked]>
Date: Wed, 10 Feb 2016 16:48:53 +0000

[email-masked] wrote:
> I'm saying there are very few use cases (ECMP and DDOS) requiring 
> parsing further than the IPv6 header in transit networks. with the 
> caveats mentioned previously I believe EHs work as well as they can 
> be made to work.

transit networks need to be able to provide anti-ddos protection.  At
the moment this is normally handled on the provider side using
customer-signaled RTBH, but in future this will need to be either
provider- or customer-signalled using flowspec. Also, all routers need
control plane protection / RE firewalling.

> I think the IETF has done (more than) enough on this topic for a 

draft-gont-v6ops-ipv6-ehs-in-real-world shows that EHs are fundamentally
unusable on the wider Internet today.  Declaring that the IETF has done
more than enough is giving up completely, and that will effectively
write off EHs as a viable technology in the long term.  I don't think
the IETF should give up, at least not at this stage.  If it does, it
should have good reasons for doing so.

Nick

---

From: Joe Touch <[email-masked]>
Date: Wed, 10 Feb 2016 10:21:48 -0800

> could be an interesting can of worms DoS-wise.

https://tools.ietf.org/html/draft-ietf-dprive-start-tls-for-dns-01

---

From: 神明達哉 <[email-masked]>
Date: Wed, 10 Feb 2016 11:24:08 -0800

> those would have to fallback to TCP.

I may misunderstand it, but DNS clients (resolvers) are not supposed
to fall back to UDP simply because its UDP query times out (because
the fragmented response is dropped in this scenario).

> the exception would be more
> like in-sequence, back to back fragments with a maximum fragment
> chain of 2.

One common practice of the DNS server is to fragment UDP responses at
the minimum MTU, and a large response with DNSSEC signatures could be
larger than several thousands of bytes, a max chain of 2 is probably
not enough (although 3 or 4 is probably okay in practice).  If you
meant the server should just truncate responses larger than that to
trigger immediate fall back to TCP by "would have to fallback to TCP"
above, then I see that.

--
JINMEI, Tatuya

---

From: Fernando Gont <[email-masked]>
Date: Wed, 10 Feb 2016 14:14:17 -0300

> how did you manage to reach that conclusion?

What I understood from your note is that you can only rely on 40B-header
packets. Anything longer than that is uncertain. Did I misinterpret you?

Clearly, there's widespread dropping of packets, and the longer the EH
chain, the worse.

If we cannot agree on at least some minimum EH length that is widely
supported, then it becomes "no EHs, or be prepared for packet drops".

> with the caveats mentioned previously I believe EHs work as well as they can be made to work.

Folks seem to be needing to filter packets, which means processing the
EH chain for reasons other than ECMP. Widespread support would fix the
drops related to ECMP, but not the ones related to the need to filter
based on layer-4 info.

>  - strongly discourage new EHs

If asked I'd also ban: multiple instances of the same EH, ban some
pathological cases such as tow back-to-back FHs, and also establish a
least common denominator of an EH-Chain length that is supposed to work
everywhere.

> I think the IETF has done (more than) enough on this topic for a while.

FWIW, my metric of "enough" is when you get stuff to work -- we're far
from there yet.

Thanks,

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 11 Feb 2016 14:21:53 +1300

Somebody said:

>> and we might be making parsing the HBH optional.

We already did that, in RFC 7045 section 2.2.

>> headers that can be set by the operator. 

We pretty much require that in RFC 7045 section 2.1:

  "...the discard policy for each standard type of extension
   header MUST be individually configurable.  The default configuration
   SHOULD allow all standard extension headers.

   Experimental IPv6 extension headers SHOULD be treated in the same way
   as standard extension headers, including an individually configurable
   discard policy.  However, the default configuration MAY drop
   experimental extension headers.

   Forwarding nodes MUST be configurable to allow packets containing
   unrecognised extension headers, but the default configuration MAY
   drop such packets."

>> drops related to ECMP, but not the ones related to the need to filter

Right. Boxes that insist on parsing the header chain but do not understand
all header types are doomed to fail. There is no solution to that except
scrapping those boxes. RFC 6564, 7045, 7112 and draft-ietf-6man-hbh-header-handling
are mitigations, but broken middleboxes are broken.

I tend to agree with Ole that on the standards front there isn't much
more we can do. Operators can require their suppliers to support the above
mitigations, hopefully.

    Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 11 Feb 2016 15:38:29 +1300

> there's no reason why we can't get to widespread usage in a relatively

Windows 7 doesn't set any value by default, and I can't find a netshell
command to switch it on. (I assume a TCP app can set it through the socket
interface, but that's beside the point.)

    Brian

---

From: Tom Herbert <[email-masked]>
Date: Thu, 11 Feb 2016 10:57:31 +0100

> more we can do. Operators can require their suppliers to support the above
Suppose we define an "EH chain length" extension header. This would
include the length of the chain starting from the first byte of the EH
and also a next header value for the header that follow the chain.
This EH could follow HBP. Network nodes can use this to skip over a
long chain to parse the transport header. The receiver would need to
validate the length and protocol are correct in order to prevent
someone from spoofing transport headers.

Tom

---

From: Philip Homburg <[email-masked]>
Date: Thu, 11 Feb 2016 12:07:51 +0100

In your letter dated Wed, 10 Feb 2016 11:24:08 -0800 you wrote:
>> >> if I would design an IPv6 firewall today I'd drop most fragments. they are
> nothing but a DOS vector (for middleboxes).

Clients have some control over fragmentation using the EDNS0 UDP payload size
option.

For example setting the payload size to 1280 minus headers should avoid
fragmentation and will be fine for just about all DNSSEC except some key
rollover with RSA 2048 bit keys.

Unfortunately, there doesn't seem to be any DNS RFC that tells exactly what to
do.

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 12 Feb 2016 13:07:43 +1300

On 11/02/2016 22:57, Tom Herbert wrote:
...
> someone from spoofing transport headers.

https://tools.ietf.org/html/draft-zhang-6man-offset-option-01

But it doesn't help, because any middlebox that insists on trying
to parse all the headers will not make use of this feature.

   Brian

---

From: Mark Andrews <[email-masked]>
Date: Fri, 12 Feb 2016 11:47:47 +1100

> Clients have some control over fragmentation using the EDNS0 UDP payload size

Which is how recursive server work around stupid firewalls. A
firewall could open a slit to let through non-initial fragments
when it opens the pin hole to let through the initial fragment /
full packet.  It doesn't have to be all fragments.

> rollover with RSA 2048 bit keys.

Lots of answers from signed zones are > 1500 let alone 1280.
Reassembly attacks on the DNS are easy enough to address with
stronger checksums in a EDNS option or a well known TSIG secret
(this devolves to a crypto strength checksum).  We already have
EDNS COOKIES to address off path spoof of single / last fragment
responses.  The checksum/TSIG bind ties the last fragment to the
rest of the response.

We already see some TLD servers deploying EDNS COOKIES (the code
points are allocated) and in the Alexa top/bottom 1000.  .GOV zones
are the only sample w/o EDNS COOKIES deployment.
https://ednscomp.isc.org/compliance/summary.html

> Unfortunately, there doesn't seem to be any DNS RFC that tells exactly what to

Why should there be? Fragments are part of IPv6. Why shouldn't the
DNS depend on them.

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Tom Herbert <[email-masked]>
Date: Fri, 12 Feb 2016 09:02:07 +0100

> to parse all the headers will not make use of this feature.
Right, but realistically if we want to send extension headers into the
Internet we can't wait an indefinite amount of time for every single
device to properly handle them. It seems that we'd want to apply a
happy eyeballs approach. If the offset option makes it more palatable
for some devices to forward packets with EH that could move things
forward by some amount. I also see a use case for this at the end
hosts, a ridiculously long chain could be a nice DOS attack and the
offset option might mitigate that. For instance, before processing the
chain we could verify if an encapsulated TCP packet refers to a valid
connection.

Tom

---

From: Nick Hilliard <[email-masked]>
Date: Fri, 12 Feb 2016 11:26:47 +0000

> Why should there be? Fragments are part of IPv6. Why shouldn't the

EHs are dropped in ~10% of cases according to Fernando's draft (ymmv,
etc).  This isn't justification for using or not using EHs, btw, just a
observation that if the DNS depends on them, operational problems are
likely to occur.

Nick

---

From: Enno Rey <[email-masked]>
Date: Fri, 12 Feb 2016 12:30:42 +0100

Hi,

> observation that if the DNS depends on them, operational problems are

DNS has plenty of ways to work around those, see https://www.insinuator.net/2015/11/some-notes-on-the-drop-ipv6-fragments-vs-this-will-break-dnssec-debate/.

best

Enno

-- 
Enno Rey

ERNW GmbH - Carl-Bosch-Str. 4 - 69115 Heidelberg - www.ernw.de
Tel. +49 6221 480390 - Fax 6221 419008 - Cell +49 173 6745902 

Handelsregister Mannheim: HRB 337135
Geschaeftsfuehrer: Enno Rey

=======================================================
Blog: www.insinuator.net || Conference: www.troopers.de
Twitter: @Enno_Insinuator
=======================================================

---

From: Philip Homburg <[email-masked]>
Date: Fri, 12 Feb 2016 13:36:04 +0100

>Lots of answers from signed zones are > 1500 let alone 1280.

As far as I can tell, it is quite possible to run DNSSEC with 2048 bit RSA
keys and have everything fit in 1280, except during some key rollovers.

The interesting thing is that for .org, a DNSKEY answer is larger than 1280,
but is not fragemented.

It seems that part of the DNS community will fragment at 1280 (to avoid PMTU
problems). Andother part fragments at 1500 or something, has PMTU problems
but avoids more problems with firewalls dropping fragmented packets.

>Why should there be? Fragments are part of IPv6. Why shouldn't the

Because when it comes to fragmentation, IPv6 is in practice worse than IPv4.
It was a nice idea at the time to only let hosts do fragmentation. But if
you look at DNS then IPv6 is clearly worse than IPv4.

Note, it is easy to tell people to just fix their firewalls to let fragments
go through, but they may just ignore you and point out that DNS works fine
on IPv4.

---

From: Philip Homburg <[email-masked]>
Date: Fri, 12 Feb 2016 13:49:51 +0100

In your letter dated Fri, 12 Feb 2016 12:30:42 +0100 you wrote:
>DNS has plenty of ways to work around those, see https://www.insinuator.net/201
>5/11/some-notes-on-the-drop-ipv6-fragments-vs-this-will-break-dnssec-debate/.

This analysis is very much incomplete.

For IPv6 you have case 4.5, response is smaller than the local MTU, but bigger
than PMTU.

DNS servers generally cannot do PMTU discovery. So these packets get dropped.

Unless the server fragments at 1280. In which case it has to set TC for
a lot of requests (in some deployment scenarios). Which results in a lot of
retries over TCP, which may be beyond what the current setup can handle.

---

From: Enno Rey <[email-masked]>
Date: Fri, 12 Feb 2016 14:05:12 +0100

Hi,

> For IPv6 you have case 4.5, response is smaller than the local MTU, but bigger

good point actually, I missed that one. question remains though: how often does that one occur in real-life?
read: is it worth to accept v6 fragments into your own network just to avoid this specific case? [note I'm talking from an enterprise/edge network perspective here, not from a transit network perspective]

> retries over TCP, which may be beyond what the current setup can handle.

I hear you. no offense, but I've yet to see those servers not being able to handle an increased amount of TCP based DNS requests coming in. 

Enno

-- 
Enno Rey

---

From: Philip Homburg <[email-masked]>
Date: Fri, 12 Feb 2016 14:34:29 +0100

In your letter dated Fri, 12 Feb 2016 14:05:12 +0100 you wrote:
>read: is it worth to accept v6 fragments into your own network just to avoid th
>is specific case? [note I'm talking from an enterprise/edge network perspective
> here, not from a transit network perspective]

Lets take the current situation at .org. A DNSKEY query for .org results in
a 1342 octet reply.

Currently this is sent as a single IPv6 packet. Depending on whether PMTU
works or not, a system with a 1280 PMTU receives nothing or receives a
fragmented response. And if you drop fragments, then such a system will not
receive any response whether PMTU works or not.

>I hear you. no offense, but I've yet to see those servers not being able to han
>dle an increased amount of TCP based DNS requests coming in. 

I have yet so see any system that consistently sets TC for all responses
bigger than 1280.

However, the DNSKEY query rate for k-root seems to be less than 80 q/s. So that
is quite doable over TCP.

---

From: Mark Andrews <[email-masked]>
Date: Sat, 13 Feb 2016 00:57:05 +1100

> is quite doable over TCP.

Infrastructure servers don't generate big DNSSEC responses. Non
delegation, positive answers get big responses.  DNSKEY and DS
responses in named atleast turns on miminal responses.  Prior to
this change DNSKEY and DS responses were much larger.

---

From: Nick Hilliard <[email-masked]>
Date: Fri, 12 Feb 2016 13:57:20 +0000

> is quite doable over TCP.

this is o/t for v6ops, but how does failover to +vc and then subsequent
tcp query affect query latency?

Nick

---

From: Fernando Gont <[email-masked]>
Date: Tue, 16 Feb 2016 05:34:58 -0300

Thanks!

Cheers,

---

From: Fernando Gont <[email-masked]>
Date: Tue, 16 Feb 2016 05:39:20 -0300

> chain we could verify if an encapsulated TCP packet refers to a valid

Other things aside, this would be done by a totally different layer of
code... layer-3 vs layer-4...

---

From: Fernando Gont <[email-masked]>
Date: Tue, 16 Feb 2016 08:11:40 -0300

> interface, but that's beside the point.)

FWIW, I seem to recall some BSDs had a bug in which during the TCP 3WHS
they would set the FL to zero, and once established it would be set to
non-zero.. not sure if this changed.

In some scenarios (think SYN-cookies) this behavior might be difficult
to avoid.

Thanks!

Cheers,

---

From: Philip Homburg <[email-masked]>
Date: Tue, 16 Feb 2016 14:27:09 +0100

>How do you skip past an unknown EH?

A local config file on the middle box. 

Something like 'safe-unknown-extension-headers: ddd'

The operator adds the number when it is determined to be perfectly
safe (that I find unlikely for new extension headers) and enough people
are speaking up that the operator is a aware of the problem.

---

From: joel jaeggli <[email-masked]>
Date: Tue, 16 Feb 2016 07:10:46 -0800

> In some scenarios (think SYN-cookies) this behavior might be difficult

right, if you want a flow label that is itself not derived from some
deterministic transform you kind of have to keep track of what it is
once you set it.

syn cookies causes a non-zero amount of breakage so it tends to be used
sparingly but it remains an extremely useful tool under duress.

---

From: Warren Kumari <[email-masked]>
Date: Tue, 16 Feb 2016 15:13:54 +0000

> are speaking up that the operator is a aware of the problem.
This also requires that the application developer makes a judgment call as
to when it is safe to be able to enable this unknown EH.
So, after all middle boxes support "safe-unknown-extension-headers" we need:
A: the new unknown EH written
B: someone to be brave enough to try using it (and have it fail in many /
most cases)
C: a large upswell of people going around an poking operators (including
Billybob, who runs the edge middlebox "protecting" Henrys Tire and Wheel
Balancing, Middleburg, VA) to get them to log onto all their devices and
add this new EH to the list of safe things
D: More applications to try using this and have it fail elegantly in some
set of conditions
E: outreach to once again poke Billybob, who replaced his middlebox with
the backup one which lives in the spares closet and wasn't turned on in
step C.
F: application developers to have enough faith that this will work 100% of
the time (keeping in mind that they got bitten in B and D) to turn it on.

This EH would need to provide some *really* compelling benefit to make this
process worthwhile. We've already seen that it is hard to get something
deployed which requires reaching out to all operators and getting everyone
to take some action without direct benefit (e.g BCP38...)

W

---

From: Fernando Gont <[email-masked]>
Date: Tue, 16 Feb 2016 12:28:09 -0300

On 02/16/2016 12:13 PM, Warren Kumari wrote:
> 
> 
> On Tue, Feb 16, 2016 at 8:27 AM Philip Homburg
> <[email-masked] <mailto:[email-masked]>> wrote:
> 
> This also requires that the application developer makes a judgment call
> as to when it is safe to be able to enable this unknown EH.
> B: someone to be brave enough to try using it (and have it fail in many
> / most cases)
> D: More applications to try using this and have it fail elegantly in
> some set of conditions
> F: application developers to have enough faith that this will work 100%
> of the time (keeping in mind that they got bitten in B and D) to turn it on.
> 
> This EH would need to provide some *really* compelling benefit to make
> this process worthwhile. 

I think I learned a long-version of the English word "never" (?). :-)

Cheers,

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 17 Feb 2016 08:31:09 +1300

> deterministic transform you kind of have to keep track of what it is

That's exactly why a stateless hash is the way to go, but certainly it
should be applied from the SYN packet onwards (in both directions).
I guess we should have specified that in RFC 6437.

   Brian

---

From: Fernando Gont <[email-masked]>
Date: Tue, 16 Feb 2016 16:40:53 -0300

> I guess we should have specified that in RFC 6437.

I did something along those lines in:
<https://tools.ietf.org/html/draft-gont-6man-flowlabel-security-03>

Cheers,

---

From: Philip Homburg <[email-masked]>
Date: Wed, 17 Feb 2016 12:23:50 +0100

>I think I learned a long-version of the English word "never" (?). :-)

I don't see anything that we do about this at the protocol level. Do you want
to allocate a not-evil bit that can be set on extension headers? "This
unknown extension header is IETF certified to be not evil, please pass it
along unchecked".

---

From: Warren Kumari <[email-masked]>
Date: Wed, 17 Feb 2016 14:23:41 +0000

On Wed, Feb 17, 2016 at 6:23 AM Philip Homburg <[email-masked]>
wrote:

> I don't see anything that we do about this at the protocol level.

I fully agree -- and yet we seem to keep trying to...

> Do you want
That sounds like a grand idea! Let's do that!

More seriously, I think that new EH (and "long" chains") simply won't fly,
that we should admit this and get on with fixing the other issues.
Yes, v6 was supposed to be the grand panacea, curing all ills and being
infinitely extensible - but it turns out that (just like for the
alchemists) the real world gets in the way. Shaking your fists at the gods
and demanding that the world change simply makes your feet sore and your
arm tired :-)

W

---

From: Tom Herbert <[email-masked]>
Date: Wed, 17 Feb 2016 08:45:43 -0800

> I guess we should have specified that in RFC 6437.
I didn't see any requirement in RFC6437 that the flow label must be
persistent for the lifetime of a flow (e.g. TCP connection). In fact,
we implemented a facility that will recompute the flow label for a
failing connection (random value assigned) in hopes of getting a
different route through ECMP. This is a nice feature in IPv6 since we
can't change the four tuple for an existing connection to get that
same effect in IPv4. If intermediate devices drop packets because the
flow label doesn't match what they think it should be this will be a
problem...

Tom

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 18 Feb 2016 11:42:38 +1300

On 18/02/2016 05:45, Tom Herbert wrote:
> persistent for the lifetime of a flow (e.g. TCP connection). 

No, nor is it forbidden to use the same flow labels for two
consecutive flows that are really part of the same application
session. But I think we should have said to mark the *first*
packet, precisely so that it will take the same path as its
followers.

> In fact,

Agreed.

   Brian