Subject: [v6ops] IPv6 MTU Flow-label.... (related to draft-v6ops-pmtud-ecmp-problem-01)

From: Jeroen Massar <[email-masked]>
Date: Mon, 10 Nov 2014 11:31:52 +0100

Hola folks (and folks in BCC ;),

With the recent Google and Akamai outages (latter still ongoing afaik),
it came to light that the cause is likely the model and problem
described here:

 https://tools.ietf.org/html/draft-v6ops-pmtud-ecmp-problem-01
which previously was:
 https://tools.ietf.org/html/draft-v6ops-jaeggli-pmtud-ecmp-problem-01

Or shortly described: terminating an IP address at different hosts and
having the balancer box not knowing where to deliver the ICMP PTBs that
get send for large packets.

One of the suggestions there is to lower the MSS for every connection by
forcing it (either on the loadbalancer or on the final host) to a value
that "works everywhere": the one for an MTU of 1280.

MSS only applies to TCP, and people like Google are coming out with QUIC
and other schemes.

As we really do not want an Internet at an MTU of 1280, why don't we
indicate in the packet what the MTU is when it is diverting from the norm?

What if we instead let a router that sources a packet from a link or is
going to transmit a packet over a link < 1500 indicate with that packet
that that packet came from/is going to is a link with a MTU < 1500.

We can't use an additional extension header, as adding anything would
mean we might hit the MTU of the packet and we have other issues.

As our least-known-used field is the FlowLabel field, we could abuse
that and have enough bits there to stuff our data.

What if we define that when the first 4 bits are set to 0xF (all one)
that the rest (16bits) defines the MTU of the link (MTU 0 - 65k)?
(We could even use a 'base of 1280' and thus 0xf0000 = 1280 MTU, but
possibly it is better to state "value of < 0xf0500 is invalid")

Thus allowing when the first 4 bits are not set to all-1 that the
flowlabel field is a "normal flowlabel" field ala RFC6437. We could even
state "Only set this MTU option when the FlowLabel field == 0" to avoid
incompatibility (though I do not expect any as I rarely see packets with
the field non-0...)

Thus given a network like:

  [H1]
2001:db8:1500::1/64
   | mtu = 1500
2001:db8:1500::a/64
  [RA]
2001:db8:1501::a/64
   | mtu = 1500
2001:db8:1501::b/64
  [RB]
2001:db8:1480::b/64
   | mtu = 1480
2001:db8:1480::c/64
  [RC]
2001:db8:1280::c/64
   | mtu = 1280
2001:db8:1280::d/64
  [RD]
2001:db8:9000::d/64
   | mtu = 9000
2001:db8:9000::2/64
  [H2]

RA receives packet, src+dst interface are MTU=1500, thus does nothing
RB receives packet, src = 1500, dst = 1480, thus sets FL = 0xf05c8
RC receives packet, src = 1480, dst = 1280, thus sets FL = 0xf0500
RD receives packet, src = 1280, dst = 9000, thus sets FL = 0xf0500
                             (again, just set is quicker than checking)

Now even if H2 is a loadbalancer, if the flow is just forwarded (without
TTL change btw...) the destination receives it correctly.

The disadvantage is of course that you lose the ability to balance based
on the FlowLabel, but if we go with "only change when not 0 then there
was one anyway. Also you got src+dst which is 256bits, which should be
pretty good already and optionally next-header + the contents of the
header if you want that.

Note that as we have no checksum in IPv6, there is little overhead to do
this kind of forwarding, HopLimit already needs updating, this is just
another field to update.

In another model from the above, we could even just let every hop set
the known lowest MTU.

In that case, H1 would set 0xf05dc in the packet, and then it gets
lowered automatically. Which would also mean that a pure 9000 path would
nicely work suddenly as everybody knows that 9000 will fit :)

Greets,
 Jeroen

---

From: Jeroen Massar <[email-masked]>
Date: Mon, 10 Nov 2014 11:58:06 +0100

On 2014-11-10 11:31, Jeroen Massar wrote:
[..]
> Thus given a network like:
[..]

Of course, there is one big problem with that example:
 network paths might be asymmetric.

As such, such a change will work fine in a network where the end-links
and not the core/transit (that might be asymmetric) is changing MTU.

Thus in the case the return path is different and there is a MTU change
there, that link will not have the correct MTU from this measurement.

But there is no easy way to solve this except for working PMTUD as that
does let the source know that that link has a wrong MTU.

Hence, this "IPv6 MTU Flow-Label" idea only solves a part of the problem
and PMTUD as per ICMP needs to remain working and operable.

But IMHO, as most well-connected mass-hosting providers are very close
to their users they likely have a non-assymetric path and this trick
will just solve their problem a little bit better.

Greets,
 Jeroen

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 11 Nov 2014 07:57:43 +1300

Er, no, you really can't use the flow label like that. RFC6437.

What you can do is use the flow label as intended, for ECMP or any
other kind of load balancing. RFC6438, RFC7098. They don't fix the
ICMPv6 problem though.

The key sentence in Joel's draft is

>    Because the PTB message is not identifiable as part of the
>    original flow by the packet header the results of the ICMPv6 ECMP
>    hash are unlikely to be hashed to the same nexthop as packets
>    matching TCP or UDP ECMP hash.

To fix that, I suspect that flow label reflection is needed.
draft-wang-6man-flow-label-reflection doesn't discuss its use for
ensuring that ICMPv6 is treated consistently, but could do so.

Regards
   Brian

On 10/11/2014 23:58, Jeroen Massar wrote:
> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> [email-masked]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
> --------------------------------------------------------------------
>

---

From: Jeroen Massar <[email-masked]>
Date: Mon, 10 Nov 2014 20:02:44 +0100

> Er, no, you really can't use the flow label like that. RFC6437.

Hence, redefining the bits :) Few implementations use them anyway, thus
we still have time to actually make good use of them.

Note that I am specifically proposing setting the first four bits to 1
aka 0xf so that the space can still be used for it's original intent too.

> other kind of load balancing. RFC6438, RFC7098. They don't fix the

It won't fix the 1-RTT issue that worries content providers either.

That is the biggest problem for them: a ICMPv6 PTB is an extra round
trip and they want speed. Hence including the MTU already in the packet.
(though as noted in my reply, it won't work fully for async networks)

> To fix that, I suspect that flow label reflection is needed.

Won't work as that packet will not be matching the loadbalancers setup
either.

What would partially work is if the ICMPv6 PTB would copy the flow-label
from the original flow that caused the PTB.

But then still, large content providers will not be happy, as it does
not satisfy the 1-RTT argument, which now causes them to just ignores
PTBs and set the MSS to something tiny (thus in the end causing more
packets, but at least all packets will go through).

Greets,
 Jeroen

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 11 Nov 2014 08:54:24 +1300

> aka 0xf so that the space can still be used for it's original intent too.

Yeah, well, 6man has consistently said no to every such proposal.

> from the original flow that caused the PTB.

Yes, that idea has come up a few times but frankly I will be amazed
if it ever catches on.

> packets, but at least all packets will go through).

Sure. Isn't that the main reason that the IPv4 network actually works
today?

    Brian

---

From: Jeroen Massar <[email-masked]>
Date: Tue, 11 Nov 2014 10:52:48 +0100

> Yeah, well, 6man has consistently said no to every such proposal.

Well, maybe it is time to finally look at such a proposal instead of
keeping those 20 bits be used for nothing useful.

At minimum there should be a document describing WHY such a change is
out of the question.

It is not because anybody used the IPv6 Flowlabel for any useful data.
Devices will always combine it with src+dst+nextheader at minimum, which
is a lot more bits than those 20 bits wasted on a flowlabel.

> Yes, that idea has come up a few times but frankly I will be amazed

Actually there is no need for it. The ICMPv6 PTB has a maximum of 1280
bytes (due to minimum MTU), but includes an IPv6 header (40 bytes), the
ICMP PTB (~20 bytes) and then a copy of the original packet.

Hence, unless devices do not trust those packets, it will contain at
least the original 40 bytes of the original IPv6 packet that caused the
PTB, which includes the source + dest + flow-label and likely even the
UDP/TCP packet.

As such, while handling ICMPv6 PTBs on a load-balancer is extra work, it
can be done.

There will be about 1200 bytes of the original packet in there, most of
that will be data, not even headers that are needed to identify the packet.

> Sure. Isn't that the main reason that the IPv4 network actually works

No. The IPv4 works as routers fragment and thus no ICMP is involved
unless we are talking going below the extremely small IPv4 MTU.

Greets,
 Jeroen

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 12 Nov 2014 07:46:13 +1300

> At minimum there should be a document describing WHY such a change is

This is exactly why we did the RFCs I mentioned. EOM.

   Brian

---

From: Mark ZZZ Smith <[email-masked]>
Date: Tue, 11 Nov 2014 14:44:49 -0800

>________________________________
> From: Jeroen Massar <[email-masked]>
>To: Brian E Carpenter <[email-masked]> 
>Cc: IPv6 Operations <[email-masked]>; 6man <[email-masked]> 
>Sent: Tuesday, 11 November 2014, 6:02
>Subject: Re: [v6ops] IPv6 MTU Flow-label.... (related to draft-v6ops-pmtud-ecmp-problem-01)
> 
>

So the "flow label" then field name wouldn't then be accurate, because an MTU value doesn't identify or contribute to the identification or labelling of a flow very well. There's a lot of troubleshooting and other value in "truth in advertising", or less cliché'd "accuracy in naming". Encoding non-flow labelling MTU information in that existing flow labelling field would therefore be a hack (MTU might be an attribute of a flow, but it isn't very specific or unique to a flow) for this one special case of stateless ECMP based load balancing. It wouldn't be useful in most or perhaps all other IP use cases. (So if this idea gained more traction, I'd start lobbying for the field to be renamed.)

It seems to me that the fundamental issue is that given how the IP protocols work (as this problem also occurs in IPv4), stateless ECMP isn't a very effective method of performing load balancing. While probably chosen for pure performance/throughput, it isn't distributing load to the LB hosts based on their current load (so it isn't actually 'balancing'), and because it isn't maintaining per-host state, isn't able to perform per-host specific processing/forwarding, which is what is needed for PMTUD to function in this scenario.

This/these methods of load balancing have generally annoyed me for a while, primarily because I have also had to troubleshoot or deal with these sorts of issues in the past. I think the fundamental issue is that to the network, the unicast LB address is looks like a single unicast destination/host, where as in reality is being shared by a number of hosts. While that might sound like the anycast problem, I think the difference is that even though with anycast multiple addresses are shared between a set of hosts, for any particular source host there is no variation in which of the anycast hosts is used over time, unless it fails (i.e., the routing system would normally consistently pick one of the anycast destinations for a particular source, as would IPv6's 'on-link' anycast method (see RFC7094 for more discussion)). In this LB case the destination LB host changes based on changes in TCP or UDP ports, which is occurring all of the time.

If high performance stateless and therefore load insensitive LB distribution is needed or wanted, then perhaps something more simple that doesn't use TCP or UDP port information might be better. For example, using SA+DA based forwarding in the LB, and then statically mapping portions of the SA address space to a particular LB host (e.g., 1/4 to LB host a, 1/4 to LB host b etc. for four LB hosts), with all LB hosts configured with the LB 'shared' unicast address (i.e., no NAT). If you want anything smarter than that, then I think the LB needs to become stateful, and start maintaining per-connection state.

>_______________________________________________
>v6ops mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/v6ops
>
>
>
>

---

From: Lorenzo Colitti <[email-masked]>
Date: Tue, 11 Nov 2014 13:25:23 -1000

On 11 Nov 2014 12:47 pm, "Mark ZZZ Smith" <[email-masked]> wrote:
> If high performance stateless and therefore load insensitive LB
distribution is needed or wanted, then perhaps something more simple that
doesn't use TCP or UDP port information might be better. For example, using
SA+DA based forwarding in the LB, and then statically mapping portions of
the SA address space to a particular LB host (e.g., 1/4 to LB host a, 1/4
to LB host b etc. for four LB hosts), with all LB hosts configured with the
LB 'shared' unicast address (i.e., no NAT). If you want anything smarter
than that, then I think the LB needs to become stateful, and start
maintaining per-connection state.

The problem is not that the TCP/UDP port changes. The problem is that the
source address of the intermediate router sending the PTB hashes
differently than the source address of the client. So SA+DA only
load-balancing won't help.

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 12 Nov 2014 09:46:41 +0100

On 2014-11-11 23:44, Mark ZZZ Smith wrote:
[..]
> So the "flow label" then field name wouldn't then be accurate, because an
> MTU value doesn't identify or contribute to the identification or
labelling
> of a flow very well.

Yes, that field name would have to get a better name indeed.

> There's a lot of troubleshooting and other value in "truth in advertising",
> or less cliché'd "accuracy in naming". Encoding non-flow labelling MTU
> information in that existing flow labelling field would therefore be a
hack
> (MTU might be an attribute of a flow, but it isn't very specific or
unique to a flow)
> for this one special case of stateless ECMP based load balancing.

It is only a hack if somebody currently does it. It would not be a hack
when it is an official RFC.... and then for sure the name would be
updated to reflect this information.

Note that I specifically propose setting the the first four bits to 1 so
that the old mechanism can keep on working.

As the flow-label does not add anything to the magic bits (src + dst)
that are already present and as it does not solve the actual problem
that loadbalancers have: associating any ICMPv6 message caused by the
'flow' to be associated; the flow label as currently specified is quite
useless anyway.

Copying the flow-label from the real flow into the flow-label of the
ICMPv6 would not help either, as the load balancer, per the current
spec, uses src+dst+flow-id and as the src == router-which-sent-icmpv6
the flow might end up at a different device.

Vendors will not like the flow-id either, as it cannot be promised to be
uniformly distributed by the sending source, thus just using flow-label
is not good enough. Ignoring the part where, if the flow-label was the
only part used for load balancing it would be useless for all current
implementations that do not set the flow-id at all.

The more correct answer to that problem is that Load Balancers MUST look
at the content of the ICMPv6 packet for making load balancing decisions
and not just do it based on src/dst(+flow-label).

> It wouldn't be useful in most or perhaps all other IP use cases.
> (So if this idea gained more traction, I'd start lobbying for the
field to be renamed.)
> 
> It seems to me that the fundamental issue is that given how the IP protocols work
> (as this problem also occurs in IPv4), stateless ECMP isn't a very
effective method
> of performing load balancing. While probably chosen for pure
performance/throughput,
> it isn't distributing load to the LB hosts based on their current load
(so it isn't
> actually 'balancing'), and because it isn't maintaining per-host
state, isn't able
> to perform per-host specific processing/forwarding, which is what is
needed for PMTUD to function in this scenario.

Just letting the load balancer peek into the ICMPv6 packet, which
includes a large portion of the packet that would be sufficient.

For performance reasons apparently these devices are not doing so.

But next to load-balancers, there are still people who think they have
the need to filter ICMPv6 completely. These devices miss out on PMTUD
altogether.

> This/these methods of load balancing have generally annoyed me for a while, primarily because
> I have also had to troubleshoot or deal with these sorts of issues in
the past.
> I think the fundamental issue is that to the network, the unicast LB
address is looks
> like a single unicast destination/host, where as in reality is being
shared by a number of hosts.
[..]

Indeed. The problem there is simply that IP was not designed with such a
system (load-balancers looking only at src/dst(/flow-label)) in mind,
especially when considering these devices have to look at ICMPv6 too,
which they clearly do not (for performance reasons, hardware/asic etc).

What is sad about this is that the vendors who built those devices did
not discover this till they deployed those devices and that when they
did, they apparently did not raise this for discussion in the community
so that a proper fix could be made.

Greets,
 Jeroen

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 12 Nov 2014 09:57:27 +0100

On 2014-11-12 00:25, Lorenzo Colitti wrote:
> 
> On 11 Nov 2014 12:47 pm, "Mark ZZZ Smith" <[email-masked]
> <mailto:[email-masked]>> wrote:
>> distribution is needed or wanted, then perhaps something more simple
>> that doesn't use TCP or UDP port information might be better. For
>> example, using SA+DA based forwarding in the LB, and then statically
>> mapping portions of the SA address space to a particular LB host (e.g.,
>> 1/4 to LB host a, 1/4 to LB host b etc. for four LB hosts), with all LB
>> hosts configured with the LB 'shared' unicast address (i.e., no NAT). If
>> you want anything smarter than that, then I think the LB needs to become
>> stateful, and start maintaining per-connection state.
> 
> The problem is not that the TCP/UDP port changes. The problem is that
> the source address of the intermediate router sending the PTB hashes

Thus what the load balancer should be doing is unfortunately special
casing the situation when the next-header (which might need an extension
header walk) is ICMPv6 and then look into the copy (first portion upto a
max of 1280 total packet size) of the original packet that caused that
ICMPv6 and actually make the load balancing decision based on that.

But it is quite likely that such a decision branch does not fit in the
ASICs how they where designed, thus all currently deployed hardware and
stuff coming out in the next years will be broken for this kind of setup.

Though, they could in theory avoid doing the extension walk, and only
handling this special case when the first header is ICMPv6. Though then
you still get magic breakage when

Even if we get such a "IPv6 MTU Label" it does not exclude that ICMPv6s
might get send. It only minimizes the chance that they happen.

What The "IPv6 MTU Label" will give us is that that Load Balancer does
not need to know about it, it can just pass the packets based on src/dst
as it has been doing anyway.

Just in case the process is possibly better described in:
http://www.ietf.org/id/draft-massar-v6man-mtu-label-00.txt

Though I was already informed of a small oversight in that only the 3rd+
packet should attempt to increase the MTU to the MTU that is discovered.

packet 1 @ 1280 max: A -> B to discover MTU on A -> B
packet 2 @ 1280 max: B -> A to discover MTU on B -> A
                            and inform A of A-B-A path
packet 3 @  MTU max

I'll prepare an update to include that detail after processing some
other comments that where received. More comments/arguments and even "We
should not do this because Y" and "it does not work because X" are very
welcome of course, those things are called Requests for Comments after
all and it is not a STD yet...

Greets,
 Jeroen

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 12 Nov 2014 10:27:14 +0100

> This is exactly why we did the RFCs I mentioned. EOM.

I assume you meant RFC6294. This does not discuss the possibility of
using the bits for the purposes of making the live of every operator
easier and avoiding PMTUD blackholes.

Also note from the introduction of that document:
 "However, it [Flow Label] is used very little in practice"

The new set of RFCs are only from later in that year and have the same
problem.

But the bigger issue is simply that for load-balancing purposes "on just
the IP layer" it does not work because of ICMPv6.

Which is documented in RFC7098:
8<----------------------------
      2.  A layer 3/4 balancer must correctly handle Path MTU Discovery
          by forwarding relevant ICMPv6 packets in both directions.
          This too is not directly affected by use of the flow label.
          It should be noted that there may be difficulty correlating an
          ICMPv6 "Packet too big" response with the session it refers
          to, but that is out of the scope of the present document.
--------------------------->8

As the layer 3/4 balancer is then looking into the lower layers anyway
(ICMPv6), they don't need the flow-label as then they can then also look
at TCP/UDP ports which are much better for uniqueness than an
arbitrarily set 20 bit label instead of a much better src & dst-port =
2x16bit = 32bits; though okay, indeed dst-port is likely always 80 or
443. Then again, src/dst should be enough to balance load, if a certain
src is creating most of your traffic, better let one node handle that
one then DoS your whole cluster.

It is also evident that the balancers out there (at least some of them
used by large corporations) only care about the src + dst pair.

The big point still remains: We can still redefine these recent RFCs
that are nowhere close to STDs and optimize something that will have a
benefit for the lifetime of IPv6.

Or we can keep living in a world where the MTU of IPv6 will defacto be 1280.

While in the proposal I made that MTU can be anything upwards from 1280.

Greets,
 Jeroen

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 12 Nov 2014 09:18:57 -1000

Before answering Jeroen, about the draft in the original subject: huh? I don't understand the problem. Why would PMTUD packets need to flow over the same path as the forward packets unless the load balancing is over multiple paths with unequal MTUs? And in the latter case: DON'T DO THAT.

On 11 Nov 2014, at 23:27, Jeroen Massar <[email-masked]> wrote:

> 2x16bit = 32bits;

That may or may not be the case, opinions about the best way to do something have no bearing on what's out there in the real world.

If I were to implement ECMP for IPv6 I would love to be able to use the flow label for this rather than overload port numbers which have a completely different purpose, aren't even available in some types of packets and aren't always found at the same offset from the beginning of the packet. (Also, it would be cool for multipath TCP to express path selection through changing the flow label.)

But the problem is that some implementations don't set the flow label.

The length of the fields is irrelevant because all of this will be hashed to a handful of bits anyway.

> While in the proposal I made that MTU can be anything upwards from 1280.

Which proposal?

IPv6 traffic is routinely 1500 bytes today and PMTUD failures are _relatively_ infrequent. Of course 1500 is nothing to write home about either, we really need to go beyond that. See:

http://tools.ietf.org/html/draft-van-beijnum-multi-mtu-04

I'll be presenting it on Friday.

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 12 Nov 2014 19:21:49 +0000

Hi,

> http://www.ietf.org/id/draft-massar-v6man-mtu-label-00.txt

This looks very similar to RFC1063, which was obsoleted by RFC1191. There
was an analysis done that more or less concluded the RFC1063 was for all
practical purposes not deployable.

> While in the proposal I made that MTU can be anything upwards from 1280.

What you are proposing is not the only way forward to accomplish this.
RFC4821 progressively grows the size of packets it sends to achieve a
more optimal size even if no PTBs are received. Also, I do not see how
what you are proposing applies to tunnels, or even more specifically
tunnels-within-tunnels. Can you say more about that?

Thanks - Fred
[email-masked]

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 13 Nov 2014 08:42:39 +1300

> Before answering Jeroen, about the draft in the original subject: huh? I don't understand the problem. Why would PMTUD packets need to flow over the same path as the forward packets unless the load…

It would imply that one path was native and the other path was tunnelled,
or something else rather strange. I agree with "don't do that" but maybe
the real world tells us that people do do that?

> But the problem is that some implementations don't set the flow label.

Right. Which is why the recent updates to the flow label standard allow
routers to set the flow label on behalf of hosts that don't. Also why
RFC 6438 "Using the IPv6 Flow Label for Equal Cost Multipath Routing and
Link Aggregation in Tunnels" was written (and my co-author on that was
a real genuine operator).

If vendors aren't implementing the most recent flow label RFCs, it's
presumably because operators aren't asking for it. I think they're even
less likely to ask for encoding MTU in the flow label to be suppported,
even if the IETF changed the standard yet again.

     Brian

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 12 Nov 2014 21:09:12 +0100

On 2014-11-12 20:18, Iljitsch van Beijnum wrote:
> Before answering Jeroen, about the draft in the original subject: huh?
>
> I don't understand the problem. Why would PMTUD packets need to flow over
> the same path as the forward packets unless the load balancing is over
> multiple paths with unequal MTUs? And in the latter case: DON'T DO THAT.

Different paths is just an example, and it indeed mostly happens on the
wild Internet, likely not behind your LB and especially not with a
different MTU there (though some people might GRE tunnel some traffic
somewhere else actually, but that is very scary).

Note that in the ECMP draft the loadbalancer does not _terminate_ the
connection and thus handle the ICMPv6 unreachable but just forwards the
packets (likely even without decreasing TTL).

This unlike for instance a TCP proxy that thus does terminate the TCP
connection (or UDP etc) and then creates a new connection to a backend
system.

> That may or may not be the case, opinions about the best way to do
> something have no bearing on what's out there in the real world.

But this is exactly what those load balancers do and why they fail at
it. Hence why there is a draft that is a WG doc out on this problem.

> If I were to implement ECMP for IPv6 I would love to be able to
> use the flow label for this rather than overload port numbers
> which have a completely different purpose, aren't even available
> in some types of packets and aren't always found at the same offset
> from the beginning of the packet. (Also, it would be cool for multipath
> TCP to express path selection through changing the flow label.)

Note that the Flow-Label set in the ICMPv6 PTB is not the same as the
original flow. Which thus causes the issue as then the ECMP decision
goes to the wrong place.

> But the problem is that some implementations don't set the flow label.

I'll have to run a check grabbing packets of the wire and analyzing the
distribution at one point. But looking at various open source stacks and
wireshark, I don't see much of labels in my local network.

> The length of the fields is irrelevant because all of this will be hashed to a handful of bits anyway.

Quite correct. Hence why bother with flow-labels at all when src+dst
gives you those bits? :)

>> While in the proposal I made that MTU can be anything upwards from 1280.

See:
http://www.ietf.org/id/draft-massar-v6man-mtu-label-01.txt

Originally in the email, now nicely in a draft so that everybody can
comment on it in a much easier way.

> IPv6 traffic is routinely 1500 bytes today and PMTUD failures are _relatively_ infrequent.

Tell that to both Google who is bandaiding this by forcing the MSS to a
small size and Akamai who are currently at a 1.5 months long outage
likely related to this problem.

> Of course 1500 is nothing to write home about either, we really need to go beyond that. See:
> 

Check:
http://www.ietf.org/id/draft-massar-v6man-mtu-label-01.txt

For a much easier solution that and also solves this for load balancer
setups.

It does not discuss direct-neighbor discovery, but it could do that too.

IMHO though on the same link it should be using the MTU configured on
the L2 (eg Ethernet) and L3 (IP) should not be second guessing that
configuration.

Greets,
 Jeroen

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 12 Nov 2014 21:15:49 +0100

> practical purposes not deployable.

Thanks for mentioning that RFC, I missed the "Obsoletes: 1063" at the
top of 1191, while 1063 has a lot more background, even though it is
from 1988. Will study that and related google-able documents to see if I
can gain some extra info out of it.

> more optimal size even if no PTBs are received.

The major problem with that sending a lot of packets is RTT.

The big content providers want to go to 0-RTT. Hence protocols like QUIC
coming into existence.

Also, at the scale of say a Google, Amazon, Akamai, you don't want to
start spraying more packets and doing probing around the world for such
a situation.

The proposed http://www.ietf.org/id/draft-massar-v6man-mtu-label-01.txt
avoids any need for that too.

> Also, I do not see how what you are proposing applies to tunnels,

Tunnels typically have smaller MTUs than native links, hence lower MTU.

But there are also other link technologies that are not tunnels that use
a non-1500 MTU. Hence why tunnels are not mentioned in the document as
it is a generic "lower MTU case"

> or even more specifically

For the IP layer does it matter how many things are nested into it?

As long as the minimum MTU for a link technology is 1280 all is fine.

If the link-technology has a lower MTU than 1280 it will have to make
sure that it can transport those packets by splitting them up in a
magical way.

Greets,
 Jeroen

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 12 Nov 2014 21:28:54 +0100

On 2014-11-12 20:42, Brian E Carpenter wrote:
>> Before answering Jeroen, about the draft in the original subject: huh? I don't
> understand the problem. Why would PMTUD packets need to flow over the
same path
> as the forward packets unless the load balancing is over multiple
paths with
> unequal MTUs? And in the latter case: DON'T DO THAT.
> 

Behind the LB, not likely though possible if one is moving datacenters
and thus forwarding packets to the new backend in another datacenter
which is linked up over a GRE tunnel to move those bits around.

But it is then a much better idea terminate the TCP conection on the
host that is the loadbalancer or directly behind it and do a TCP or
application-specific proxy.

The document (draft-v6ops-pmtud-ecmp-problem-01) mentions problems with
ICMP because the *internet* is inherently asymmetric.

At least I rarely see a symetric link when crossing an ocean.

> routers to set the flow label on behalf of hosts that don't.

In that case if we have:

A - R1 - R2 - R3 - B

A sets FL = 0
R1 passes it
R2 "updates" the FL to <randA>
R3 passes it
B receives <randA>

B sends back a packet with FL=<randA>
...
A receives a unknown flow with FL=<randA>

I don't see how that can work if some people think to be smart and start
firewalling on the FL...

Or what about a IPFIX export from R1, those flows suddenly are
different, which completely defies having a unique label.

More importantly though as the label is randomly set it is unknown if it
is 0 (common) or a random label, or changed by an intermediary.

Hence for IPFIX/analysis functions, for loadbalancing and firewalling
that label as defined is IMHO completely useless..

> Also why

(https://tools.ietf.org/html/rfc6438)

As draft-v6ops-pmtud-ecmp-problem-01 shows, there is a major oversight
called ICMP in that system.

There is also no reference to ICMP or Packets Too Big in that doc.

IMHO, that is a major Errata....

Note that it appears not many people thought/realized the
'pmtud-ecmp-problem', hence why that is a majorly awesome draft that
definitely should become an RFC.

> even if the IETF changed the standard yet again.

Possibly because they don't see the value of such a Flow Label as
nothing uses it.

I am fairly confident that my "MTU Label" proposal though IS valuable to
solve a lot of real life problems AND as a bonus allows nodes to
negotiate a Internet wide high MTU.

Google and likely Akamai (as we don't know their problem yet...) would
have likely not have an outage if we had this in place.

With all the Fiber deployments coming out, there is a major difference
between a 1500 MTU and a 9000 MTU.... even bigger as the 18% for
1280->1500, 1500->9000 is a factor 5 less packets to send (and that is
while ignoring IP, TCP headers etc).

Greets,
 Jeroen

---

From: Mark Andrews <[email-masked]>
Date: Thu, 13 Nov 2014 04:03:45 +1100

> load-balancing won't help.

The PTB MUST have the source and destination addresses of the original
packet to work.

Mark
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Mark Andrews <[email-masked]>
Date: Thu, 13 Nov 2014 08:11:30 +1100

If one has a load balancer that cannot look into the ICMPv6 packet.
Take it out of the rack.  Put it in the back of the pickup and take
it to the recycler.  It is garbage and need to be treated as such.

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 12 Nov 2014 23:30:11 +0000

Hi Jeroen,

> -----Original Message-----
> From: Jeroen Massar [mailto:[email-masked]]
> Sent: Wednesday, November 12, 2014 12:16 PM
> To: Templin, Fred L; Brian E Carpenter
> Cc: IPv6 Operations; 6man

Good luck. You might want to consider digging the TCP-IP mailing list archives from
the 1982-1991 timeframe:

http://www-mice.cs.ucl.ac.uk/multimedia/misc/tcp_ip/

I dug through this years and years ago, but have since lost track of my notes.

> The major problem with that sending a lot of packets is RTT.

It's not a lot of packets. And, who cares about RTT? A flow can start off sending
smalls (1500 or less) and then gradually work its way up to sending bigs (bigger
than 1500). But, communications are continuous and not in any way held up
waiting for an MTU probe reply.

> start spraying more packets and doing probing around the world for such

It is by no means a lot of packets - only an occasional probe to see if you can
do better than 1280.

> avoids any need for that too.

Check the RFC1063 because I think you are proposing roughly the same thing.

> Tunnels typically have smaller MTUs than native links, hence lower MTU.

Well, but how big is "lower"? Once the IPv6 packet is encapsulated in IPv4 and
shipped out across the Internet, that IPv6 flow label is no longer accessible and
legacy IPv4 routers would not honor it even if it were. There are only two ways
of policing the tunnel MTU - "clamp" it to something smaller than 1500, or set
it to unlimited. In the latter case, make sure everything 1500 and smaller gets
through even if a small amount of fragmentation is necessary, but let everything
1501 and larger in unfragmented and let them sink or swim on their own.

> it is a generic "lower MTU case"

You don't know how low "lower" is without some kind of tunnel MTU assurance,
the problem is worse for tunnels-within-tunnels.

> As long as the minimum MTU for a link technology is 1280 all is fine.

If you start out with a 1300 MTU tunnel, then you can run another
1280 tunnel through that. But, the next tunnel in line would see only
1260 and recursion hits a brick wall.

> sure that it can transport those packets by splitting them up in a

For native IPv6 links, link-specific adaptation yes (see: 6lowpan). For tunnels,
there's nothing magic about IPv6 fragmentation - it just works.

Thanks - Fred

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 13 Nov 2014 12:59:51 +1300

...

>From what I learned during that discussion, an ISP doing that kind
of tunnel would simply configure MTU >>1500 and assume that nobody tries
a PMTU >1500. Any PTBs from further downstream could return on any
ECMP path, of course.

> Possibly because they don't see the value of such a Flow Label as

Well, yes, it's a chicken/egg problem of course. RFC 6437 is
a SHOULD in RFC 6434 "IPv6 Node Requirements". But these things
take years to get into products.

   Brian

---

From: Jeroen Massar <[email-masked]>
Date: Thu, 13 Nov 2014 20:05:30 +0100

Hola,

Or hmmm this week likely Aloha for most readers ;)

A new -02 version of the draft is available:

http://datatracker.ietf.org/doc/draft-massar-v6man-mtu-label/

More comments/arguments/etc are of course very welcome.

some comments to the thread inline.

> a PMTU >1500. Any PTBs from further downstream could return on any

The ICMPv6 PTB would not be on the internet network but come from this
magic out of control part called the Internet: where the users are.

Indeed in their own network the ISP can mostly solve this problem with a
variety of tricks like using larger local MTUs.

> take years to get into products.

Which is exactly why we should push this through while we still can.

Unless somebody comes up with an argument why the idea is totally flawed
of course.

Greets,
 Jeroen

---

From: Mark ZZZ Smith <[email-masked]>
Date: Thu, 13 Nov 2014 15:55:58 -0800

----- Original Message -----
> From: Mark ZZZ Smith <[email-masked]>
> To: Jeroen Massar <[email-masked]>; Brian E Carpenter <[email-masked]>
> Sent: Wednesday, 12 November 2014, 9:44
<snip>
> 
> If high performance stateless and therefore load insensitive LB distribution is 
> needed or wanted, then perhaps something more simple that doesn't use TCP or 
> UDP port information might be better. For example, using SA+DA based forwarding 
> in the LB, and then statically mapping portions of the SA address space to a 
> particular LB host (e.g., 1/4 to LB host a, 1/4 to LB host b etc. for four LB 
> hosts), with all LB hosts configured with the LB 'shared' unicast 
> address (i.e., no NAT). If you want anything smarter than that, then I think the 
> LB needs to become stateful, and start maintaining per-connection state.
> 

Actually, I've remembered more details of the problem, and the above doesn't fix it.

I think the fundamental problem is that a unicast address is expected to uniquely identify a single reachable destination from the position of the sender (i.e., a single host, or a single one of the hosts sharing an anycast address, with the routing system selecting the single destination anycast host to deliver it to.).

The trouble here is that the unicast destination address in the PTB message doesn't uniquely identify a single reachable destination, and therefore the simple stateless ECMP method of load balancing may not select the right destination. The source address of the PTB doesn't help either, because it is of the router generating the PTB, so it isn't specific to the session either (which is my simpler stateless method above doesn't fix the problem.)

---

From: Mark ZZZ Smith <[email-masked]>
Date: Thu, 13 Nov 2014 16:07:39 -0800

----- Original Message -----
> To: Mark ZZZ Smith <[email-masked]>
> Sent: Wednesday, 12 November 2014, 19:46
<snip>
> so that a proper fix could be made.

So I'd argue that protocols shouldn't have to be changed in reaction to lack of proper product design. I wouldn't support using the flow label for this purpose. Doing so would seem to me to be expecting the protocols to be band-aided to fix what is a problem with some vendors products, rather than products being designed to properly work with the protocols as they're defined.

I think the fundamental conflict is that the Internet protocols assume a unicast destination address in a packet identifies a single unicast destination for the forwarding system to deliver to. Stateless LBs are violating that assumption, which is why the protocols built on that assumption then fail.

---

From: Jeroen Massar <[email-masked]>
Date: Fri, 14 Nov 2014 01:35:19 +0100

On 2014-11-14 01:07, Mark ZZZ Smith wrote:
[..]

> So I'd argue that protocols shouldn't have to be changed in reaction
> to lack of proper product design. I wouldn't support using the flow
> label for this purpose.

Flow Label cannot work the way that it is defined.

The reason for having a Flow Label is so that for instance a Load
Balancer does not have to look further than the first 40 bytes of packet
(thus just the IPv6 Header).

But, as every node MUST look at ICMPv6, that is thus automatically false.

Hence, the Flow Label is useless for the purposes of identifying a flow.

Note that an IPv6 'error', thus an ICMPv6 packet, for instance an PTB,
but also "port unreachable" etc is part of the flow as it indicates an
error status that relates to that flow.

Obsoleting the Flow Label and using the bits for a problem we did not
have in 1996: Load Balancing, proper MTU discovery etc is thus a good idea.

Note that back then nobody really cared about 0-RTT either. This
proposal CAN make that work, which will make a lot of content providers
very very happy.

Oh, and it solves most cases of the Load Balancer issue too (as
described in the pmtud-ecmp draft). Though yes, it still will need to
inspect a ICMPv6 PTB, at least it is not misled to believe the Flow
Label is useful in any way.

[..]
> I think the fundamental conflict is that the Internet protocols
> assume a unicast destination address in a packet identifies a single
> unicast destination for the forwarding system to deliver to.
> Stateless LBs are violating that assumption, which is why the
> protocols built on that assumption then fail.

That is mostly correct, but that is not the primary reason.

The big problem is that even if the Load Balancer supports parsing the
ICMPv6 PTB and can associate it with the correct "flow" and thus the
right backend host, it won't help as there are a lot of other systems
out there that are on purpose filtering out ICMPv6.

By including the MTU in the packet though, it won't be filtered (or at
least set to 1280, which is the minimum which just makes them hurt their
own performance, but it won't break things).

Greets,
 Jeroen

---

From: Jeroen Massar <[email-masked]>
Date: Fri, 14 Nov 2014 01:37:52 +0100

On 2014-11-14 00:55, Mark ZZZ Smith wrote:
[..]
> The trouble here is that the unicast destination address in the PTB
> message doesn't uniquely identify a single reachable destination, and
> therefore the simple stateless ECMP method of load balancing may not
> select the right destination. The source address of the PTB doesn't
> help either, because it is of the router generating the PTB, so it
> isn't specific to the session either (which is my simpler stateless
> method above doesn't fix the problem.)

Having a Flow Label is thus useless for this situation.

And as the Flow Label is there to make work for Load Balancers and other
'flow based decision makers' easier, ... well it fails at that.

Hence, lets just get rid of it as it is broken by design anyway.

Greets,
 Jeroen

---

From: Ivan Pepelnjak <[email-masked]>
Date: Fri, 14 Nov 2014 08:10:46 +0100

Gentlemen,

I will risk wasting everyone else’s time trying to explain yet again what the real problem is and why all the ranting about load balancers inspecting or not inspecting ICMPv6 messages won’t bring the world peace.

Some environments are simply too big for stateful load balancing. The only workaround those environments have (based on how badly we crippled TCP/IP stack with lack of session layer, leaky abstractions and DNS pinning) is to insert a layer of stateless anycast load balancing in front of the stateful load balancing (or proxies or whatever they use).

The stateless anycast load balancing at speeds these environments need can only be done with a layer 3 switch (device formerly known as router), which is not expected to look deep into the packets (at least that’s the theory that should be promoted on IETF mailing lists AFAIK).

A client request landing at this L3 switch is hashed based on SRC+DST IP (port numbers may be used or not) and sent to one of the anycast destinations. A PMTUD ICMP message arrives from an intermediate hop, not from the client, and is thus hashed based on Router IP+DST IP. Nobody in his right mind will waste the silicon to inspect random fields deep into the ICMPv6 payload, or punt the transit ICMPv6 packets to the CPU just because.

Also, the flow label approach will not solve this problem. You’re just reinventing cookie-based stickiness and all its properties that anyone operating a load balancer immensely loves with flow labels.

Ivan Pepelnjak
www.ipSpace.net / blog.ipSpace.net
Need a quick expert advice? Try ExpertExpress (www.ipspace.net/ExpertExpress)

On 14 Nov 2014 at 01:35:41 , Jeroen Massar ([email-masked]) wrote:

[..]  

Greets,  
Jeroen

---

From: Jeroen Massar <[email-masked]>
Date: Fri, 14 Nov 2014 09:36:48 +0100

On 2014-11-14 08:10, Ivan Pepelnjak wrote:
[..]
> The stateless anycast load balancing at speeds these environments need
> can only be done with a layer 3 switch (device formerly known as
> router), which is not expected to look deep into the packets (at least
> that’s the theory that should be promoted on IETF mailing lists AFAIK).

For the fast path packets this is indeed the case.
I've not seen anybody mentioning anything against this yet in these threads.

> A client request landing at this L3 switch is hashed based on SRC+DST IP
> (port numbers may be used or not) and sent to one of the anycast
> destinations.

Thus you are stating that the "IPv6 Flow Label" is not used?
(I am completely fine with that btw...)

Is there a specific vendor you are talking about here?

> A PMTUD ICMP message arrives from an intermediate hop, not
> from the client, and is thus hashed based on Router IP+DST IP. Nobody in
> his right mind will waste the silicon to inspect random fields deep into
> the ICMPv6 payload, or punt the transit ICMPv6 packets to the CPU just
> because.

Unfortunately you will have to be in your right mind to do this.
As not doing looking at ICMPv6 breaks IPv6.

Please read:
 http://datatracker.ietf.org/doc/draft-v6ops-pmtud-ecmp-problem/

And then read:

The first document this issue (mostly the same like you describe it).

The second proposes a way to minimize the chance of ICMPv6 PTBs being
generated by redefining the "IPv6 Flow Label" field to become an "MTU
Label" (better name still looked for).

Handling the remaining PTBs still have to be done in such a device, but
could be done slow-path, or heck, just forwarding it all to a specific
device when proto == ICMPv6 and letting that forward it to the right
hosts if needed.

Looking forward to your comments to those documents.

Greets,
 Jeroen

---

From: Ivan Pepelnjak <[email-masked]>
Date: Fri, 14 Nov 2014 11:03:06 +0100

On 14 Nov 2014 at 09:36:52 , Jeroen Massar ([email-masked]) wrote:
On 2014-11-14 08:10, Ivan Pepelnjak wrote: 
[..] 

For the fast path packets this is indeed the case. 
… and this is the problem some people have to solve. Slow path is not an option for them.

Is there a specific vendor you are talking about here? 
I haven’t noticed anyone mentioning it in their documentation, but I’ve been wrong before ;) Would all major vendors doing per-flow-label load balancing on high-speed hardware switching gear please raise their hands?

Unfortunately you will have to be in your right mind to do this. 
As not doing looking at ICMPv6 breaks IPv6. 
Interim nodes are not required to look into ICMPv6 (or we would have a nice attack vector), and the L3 switch doing anycast ECMP is an interim node, so could we please stop this thread?

Please read: 

And then read: 

The first document this issue (mostly the same like you describe it). 
OK, so we’re in sync that there is a real-life problem experienced on real-life gear used in real-life networks. Good.

The second proposes a way to minimize the chance of ICMPv6 PTBs being 
generated by redefining the "IPv6 Flow Label" field to become an "MTU 
Label" (better name still looked for). 
And you might eventually get a solution that will be implemented in 3-5 years. In the meantime, we supposedly ran out of IPv4 addresses, and the “too big to load balance statefully” web properties exist and have to cope with real-life issues. 

How about figuring out how to solve them with the tools that have a fighting chance of being useful when we need them (= now)?

Handling the remaining PTBs still have to be done in such a device, but 
could be done slow-path, or heck, just forwarding it all to a specific 
device when proto == ICMPv6 and letting that forward it to the right 
hosts if needed. 
Now, this is an interesting one. It could be done if the ICMPv6 redirector would know where to send the ICMPv6 packet based solely on headers in the payload, which would require it to know the hashing tables in the L3 switch, which could be made deterministic with OpenFlow, and a controller that would monitor the flow counters and adjust the OpenFlow-based hashing tables in real time. Wait, hang on for just a second, I have Rube Goldberg on the other line ...

Greets, 
Jeroen

---

From: Jeroen Massar <[email-masked]>
Date: Fri, 14 Nov 2014 11:31:12 +0100

On 2014-11-14 11:03, Ivan Pepelnjak wrote:
[..]
> Interim nodes are not required to look into ICMPv6 (or we would have a
> nice attack vector), and the L3 switch doing anycast ECMP is an interim
> node, so could we please stop this thread?

Unfortunately Load Balancers are special cases.
They are NOT "interim routers" and typically they do not even lower the
HopLimit.

Yes, that is a design oversight when IPv6 was made. Back then likely
even the idea of these kind of Load Balancers did not exist yet and thus
it was not accounted for. If they had existed ICMP would have been inline.

Note that even for real 'routers' they are supposed to handle Hop-By-Hop
options which have the Router Alert flag set. Now I know that my code
does not do that for instance, I am sure others do not either.

> OK, so we’re in sync that there is a real-life problem experienced on
> real-life gear used in real-life networks. Good.

Most people on this thread are, as they are aware of the first doc.

> And you might eventually get a solution that will be implemented in 3-5
> years.

The -03 version of the draft (coming soon to a server near you)
describes the option of sticking to an MTU of 1280 when either direction
on the path has no MTU Label.

This way it can be gradually deployed without any MTU issues for any
protocol while when sufficient support (both src + dst understand it) is
reached it will automatically go up again.

This in contrast to MSS clamping which stick the MTU to 1280.

> In the meantime, we supposedly ran out of IPv4 addresses, and the
> “too big to load balance statefully” web properties exist and have to
> cope with real-life issues. 
> 
> How about figuring out how to solve them with the tools that have a
> fighting chance of being useful when we need them (= now)?

The users of these kind of load balancers are using a temporary bandaid:
clamping MSS.

Hence. There is no extreme urgency for those people at the moment as
they mostly just do TCP.

The problem comes though when they move on to non-TCP protocols.

> Now, this is an interesting one. It could be done if the ICMPv6
> redirector would know where to send the ICMPv6 packet based solely on
> headers in the payload, which would require it to know the hashing
> tables in the L3 switch, which could be made deterministic with
> OpenFlow, and a controller that would monitor the flow counters and
> adjust the OpenFlow-based hashing tables in real time. Wait, hang on for
> just a second, I have Rube Goldberg on the other line ...

Handling it in the slow path is thus an easer option.

With the MTU Label in place though one would receive very very few of
ICMP PTBs.

And if one is then still afraid of performance one could:
 - rate limit handling them (might break some things...)
 - only handle ICMPv6 headers directly following the IPv6 header

Greets,
 Jeroen

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 15 Nov 2014 06:57:14 +1300

> (I am completely fine with that btw...)

The point is that the flow label is now defined in a way that
makes a stateless hash based on {src,dst,flow-label} more useful
than just {src,dst} but still using strictly fixed fields in the
beginning of the packet. It is a fact that this is fairly new
in the standards and if you only read RFC 2460 you will miss it.
But when we redesigned the flow label model we did talk to load
balancer implementors, believe it or not. I hope it will show up
in products one day.

We didn't solve the ICMPv6-PTB problem because it's pretty much
insoluble. OK, we could write rules about the boxes that generate
PTB having to insert the appropriate flow label (but that only
works if flow label reflection is universally deployed, so
that will never work in the real world).

   Brian

> ------------------------------------------------------------------------
>

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 14 Nov 2014 09:06:19 -1000

[snip]

Ok, good explanation on why this is useful.

Would one solution be to simply duplicate these too big packets over ALL ECMP paths?

---

From: joel jaeggli <[email-masked]>
Date: Fri, 14 Nov 2014 09:55:24 -1000

> Would one solution be to simply duplicate these too big packets over ALL ECMP paths?

That's actually my mitigation  on the edge (subject to rate limits)

joel

---

From: Ian Cox <[email-masked]>
Date: Fri, 14 Nov 2014 21:50:10 +0000

From: v6ops [mailto:[email-masked]] On Behalf Of Ivan Pepelnjak
Sent: Friday, November 14, 2014 2:03 AM
To: Jeroen Massar
Cc: IPv6 Operations; 6man
Subject: Re: [v6ops] IPv6 MTU Flow-label.... (related to draft-v6ops-pmtud-ecmp-problem-01)

On 14 Nov 2014 at 09:36:52 , Jeroen Massar ([email-masked]<mailto:[email-masked]>) wrote:
On 2014-11-14 08:10, Ivan Pepelnjak wrote:
[..]

This option has existed since Trident+ in Broadcom devices.

If the Flow Label is non-zero there is an option to include the IPv6 Flow Label  into the vector that goes to the hash for ECMP. Placing IPv6 Flow Label into vector is not the default.  The  ICMP and TCP/UDP are going to end up taking different paths since the TCP/UDP port numbers can also be admitted vector that goes into the hash, and a lot of implementations I’m aware of have this behavior as default.

Ian

---

From: Jeroen Massar <[email-masked]>
Date: Fri, 14 Nov 2014 23:16:04 +0100

On 2014-11-14 22:50, Ian Cox wrote:
> Ivan Pepelnjak wrote:
>> I haven’t noticed anyone mentioning it in their documentation, but
>> I’ve been wrong before ;) Would all major vendors doing
>> per-flow-label load balancing on high-speed hardware switching gear
>> please raise their hands?
> 
> If the Flow Label is non-zero there is an option to include the IPv6
> Flow Label  into the vector that goes to the hash for ECMP. Placing IPv6
> Flow Label into vector is not the default.

When it is not the default, it is not being used.

The good thing is that when those 20 bits are re-purposed it does not
hurt that platform.

> The  ICMP and TCP/UDP are
> going to end up taking different paths since the TCP/UDP port numbers
> can also be admitted vector that goes into the hash, and a lot of
> implementations I’m aware of have this behavior as default.

As you state: The IPv6 Flow Label is not being used.

Also, you are stating that there are better and more reliable
information sources (TCP/UDP port numbers) to mix into the hash that can
be used in apparently hardware.

Greets,
 Jeroen

---

From: Jeroen Massar <[email-masked]>
Date: Fri, 14 Nov 2014 23:24:26 +0100

On 2014-11-14 18:57, Brian E Carpenter wrote:
[..]

> insoluble.

But because that is not solved the Flow Label is also useless as the
intended audience is broken when it does not want to bother with ICMPv6
as the Flow Label makes a promise of not having to peek further than the
IPv6 header itself.

Load Balancers that do properly peek further than the IPv6 Header will
find TCP/UDP/SCTP ports but also the ICMPv6 PTB. But these do not need a
Flow Label anymore as they have lots of good bits there to hash.

> OK, we could write rules about the boxes that generate

You don't need flow reflection for that.

You do need every box that is sending ICMPv6 errors (PTBs and anything
else, including port unreach) to copy the Flow Label from the original
packet into the error packet.

Note the Port Unreach note above, or heck Network Unreachable: the
router that receives the packet indicates that the destination is
unreachable to the sender of the packet. As most Load Balancers are
HTTP-based and thus TCP, one of the servers actually doing that TCP
connection will have a nice timing-out TCP connection when during a TCP
connection a network route flips over that causes network unreachables.

This though more shows that it is simply a requirement to parse ICMPv6
properly in even those funny load balance boxes...

ICMPv6 is still a node requirement afaik, but the Load Balancers just
ignore that requirement.

The sudden network unreach is insolvable though, but MTU is something we
can solve by getting that message to the Load Balancer.

Greets,
 Jeroen

---

From: Mark ZZZ Smith <[email-masked]>
Date: Tue, 18 Nov 2014 01:10:46 +0000 (UTC)

----- Original Message -----
> Sent: Friday, 14 November 2014, 11:35

Agree.

> But, as every node MUST look at ICMPv6, that is thus automatically false.

I don't think that really needs to be the case, and I think it actually is somewhat contrary to the end-to-end principle, which I think is really just a case of "if you want something done properly, you need to do it yourself."

ICMP isn't reliable, as they're not acknowledged and therefore aren't and can't be retransmitted if they're lost. Additionally, they're send rate limited by implementations. Consequently, they're really only a hint from the network rather than something that is essential and therefore can be relied upon. For the purposes of PMTUD, which is something hosts would want done properly (so they need to do it themselves), they should be using ICMP messages as hints if they're received, but use other methods to discover the PMTU if they're not. This is why I think the method of PMTUD described in RFC4821 is better, perhaps also using ICMP PTBs if they're received too.

> have in 1996: Load Balancing, proper MTU discovery etc is thus a good idea.

Actually, I really think this form of load balancing at layer 3 isn't a good idea at all. I think it is being implemented at the wrong layer and in the wrong place.

It seems to me that the fundamental problem that 'network' load balancing is trying to solve is to be able to scale up the processing capacity of an application or a set of applications. I think that is an operating system or application layer problem, not a networking problem.

As an operating system problem, it is solved in using techniques such as multi-processing, threads and load aware resource schedulers. Doing that well and effectively means keeping a lot of state, which is what operating systems do and do well. Network located and stateless LB isn't going to be anywhere near as effective, and is also going to encounter the issues described because of lack of enough (or any) state.

I think it is quite reasonable and desirable to cluster together many commodity machines to solve the application processing capacity scaling problem. However, I think the much better model is to use an operating system that can properly cluster together their resources such that to the network it looks like a single and multi-homed host. I have no experience with them, however I first came across this idea back in 1995/96 at a presentation on SGI's Origin system, which could scale up to 128 nodes (which were stand alone computers by themselves). The OpenSSI (http://openssi.org/) project seemed to be doing the same thing with Linux as the base operating system.

IOW, I think application processing capacity scaling is a host or end-system function, not a networking function.

Another way to answer the question of whether it is a network or host function is to ask where the packet payload 'care/don't care' boundary is. I think if a device is just processing network layer packet headers and doesn't care what the payload is then is a networking device; if the device does care what the payload is (which includes the TCP/UDP headers and above), then it really is a host or end-system device (or at least performing one of the host or end-system functions). Another way to answer this question is whether an encrypted payload matters or not. To networking devices it doesn't, to devices 'in the network' that are trying to perform host or end-system functions, it does, as they will fail when faced with payload encryption.

So I think an LB is part of the host or end-system, and therefore it should own the IP address and be the end point for that IP address. The ICMP PTB problem then disappears, because there is no ambiguity as to which device is processing the PTB for the IP address because there is only one, or in the case of anycast, only one device reachable with that IP address, chosen by the routing system, from a particular source.

---

From: Jeroen Massar <[email-masked]>
Date: Tue, 18 Nov 2014 03:40:34 +0100

On 2014-11-18 02:10, Mark ZZZ Smith wrote:
[..]

> For the purposes of PMTUD, which is
> something hosts would want done properly (so they need to do it
> themselves), they should be using ICMP messages as hints if they're
> received, but use other methods to discover the PMTU if they're not.
> This is why I think the method of PMTUD described in RFC4821 is
> better, perhaps also using ICMP PTBs if they're received too.

Yep. This is why I am trying to come up with a proper way of putting the
MTU in the IPv6 Header instead of the Flow Label.

I'm in the process of rewriting the doc with a better name and details,
new name: "The 'IPv6 Maximum Payload Length' (IPv6 MPL) Field"

Thus no more mix of "Label" in there, it will just completely wipe out
the IPv6 Flow Label as I cannot find anybody who can define a real use
for it. The Flow Label is a fun idea, but the standard five tuple covers
them just fine, thus no need to waste those bits there.

[..]
>> Obsoleting the Flow Label and using the bits for a problem we did
>> not have in 1996: Load Balancing, proper MTU discovery etc is thus
>> a good idea.
>> 
> 
> Actually, I really think this form of load balancing at layer 3 isn't
> a good idea at all. I think it is being implemented at the wrong
> layer and in the wrong place.

>From an IP perspective I agree. The problem though is that the folks
doing high bandwidth setups require this today, and thus do it like that.

They went through the HTTP proxy thing already and terminating millions
of TCP sessions on a single IP does not scale.

Their only other alternative would be providing a different IPv6 address
per user, but that does not scale either.

> It seems to me that the fundamental problem that 'network' load
> balancing is trying to solve is to be able to scale up the processing
> capacity of an application or a set of applications. I think that is
> an operating system or application layer problem, not a networking
> problem.

It is not an application thing. It already breaks at TCP.

Note that a standard HTTP proxy needs to accept a TCP session and then
also forward it.

This is prone for DDoS.

With the current 'transparent load balancer' which just looks at
src/dst, puts it in a hash and picks out a backend box at random, there
is no state, hence there is nothing to DDoS except the 1000's of hosts
that are really servicing that IP address.

> As an operating system problem, it is solved in using techniques such
> as multi-processing, threads and load aware resource schedulers.
[..]
> IOW, I think application processing capacity scaling is a host or
> end-system function, not a networking function.

nginx on a BSD box is a lot of fun, but still, that is not enough.

Think Google, Akamai and Facebook etc scales.

They don't think at the level of 1000s they think at the level of
Googles of users and servers (which is actually why it is likely quite a
bit of fun to be working on solving those problems there; just a shame
that they are ad companies; good that they have lots of bright minds
that do not mind that)

The big problem is that they are solving their problems

Greets,
 Jeroen

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 18 Nov 2014 16:33:34 +1300

Jeroen,

On 18/11/2014 15:40, Jeroen Massar wrote:
...
> The Flow Label is a fun idea, but the standard five tuple covers
> them just fine, 

Unfortunately not, for several reasons:

1. Only works for known transport protocols. (Pragmatically,
that is probably OK, but mentioned for completeness.)

2. Fails, or kicks you to slow path, for packets with extension
headers.

3. Fails with fragments after the first one (unless the balancer
reassembles fragments itself).

4. Fails with IPsec-encrypted payload.

We did RFC 7098 because it allows IPv6 to mitigate those
problems, which is impossible with IPv4 since there is no flow
label. Stateless server LB may have other problems too, of course.

    Brian

---

From: Mark ZZZ Smith <[email-masked]>
Date: Tue, 18 Nov 2014 06:42:25 +0000 (UTC)

----- Original Message -----
> Sent: Tuesday, 18 November 2014, 13:40

> for it.

You should read the following then, with the later three being real uses:

6436 Rationale for Update to the IPv6 Flow Label Specification. S.
Amante, B. Carpenter, S. Jiang. November 2011. (Format: TXT=28062
bytes) (Status: INFORMATIONAL)

6437 IPv6 Flow Label Specification. S. Amante, B. Carpenter, S. Jiang,
J. Rajahalme. November 2011. (Format: TXT=35269 bytes) (Obsoletes
RFC3697) (Updates RFC2205, RFC2460) (Status: PROPOSED STANDARD)

6438 Using the IPv6 Flow Label for Equal Cost Multipath Routing and
Link Aggregation in Tunnels. B. Carpenter, S. Amante. November 2011.
(Format: TXT=20999 bytes) (Status: PROPOSED STANDARD)

7098 Using the IPv6 Flow Label for Load Balancing in Server Farms. B.
Carpenter, S. Jiang, W. Tarreau. January 2014. (Format: TXT=30884
bytes) (Status: INFORMATIONAL)

---

From: Mark ZZZ Smith <[email-masked]>
Date: Tue, 18 Nov 2014 06:47:14 +0000 (UTC)

----- Original Message -----
> To: Jeroen Massar <[email-masked]>
> Sent: Tuesday, 18 November 2014, 17:42

<snip>
> bytes) (Status: INFORMATIONAL)

This one didn't come through for some reason:
Enhancing Virtual Network Encapsulation with IPv6
http://tools.ietf.org/html/draft-smith-enhance-vne-with-ipv6-05

<snip>

---

From: Jeroen Massar <[email-masked]>
Date: Tue, 18 Nov 2014 09:00:52 +0100

On 2014-11-18 07:42, Mark ZZZ Smith wrote:
[..]
> You should read the following then, with the later three being real uses:

They all describe 1 use case: Load Balancing

Writing it down in a lot of documents does not make it work...

> bytes) (Status: INFORMATIONAL)

"The flow label is hardly used in practice in widespread IPv6
   implementations, although some operating systems do set it
   [McGann05]."

Only usage case: Load Balancing

Which does not work unless poking at lower headers too for ICMPv6.
At which one can also look simply at TCP/UDP.

> RFC3697) (Updates RFC2205, RFC2460) (Status: PROPOSED STANDARD)

"The flow label could be used in both stateless and stateful
   scenarios."

Does not work, as ICMPv6 is a Node Requirement.

"A specific goal is to enable and encourage the use of the
   flow label for various forms of stateless load distribution,
   especially across Equal Cost Multi-Path (ECMP) and/or Link
   Aggregation Group (LAG) paths.  ECMP and LAG are methods"

Again, usage case: Load Balancing.

> (Format: TXT=20999 bytes) (Status: PROPOSED STANDARD)

"Two such techniques are known as equal cost multipath (ECMP) routing
and link aggregation (LAG)"

aka Load Balancing for Tunnels.

Same thing that does not work because of this little ICMPv6 thing.

> bytes) (Status: INFORMATIONAL)

And guess what that is about: Load Balancing.

Please provide a use case where Flow Labels are useful and functional.
As I cannot find them.

Greets,
 Jeroen

---

From: Jeroen Massar <[email-masked]>
Date: Tue, 18 Nov 2014 09:04:10 +0100

On 2014-11-18 07:47, Mark ZZZ Smith wrote:
[..]
[..]
> http://tools.ietf.org/html/draft-smith-enhance-vne-with-ipv6-05

"Carrying the Virtual Network Context ID in the Flow Label Field"

Ah redefining the Flow Label as another number! :)

Indeed, that is what we need to do: drop the Flow Label field and use it
for something proper that actually works.

As for the draft itself, sorry, use fields below the IP stack. There is
no need for every node in the world to know about those kind of tunnels.

Greets,
 Jeroen

---

From: Jeroen Massar <[email-masked]>
Date: Tue, 18 Nov 2014 09:13:39 +0100

> that is probably OK, but mentioned for completeness.)

What extra information does the Flow Label (a random number that can
change anyway) to entities looking at things?

> 2. Fails, or kicks you to slow path, for packets with extension

That completely depends on the hardware and design.

For sixxsd the extension header parsing thing is only 250 lines of C
including a lot of comments and white space and most of that is error
handling. I am fairly certain that this 'slow path' fits easily in any
FPGA and definitely in the cache of any CPU since 1980...

> reassembles fragments itself).

Right. But as Fragments should not exist in the first place, is that a
big problem?

What if send millions of packets, all fragmented and with different Flow
Labels, how is Flow Label helping this situation?

Why do Fragments exist anyway? :)

We really need to resurrect Ron Bonica's No Fragment draft to resolve
this. Flow Labels do not help the situation.

> 4. Fails with IPsec-encrypted payload.

How does it fail? The ID is simply: src + dst + proto=encrypted

How is that different from: src + random-number?

As the Flow Label is not 'reflected', dst does not come into play anyway.

> label. Stateless server LB may have other problems too, of course.

While it is great that the Flow Label is defined, it unfortunately is a
faulty definition that cannot be used.

Lets learn from it, get rid of it and move on.

Greets,
 Jeroen

---

From: Mark ZZZ Smith <[email-masked]>
Date: Tue, 18 Nov 2014 11:15:34 +0000 (UTC)

----- Original Message -----
> Sent: Tuesday, 18 November 2014, 19:00

That is not correct. You're confusing stateless load balancing across links within the path through the network with (attempted) stateless load balancing across multiple hosts with the same active, non-anycast, unicast destination address.

In the former case, there is either only *one host* with the destination IP address, or if the hosts are sharing a unicast address via anycast, the routing system, including the last hop router, will select a constant *one* of them to send the all traffic to. Within the routing system the selected anycast destination host will be the closest, and if the are multiple hosts on last hop sharing the anycast address, the first one that responds to the multicast ND NS will be the one chosen, as the Override flag is switched off in all of the ND NAs from the anycast address assigned hosts. From that point on, all traffic towards that destination address will be delivered to the same and single host.

There is no ambiguity in this case - a unicast destination address in a packet results in that packet being delivered to a single host in both the anycast and non-anycast scenarios, with no extra context information needed. ICMP PTB will work correctly in this scenario.

In your LB host scenario, some other context is needed to uniquely identify which of the hosts, all actively sharing the same address, is the right one to deliver the specific PTB message to. Stateless LB doesn't record that extra context (state) to uniquely distinguish the hosts, and that is why PTB breaks.

---

From: Mark ZZZ Smith <[email-masked]>
Date: Tue, 18 Nov 2014 11:38:26 +0000 (UTC)

----- Original Message -----
> Sent: Tuesday, 18 November 2014, 19:04

The IPv6 underlay network is ignorant of how the flow label values are chosen at the source NVE and what the received value be used for at the destination. The packet source and destination can chose to place more semantic meaning on those values if useful, and it doesn't effect in anyway the use of the flow label for network load balancing across the IPv6 underlay network. Standard RFC6438 flow label processing applies within the IPv6 underlay network.

The only requirement is that the flow label "values should be chosen such that their bits exhibit a high degree of variability, making them suitable for use as part of the input to a hash function used in a load distribution scheme." (RFC6437). You'll notice I've advised that Virtual Network Context IDs should be uniformly distributed to suit this flow label field value requirement, if the flow label field is carrying or carrying a copy of the Virtual Network Context ID.

> for something proper that actually works.

So you're proposal is quite young, has had very little review, and yet you're confident it 'actually works'?

I think you need to *prove* what is currently specified doesn't work if you're going to make statements like that.

> no need for every node in the world to know about those kind of tunnels.

Huh? Only NVEs need to know anything about what I've proposed in that draft, and NVEs are not 'every node in the world'.

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 18 Nov 2014 16:58:57 +0000

> Right. But as Fragments should not exist in the first place, is that a

That's not right; tunnels need fragmentation to support an assured 1500 and also
to support tunnels within tunnels. (I am speaking here of IPv6 fragmentation, and
not IPv4 fragmentation which is unsafe at high speeds when there are no NATs
and unsafe at any speed above a slow crawl when there is a NAT.)

Thanks - Fred

---

From: Jeroen Massar <[email-masked]>
Date: Tue, 18 Nov 2014 18:03:21 +0100

> and unsafe at any speed above a slow crawl when there is a NAT.)

Please handle your fragmentation in your tunneling protocol and don't
settle all the IPv6 nodes with this problem

Greets,
 Jeroen

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 18 Nov 2014 17:16:25 +0000

Hi Jeroen,

> Sent: Tuesday, November 18, 2014 9:03 AM
> Cc: 6man; IPv6 Operations

That's not right either; large UDP packets need fragmentation too. Fragmentation
is a core component of the IPv6 protocol, and getting rid of it would be just another
example of how we are dumbing down the protocol and painting ourselves into a
corner with no hope for future extensibility.

Thanks - Fred

---

From: Jeroen Massar <[email-masked]>
Date: Tue, 18 Nov 2014 18:22:06 +0100

On 2014-11-18 18:16, Templin, Fred L wrote:
> That's not right either; large UDP packets need fragmentation too.

UDP does not need for IP to handle the fragmentation.

The application needs to be informed of the path MTU and the application
should then send packets in sizes of the given MTU.

> Fragmentation is a core component of the IPv6 protocol, and getting rid of it would be just another

Dumb is good in the IP layer.

Fragmentation can be handled by the protocol/application. It does not
need to be on the IP layer.

And I fully agree we should not force things into IPv6 that we know are
broken already in IPv4 and that causes issues there for a long time.

Or do you like those Fragmentation Attacks on your infrastructure?

Not having this handled in IP makes IP easy to implement with little
problems. That also means that it can scale much better.

Greets,
 Jeroen

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 18 Nov 2014 17:44:46 +0000

Hi Jeroen,

> Sent: Tuesday, November 18, 2014 9:22 AM

The application can only be informed of the path MTU if it sends more than just
one or a couple of packets. Single, isolated UDP packets know only one safe IPv6
size, and that is 1280.

> broken already in IPv4 and that causes issues there for a long time.

That is not what I said; IPv6 has a 32bit Identification field, which avoids
the RFC4963 issues. 

> problems. That also means that it can scale much better.

The IP layer provides facilities for 1) routing, 2) addressing and a 3) fragmentation
and reassembly capability for accommodating large packets for which the size
cannot be reduced. That is core in nature to the design of the protocol, and if
there are problems we are just going to have to fix them.

Thanks - Fred

---

From: Jeroen Massar <[email-masked]>
Date: Tue, 18 Nov 2014 18:51:00 +0100

[Subject updated]

On 2014-11-18 18:44, Templin, Fred L wrote:
[..]
> one or a couple of packets. Single, isolated UDP packets know only one safe IPv6

Indeed if you just sent packets one way you will basically be limited to
1280.

Is that a problem?

> The IP layer provides facilities for 1) routing, 2) addressing and a

Actually only addressing. Routing is handled by the implementations.

This is also why people tend to say that "IPv6 is just IPv4 with more
addresses".

> 3) fragmentation and reassembly capability for accommodating large
packets for which the size
> cannot be reduced.

While Fragmentation is currently in the spec, there really is no reason
why it should stay.

> That is core in nature to the design of the protocol, and if

Please provide the magic for "fixing a fragmentation attack".

Greets,
 Jeroen

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 18 Nov 2014 17:57:39 +0000

Hi Jeroen,

> Sent: Tuesday, November 18, 2014 9:51 AM
> Subject: Deprecating Fragmentation (Was: IPv6 MTU Flow-label....)
> 
> [Subject updated]
> 

I have heard that DNS and certain security protocols sometimes need to send
large UDP packets with no prior knowledge of path MTU, and that fragmentation is
used. I understand that at least one of those security protocols has been updated
to include a UDP-level fragmentation, but I don't know about others.

> While Fragmentation is currently in the spec, there really is no reason

RFC2473 depends on IPv6 fragmentation, for one.

> Please provide the magic for "fixing a fragmentation attack".

That's not my job - but consider that IPv6 fragmentation within an enterprise
may be different than fragmentation that crosses an enterprise boundary as
just one example where this concern might not apply.

Thanks - Fred

---

From: Jeroen Massar <[email-masked]>
Date: Tue, 18 Nov 2014 19:08:00 +0100

On 2014-11-18 18:57, Templin, Fred L wrote:
> I have heard that DNS

Per default DNS messages (over UDP) are limited to 512 bytes unless they
use EDNS0.

See also:
https://tools.ietf.org/html/draft-andrews-dnsext-udp-fragmentation-01
and:
https://www.nlnetlabs.nl/blog/2013/06/04/pmtud4dns/

and then also please read:
http://nlnetlabs.nl/downloads/publications/pmtu-black-holes-msc-thesis.pdf
and:
https://ripe65.ripe.net/presentations/167-20120926_-_RIPE65_-_Amsterdam_-_DNSSEC_reco_draft.pdf

For more details on why this is already not working.

> and certain security protocols sometimes need to send
> large UDP packets with no prior

Which "certain security protocols" are these?

> knowledge of path MTU, and that fragmentation is

If that "one of those" can be upgraded, others can be too as they should.

Fragmentations are not going to fly anymore.

[..]
> RFC2473 depends on IPv6 fragmentation, for one.

Just like proto-41 based tunnels GRE tunnels can easily be configured to
a limited MTU. Hence packets going through it do not need to be
fragmented as they are not large enough anyway.

> just one example where this concern might not apply.

They are very likely already exactly the locations that are employing
devices which block fragmentation completely.

Greets,
 Jeroen

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 18 Nov 2014 18:17:39 +0000

Hi Jeroen,

> Sent: Tuesday, November 18, 2014 10:08 AM
> Subject: Re: Deprecating Fragmentation (Was: IPv6 MTU Flow-label....)
> 

It is funny you should say "fly", because the domain I work in does not always
have "GigE-everywhere", and some resource-constrained links are lucky to do
1280 let alone 1500 or larger.

> fragmented as they are not large enough anyway.

I'm not sure to make of what you just said, but I am basing my statement
on Section 7 of RFC2473.

> devices which block fragmentation completely.

What makes you say this? What gives you insight into the way in which IPv6
is being used in all enterprises? And, all "enterprises" are not created equal;
some might be as large as a major corporation or as small as a plane flying
around in the sky.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Tue, 18 Nov 2014 10:51:17 -0800

> While Fragmentation is currently in the spec, there really is no reason

Reason #1: UDP (which is more than just DNS)

Reason #2: tunnels

Finally, this is V6OPS, and changing specs is out of scope, so if this
issue is intended to be considered, it needs to be taken to the
appropriate venue.

Joe

---

From: Ole Troan <[email-masked]>
Date: Tue, 18 Nov 2014 11:50:39 -0800

Fred,

> and unsafe at any speed above a slow crawl when there is a NAT.)

1500-1280 is for tunnel encapsulation overhead.

any reassembly in the network is tricky.

cheers,
Ole

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 18 Nov 2014 20:18:25 +0000

Hi Ole,

> From: Ole Troan [mailto:[email-masked]]
> Sent: Tuesday, November 18, 2014 11:51 AM
> To: Templin, Fred L
> Cc: Jeroen Massar; Brian E Carpenter; IPv6 Operations; 6man WG

That doesn't work for sufficiently nested tunnels-within-tunnels and requires careful
coordination of MTUs at each tunnel ingress within the nesting. It also exposes a
degenerate MTU to the original source, i.e. an MTU less than 1500.

I would like to introduce a new term:

  "MTU Clamping - the act of setting a tunnel MTU smaller than 1500 bytes"

We want to avoid MTU clamping so original sources will not experience an
MTU underrun, i.e., a path along which 1500's don't get through and for
which PTB messages may be lost.

> any reassembly in the network is tricky.

That depends greatly on where in the network you are doing the reassembly.
In core routers, of course it would be a burden. But, in routers near the edges
maybe not as much - and especially since we are only expecting 2-fragment
fragmented packets and not N-fragment.

Please do not misunderstand; sustained fragmentation and reassembly is not
a desired steady-state condition. Rather, it is a condition to be detected and
tuned out as soon as possible. Fostering deployment of links with MTUs
sufficiently larger than 1500 bytes (and allowing tunnels to accommodate
the larger sizes) is therefore the desired way forward.

Thanks - Fred

---

From: Tore Anderson <[email-masked]>
Date: Wed, 19 Nov 2014 07:06:26 +0100

* Joe Touch

> > While Fragmentation is currently in the spec, there really is no
> > reason why it should stay.
> 

#3: OSPF

Tore

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 19 Nov 2014 11:55:22 +0100

>> Reason #1: UDP (which is more than just DNS)

Seems indeed that that protocol abuses the fragmentation on the IP layer.

>From the below message it also seems there is effort to make UDP handle
fragmentation. That work is being done in the Transport Area (yeah for
multiple groups working on the same thing so that unless you are
everywhere you miss out on some of it...)

When those UDP extensions are in place, IPv6 Frags can go.

I think though it would be prudent to start advising that new protocols
do not rely on IPv6 Fragmentations working, as Ron describes below, as
various network are dropping them.

Greets,
 Jeroen

--
From: Ronald Bonica <[email-masked]>
To: Erik Nordmark <[email-masked]>, IETF IPv6 <[email-masked]>,
        Ron Bonica
 <[email-masked]>
Subject: RE: Responding to Ron's comment about removing fragmentation
Date: Tue, 18 Nov 2014 21:32:22 +0000

Erik,

Yikes! When I made that comment at the mike, I didn't realize that it
would reinvigorate a larger debate concerning fragmentation. Had I known
that, I would have kept my seat ;-)

Having stirred the pot, I offer the following observations:

IPv6 fragmentation isn't working well. Studies show that fragments are
discarded in a significant number of IPv6 paths. At every IETF, we hear
about a new security issue associated with IPv6 fragmentation. So, our
options are a) fix the security issues, b) ignore the security issues or
c) deprecate IPv6 fragmentation.

All three options are painful. Pick your pain!

If the IETF were to deprecate IPv6 fragmentation, many applications
would not be impacted at all. For example, an application that runs over
TCP wouldn't be impacted, because TCP adjusts its MSS accommodate the
PMTU. However, many applications might be adversely impacted. An
examples are:

- Applications that run directly over IP (e.g., OSPF, RSVP, SIIT)
- Applications that run over UDP (e.g., DNSSEC)
- Tunnels

Of the application that run directly over IP, some rely on IPv6
fragmentation (e.g., SIIT), while others do not (OSPF, RSVP). The ones
that do not rely on fragmentation run over a single link and never emit
packets larger than the link MTU.

Some applications that run over UDP rely on IPv6 fragmentation (e.g.
DNSSEC). To the extent that many IPv6 paths discard fragments, these
applications are broken today. As Mike Heard suggests, this problem
could be fixed by extending UDP to support fragmentation at the
transport layer. I think that we should do this, regardless of whether
the IETF deprecates IPv6 fragmentation. We need these extensions to fix
current  brokenness.

I don't think that tunnels are a problem. Tunnels shouldn't present a
problems so long as:

- tunnel interior nodes can deliver a PTB to the tunnel ingress
- the tunnel ingress executes PMTUD procedures
- the tunnel ingress router can deliver a PTB to the packet source.

In conclusion, I don't think that we should even discuss deprecating
IPv6 fragmentation today. We can't deprecate IPv6 fragmentation  until
Mike Heard's UDP extensions have been specified and widely deployed.
However, the Transport Area should start work on Mike's UDP extensions ASAP.

    Ron

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 19 Nov 2014 12:09:49 +0100

> Reason #1: UDP (which is more than just DNS)

UDP is only an issue for protocols that just sent large chunks of data.
That is never an optimal situation and likely they should not be using
UDP for that then anyway.

UDP can be lossy (there is no retransmit) thus if you send a 30k
"packet" on a link with a MTU of 1500 you will lose all 30k of packets
when just one goes missing. The receiver has to then decide when the get
of those other 29 packets because the 30th is not coming, the
application that was supposed to receive them will never know about it
either.

Hence IMHO there really is no reason for the ability to send packets > PMTU.

> Reason #2: tunnels

Tunnels is not a reason at all. They can use a smaller MTU and otherwise
handle fragmentation properly inside the protocol.

This is what is done for proto-41 tunnels (MTU = 1280 unless the path
allows bigger) and for example OpenVPN handles fragmentation to be able
to send 1500 byte packets anyway.

And in case you have:
 A - R1 - R2 - R3 - B

where R1 tunnels the packets, wrapping IPSEC
and R3 unwraps the IPSEC.

You just have to make sure that the R1 - R2 - R2 at least has a MTU of
1280 + IPSEC overhead, or use a different tunneling protocol.

> issue is intended to be considered, it needs to be taken to the

Hence the cc's to ipv6@ which would be more appropriate.

But the comments that Ron Bonica made there about getting UDP to support
fragging, thus allowing protocols that require that (even if it silly in
the light of the above 30k with one packet loss example) to keep on
making that mistake. But we'll have that discussion there.

Greets,
 Jeroen

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 19 Nov 2014 12:14:51 +0100

On 2014-11-18 19:17, Templin, Fred L wrote:
[..]
> 1280 let alone 1500 or larger.

As per the IPv6 spec: Mediums that cannot handle packets of an MTU of
1280 need to handle this problem.

Those pigs will fly perfectly fine that way.

Check 6lowpan as an example as used in a lamp near you.

Greets,
 Jeroen

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 19 Nov 2014 16:47:40 +0000

> Sent: Wednesday, November 19, 2014 3:15 AM
> To: Templin, Fred L

Wrong context and wrong answer. This concerns tunneling over links that
support a 1280 MTU such that the tunnel ingress sees 1240. The one and
only solution in that case is IP fragmentation.

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 19 Nov 2014 17:32:36 +0000

Related to this discussion, here is a draft on tunneling over ip-proto-44:

https://datatracker.ietf.org/doc/draft-templin-aeromin/

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Wed, 19 Nov 2014 10:39:41 -0800

> When those UDP extensions are in place, IPv6 Frags can go.

That would require:

	- widescale, overwhelming support for a variant of UDP that
	hasn't been developed yet

	- widescale, overwhelming support for PLMTUD in TCP and SCTP

We're a long way from either one.

> various network are dropping them.

It would be prudent to start advising operators to fix broken deployments.

Not all operational problems are the fault of the user.

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 19 Nov 2014 10:44:22 -0800

> UDP is only an issue for protocols that just sent large chunks of data.

Where large == MTU+1, yes.

> UDP for that then anyway.

Users have their view of optimality, and operators have theirs.

Users expect the network to support existing requirements.

Operators should not expect otherwise.

> when just one goes missing.

Yes, but UDP doesn't have to be lossy. It can be low-rate, somewhat
larger than one MTU, and still be very efficient by relying on network
layer fragmentation.

...
> Hence IMHO there really is no reason for the ability to send packets > PMTU.

Reason #1 that trumps all others: it's currently a requirement

The reality is that they already do this. There are more protocols than
the ones you see commonly in the WAN.

> handle fragmentation properly inside the protocol.

Again, not all tunnels work the way you - or I - want them to. Until
they do, this is a reality.

...
> making that mistake. But we'll have that discussion there.

You also need to find a way to get PLMTUD out there. It isn't right now,
not at the levels that would be needed to start a conversation about
deprecating fragmentation.

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 19 Nov 2014 10:47:10 -0800

> only solution in that case is IP fragmentation.

Tunnels should be "seeing" the ingress-to-egress MTU, which is 1500 for
IPv6, not the link MTU, which is 1280.

That's the crux of the error.

Joe

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 19 Nov 2014 20:35:12 +0100

> https://datatracker.ietf.org/doc/draft-templin-aeromin/

As you are well aware about the Subject: title, you might want to
consider not doing things those ways.

Also, something about actually running code would be great.

Greets,
 Jeroen

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 19 Nov 2014 20:38:45 +0100

> only solution in that case is IP fragmentation.

That is _a_ solution. But a bad one.

The proper solution is to handle the fragmentation in the protocol that
runs below it.

This rarely used protocol called TCP does this as a great example.

Greets,
 Jeroen

---

From: Joe Touch <[email-masked]>
Date: Wed, 19 Nov 2014 11:55:57 -0800

> This rarely used protocol called TCP does this as a great example.

There are plenty of places where TCP isn't the right solution and even
TCP's two solutions (PMTUD via ICMP, PLMTUD probing) have problems (ICMP
blocking, time delays for probing).

Sometimes the best thing for the user and the network is to just send a
message and let the system frag/reassemble it. That's how ATM works,
that's how recursive tunnels (X over X) have to work, and so I don't see
a good reason to go out of our way to endorse disabling of this key
feature of the Internet.

Joe

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 19 Nov 2014 21:10:49 +0100

On 2014-11-19 20:55, Joe Touch wrote:
> There are plenty of places where TCP isn't the right solution

I don't disagree. I only gave TCP as an example that a lower-layer
protocol CAN handle the actual fragmenting of packets.

> and even

Your forgetting MSS, which is being clamped by large providers (see
horrible threads of last week) because they don't do ICMPv6 in their
Load Balancers.

But please note that any problem you describe above (ICMPv6 PTB being
ignored/dropped being the primary one) will also affect any system that
is fragmenting packets as it will not know the right MTU to frag at.

For UDP and anything else that thus just means your packets disappear.

> message and let the system frag/reassemble it.

When that system does not know what to frag it at though, you have problems.

> That's how ATM works,

Fortunately IP has very little to do with ATM from a design perspective
and we do not have to take properties of it to IP either, otherwise we
should just be using ATM...

> that's how recursive tunnels (X over X) have to work

These can just work by lowering the inner MTU.

And if you can't anymore, then use OpenVPNs refragmetation.

> and so I don't see

Why should anybody go out of their way to support it.

Please also see Ron Bonica's excellent reasons why fragging is not a
good idea to do in the IP layer, see Subject:

"E: Responding to Ron's comment about removing fragmentation"

on the [email-masked] list.

Greets,
 Jeroen

---

From: Joe Touch <[email-masked]>
Date: Wed, 19 Nov 2014 12:15:07 -0800

On 11/19/2014 12:10 PM, Jeroen Massar wrote:
...
> on the [email-masked] list.

Suggesting that users avoid it is not the same as disabling it as a
capability.

The Internet works because it historically traded capability for
efficiency. Fragmentation isn't efficient, but I don't think there's a
good case to get rid of it altogether.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 19 Nov 2014 20:26:33 +0000

Hi Jeroen,

> Sent: Wednesday, November 19, 2014 11:35 AM
> To: Templin, Fred L

I am running AERO over ip-proto-41 in my enterprise  network today, and
I haven't found paths yet where it is blocked. The new draft proposes
running AERO over ip-proto-44, and I haven't done that yet but also expect
to see it get through on most if not all paths that ip-proto-41 gets through.

> Also, something about actually running code would be great.

I have running code for AERO on both UDP port 8060 and ip-proto-41
running inside a major corporate enterprise network.

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 19 Nov 2014 20:34:34 +0000

Hi Jeroen,

> That is _a_ solution. But a bad one.

No, it is the only solution. Consider the IPv6 path:

  A  -> (1500) -> (1500) -> (1280) -> (1500) -> (1500) -> B

Now, A wants to set up an ip-in-ipv6 tunnel to B. It sets the MTU to
1500 minus 40 bytes for the IPv6 header (i.e., 1460). Its packets get
through the first two links (both 1500) but then get dropped at the
third link (1280) and a PTB message returned. A now has to reduce
the size of packets it sends to 1240 - but it can't, because IPv6 needs
to see a 1280 MTU. The only solution is IPv6 fragmentation of the
tunneled packet

> The proper solution is to handle the fragmentation in the protocol that

I know about link adaptation, but it applies only to a single link; not to
a path comprised of multiple joined together by IPv6 routers.

> This rarely used protocol called TCP does this as a great example.

This is a completely out of context comment wrt the discussion above.

Thanks - Fred

---

From: Jeroen Massar <[email-masked]>
Date: Thu, 20 Nov 2014 10:50:29 +0100

On 2014-11-19 21:34, Templin, Fred L wrote:
> 1500 minus 40 bytes for the IPv6 header (i.e., 1460).

Then you have misconfigured the tunnel.

Do a tracepath6 before you do it and you will reveal that the path is
only 1280.

Hence, you will need to chose a tunneling protocol that can handle
chunking up your packet smaller to be able to send it over the link.

You should btw be receiving ICMPv6 PTBs for this.

> Its packets get

OpenVPN as a great example works perfectly fine over that link as it
handles the fragmentation.

Greets,
 Jeroen

---

From: Jeroen Massar <[email-masked]>
Date: Thu, 20 Nov 2014 10:55:23 +0100

> Suggesting that users avoid it is not the same as disabling it as a

Users do not chose this, developers & protocol designers do.

And IMHO there MUST ( ;) ) be at least a documentation stating:
"New protocols SHOULD avoid IP fragmentation"

Especially as there are many networks that block it and thus it gives
unpredictable results. But see Ron's mail for a longer list.

> good case to get rid of it altogether.

And historically we have also seen many attacks using those 'capabilities'.

As the Internet needs to be a stable thing though that works, we might
want to shift some features elsewhere.

Greets,
 Jeroen

---

From: Jeroen Massar <[email-masked]>
Date: Thu, 20 Nov 2014 10:57:24 +0100

On 2014-11-19 19:39, Joe Touch wrote:
[..]
> It would be prudent to start advising operators to fix broken deployments.

The operators who are filtering ICMPv6 and fragments are doing so
because they chose to do that. It is typically a little switch in their
'firewall' options and they are like "do not need".

Those are the people who do not understand the implications but do make
the choices in large networks to make those changes and break things.

> Not all operational problems are the fault of the user.

(end)-users typically have little to do with it.

Greets,
 Jeroen

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 20 Nov 2014 16:39:14 +0000

Hi Jeroen,

> Sent: Thursday, November 20, 2014 1:50 AM
> To: Templin, Fred L

Wrong.

> Do a tracepath6 before you do it and you will reveal that the path is

Maybe for static tunnels, but not efficient for automatic tunnels.

> chunking up your packet smaller to be able to send it over the link.

If my only choice is RFC2473, then I am stuck with IPv6 fragmentation.
Please don't try to navigate away from this indisputable point again;
you are wasting bandwidth and people's time.

Thanks - Fred

---

From: Jeroen Massar <[email-masked]>
Date: Thu, 20 Nov 2014 17:44:28 +0100

>> Then you have misconfigured the tunnel.

Nothing wrong about that statement.

You chose to use a tunneling technology that did not work in the
environment you intended it for.

Hence, pick a better one that matches your requirements.

> Maybe for static tunnels, but not efficient for automatic tunnels.

OpenVPN, as a good example, works fine in those cases...

> If my only choice is RFC2473, then I am stuck with IPv6 fragmentation.

Then get better gear so you have more choice. Don't restrict yourself.

Greets,
 Jeroen

---

From: Joe Touch <[email-masked]>
Date: Thu, 20 Nov 2014 08:48:32 -0800

> "New protocols SHOULD avoid IP fragmentation"

That's impossible because it's not under the control of the new protocol
designer. A designer can ask for the MTU and even probe, but nothing
says that either value is static. And the reported MTU doesn't include
IP options, which can vary on a per-packet basis.

I agree that deliberately relying on IP fragmentation over long, lossy
paths is a bad idea, but it's only bad for that application. Let's let
that be the decision of the protocol designer.

> unpredictable results. 

And there are networks that block IP packets altogether (i.e., non-IP
networks). Does that mean we should recommend that protocol designers
SHOULD avoid IP?

At some point, there are requirements and networks that don't meet them
aren't IP.

> want to shift some features elsewhere.

That's a nice goal, but we're not there yet. Let's not shutoff a
capability until we have a replacement ready.

Joe

---

From: Joe Touch <[email-masked]>
Date: Thu, 20 Nov 2014 08:50:16 -0800

> 'firewall' options and they are like "do not need".

They are like "I don't support IP".

> the choices in large networks to make those changes and break things.

A fine opportunity to educate operators - maybe that ought to be part of
the charter of V6OPS.

Joe

---

From: Joe Touch <[email-masked]>
Date: Thu, 20 Nov 2014 08:53:47 -0800

On 11/20/2014 1:50 AM, Jeroen Massar wrote:
...
> Do a tracepath6 before you do it and you will reveal that the path is

Tracepath6 tells you the current path. It won't tell you what the path
is when routing updates or links go up or down.

> You should btw be receiving ICMPv6 PTBs for this.

PLMTUD exists because ICMPs don't reach origin hosts. The same problem
exists within tunnels and between tunnels and the traffic source.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 20 Nov 2014 17:21:11 +0000

> Sent: Thursday, November 20, 2014 8:44 AM
> To: Templin, Fred L

Arrggh, still wasting bandwidth.

> OpenVPN, as a good example, works fine in those cases...

There are cases for tunnels that come up, send one or a few packets, and
then go away again. I don't want to send 10's or 100's of control packets just
to get one or a few data packets across.

> Then get better gear so you have more choice. Don't restrict yourself.

If by better gear you mean some that stuffs even more wasted bytes into
the headers of packets that need to traverse a low-end data link, no thanks!

Thanks - Fred

---

From: Jeroen Massar <[email-masked]>
Date: Fri, 21 Nov 2014 14:03:39 +0100

On 2014-11-20 18:21, Templin, Fred L wrote:
[..]
> Arrggh, still wasting bandwidth.

I'll do that by simply pointing you to:
 https://tools.ietf.org/html/rfc5405 section 3.2

There is a lot more in that document, but this is a nice exerpt:
8<----------
   Due to these issues, an application SHOULD NOT send UDP datagrams
   that result in IP packets that exceed the MTU of the path to the
   destination.  Consequently, an application SHOULD either use the path
   MTU information provided by the IP layer or implement path MTU
   discovery itself [RFC1191][RFC1981][RFC4821] to determine whether the
   path to a destination will support its desired message size without
   fragmentation.
----------->8

And also:
 https://tools.ietf.org/html/rfc4963

which is referenced from it.

Note that rfc5405 as also known as BCP145.

Greets,
 Jeroen

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 21 Nov 2014 14:21:13 +0000

Jeroen,

IPv6 needs to see a minimum MTU of 1280 no matter what it is tunneled over,
i.e., IPv6-over-foo-over-IP. "foo" can be any crazy combination of encapsulations
that you'd like (or none at all), but IPv6 must still see 1280 per RFC2460.

Thanks - Fred

---

From: Jeroen Massar <[email-masked]>
Date: Fri, 21 Nov 2014 17:22:33 +0100

> that you'd like (or none at all), but IPv6 must still see 1280 per RFC2460.

I am very well aware of this, having probably built and running the most
IPv6 tunnels in the world (bold claim, but the publicly available stats
quite indicate the truth; note also 'probably').

As per those RFCs it is the responsibility of the underlying medium (be
that Ethernet, ATM, Firewire, a tunnel or a tunnel over a tunnel over a
tunnel) to make sure that those 1280 bytes can be transmitted.

You have been stating that you require IPv6 to fragment your packets
because of that. There is no need for that. Just chunk your packets in
chunks of 1280, the medium below that will take care of it.

If you think differently, please rewrite your statements to clarify what
you are actually trying to comment on. (Removing all context for that
matter does not help a bit...)

Greets,
 Jeroen

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 21 Nov 2014 16:37:36 +0000

Jeroen,

> chunks of 1280, the medium below that will take care of it.

For IPv6-in-IPv6, the outer IPv6 layer *is* the medium below. It may be
conveyed over many underlying links between the ingress and egress
connected by IPv6 routers. If any of those links configures a too-small MTU,
the inner IPv6 layer would see an MTU smaller than 1280. Therefore, the
outer IPv6 layer MUST fragment in order for the inner layer to see at
least 1280.

This is clearly specified in RFC2473, Section 7. Your messages are starting
to get tiresome to the point of badgering. In order for your assertions
to be validated, you would need to update either RFC2473 or RFC2460
or both.

Thanks - Fred

---

From: Jeroen Massar <[email-masked]>
Date: Fri, 21 Nov 2014 17:49:06 +0100

On 2014-11-21 17:37, Templin, Fred L wrote:
> For IPv6-in-IPv6, the outer IPv6 layer *is* the medium below.

That is what I state above, note "tunnel over a tunnel".

> It may be
> the inner IPv6 layer would see an MTU smaller than 1280.

Then that medium, the tunnel, should be taking care of fragmenting the
chunks. Note that is what I write above.

This is also why I have been referring to a tunneling mechanism like
OpenVPN, as just an example, which is perfectly able to do so.

Greets,
 Jeroen

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 21 Nov 2014 16:53:50 +0000

Jeroen,

There is no basis for further discussion if you continue to refuse to acknowledge
RFC2473, Section 7 where IPv6 fragmentation is absolutely mandated.

Thanks - Fred

---

From: Jeroen Massar <[email-masked]>
Date: Fri, 21 Nov 2014 17:58:51 +0100

> RFC2473, Section 7 where IPv6 fragmentation is absolutely mandated.

Yes, that old document states that IPv6 will do fragmentation.

That does not mean that it is a good thing to keep on doing so.

I've already referenced you to:

Which shows plenty of examples why not to do it. Hence there is no
"discussion" as it is already documented as a nice BCP that one should
not be depending on fragmentation.

Which is what I've been trying to point out all along...

You can of course chose to ignore that BCP at your own peril.

Greets,
 Jeroen

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 21 Nov 2014 17:13:48 +0000

Jeroen,

> Sent: Friday, November 21, 2014 8:59 AM
> To: Templin, Fred L

Good - thanks for the acknowledgement.

> That does not mean that it is a good thing to keep on doing so.

I have already told you that there are use cases in which we need to minimize
encapsulation overhead over low-end data links. That is at least one  use case
for RFC2473 encapsulation in which IPv6 fragmentation might ensue. 

Everything else you have written below is completely irrelevant wrt RFC2473.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Fri, 21 Nov 2014 09:57:16 -0800

On 11/21/2014 8:22 AM, Jeroen Massar wrote:
...
> chunks of 1280, the medium below that will take care of it.

You have just deprecated IPv6-over-IPv6 tunnels in the absence of any
other layering.

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 21 Nov 2014 09:57:09 -0800

On 11/21/2014 8:58 AM, Jeroen Massar wrote:
> Which shows plenty of examples why not to do it.

That's a SHOULD, not a MUST.

RFC2119 explains the difference ;-)

Joe