Subject: Fun and surprises with IPv6 fragmentation

From: Christian Huitema <[email-masked]>
Date: Fri, 2 Mar 2018 21:02:03 -0800

Yesterday, I was mentioning bugs of the interop. This morning, I woke up
to find an interesting message from Patrick McManus. Something is weird,
he said. The first data message that your server sends, with sequence
number N, always arrives before the final handshake message, with
sequence number N-1. That inversion appears to happen systematically.

It took us the best part of a day to explore blind alleys and finally
understand what was happening. The exchange was over IPv6. Upon
receiving a connection request from Patrick’s implementation, Picoquic
was sending back a handshake packet. Immediately after that, Picoquic
was sending its first data packet, which happens to be an MTU probe. And
it turns out that the probe was 1518 bytes, a bit longer than what the
AWS routers could accept. So some router inserted an IPv6 fragmentation
header and split the packet in two: a large initial fragment, 1496 byte
long, and a small second fragment 78 bytes long. You could think that
this is no big deal, since fragments would just be reassembled at the
destination, but you would be wrong.

Some routers on the path try to be helpful. They have learned from past
experience that short packets often carry important data, and so they
try to route them faster than long data packets. And here is what
happens in our case:

·         * The server prepares and send a Handshake packet, 590 bytes long.

·         * The server then prepares the MTU probe, 1518 bytes long.

·         * The MTU probe is split into fragment 1, 1496 bytes, and
fragment 2, 78 bytes.

·         * The handshake and the long fragment are routed on the normal
path, but the small fragment is routed at a higher priority level.

·         * The Linux driver at the destination receives the small
fragment first. It queues everything behind that until it receives the
long fragment.

·         * The Linux driver passes the reassembled packet to the
application, which cannot do anything with it because the encryption
keys can only be obtained from the handshake packet.

·         * The Linux driver then passes the handshake packet to the
application.

Which confirms an old opinion. When routers try to be smart and helpful,
they end up being dumb and harmful. Please just send the packets in the
order you get them!

I tried to work around the issue by setting the "don't fragment" bit on
the socket, but somehow that doesn't work. So I simply programmed the
server to not use payloads larger than 1440 bytes. Still, I can see that
pattern happening in other circumstances, such as a long Connection
Initial message followed by a short 0-RTT packet. isn't networking fun?

-- Christian Huitema

---

From: Ryan Hamilton <[email-masked]>
Date: Fri, 2 Mar 2018 21:09:57 -0800

I'm sorry if this is a dumb question, but I understood that in IPv6 routers
could not fragment IPv6 packets, only endpoints.

Unlike in IPv4, IPv6 routers never fragment IPv6 packets. Packets exceeding
the size of the maximum transmission unit of the destination link are
dropped and this condition is signaled by a Packet too Big ICMPv6 type 2
message to the originating node, similarly to the IPv4 method when the
Don't Fragment bit is set.[1]

End nodes in IPv6 are expected to perform path MTU discovery to determine
the maximum size of packets to send, and the upper-layer protocol is
expected to limit the payload size. However, if the upper-layer protocol is
unable to do so, the sending host may use the Fragment extension header in
order to perform end-to-end fragmentation of IPv6 packets.

https://en.wikipedia.org/wiki/IPv6_packet#Fragmentation

How sure are you that it's a router and not the sending host that's doing
the fragmentation.

Cheers,

Ryan

On Fri, Mar 2, 2018 at 9:02 PM, Christian Huitema <[email-masked]>
wrote:

> to find an interesting message from Patrick McManus. Something is weird, he
> said. The first data message that your server sends, with sequence number
> N, always arrives before the final handshake message, with sequence number
> N-1. That inversion appears to happen systematically.
>
> understand what was happening. The exchange was over IPv6. Upon receiving a
> connection request from Patrick’s implementation, Picoquic was sending back
> a handshake packet. Immediately after that, Picoquic was sending its first
> data packet, which happens to be an MTU probe. And it turns out that the
> probe was 1518 bytes, a bit longer than what the AWS routers could accept.
> So some router inserted an IPv6 fragmentation header and split the packet
> in two: a large initial fragment, 1496 byte long, and a small second
> fragment 78 bytes long. You could think that this is no big deal, since
> fragments would just be reassembled at the destination, but you would be
> wrong.
>
> experience that short packets often carry important data, and so they try
> to route them faster than long data packets. And here is what happens in
> our case:
>
> fragment first. It queues everything behind that until it receives the long
> fragment.
>
> application, which cannot do anything with it because the encryption keys
> can only be obtained from the handshake packet.
>
> pattern happening in other circumstances, such as a long Connection Initial
> message followed by a short 0-RTT packet. isn't networking fun?
>
> -- Christian Huitema
>
>

---

From: Christian Huitema <[email-masked]>
Date: Fri, 2 Mar 2018 21:23:41 -0800

On 3/2/2018 9:09 PM, Ryan Hamilton wrote:
> I'm sorry if this is a dumb question, but I understood that in IPv6
> routers could not fragment IPv6 packets, only endpoints.
>
>     Unlike in IPv4, IPv6 routers never fragment IPv6 packets. Packets
>     exceeding the size of the maximum transmission unit of the
>     destination link are dropped and this condition is signaled by a
>     Packet too Big ICMPv6 type 2 message to the originating node,
>     similarly to the IPv4 method when the Don't Fragment bit is set.[1]
>
>     End nodes in IPv6 are expected to perform path MTU discovery to
>     determine the maximum size of packets to send, and the upper-layer
>     protocol is expected to limit the payload size. However, if the
>     upper-layer protocol is unable to do so, the sending host may use
>     the Fragment extension header in order to perform end-to-end
>     fragmentation of IPv6 packets.  
>
> How sure are you that it's a router and not the sending host that's
> doing the fragmentation.

Yes, you would think that. And you would be right in theory. But then,
in theory, theory and practice are the same...

-- Christian Huitema

---

From: Christian Huitema <[email-masked]>
Date: Fri, 2 Mar 2018 21:29:03 -0800

On 3/2/2018 9:23 PM, Christian Huitema wrote:
>>     determine the maximum size of packets to send, and the
>>     upper-layer protocol is expected to limit the payload size.
>>     However, if the upper-layer protocol is unable to do so, the
>>     sending host may use the Fragment extension header in order to
>>     perform end-to-end fragmentation of IPv6 packets.  
>>

And to answer the "how do you know" question, here is what the server
says about itself:

ubuntu@ip-172-31-61-32:~$ ifconfig
eth0      Link encap:Ethernet  HWaddr 12:40:99:30:a3:86
          inet addr:172.31.61.32  Bcast:172.31.63.255  Mask:255.255.240.0
          inet6 addr: 2600:1f18:2310:d230:5103:7d9e:7d75:374f/128
Scope:Global
          inet6 addr: fe80::1040:99ff:fe30:a386/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9001  Metric:1
          RX packets:32757 errors:0 dropped:0 overruns:0 frame:0
          TX packets:15697 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:8025531 (8.0 MB)  TX bytes:3622487 (3.6 MB)

Note the "MTU:9001". I don't think that a 1518 bytes packet would be
fragmented by that particular end point. Something else did it.

-- Christian Huitema

---

From: Ryan Hamilton <[email-masked]>
Date: Fri, 2 Mar 2018 21:38:15 -0800

On Fri, Mar 2, 2018 at 9:29 PM, Christian Huitema <[email-masked]>
wrote:

> exceeding the size of the maximum transmission unit of the destination link
> are dropped and this condition is signaled by a Packet too Big ICMPv6 type
> 2 message to the originating node, similarly to the IPv4 method when the
> Yes, you would think that. And you would be right in theory. But then, in
> theory, theory and practice are the same...
>
>
> And to answer the "how do you know" question, here is what the server says
> about itself:
>

Middle boxes! *sigh* ​That does seem compelling, I agree. I don't suppose
you have a packet dump from the sending host?

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Sat, 3 Mar 2018 01:26:55 -0500

Excellent writeup - almost like a Mars Rover gone AWOL.

Could it be that IPV6 is tunnelled over IPV4 and reassembled within the
tunnel rather than at the receiving endpoint?

On 3 March 2018 at 06.38.27, Ryan Hamilton (rch=[email-masked])
wrote:

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Sat, 3 Mar 2018 01:26:55 -0500

---

From: Christian Huitema <[email-masked]>
Date: Fri, 2 Mar 2018 23:02:53 -0800

Mikkel, we have a PCAP capture showing the fragments arriving at the client, out of order of course.

-- Christian Huitema 

> On Mar 2, 2018, at 10:26 PM, Mikkel Fahnøe Jørgensen <[email-masked]> wrote:
> 
> Could it be that IPV6 is tunnelled over IPV4 and reassembled within the tunnel rather than at the receiving endpoint?
> 
> 
> 
>> On 3 March 2018 at 06.38.27, Ryan Hamilton (rch=[email-masked]) wrote:
>> 
>> On Fri, Mar 2, 2018 at 9:29 PM, Christian Huitema <[email-masked]> wrote:
>>>>> I'm sorry if this is a dumb question, but I understood that in IPv6 routers could not fragment IPv6 packets, only endpoints.
>>>>> 
>>>>> Unlike in IPv4, IPv6 routers never fragment IPv6 packets. Packets exceeding the size of the maximum transmission unit of the destination link are dropped and this condition is signaled by a Packet too Big ICMPv6 type 2 message to the originating node, similarly to the IPv4 method when the Don't Fragment bit is set.[1]
>>>>> 
>>>>> End nodes in IPv6 are expected to perform path MTU discovery to determine the maximum size of packets to send, and the upper-layer protocol is expected to limit the payload size. However, if the upper-layer protocol is unable to do so, the sending host may use the Fragment extension header in order to perform end-to-end fragmentation of IPv6 packets.  
>>>>> 
>>>>> How sure are you that it's a router and not the sending host that's doing the fragmentation.
>>>> 
>>>> Yes, you would think that. And you would be right in theory. But then, in theory, theory and practice are the same...
>>> 
>>> And to answer the "how do you know" question, here is what the server says about itself:
>>> 
>>>           inet6 addr: 2600:1f18:2310:d230:5103:7d9e:7d75:374f/128 Scope:Global
>>> Note the "MTU:9001". I don't think that a 1518 bytes packet would be fragmented by that particular end point. Something else did it.
>>> 
>>> -- Christian Huitema
>> 
>> Middle boxes! *sigh* ​That does seem compelling, I agree. I don't suppose you have a packet dump from the sending host?
>>

---

From: Christian Huitema <[email-masked]>
Date: Fri, 2 Mar 2018 23:02:53 -0800

-- Christian Huitema

---

From: Patrick McManus <[email-masked]>
Date: Sat, 3 Mar 2018 10:23:49 -0500

> routers could not fragment IPv6 packets, only endpoints.
I know! fun. Honestly the Internet is just so interesting - you wonder how
it works at all.

The pcap is in the #picoquic slack channel. (as always anyone reading this
can just ask anyone on the slack, like me, for an invite). search for
octopus.pcap

> How sure are you that it's a router and not the sending host that's doing
It seems unlikely to be in the core.. the recv host does have a mtu of 1500
but its hard to imagine a recv stack fragmenting (and then reassembling and
reordering!) things.

one of the interesting tidbits here is that it isn't just the small
fragment that moves ahead in the queue - its both fragments of the big
packet.

---

From: Patrick McManus <[email-masked]>
Date: Sat, 3 Mar 2018 10:23:49 -0500

---

From: Praveen Balasubramanian <[email-masked]>
Date: Sat, 3 Mar 2018 18:51:46 +0000

This is why TCP goes out of its way to avoid fragmentation. Fragments are known to take a different (and usually a slower) path through the network. For UDP, if ECMP is being done on 4-tuple then only the first fragment will take one path and subsequent fragments may take another path (since the hash will be on 2-tuple). The right way to deal with this is to detect MF bit and start doing 2-tuple hashing instead of 4-tuple but I have my doubts if many routers do that. Surprising  that a router fragmented IPv6.

From: QUIC [mailto:[email-masked]] On Behalf Of Patrick McManus
Sent: Saturday, March 3, 2018 7:24 AM
To: Ryan Hamilton <rch=[email-masked]>
Cc: [email-masked]; huitema <[email-masked]>
Subject: Re: Fun and surprises with IPv6 fragmentation

On Sat, Mar 3, 2018 at 12:09 AM, Ryan Hamilton <rch=[email-masked]<mailto:rch=[email-masked]>> wrote:
I'm sorry if this is a dumb question, but I understood that in IPv6 routers could not fragment IPv6 packets, only endpoints.

I know! fun. Honestly the Internet is just so interesting - you wonder how it works at all.

The pcap is in the #picoquic slack channel. (as always anyone reading this can just ask anyone on the slack, like me, for an invite). search for octopus.pcap

https://en.wikipedia.org/wiki/IPv6_packet#Fragmentation<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FIPv6_packet%23Fragmentation&data=04%7C01%7Cpravb%40microsoft.com%7Ca78a3514879f4032cde508d5811acaaa%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636556874442590743%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwifQ%3D%3D%7C-1&sdata=Eyq1%2Bv8F7Q9CF%2BWtT0F06M3xwcOnRPhET77cSeJE9XY%3D&reserved=0>

It seems unlikely to be in the core.. the recv host does have a mtu of 1500 but its hard to imagine a recv stack fragmenting (and then reassembling and reordering!) things.

one of the interesting tidbits here is that it isn't just the small fragment that moves ahead in the queue - its both fragments of the big packet.

Cheers,

Ryan

On Fri, Mar 2, 2018 at 9:02 PM, Christian Huitema <[email-masked]<mailto:[email-masked]>> wrote:
Yesterday, I was mentioning bugs of the interop. This morning, I woke up to find an interesting message from Patrick McManus. Something is weird, he said. The first data message that your server sends, with sequence number N, always arrives before the final handshake message, with sequence number N-1. That inversion appears to happen systematically.
It took us the best part of a day to explore blind alleys and finally understand what was happening. The exchange was over IPv6. Upon receiving a connection request from Patrick’s implementation, Picoquic was sending back a handshake packet. Immediately after that, Picoquic was sending its first data packet, which happens to be an MTU probe. And it turns out that the probe was 1518 bytes, a bit longer than what the AWS routers could accept. So some router inserted an IPv6 fragmentation header and split the packet in two: a large initial fragment, 1496 byte long, and a small second fragment 78 bytes long. You could think that this is no big deal, since fragments would just be reassembled at the destination, but you would be wrong.
Some routers on the path try to be helpful. They have learned from past experience that short packets often carry important data, and so they try to route them faster than long data packets. And here is what happens in our case:

•         * The server prepares and send a Handshake packet, 590 bytes long.

•         * The server then prepares the MTU probe, 1518 bytes long.

•         * The MTU probe is split into fragment 1, 1496 bytes, and fragment 2, 78 bytes.

•         * The handshake and the long fragment are routed on the normal path, but the small fragment is routed at a higher priority level.

•         * The Linux driver at the destination receives the small fragment first. It queues everything behind that until it receives the long fragment.

•         * The Linux driver passes the reassembled packet to the application, which cannot do anything with it because the encryption keys can only be obtained from the handshake packet.

•         * The Linux driver then passes the handshake packet to the application.
Which confirms an old opinion. When routers try to be smart and helpful, they end up being dumb and harmful. Please just send the packets in the order you get them!
I tried to work around the issue by setting the "don't fragment" bit on the socket, but somehow that doesn't work. So I simply programmed the server to not use payloads larger than 1440 bytes. Still, I can see that pattern happening in other circumstances, such as a long Connection Initial message followed by a short 0-RTT packet. isn't networking fun?
-- Christian Huitema

---

From: Praveen Balasubramanian <[email-masked]>
Date: Sat, 3 Mar 2018 18:51:46 +0000

Cheers,

Ryan

---

From: "Eggert, Lars" <[email-masked]>
Date: Sat, 3 Mar 2018 21:08:21 +0000

Actually, a few years back Linux had issues with reordering in the receive stack. One would hope those kernels are out of service by now.

--
Sent from a mobile device; please excuse typos.
+49 151 120 55791

On Mar 3, 2018, at 16:24, Patrick McManus <[email-masked]<mailto:[email-masked]>> wrote:

Cheers,

Ryan

---

From: "Eggert, Lars" <[email-masked]>
Date: Sat, 3 Mar 2018 21:08:21 +0000

Cheers,

Ryan

---

From: Erik Kline <[email-masked]>
Date: Sun, 4 Mar 2018 20:59:57 +0900

> I tried to work around the issue by setting the "don't fragment" bit on the
> socket, but somehow that doesn't work.

I don't see how an app can do any sort of PathMTU on its own without following:

    https://tools.ietf.org/html/rfc3542#section-11.2

Without this, the sender OS can (by default) generate fragments if it
receives an ICMPv6 PTB.  I.e. IPV6_DONTFRAG seems to me like a
requirement for MTU detection.

Put another way, I think you probably /always/ want to perform the
step you describe, whenever probing for MTU.

You said (elsewhere) you had a pcap showing fragments arriving at the
client, but have you checked that the sender isn't receiving PTBs and
reacting accordingly?

If PTBs are being received and the fact fragments are still being
generated with IPV6_DONTFRAG, that could point to a bug in the sender
OS.

If something in the Amazon infrastructure or along the path is doing
IPv6 fragmentation of its own accord it's violating the text at the
top of:

    https://tools.ietf.org/html/rfc8200#section-4.5

even that text isn't particularly strongly worded.  (I looked for a
MUST or MUST NOT elsewhere, but haven't found such text yet.)  Devices
doing this should probably be identified and expurgated from the
network.

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Sun, 4 Mar 2018 09:55:54 -0500

It would be interesting to see what a raw socket interface would do, sender
side.

I’m also wondering if the hypervisor might be interfering, though it should
act as middleware.

> I tried to work around the issue by setting the "don't fragment" bit on
the
> socket, but somehow that doesn't work.

I don't see how an app can do any sort of PathMTU on its own without
following:

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Sun, 4 Mar 2018 09:55:54 -0500

the

---

From: "Lubashev, Igor" <[email-masked]>
Date: Sun, 4 Mar 2018 17:30:41 +0000

Depends on what you mean by raw socket. A raw IP socket will have sender kernel send IP fragments per its interface MTU or destination MTU cache, if available. A raw Packet socket will error out, if you try to send larger than interface MTU.

-----Original Message-----
From: Mikkel Fahnøe Jørgensen [[email-masked]]
Received: Sunday, 04 Mar 2018, 9:56AM
To: Erik Kline [ek=[email-masked]]; Christian Huitema [[email-masked]]
CC: [email-masked] [[email-masked]]
Subject: Re: Fun and surprises with IPv6 fragmentation

It would be interesting to see what a raw socket interface would do, sender side.

I'm also wondering if the hypervisor might be interfering, though it should act as middleware.

https://tools.ietf.org/html/rfc3542#section-11.2<https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_rfc3542-23section-2D11.2&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=0_EwNIr1oLcj1zLJnxR98mB2q-UQbuPwlhQ6wAvsvqk&s=gLmtYGMeHbWnHwRmuTzdlBs7j3QbbtXjcHhmU9evKeI&e=>

https://tools.ietf.org/html/rfc8200#section-4.5<https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_rfc8200-23section-2D4.5&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=0_EwNIr1oLcj1zLJnxR98mB2q-UQbuPwlhQ6wAvsvqk&s=TEunT0TN8uqVxX5LC9IAIH7dkUkHZZ3EJ2i6NY-0NRY&e=>

---

From: "Lubashev, Igor" <[email-masked]>
Date: Sun, 4 Mar 2018 17:30:41 +0000

---

From: Christian Huitema <[email-masked]>
Date: Sun, 4 Mar 2018 10:45:01 -0800

Thanks to all of you for the comments. I updated my blog entry on the
subject
(https://huitema.wordpress.com/2018/03/03/having-fun-and-surprises-with-ipv6/)
with the following new conclusion:

/It seems clear now that the fragmentation happened at the source, in
the Linux kernel. This leaves one remaining issue, the out of order
delivery./

/There are in fact two separate out of order delivery issues. One is
having the second fragment arrive between the first one, and the second
is having the MTU probe arrive before the previously sent Handshake
packet. The inversion between the segments may be due to code in the
Linux kernel that believes that sending the last segment first speeds up
reassembly at the receiver. The inversion between the fragmented MTU
probe and the Handshake packet has two plausible causes:/

  * /Some router on the path may be forwarding the small fragment at a
    higher priority level./
  * /Some routers may be implementing equal cost multipath, and then
    placing fragmented packets and regular packets into separate hash
    buckets/

/The summary for developers, and for QUIC in particular, is that we
should really avoid triggering IPv6 fragmentation. It can lead to packet
losses when NATs and firewalls cannot find the UDP payload type and the
port numbers in the fragments. And it can also lead to out of order
delivery as we just saw. And for my own code, the lesson is simple. I
really need to set up the IPv6 Don’t Fragment option when sending MTU
probes, per section 11.2 of RFC 3542
<https://tools.ietf.org/html/rfc3542#section-11.2>.

/-- Christian Huitema

---

From: Christian Huitema <[email-masked]>
Date: Sun, 4 Mar 2018 10:45:01 -0800

/-- Christian Huitema

---

From: Ryan Hamilton <[email-masked]>
Date: Tue, 13 Mar 2018 11:16:37 -0700

I missed this summary when you sent it out earlier. Thanks for getting to
the bottom of this, and for the detailed explanation. Your conclusion fo
always set the DF option when sending makes a ton of sense. IP fragments
are ... not awesome!

On Sun, Mar 4, 2018 at 10:45 AM, Christian Huitema <[email-masked]>
wrote:

> subject (https://huitema.wordpress.com/2018/03/03/having-fun-and-
> surprises-with-ipv6/) with the following new conclusion:
>
> *It seems clear now that the fragmentation happened at the source, in the
> Linux kernel. This leaves one remaining issue, the out of order delivery.*
>
> *There are in fact two separate out of order delivery issues. One is
> having the second fragment arrive between the first one, and the second is
> having the MTU probe arrive before the previously sent Handshake packet.
> The inversion between the segments may be due to code in the Linux kernel
> that believes that sending the last segment first speeds up reassembly at
> the receiver. The inversion between the fragmented MTU probe and the
> Handshake packet has two plausible causes:*
>
>    - *Some router on the path may be forwarding the small fragment at a
>    higher priority level.*
>    - *Some routers may be implementing equal cost multipath, and then
>    placing fragmented packets and regular packets into separate hash buckets*
>
>
>
> *The summary for developers, and for QUIC in particular, is that we should
> really avoid triggering IPv6 fragmentation. It can lead to packet losses
> when NATs and firewalls cannot find the UDP payload type and the port
> numbers in the fragments. And it can also lead to out of order delivery as
> we just saw. And for my own code, the lesson is simple. I really need to
> set up the IPv6 Don’t Fragment option when sending MTU probes, per section
> 11.2 of RFC 3542 <https://tools.ietf.org/html/rfc3542#section-11.2>. *--
> Christian Huitema
>

---

From: Ryan Hamilton <[email-masked]>
Date: Tue, 13 Mar 2018 11:16:37 -0700