Subject: draft-van-beijnum-multi-mtu-05.txt

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 21 Mar 2016 22:46:52 +0100

(Please note: crossposted to 6man and intarea, please prune as desired.)

I've been working on a specification that allows for hosts on the same subnet to be configured with different jumboframe MTUs and then use the largest packet size supported by the sender and the receiver as well as the switches in the middle.

This has been discussed in intarea in the past, but I'm also sending the announcement of the new version of the draft to 6man because there's a new ND option and a change to the RA MTU option.

Feedback is highly appreciated.

Iljitsch

> Begin forwarded message:
> 
> From: [email-masked]
> Date: 21 March 2016 at 22:33:22 +0100
> To: <[email-masked]>
> Subject: I-D Action: draft-van-beijnum-multi-mtu-05.txt
> Reply-To: [email-masked]
> 
> 
> A New Internet-Draft is available from the on-line Internet-Drafts directories.
> 
> 
>        Title           : Extensions for Multi-MTU Subnets
>        Author          : Iljitsch van Beijnum
> 	Filename        : draft-van-beijnum-multi-mtu-05.txt
> 	Pages           : 24
> 	Date            : 2016-03-21
> 
> Abstract:
>   In the early days of the internet, many different link types with
>   many different maximum packet sizes were in use.  For point-to-point
>   or point-to-multipoint links, there are still some other link types
>   (PPP, ATM, Packet over SONET), but multipoint subnets are now almost
>   exclusively implemented as Ethernets.  Even though the relevant
>   standards mandate a 1500 byte maximum packet size for Ethernet, more
>   and more Ethernet equipment is capable of handling packets bigger
>   than 1500 bytes.  However, since this capability isn't standardized,
>   it is seldom used today, despite the potential performance benefits
>   of using larger packets.  This document specifies mechanisms to
>   negotiate per-neighbor maximum packet sizes so that nodes on a
>   multipoint subnet may use the maximum mutually supported packet size
>   between them without being limited by nodes with smaller maximum
>   sizes on the same subnet.
> 
> 
> The IETF datatracker status page for this draft is:
> https://datatracker.ietf.org/doc/draft-van-beijnum-multi-mtu/
> 
> There's also a htmlized version available at:
> https://tools.ietf.org/html/draft-van-beijnum-multi-mtu-05
> 
> A diff from the previous version is available at:
> https://www.ietf.org/rfcdiff?url2=draft-van-beijnum-multi-mtu-05
>

---

From: Philip Homburg <[email-masked]>
Date: Wed, 23 Mar 2016 13:32:02 +0100

Hi Iljitsch,

I have a few remarks:
- I think it is best to drop all references to IPv4. 
- Some discussion is required why using UDP is better than ICMP for testing
  whether the link supports the desired MTU.
- The TCP MSS option needs more discussion. What is supposed to happen if the
  MSS is set to 9000 and the link only supports 1500.
- I wonder if the complexity of the HintMTU field is warranted. Having a
  fixed length option is better than one with a variable length.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 23 Mar 2016 23:29:25 +0100

Hi Philip,

> - I think it is best to drop all references to IPv4. 

Hm, there actually are still a few people out there that use v4, you know.  :-)

Me being one of them as my ISP turned off v6 yesterday for some unknown reason.  :-(

>  whether the link supports the desired MTU.

Do you have any reason why ICMP is better?

The advantage of UDP is that the whole thing can be implemented in a userspace daemon easily, so we don't have to wait for the OS maker to implement the protocol before we can gain some experience.

> - The TCP MSS option needs more discussion. What is supposed to happen if the MSS is set to 9000 and the link only supports 1500.

What we do is set the MSS to the maximum size and then send smaller packets if we have to. A better way would be to set the MSS to a safe value and send larger packets if we can. But unfortunately, that won't work because once you've set the MSS to a certain value you can never have packets larger than that during the lifetime of the TCP session.

The way I plan to implement this is adjust the MTU for a cloned route towards a neighbor on the local subnet once we've discovered the maximum packet size that is supported.

This may mean that some of the first large packets of a TCP session are lost if the MTU supported by the layer 2 infrastructure is smaller than the maximum supported by the nodes in question.

A more advanced implementation may limit the size of TCP packets while MTU probing is still ongoing.

>  fixed length option is better than one with a variable length.

Hm, if we manage to figure out the padding to make the options align on 64-bit boundaries then this part should be easy...

It would be possible to make the option always the same length by always including the HintMTU even if we don't really have a good value for it.

The idea behind advertising the HintMTU is that if we (for instance) learned that the layer 2 network can handle a 1982 MTU between nodes A and B, then if C starts to talk to A, it may be helpful to test for 1982 immediately rather than test a much larger number of values. Maybe that's not really all that helpful... I don't know.

Thanks for the feedback,

Iljitsch

---

From: Philip Homburg <[email-masked]>
Date: Thu, 24 Mar 2016 13:55:19 +0100

>On 23 Mar 2016, at 13:32, Philip Homburg wrote:
>

Maybe it is an idea to create separate versions for IPv4 and IPv6. It seems
to me that an IPv6-only version could be cleaned up a lot.

>> - Some discussion is required why using UDP is better than ICMP for =
>testing

ND is already ICMP so you don't have to add a dependency on UDP to perform
ND.

For IPv6, you only need to verify reachability (after receiving a neighbor's
NODEMTY option). For that ECHO is fine. 

>The advantage of UDP is that the whole thing can be implemented in a 
>userspace daemon easily, so we don't have to wait for the OS maker to 
>implement the protocol before we can gain some experience.

ICMP is routinely done from user space. Though I have no idea how you
want to send packets bigger than the link MTU in existing implementations.

>> - The TCP MSS option needs more discussion. What is supposed to happen 
>if the MSS is set to 9000 and the link only supports 1500.
>
>What we do is set the MSS to the maximum size and then send smaller 
>packets if we have to. A better way would be to set the MSS to a safe 
>value and send larger packets if we can. But unfortunately, that won't 
>work because once you've set the MSS to a certain value you can never 
>have packets larger than that during the lifetime of the TCP session.

I guess PMTU discovery should already deal with an MTU smaller than the MSS,
so setting the MSS to the maximum that the host will support should be
safe.

>MTU probing is still ongoing.

I guess this make most sense.

>> - I wonder if the complexity of the HintMTU field is warranted. Having =
>a
>Hm, if we manage to figure out the padding to make the options align on 
>64-bit boundaries then this part should be easy...

Do you really need MTUs bigger than 64KB? Why not use the space that is
now called reserved for the NodeMTU and have 32-bits of reserved space
after that. That should be enough for future flags, scaling factors and
a HintMTU.

>The idea behind advertising the HintMTU is that if we (for instance) 
>learned that the layer 2 network can handle a 1982 MTU between nodes A 
>and B, then if C starts to talk to A, it may be helpful to test for 1982 
>immediately rather than test a much larger number of values. Maybe 
>that's not really all that helpful... I don't know.

My guess would be that if A knows about the 1982, then C will probably now
as well. Maybe not immediately after boot, but during most of the
uptime.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 24 Mar 2016 16:36:56 +0100

Hi Philip,

> to me that an IPv6-only version could be cleaned up a lot.

I reread the draft just to make sure (and caught a bunch of little stuff that needs to be fixed...) but I don't agree with your statement. IPv4 is mentioned 15 times and IPv6 24 times. Despite that, the draft is pretty IP version agnostic. There's just some details that differ, such as the fact that we have ND options for IPv6 and not IPv4, and the total length / payload length difference.

> ND is already ICMP so you don't have to add a dependency on UDP to perform ND.

> NODEMTY option). For that ECHO is fine. 

Hm, using ICMP echo for this would have one very nice advantage: now we can test whether nodes that don't run the test code support larger MTUs. For example, routers.

I'll see if I can change the test implementation to use ICMP echo. (I haven't tried IPv6 for the test program, but with a different project this gave me a lot of trouble because IPv6 doesn't support raw sockets the way IPv4 does.)

> Though I have no idea how you

Hm, isn't that also the physical interface MTU? Sending packets larger than that would indeed be a neat trick.

But even sending packets larger than those determined by PTMUD is hard or impossible. Hence the only solution is to assume a large MTU and reduce it in the user space implemenation, the opposite is not possible without fundamental changes to the system.

> I guess PMTU discovery should already deal with an MTU smaller than the MSS, so setting the MSS to the maximum that the host will support should be
> safe.

If the path MTU is limited by the MTU of a switch you won't be receiving any ICMP too big messages, so no, PMTUD won't always catch all issues. However, if testing fails the neighbor MTU will be reduced quickly and retransmissions will use smaller packets.

> after that. That should be enough for future flags, scaling factors and

The problems we're having now is because someone typed "1500" back in 1980 or so. I really don't want to introduce any limitations that may come back to bite us in the future.

However, I think the HintMTU can be removed from the ND option as it will be exchanged during testing anyway.

Iljitsch