Subject: [Int-area] Probe sizes for MTU discovery

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 14 Nov 2014 15:03:32 -1000

I got the suggestion to not reinvent the wheel and look at RFCs 7383 and 4821 for how probing is done and what size probes to use.

However, a quick look at these two doesn't show a very specific probing procedure as far as the probe size goes. RFC 1191, the original PMTUD spec, on the other hand, has a list of common MTU sizes and suggests a number of probe sizes (page 17).

I didn't reuse much of that, because a lot of it is very much out of date. Instead, I found this document at Cisco, which lists a bunch of MTU sizes supported by their switches:

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6000-series-switches/24048-148.html

I added to this some sizes that I encountered myself, leading to this list (a few more added since -04):

     # hint sources:
     #   576: common PPP low delay
     #  1492: PPP over Ethernet [RFC2516]
     #  1500: Ethernet II
     #  1982: IEEE Std 802.3as-2006
     #  2304: 802.11
     #  2482: Fibre Channel over Ethernet (FCoE)
     # [CATALYST]:
     # 9216, 8092, 1600, 1998, 2000, 1546, 1530, 17976, 2018
     # sizes observed by the author:
     # 576, 4070, 9000, 16384, 64000
     let Hints = 576, 1492, 1530, 1982, 2304, 4070, 8092, 9000, 16384, 32000, 64000

If I read correctly (I'll go back to make sure next week) RFC 7383 suggests starting to probe from high on down, because starting low and going up wouldn't work because of the short lived transfers involved.

My logic was err on the side of sending more smaller packets rather than send big packets that don't make it and have to back off. I suspect some implementations may react badly if too many too large packets come too quickly. Doing a first round where the packet size is doubled each time and then trying a few well-known (or close to well-known) sizes seemed like a good way to accomplish this.

Another way to do this would be a binary search, which would need a smaller total number of packets, but more of them would be ones that are too big that are lost so a backoff time needs to be observed, so the probing procedure would probably take longer.

However, I'm open to looking at other approaches.

---

From: "Templin, Fred L" <[email-masked]>
Date: Sun, 16 Nov 2014 02:00:12 +0000

Hi Iljitsch,

> -----Original Message-----
> From: Int-area [mailto:[email-masked]] On Behalf Of Iljitsch van Beijnum
> Sent: Friday, November 14, 2014 5:04 PM
> To: [email-masked]
> Subject: [Int-area] Probe sizes for MTU discovery
> 

But, it is the *original source* that does the probing; not the router at the head end
of the multi-MTU link.

> However, a quick look at these two doesn't show a very specific probing procedure as far as the probe size goes. RFC 1191, the original
> PMTUD spec, on the other hand, has a list of common MTU sizes and suggests a number of probe sizes (page 17).

AKA the "plateau table", sure.

> If I read correctly (I'll go back to make sure next week) RFC 7383 suggests starting to probe from high on down, because starting low
> and going up wouldn't work because of the short lived transfers involved.

I think it would be totally nuts to try to probe for all of these sizes - especially
from a network middlebox. If an 8KB probe works, for example, why bother
probing for 9KB even if that size would work too, because 8KB is already good
enough. You will never be able to probe down to the byte level, e.g., probe
for 7983 bytes followed by 7984 bytes followed by etc. So, maybe just probe
for 1500, 2K, 4K and 8K and take the best answer you get and call it good?

> My logic was err on the side of sending more smaller packets rather than send big packets that don't make it and have to back off. I
> suspect some implementations may react badly if too many too large packets come too quickly. Doing a first round where the packet
> size is doubled each time and then trying a few well-known (or close to well-known) sizes seemed like a good way to accomplish this.
> 
> Another way to do this would be a binary search, which would need a smaller total number of packets, but more of them would be
> ones that are too big that are lost so a backoff time needs to be observed, so the probing procedure would probably take longer.
> 

Remember about RFC1191 plateau table MTU determination that it is the *source host*
and not some router in the middle that does the probing. MTU probing is an end-to-end
consideration; not hop-by-hop. That is why we need to promote RFC4821.

Thanks - Fred
[email-masked]
> _______________________________________________
> Int-area mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/int-area

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sun, 16 Nov 2014 10:24:51 -0800

Hi Fred,

> for 1500, 2K, 4K and 8K and take the best answer you get and call it good?

I agree that it would be excessive to test for all these sizes. Assuming that the size supported by the two nodes in question isn't supported by the switches, the probing sizes are as follows. I'm using 9216 as the size the two nodes support and 9000 as what the switches support.

probe = 9216: fail
probe = 320: success (only for IPv4, 1280 is mandatory for IPv6)
probe = 640: success (only for IPv4, 1280 is mandatory for IPv6)
probe = 1280: success (only for IPv4, 1280 is mandatory for IPv6)
probe = 2560: success

At this point the next doubling of the probe size would exceed the 9216 that has already failed, so we stop the doubling here and instead start using the hints:

probe = 4070: success
probe = 8092: success
probe = 9000: success

Done. Our supported MTU is 9000. This took 8 probe packets for IPv4, 5 for IPv6.

>>     let Hints = 576, 1492, 1530, 1982, 2304, 4070, 8092, 9000, 16384, 32000, 64000

you'll see that it skips many sizes, especially in the 1500 - 2018 byte range. Being able to do 1528 bytes (an IPv4 + a UDP header) can be very helpful and nearly 2k could be a modest efficiency improvement over 1500, but 2018 vs 1982 is probably not worth the extra probing.

(If a network supports 2018 or 4464 or 8192 then a system administrator may add that size as a hinted size to be probed on the systems in question.)

> Remember about RFC1191 plateau table MTU determination that it is the *source host* and not some router in the middle that does the probing.

I do remember that, but I'm not sure why you keep stressing that:

> MTU probing is an end-to-end consideration; not hop-by-hop.

Last but not least:

> That is why we need to promote RFC4821.

I agree. However, I think RFC 4821 and my variable MTU subnet mechanism would complement each other.

In an earlier version of the draft I skipped the probing if a system supports RFC 4821, but I removed that as there were just too many ways for things to fall through the cracks that way.

Iljitsch

---

From: "Black, David" <[email-masked]>
Date: Mon, 17 Nov 2014 22:29:49 +0000

Hi Iljitsch,

> In an earlier version of the draft I skipped the probing if a system supports
> RFC 4821, but I removed that as there were just too many ways for things to
> fall through the cracks that way.

Here are a couple of comments, in an attempt to summarize what I said at the
mike in Honolulu:

(1) RFC 4821 should be the basis for the algorithm used for probing, even if
a new protocol carries the probes.  RFC 4821 allows roughly what is desired
(quoting from Section 7.3):

   As an optimization, it may be appropriate to probe at certain common
   or expected MTU sizes, for example, 1500 bytes for standard Ethernet,
   or 1500 bytes minus header sizes for tunnel protocols.

Referencing section 7 of RFC 4821 for the probe algorithm avoids reinventing
that wheel ;-), even if the probes are not end-to-end, and there's nothing
wrong with making "intelligent guesses" on probe sizes.

(2) If for some reason, Section 7 of RFC 4821 doesn't get the job done, it's
important to explain why not.  A worked example of such an explanation can
be found in Section 2.5.2 of RFC 7383, which is based on requirements
specific to the IKEv2 key exchange protocol.

To head off a few possible misimpressions, I recommend:

- Start from RFC 4821, although it's not necessary to do exactly what it
	says.  OTOH, if RFC 4821 ain't broken for this usage, please don't
	fix it ;-).
- Serious deviations from RFC 4821 do need to be justified - such deviations
	may be acceptable, as demonstrated by Section 2.5.2 of RFC 7383.
- I view section 2.5.2 of RFC 7383 an example only, not something to be
	copied/reused, as I'm reasonably certain that the approach described
	there is not applicable to the draft that we're discussing.

I hope that helps, and apologize for any confusion - I'm not always in
possession of all my wits on Friday morning after an intense IETF week.

Thanks,
--David

> Sent: Sunday, November 16, 2014 1:25 PM
> To: Templin, Fred L
> Cc: [email-masked]
> Subject: Re: [Int-area] Probe sizes for MTU discovery
> 
> Hi Fred,
> 
> I agree that it would be excessive to test for all these sizes. Assuming that
> the size supported by the two nodes in question isn't supported by the
> switches, the probing sizes are as follows. I'm using 9216 as the size the two
> nodes support and 9000 as what the switches support.
> 
> probe = 9216: fail
> At this point the next doubling of the probe size would exceed the 9216 that
> has already failed, so we stop the doubling here and instead start using the
> hints:
> 
> probe = 4070: success
> probe = 8092: success
> probe = 9000: success
> 
> you'll see that it skips many sizes, especially in the 1500 - 2018 byte range.
> Being able to do 1528 bytes (an IPv4 + a UDP header) can be very helpful and
> nearly 2k could be a modest efficiency improvement over 1500, but 2018 vs 1982
> is probably not worth the extra probing.
> 
> (If a network supports 2018 or 4464 or 8192 then a system administrator may
> add that size as a hinted size to be probed on the systems in question.)
> 
> > Remember about RFC1191 plateau table MTU determination that it is the
> *source host* and not some router in the middle that does the probing.
> 
> In an earlier version of the draft I skipped the probing if a system supports
> RFC 4821, but I removed that as there were just too many ways for things to
> fall through the cracks that way.
> 
> Iljitsch

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 17 Nov 2014 22:38:59 +0000

Hi David,

> (1) RFC 4821 should be the basis for the algorithm used for probing,

While Iljitsch is considering you message, can you tell us if you have any ideas on
how we can encourage more widespread deployment of RFC4821?

Thanks - Fred

> From: Int-area [mailto:[email-masked]] On Behalf Of Black, David
> Sent: Monday, November 17, 2014 2:30 PM
> To: Iljitsch van Beijnum
> Cc: [email-masked]

---

From: "Black, David" <[email-masked]>
Date: Mon, 17 Nov 2014 22:48:50 +0000

Hi Fred,

I'm not sure - I see a Catch-22 scenario - there needs to be widespread
deployment of environments in which RFC 4821 implementation provides
benefits, which of course presumes widespread deployment of RFC 4821.

The increasing unreliability of ICMP in the presence of middleboxes and
tunnels that are just encapsulations (i.e., aren't links, hence may not
return ICMP too big responses from ingress) may help, but a simple guess
that the MTU is 1500 bytes works entirely too well ...

Thanks,
--David

> From: Templin, Fred L [mailto:[email-masked]]
> Sent: Monday, November 17, 2014 5:39 PM
> To: Black, David; Iljitsch van Beijnum
> Cc: [email-masked]
> Subject: RE: [Int-area] Probe sizes for MTU discovery
> 
> Hi David,
> 
> > > I agree. However, I think RFC 4821 and my variable MTU subnet mechanism
> would
> > > complement each other.
> > >
> > > From: Int-area [mailto:[email-masked]] On Behalf Of Iljitsch
> van
> > > Beijnum
> > > > I think it would be totally nuts to try to probe for all of these sizes
> -
> > > especially
> > > I agree. However, I think RFC 4821 and my variable MTU subnet mechanism
> would
> > > complement each other.
> > >

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 17 Nov 2014 23:00:57 +0000

Hi David,

> From: Black, David [mailto:[email-masked]]
> Sent: Monday, November 17, 2014 2:49 PM
> To: Templin, Fred L; Iljitsch van Beijnum
> Cc: [email-masked]

Can you get behind a BCP statement such as:

  "Hosts that send packets larger than 1500 bytes SHOULD implement RFC4821."

?

> that the MTU is 1500 bytes works entirely too well ...

ACK, except I don't see how an encapsulation of IP-over-foo could be seen
as anything other than a link-layer from the perspective of IP, i.e., with IP
as the network-layer and foo as the link-layer. (Then again, there is the
possibility for a "hybrid" link/non-link tunnel that accommodates everything
up to 1500 deterministically and returns PTBs on a "best effort" basis for
anything else that is too large to make it through.)

Thanks - Fred

---

From: "Black, David" <[email-masked]>
Date: Mon, 17 Nov 2014 23:11:09 +0000

Fred,

>   "Hosts that send packets larger than 1500 bytes SHOULD implement RFC4821."

Yes.

> anything else that is too large to make it through.)

That depends on what one believes "link" means - there are a lot of tunnel
specs and deployments that don't get ICMP handling right for the tunnel MTU;
that's one of the reasons that RFC 4821 exists, as it has no ICMP dependency.

Some of the encapsulations aren't directly of IP - MPLS/UDP and GRE/UDP are
two in-progress examples.

Thanks,
--David

> Sent: Monday, November 17, 2014 6:01 PM
> > > > > RFC 4821, but I removed that as there were just too many ways for
> things
> > > to
> even if
> > > > > >>     let Hints = 576, 1492, 1530, 1982, 2304, 4070, 8092, 9000,
> 16384,
> > > > > 32000, 64000
> > > > >
> > > > > > probing for 9KB even if that size would work too, because 8KB is
> already
> > > > > good
> > > > > > enough. You will never be able to probe down to the byte level,
> e.g.,
> > > probe
> > > > > > for 7983 bytes followed by 7984 bytes followed by etc. So, maybe
> just
> > > probe
> > > > > I agree that it would be excessive to test for all these sizes.
> Assuming
> > > that
> > > > > switches, the probing sizes are as follows. I'm using 9216 as the size
> the
> > > two
> > > > > At this point the next doubling of the probe size would exceed the
> 9216
> > > that
> > > > > has already failed, so we stop the doubling here and instead start
> using
> > > the
> > > > > hints:
> > > > >
> > > > > probe = 4070: success
> > > > > probe = 8092: success
> > > > > probe = 9000: success
> > > > >
> > > > > Done. Our supported MTU is 9000. This took 8 probe packets for IPv4, 5
> for
> > > > > IPv6.
> > > > >
> > > > > >>     let Hints = 576, 1492, 1530, 1982, 2304, 4070, 8092, 9000,
> 16384,
> > > > > 32000, 64000
> > > > >
> > > > > you'll see that it skips many sizes, especially in the 1500 - 2018
> byte
> > > range.
> > > > > Being able to do 1528 bytes (an IPv4 + a UDP header) can be very
> helpful
> > > and
> > > > > nearly 2k could be a modest efficiency improvement over 1500, but 2018
> vs
> > > 1982
> > > > > (If a network supports 2018 or 4464 or 8192 then a system
> administrator
> > > may
> > > > > RFC 4821, but I removed that as there were just too many ways for
> things
> > > to

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 17 Nov 2014 23:17:56 +0000

Hi David,

> Sent: Monday, November 17, 2014 3:11 PM

Regardless of what we may/may not agree on wrt "link", for now I am happy
to meet you in the middle and amplify this point. But, this means that tunnels
have to deterministically pass 1500's, and I am prepared to accept that.

Thanks - Fred

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 18 Nov 2014 14:13:57 +0100

Hi David,

> wrong with making "intelligent guesses" on probe sizes.

The trouble is, RFC 4821 is quite vague about the specifics of the probing algorithm. (Even when rereading after getting a good night's sleep.) The portion you quoted is the most concrete thing in there; no other values than "1500" are mentioned.

RFC 1191 is better in this regard, but still allows for a lot of leeway. Based on RFC 1191, you would probe the following sizes: 508, 1006, 1492, 2002, 4352, 8166, 17914, 32000, 65535. However, of these sizes 1492 is the only one you'd still find in real-world usage today—and that's only because a newer protocol (PPPoE) happens to use the same size as an old one (IP over 802.3 with LLC/SNAP encapsulation).

As I'm also implementing the whole thing, I need to decide how exactly to execute the probing, but that doesn't necessarily mean that all those details need to go into the draft. So would it be better to leave the details up to individual implementations (like RFC 4821 does) or specify them and thus have different implementations behave more similarly?

Apart from the different plateau (I call them hint) values than in RFC 1191, the only real difference in how I do the probing is by doing it in two rounds, where the first round doubles the size of the probes and the second round uses the plateaus/hints to fine tune. This trades off a slightly higher number of failing probes against a smaller number of working probes, which I think is reasonable because failing probes are less of a problem because they won't be easily confused with congestion losses (like failing transport based data carrying probes) but probes carry no data so sending a lot of them is less desirable.

Iljitsch

---

From: "Black, David" <[email-masked]>
Date: Tue, 18 Nov 2014 15:14:04 +0000

> The trouble is, RFC 4821 is quite vague about the specifics of the probing algorithm.
> (Even when rereading after getting a good night's sleep.) The portion you quoted is
> the most concrete thing in there; no other values than "1500" are mentioned.

There are two sorts of specifics - RFC 4821 is deliberately vague on the specific 
probe values to use, but Section 7 contains a lot of detail on how to manage the
range of possible MTUs from which the probe values are picked and how to update
that range based on probe results.  It would be good to avoid reinventing that.

Thanks,
--David

> From: Iljitsch van Beijnum [mailto:[email-masked]]
> Sent: Tuesday, November 18, 2014 8:14 AM
> To: Black, David
> Cc: [email-masked]
> The trouble is, RFC 4821 is quite vague about the specifics of the probing
> algorithm. (Even when rereading after getting a good night's sleep.) The
> portion you quoted is the most concrete thing in there; no other values than
> "1500" are mentioned.
> 
> RFC 1191 is better in this regard, but still allows for a lot of leeway. Based
> on RFC 1191, you would probe the following sizes: 508, 1006, 1492, 2002, 4352,
> 8166, 17914, 32000, 65535. However, of these sizes 1492 is the only one you'd
> still find in real-world usage today—and that's only because a newer protocol
> (PPPoE) happens to use the same size as an old one (IP over 802.3 with
> LLC/SNAP encapsulation).
> 
> As I'm also implementing the whole thing, I need to decide how exactly to
> execute the probing, but that doesn't necessarily mean that all those details
> need to go into the draft. So would it be better to leave the details up to
> individual implementations (like RFC 4821 does) or specify them and thus have
> different implementations behave more similarly?
> 
> Apart from the different plateau (I call them hint) values than in RFC 1191,
> the only real difference in how I do the probing is by doing it in two rounds,
> where the first round doubles the size of the probes and the second round uses
> the plateaus/hints to fine tune. This trades off a slightly higher number of
> failing probes against a smaller number of working probes, which I think is
> reasonable because failing probes are less of a problem because they won't be
> easily confused with congestion losses (like failing transport based data
> carrying probes) but probes carry no data so sending a lot of them is less
> desirable.
> 
> Iljitsch

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 18 Nov 2014 23:21:07 +0100

> that range based on probe results.  It would be good to avoid reinventing that.

All I can tell you is that I read RFC 1191 and RFC 4821 and this is what I came up with:

- see if min(MTU side A, MTU side B) works
- if not, start at 320 (IPv4) or 2560 (IPv6) and double until a probe fails or min(A, B) is reached
- probe values from the list of hints between the highest successful probe size and the lowest unsuccessful probe size

This is not precisely what 1191 and 4821 suggest. It is also not in obvious disagreement. Reinvention or filling in unspecified details and adapting to a different use case? I'm not sure if I can answer that.

Note that all of this is mostly unimportant detail. Start with small probes and work your way up, start with big ones and work your way down, binary search... it'll all produce the same result in the end. There are much more important decisions, such as whether we can assume that MTU = MRU and whether it's reasonable to require that routers keep track of which neighbors they've sent jumboframes to recently.

---

From: Jeffery Hammond <[email-masked]>
Date: Fri, 21 Nov 2014 12:57:39 +0000

I've been catching up on the various RFC's and drafts mentioned. I haven't really looked into PMTUD since my server days when jumbo packets and direct link switching was starting up. 

Iljitsch;
	I'm very interested in your experience implementing PMTUD as I will soon be implementing this also.

	My thoughts on the probe algorithm are that even the best one we develop will need some type of modification as it's implemented to increase efficiency.  Always seems to work that way. Maybe something simple to start (use well known MTU's, then a binary search between the hi and low points) yet open enough to consider the concerns below and others (LAG groups,etc).

	Early adoption and implementation seem to be the sticking points. But until we come up a set of recommendations there can't be implementation.

	These are just my opinion's. There a lot of folks smarter then me on this list that are very capable of making the proper decisions as a group. I see this becoming a rapidly bigger issue if not addressed as soon as possible.

Jeff Hammond
Samsung System Engineer

-----Original Message-----
From: Int-area [mailto:[email-masked]] On Behalf Of Iljitsch van Beijnum
Sent: Tuesday, November 18, 2014 4:21 PM
To: Black, David
Cc: [email-masked]
Subject: Re: [Int-area] Probe sizes for MTU discovery

On 18 Nov 2014, at 16:14, Black, David <[email-masked]> wrote:

> There are two sorts of specifics - RFC 4821 is deliberately vague on 
> the specific probe values to use, but Section 7 contains a lot of 
> detail on how to manage the range of possible MTUs from which the 
> probe values are picked and how to update that range based on probe results.  It would be good to avoid reinventing that.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 24 Nov 2014 16:01:07 +0100

> 	I'm very interested in your experience implementing PMTUD as I will soon be implementing this also.

This is not exactly path MTU discovery, although there is overlap.

Originally, "traditional" PMTUD (i.e., RFC 1191, not RFC 4821 PMTUD) needed to perform probing with different packet sizes, but I think that these days all routers include the MTU size in their too big packets. (This was not originally the case before RFC 1191 was released.) So the way to implement RFC 1191 today is to start sending large packets and reduce the packet size based on the ICMP messages that are returned.

My system as well as RFC 4821 need to work differently because there are no ICMP messages when a packet is too big.

> 	My thoughts on the probe algorithm are that even the best one we develop will need some type of modification as it's implemented to increase efficiency.  Always seems to work that way. Maybe somet…

If we know what the cost of sending probes is, we can optimize to reduce that cost. A binary search optimizes for the smallest number of probes.

Intuitively, sending smaller probes is less costly, because it uses fewer resources. Also, the chances of success are higher, and a successful probe means we get to start sending larger data packets immediately. On the other hand, the potential benefits of smaller probes are also smaller. Sure, it's good to know we can send 1508-byte packets, but it would be much better to know that we can send 9000-byte packets.

The thing I worry about, probably for no good reason, is the situation where we send a 9000-byte packet into a repeater hub, and by the 2500th byte or so the hub decides something really bad is happening and it reinitializes the port in question. (I guess it couldn't hurt to only send jumboframes on ports that are operating in full duplex.)

A third consideration is that we know there's not 7499 different supported MTU sizes between 1500 and 9000 out there in the world, and even if there are switches that support 1982 and switches that support 2000, is it really worth the trouble to probe for 2000 if we know that 1982 works but 2018 doesn't?

There are a few things we can try in any event:

- Be optimistic and probe the largest possible size. If this works, we're done in one round.

- Be pessimistic and probe the smallest possible or usable size jumboframe size. (1508 would be the first 32-bit aligned value that exceeds the Ethernet_II MTU plus the size of a VLAN header and could do 1500 bytes + a PPPoE header, 1520 can handle 1500 bytes + an IPv4 header.) If this fails, we're done in one round.

- Observe what has worked before on the same port. If another neighbor (or maybe the same neighbor) supports 9000 but only 1982-byte packets worked, it's likely the same situation that limited us to 1982 before is still limiting us to 1982 now.

- Take administrative configuration into account. Perhaps a system administrator knows what the switches support and has put that value in a configuration file.

I have no obvious way to order the four options above.

But once probes based on those are sent with inconclusive results, we still have some range of values to probe. I can think of a few strategies to do this:

1. Start high and divide by some factor after each unsuccessful probe. Advantages: ? Disadvantages: many large probes that are lost before we learn a new MTU size that can be used for data packets.

2. Start low, multiply by some factor after each successful probe. Advantages: quickly narrows the search space with only one unsuccessful probe. Disadvantages: may probe sizes that make little sense, final (lost) probe is relatively large.

3. Start low, add a fixed number after each successful probe. Advantages: ? Disadvantages: see above + number of probes gets much larger as size increases.

4. Binary search. Advantages: limited number of probes. (Probably need to stop probing after MTU size is found to be within some range to avoid sending too many probes of almost the same size.) Disadvantages: relatively many unsuccessful probes, may lead to undeterministic probe sizes depending on initial neighbor MTU.

Plateau search, where there is a list of common MTU sizes or MTU sizes that represent a group of common sizes (see RFC 1191). Advantages are a reduced number of probes and avoiding sending a probe that's just a tiny bit too large. Disadvantages: need to compile a list of common MTU sizes, which may get out of date (see RFC 1191 again). This can be done in a variety of ways:

5. Plateau search, low to high: relatively many probes, only the last one fails.

6. Plateau search, binary: relatively few probes, but more failures. Relatively complex implementation.

7. Plateau search, in order of prevalence: very few probes in the common case, but relatively complex implementation and now even more information that needs to be compiled.

My current implementation combines 2 and 5. I'm now thinking of changing this to 7 (and 2 when we run out of predefined plateaus), where the plateaus and order in which they're tried can be configured using a config file, so basically any desired probing strategy can be implemented simply by changing the plateaus in the config file.

> 	These are just my opinion's. There a lot of folks smarter then me on this list that are very capable of making the proper decisions as a group. I see this becoming a rapidly bigger issue if not ad…

All of this should be relatively easy to simulate or model. If anyone has ideas about scenarios that could/should be tested that way, that would be helpful.

Iljitsch

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 24 Nov 2014 16:17:00 +0000

Hi Iljitsch,

You are providing a convincing argument that all of this is way too much "intelligence"
to incorporate on a hop-by-hop basis.

About plateau tables, you can maybe name a few values that work "most of the time",
but in the world of configurable MTUs there are no guarantees. Anything that can be
configured can be misconfigured, so you never know when someone is going to set
an oddball MTU like 2592, 3379, 5123, etc.

Finally, I would ask how it works with RFC4821 and try to do the same thing here.
Like, maybe start with 1280 then try 1500 then try 2K, then 4K then 8K then done?
Or, if you think you can come up with a better way than RFC4821 maybe consider
updating that spec?

Thanks - Fred

> Sent: Monday, November 24, 2014 7:01 AM
> To: Jeffery Hammond
> Cc: [email-masked]
> Originally, "traditional" PMTUD (i.e., RFC 1191, not RFC 4821 PMTUD) needed to perform probing with different packet sizes, but I
> think that these days all routers include the MTU size in their too big packets. (This was not originally the case before RFC 1191 was
> released.) So the way to implement RFC 1191 today is to start sending large packets and reduce the packet size based on the ICMP
> messages that are returned.
> 
> > 	My thoughts on the probe algorithm are that even the best one we develop will need some type of modification as it's
> implemented to increase efficiency.  Always seems to work that way. Maybe something simple to start (use well known MTU's, then a
> binary search between the hi and low points) yet open enough to consider the concerns below and others (LAG groups,etc).
> 
> Intuitively, sending smaller probes is less costly, because it uses fewer resources. Also, the chances of success are higher, and a
> successful probe means we get to start sending larger data packets immediately. On the other hand, the potential benefits of smaller
> probes are also smaller. Sure, it's good to know we can send 1508-byte packets, but it would be much better to know that we can
> send 9000-byte packets.
> 
> The thing I worry about, probably for no good reason, is the situation where we send a 9000-byte packet into a repeater hub, and by
> the 2500th byte or so the hub decides something really bad is happening and it reinitializes the port in question. (I guess it couldn't
> hurt to only send jumboframes on ports that are operating in full duplex.)
> 
> A third consideration is that we know there's not 7499 different supported MTU sizes between 1500 and 9000 out there in the world,
> and even if there are switches that support 1982 and switches that support 2000, is it really worth the trouble to probe for 2000 if we
> know that 1982 works but 2018 doesn't?
> 
> - Be pessimistic and probe the smallest possible or usable size jumboframe size. (1508 would be the first 32-bit aligned value that
> exceeds the Ethernet_II MTU plus the size of a VLAN header and could do 1500 bytes + a PPPoE header, 1520 can handle 1500 bytes +
> an IPv4 header.) If this fails, we're done in one round.
> 
> - Observe what has worked before on the same port. If another neighbor (or maybe the same neighbor) supports 9000 but only 1982-
> byte packets worked, it's likely the same situation that limited us to 1982 before is still limiting us to 1982 now.
> 
> 1. Start high and divide by some factor after each unsuccessful probe. Advantages: ? Disadvantages: many large probes that are lost
> before we learn a new MTU size that can be used for data packets.
> 
> 2. Start low, multiply by some factor after each successful probe. Advantages: quickly narrows the search space with only one
> unsuccessful probe. Disadvantages: may probe sizes that make little sense, final (lost) probe is relatively large.
> 
> 4. Binary search. Advantages: limited number of probes. (Probably need to stop probing after MTU size is found to be within some
> range to avoid sending too many probes of almost the same size.) Disadvantages: relatively many unsuccessful probes, may lead to
> undeterministic probe sizes depending on initial neighbor MTU.
> 
> Plateau search, where there is a list of common MTU sizes or MTU sizes that represent a group of common sizes (see RFC 1191).
> Advantages are a reduced number of probes and avoiding sending a probe that's just a tiny bit too large. Disadvantages: need to
> compile a list of common MTU sizes, which may get out of date (see RFC 1191 again). This can be done in a variety of ways:
> 
> 7. Plateau search, in order of prevalence: very few probes in the common case, but relatively complex implementation and now even
> more information that needs to be compiled.
> 
> My current implementation combines 2 and 5. I'm now thinking of changing this to 7 (and 2 when we run out of predefined plateaus),
> where the plateaus and order in which they're tried can be configured using a config file, so basically any desired probing strategy can
> be implemented simply by changing the plateaus in the config file.
> 
> > 	These are just my opinion's. There a lot of folks smarter then me on this list that are very capable of making the proper
> decisions as a group. I see this becoming a rapidly bigger issue if not addressed as soon as possible.
>