Subject: Re: [v6ops] PMTUD issue discussion

From: joel jaeggli <[email-masked]>
Date: Mon, 01 Sep 2014 09:35:13 -0700

I think this is worthwhile and I will work to incorporate the observation.

On 8/26/14 2:50 AM, Andrew 👽  Yourtchenko wrote:
> Joel,
> 
> On 8/25/14, joel jaeggli <[email-masked]> wrote:
>> On 8/25/14 1:49 PM, Joe Touch wrote:
>>> Hi, all,
>>>
>>> Speaking from TCPM-land, I would observe the following:
>>>
>>> - PMTUD already has many known problems, which is why PLMTUD is
>>> recommended instead
>>
>> I agree, operationally however I'm trying not to break existing devices
>> attempting to connect to me, is the motivation for the note.
>>
> 
> During the presentation in the v6ops session I made a comment about
> sysctl net.ipv4.tcp_mtu_probing. Since then I had a chance to test it
> - and indeed, it affects the behavior of TCP over IPv6 as well.
> 
> However, the value of 1 causes a large value of MSS to be used,
> resulting in about 1.5 second hiccup. So, yeah it "works" but for
> barely acceptable values of "works"
> 
> Setting the value to 2 causes the initial MSS to be 512, which
> "mitigates" the problem and avoided the hiccups on a 100kb file I was
> testing with - at the expense of almost 3x more packets of course.
> 
> I think it would be useful to see the discussion of this measure and
> its applicability/tradeoffs in the draft.
> 
> --a
>

---

From: joel jaeggli <[email-masked]>
Date: Mon, 01 Sep 2014 09:57:52 -0700

On 8/30/14 11:36 PM, Ray Hunter wrote:
>> Matthew Petach <mailto:[email-masked]>
...

> Not necessarily.
> 
> What might help is a realistic and early estimate of the minimum MTU 
> we're likely going to encounter on the path,. even if this is just
> used as a hint for the PLMTUD learning process that is communicated
> from the local router to local hosts.

The case where we would see this occur typically involved encapsulation,
It would be the tunnel ingress on the way to the client  that sends the
PTB message.  The client is practically speaking unaware that the MTU is
constrained (even if the immediate upstream device is aware (because it
terminates the tunnel). in that sense it may be a lot simpler to inform
the client that the mtu is constrained e.g. by setting it in the RA even
though the lan segment can in fact support something larger (e.g. rfc
4861s observation about heterogeneous networks).

You as the client could be several hops from your end of the tunnel I
suppose.

> MTU is configured on routers. Routers determine the path (based on 
> routes). Routes are based on prefixes. So there is certainly a
> relationship between MTU and prefixes.
> 
> Your concern seems to be how good that correlation is.
> 
> IMVHO I believe we could generate a single minimum MTU per path
> (based on prefixes) that is a pretty good first approximation (whilst
> avoiding blowing up the routing table).

So In my specific case I actually do have a full table on servers so
that I can do outbound path selection, per prefix/as statistics for the
application and so on, that's maybe  an unnecessary set of information
for most "servers" e.g. devices that receive incoming connections.

> In EIGRP, a lot of AS'es don't even use summary routes. So in this
> case there's a very good correlation between MTU, path, and prefix,
> at least for traffic within the AS. So I could see this working quite
> well within an individual AS, like a corporate network.
> 
> In BGP4, the path is a set of AS'es to be traversed.
> 
> The minimum MTU of an individual AS would be the minimum MTU of all 
> individual paths encountered when traversing that AS. e.g. the
> minimum of all MTUs learned in the IGP, or a static manually 
> configured number in the AS border router.
> 
> The minimum MTU of a BGP path would be the minimum of the individual
> AS MTUs.
> 
> I can see this being potentially problematic, but is it meaningfully 
> better than a single global constant (based on a minimum of the
> whole Internet)?
> 
> Or is a search for the path MTU for every host for every session
> already a good enough solution?

connections are frequently pretty short-lived I personally like to
minimize the number of times that we need to engage in negotiation when
setting up a tcp connection. fortunately as I noted this case is
actually kind of infrequent and hopefully even less common as the
network matures.

---

From: Tom Perrine <[email-masked]>
Date: Tue, 02 Sep 2014 13:22:14 -0700

What MTUs are actually (commonly) seen in the wild?

I can only think of less than a dozen that I would expect to see, 
assuming that the actual MTU is based solely on the underlying network 
technology.

Is planning and assuming that we'll see each and every possible MTU 
actually necessary?

I agree that we wouldn't want to artificially limit MTUs to only a few 
common ones, I mean look how /64s became embedded in silicon...

---

From: Tom Perrine <[email-masked]>
Date: Tue, 02 Sep 2014 13:27:01 -0700

Sorry,  total brain fart.

Encapsulation and all the other cases.

Ignore me, I do :-)

On 9/2/14 1:22 PM, Tom Perrine wrote:
> _______________________________________________
> v6ops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/v6ops

---

From: Joe Touch <[email-masked]>
Date: Tue, 02 Sep 2014 13:30:20 -0700

On 8/30/2014 10:50 AM, Ray Hunter wrote:
> Paths are learned via routing. What if Path MTU was also learned via
> routing, rather than being probed for by individual end hosts?

Not all devices that forward packets participate in routing protocols.

Using this as a hint may or may not be worthwhile, but it's at most just 
a hint.

Joe

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Thu, 4 Sep 2014 20:22:43 +0000

On Sep 3, 2014, at 6:22 AM, Tom Perrine <[email-masked]> wrote:

> I can only think of less than a dozen that I would expect to see, assuming that the actual MTU is based solely on the underlying network technology.

There are a number of magic numbers in the literature, notably the values in RFC 2470, which is for IEEE 802.5 Token Ring. As a practical matter today, I think the magic numbers are 1500 and 9000, and those numbers reduced by the sizes of various encapsulations. I’m not quite sure of the math that led us to select the number 1280, but I’ll certainly agree that any encapsulation I can readily think of would work with a packet of that size.

---

From: Jared Mauch <[email-masked]>
Date: Thu, 4 Sep 2014 18:09:59 -0400

We configure customer links at 1500 (default), 4470 and 9000.

- Jared

> On Sep 2, 2014, at 4:22 PM, Tom Perrine <[email-masked]> wrote:
> 
> Is planning and assuming that we'll see each and every possible MTU actually necessary?
> 
> I agree that we wouldn't want to artificially limit MTUs to only a few common ones, I mean look how /64s became embedded in silicon...
> 
>

---

From: joel jaeggli <[email-masked]>
Date: Thu, 04 Sep 2014 16:29:31 -0700

> We configure customer links at 1500 (default), 4470 and 9000.

in the cases that I experienced I have my mss set to 1440. So clients
with a higher MTU than 1500 are no so much of an issue. almost all of
the cases that I have have examined were due to encapsulation.

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 05 Sep 2014 12:00:37 +1200

> the cases that I have have examined were due to encapsulation.

Yes, and MSS negotiation down to 1220 doesn't always work, so
with 1280 tunnels still around, even 1440 can cause failures.

   Brian

> ------------------------------------------------------------------------
>

---

From: Fernando Gont <[email-masked]>
Date: Thu, 04 Sep 2014 20:33:26 -0300

On 09/04/2014 05:22 PM, Fred Baker (fred) wrote:
>> assuming that the actual MTU is based solely on the underlying
>> network technology.
> 
> There are a number of magic numbers in the literature, notably the
> values in RFC 2470, which is for IEEE 802.5 Token Ring. As a
> practical matter today, I think the magic numbers are 1500 and 9000,
> and those numbers reduced by the sizes of various encapsulations. I’m
> not quite sure of the math that led us to select the number 1280, but
> I’ll certainly agree that any encapsulation I can readily think of
> would work with a packet of that size.

FWIW, MTUs of around 296 bytes are reportedly quite usual in some radio
networks. That's why some operating systems enforce that as the minimum
acceptable reported MTU in ICMPv4 "frag needed but DF bit set".

What happens when/if such lins employ v6 (whether they really offer a
"virtual" MTU >=1280).. I just don't know.

Thanks,
-- 
Fernando Gont
e-mail: [email-masked] || [email-masked]
PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Fri, 5 Sep 2014 00:24:48 +0000

> "virtual" MTU >=1280).. I just don't know.

802.15.4?

---

From: Jared Mauch <[email-masked]>
Date: Thu, 4 Sep 2014 20:46:28 -0400

> On Sep 4, 2014, at 8:24 PM, Fred Baker (fred) <[email-masked]> wrote:
> 
> 

I really want to drag anyone who thinks breaking p-mtu by their improper understanding of standards to do tech support for grandmas impacted by their issues.

The big problems we have is IETF wrote down 1280 when most links were already 1500 to be ‘safe’ and allowed this broken behavior to drive average packet sizes down.  I have equipment from vendors that can’t do average packet sizes lower than 330 bytes, so I really need that higher end packet size to bring the average up as a result.

We need to make P-MTU mandatory and not enable those who build broken networks any more.

- Jared

---

From: Fernando Gont <[email-masked]>
Date: Thu, 04 Sep 2014 21:47:10 -0300

On 09/04/2014 09:24 PM, Fred Baker (fred) wrote:
>> What happens when/if such lins employ v6 (whether they really
>> offer a "virtual" MTU >=1280).. I just don't know.
> 
> 802.15.4?

No. I seem to recall that it was more of a packet radio sort of thing
(something predating 802.15.4). I could ask some folks if interested
in more details...

Thanks,
-- 
Fernando Gont
SI6 Networks
e-mail: [email-masked]
PGP Fingerprint: 6666 31C6 D484 63B2 8FB1 E3C4 AE25 0D55 1D4E 7492

---

From: joel jaeggli <[email-masked]>
Date: Thu, 04 Sep 2014 17:52:35 -0700

> with 1280 tunnels still around, even 1440 can cause failures.

sure that's kinda I find it necessary make sure the packet makes it when
i get one. I suspect even 1220 can hose you in some corner case... I'd
rather have my extra 220 bytes of pdu in any event.

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Fri, 5 Sep 2014 00:58:16 +0000

On Sep 5, 2014, at 10:47 AM, Fernando Gont <[email-masked]> wrote:

> in more details…

I think you missed my point. We have a case in point today, which is IPv6 in an IPv6 tunnel encapsulation running on a packet network whose MTU is 127 bytes. We call it “6lowpan”, and it runs on IEEE 802.15.4.

---

From: Fernando Gont <[email-masked]>
Date: Thu, 04 Sep 2014 22:50:52 -0300

On 09/04/2014 09:58 PM, Fred Baker (fred) wrote:
>>> 802.15.4?
>> 
>> No. I seem to recall that it was more of a packet radio sort of
>> thing (something predating 802.15.4). I could ask some folks if
>> interested in more details…
> 
> I think you missed my point. We have a case in point today, which is
> IPv6 in an IPv6 tunnel encapsulation running on a packet network
> whose MTU is 127 bytes. We call it “6lowpan”, and it runs on IEEE
> 802.15.4.

I expected 6lowpan to offer a virtual MTU >= 1280 (don't they), but
older stuff to not take care of that -- hence my comment.

---

From: Ray Hunter <[email-masked]>
Date: Fri, 05 Sep 2014 10:07:22 +0200

> the cases that I have have examined were due to encapsulation.
Which would seem to assume that everyone is happy with "eyeball 
reachability".

In which case 1280 will almost certainly become de facto hard(ware) 
coded, and anything longer is only going to be used for encapsulation 
mechanisms.

I'm also interested in the people who could significantly benefit from 
larger MTU's all the way to the end device (for performance reasons) 
e.g. FCIP.
>> - Jared
>>
>>> On Sep 2, 2014, at 4:22 PM, Tom Perrine<[email-masked]>  wrote:
>>>

-- 
Regards,
RayH

---

From: Joe Touch <[email-masked]>
Date: Fri, 05 Sep 2014 06:59:46 -0700

On 9/4/2014 5:58 PM, Fred Baker (fred) wrote:
> IPv6 in an IPv6 tunnel encapsulation running on a packet network whose
> MTU is 127 bytes. We call it “6lowpan”, and it runs on IEEE 802.15.4.

That's not IPv6. I don't know what it is, but RFC2460 is clear on the 
minimum MTU for IPv6.

Anyone who pays money for such a thing should return it.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 5 Sep 2014 14:48:06 +0000

> -----Original Message-----
> From: v6ops [mailto:[email-masked]] On Behalf Of Joe Touch
> Sent: Friday, September 05, 2014 7:00 AM
> To: Fred Baker (fred); Fernando Gont
> Cc: [email-masked]; Tom Perrine; Fernando Gont
> Subject: Re: [v6ops] PMTUD issue discussion
> 
> 
> 
> > MTU is 127 bytes. We call it "6lowpan", and it runs on IEEE 802.15.4.
> 

Joe is right - for IPv6 there is only one magic number (1280). Beyond
that, any size up to and including jumbograms must be accommodated,
subject to path MTU limitations.

Thanks - Fred
[email-masked]

---

From: Owen DeLong <[email-masked]>
Date: Fri, 5 Sep 2014 11:50:19 -0700

On Sep 5, 2014, at 6:59 AM, Joe Touch <[email-masked]> wrote:

> That's not IPv6. I don't know what it is, but RFC2460 is clear on the minimum MTU for IPv6.
> 

It is, actually, IPv6 for Low Power Area Networks and it does provide a “virtual MTU” of 1280 or better at layer 2 in order to facilitate moving IPv6 data over those networks. It does, however, provide additional optimizations to support the low power applications where these networks are common.

I recommend reading the RFC before saying that 6lowpan isn’t IPv6.

Owen

---

From: Joe Touch <[email-masked]>
Date: Fri, 05 Sep 2014 12:07:02 -0700

Let me be more clear:

> I recommend reading the RFC before saying that 6lowpan isn’t IPv6.

Anything that doesn't support an IPv6 MTU of at least 1280 isn't IPv6.

That requirement has no impact on the L2 MTU, but the L2 must never 
return an ICMP error of "too big" for IPv6 MTUs that are valid.

Joe

---

From: Owen DeLong <[email-masked]>
Date: Fri, 5 Sep 2014 14:16:50 -0700

> Anything that doesn't support an IPv6 MTU of at least 1280 isn't IPv6.

Correct. 6lowpan is IPv6 with an IPv6 MTU of 1280 or better over radio links that have a 127 octet MTU.

> That requirement has no impact on the L2 MTU, but the L2 must never return an ICMP error of "too big" for IPv6 MTUs that are valid.

Also correct, sort of. The requirement does have impact on L2 devices/media/etc. that have an MTU that doesn’t allow a ≥1280 octet L3 payload in that it requires L2 SAR or an L2 adaptation layer that provides SAR. The 6lowpan specification provides such SAR and is also being adapted to allow 6lowpan over BLE.

Owen

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 06 Sep 2014 13:41:25 +1200

> subject to path MTU limitations.

Actually jumbograms, according to RFC 2675, are only available
when the *link* MTU is big enough:

"  The Jumbo Payload option is relevant only for IPv6 nodes that may be
   attached to links with a link MTU greater than 65,575 octets (that
   is, 65,535 + 40, where 40 octets is the size of the IPv6 header).
   The Jumbo Payload option need not be implemented or understood by
   IPv6 nodes that do not support attachment to links with MTU greater
   than 65,575."

Note that it says "nodes", so it applies to routers as well as hosts.
My recollection is that efficent use of HIPPI within data centres
was the main motivation for jumbograms, and there is no expectation
that they will work on the Internet in general. Running jumbograms
over an adaptation layer would be craziness.

Also
"  The Jumbo Payload option must not be used in a packet that carries a
   Fragment header."

So I think it's "any size up to 65,575 octets" that must be
accomodated. Jumbograms are optional.

  Brian

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 8 Sep 2014 18:47:45 +0000

Hi Brian,

> From: Brian E Carpenter [mailto:[email-masked]]
> Sent: Friday, September 05, 2014 6:41 PM
> To: Templin, Fred L
> Cc: Joe Touch; Fred Baker (fred); Fernando Gont; [email-masked]; Tom Perrine; Fernando Gont

The key phrase in my text that you quoted was "subject to path MTU
limitations". For example, if the source's IPv6 layer presents a 4MB
packet, and if all links in the path support an MTU that is at least
that large, then the packet will be forwarded to the destination just
fine. If the path MTU is deficient at any hop along the path, however,
the packet is dropped and a PTB message returned.

We may be a long way off from having paths in the Internet (or even
in private networks) that supports such large sizes, but the protocols
will be ready when and if such links come online.

Thanks - Fred

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 09 Sep 2014 10:11:33 +1200

Fred,

> will be ready when and if such links come online.

The key phrase in my comment was "Jumbograms are optional" (which
is explicit in RFC 6434). So the only Internet-wide expectation
is a maximum of 65,575, and that is the largest size that can ever
be fragmented.

   Brian

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 8 Sep 2014 22:45:39 +0000

Hi Brian,

> Sent: Monday, September 08, 2014 3:12 PM
> To: Templin, Fred L
> is explicit in RFC 6434).

OK, thanks for that reference - much appreciated.

> So the only Internet-wide expectation

Of that, the specs leave no doubt.

Thanks - Fred

---

From: Mikael Abrahamsson <[email-masked]>
Date: Tue, 9 Sep 2014 09:56:47 +0200 (CEST)

> What MTUs are actually (commonly) seen in the wild?

My reply is late, but let me write some of my experience as I did some 
investigation into what's possible. All values presented here are IP MTU, 
not L2 MTU. Most people mix these values and I've seen a lot of broken 
deployments due to this.

The major end system IP MTU seen in the world is 1500 (huge majority).

The path MTUs seen might be slightly under due to PPPoE, GRE, GTP etc 
encapsulations on the access layer.

Going larger than this (speaking both of path MTU and end system MTUs), 
most opt for 4470, 9000 or 9180 due to older protocols such as AAL5 ATM, 
SMDS, FDDI, POS etc. Unfortunately Juniper MX doesn't support 9180, but 
instead one has to choose one value of 9178, 9174 or 9170 depending on 
number of vlan tags (or MPLS labels I guess) one wants to support, so 
these are also seen.

Personally I think the balanced approach would be to make your core 
support something around the 9180 figure, and give 9000 to end users.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 9 Sep 2014 14:38:29 +0000

Hi Mikael,

> From: v6ops [mailto:[email-masked]] On Behalf Of Mikael Abrahamsson
> Sent: Tuesday, September 09, 2014 12:57 AM
> To: Tom Perrine
> Cc: [email-masked]

Right; the vast majority of end systems currently see 1500.

> encapsulations on the access layer.

Which can be problematic when PMTUD is not functioning correctly. 

> support something around the 9180 figure,

Fully support the core going to larger MTUs.

> and give 9000 to end users.

This could be problematic when PMTUD is not functioning correctly.
As a result, IMHO any end system that wants to try for packet sizes
larger than 1500 really should be using RFC4821 (maybe even MUST?).

Thanks - Fred

---

From: Mikael Abrahamsson <[email-masked]>
Date: Tue, 9 Sep 2014 17:07:43 +0200 (CEST)

On Tue, 9 Sep 2014, Templin, Fred L wrote:

> This could be problematic when PMTUD is not functioning correctly. As a 
> result, IMHO any end system that wants to try for packet sizes larger 
> than 1500 really should be using RFC4821 (maybe even MUST?).

Well, even that might not solve it as there might be devices along the 
path that re-assemble the packets (if they're fragmented) and then 
fragment them again, which will break things anyway.

But yes, supporting 4821 or PMTUD blackhole detection should definitely be 
a requirement.

One problem is that a lot of people don't even know that sending a 16k UDP 
packet fragmented into a bunch of smaller fragments, is a perfectly valid 
use of IP. They're so used to TCP and its MSS that this is all they've 
seen.

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 9 Sep 2014 15:45:55 +0000

Hi Mikael,

> From: Mikael Abrahamsson [mailto:[email-masked]]
> Sent: Tuesday, September 09, 2014 8:08 AM
> To: Templin, Fred L
> Cc: Tom Perrine; [email-masked]
> Subject: RE: [v6ops] PMTUD issue discussion
> 

I think RFC4821 only applies to whole packets with DF=1 (i.e., where
all IPv6 packets implicitly have DF=1), but I get your point. 

> But yes, supporting 4821 or PMTUD blackhole detection should definitely be

OK.
 
> use of IP. They're so used to TCP and its MSS that this is all they've

Yes, isolated large UDP packets with DF=0 must be dealt with.

Thanks - Fred

---

From: Mark Andrews <[email-masked]>
Date: Wed, 10 Sep 2014 07:19:47 +1000

> Yes, isolated large UDP packets with DF=0 must be dealt with.

Actually there is plenty of fragmented UDP traffic.  Pick just about
any IPv6 DNSSEC aware recursive nameserver and you will almost
certainly see fragmented IPv6 UDP responses.  This traffic is only
increasing as more zones become signed pushing response size well
over 1500.

Below is from BIND 9.10 which tracks UDP response sizes.  Find a
busy resolver and you will find a lot more.  The EDNS field shows
timeouts at various EDNS UDP buffer sizes.  A timeout at a small
size is counted against larger sizes as well as named tries to
workout the DNS/UDP mtu which can be as low as 512 as some firewalls
"know" that DNS/UDP payloads "never" exceeded 512 bytes.

This nameserver is behind a HE tunnel so it also has to deal with
authorative nameservers that block PTB to them.  Named also forces
fragmentation at 1280 on all sockets if it can.  The savings for a
nameserver in sending bigger packets just aren't there.

[edns success/4096 timeout/1432 timeout/1232 timeout/512 timeout]

[rock:~/ports] marka% grep srtt /var/named/named_dump.db | grep udpsize | grep -v "udpsize 512" | grep :
;	2001:500:31::63 [srtt 119657] [flags 00806000] [edns 2/0/0/0/0] [plain 0/0] [udpsize 965] [ttl 1230]
;	2001:500:2c::254 [srtt 214672] [flags 00806000] [edns 2/0/0/0/0] [plain 0/0] [udpsize 1676] [ttl 1222]
;	2001:13c7:7002:3000::11 [srtt 242467] [flags 00806000] [edns 2/0/0/0/0] [plain 0/0] [udpsize 513] [ttl 1229]
;	2001:500:60::30 [srtt 229183] [flags 00806000] [edns 4/0/0/0/0] [plain 0/0] [udpsize 1816] [ttl 1221]
;	2001:4f8:0:2::19 [srtt 82726] [flags 00806000] [edns 2/0/0/0/0] [plain 0/0] [udpsize 1626] [ttl 1221]
;	2001:500:13::108 [srtt 115528] [flags 00806000] [edns 2/0/0/0/0] [plain 0/0] [udpsize 1991] [ttl 1229]
[rock:~/ports] marka% 

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 10 Sep 2014 14:33:42 +0000

Hi Mark,

> From: Mark Andrews [mailto:[email-masked]]
> Sent: Tuesday, September 09, 2014 2:20 PM
> To: Templin, Fred L
> Cc: Mikael Abrahamsson; [email-masked]; Tom Perrine

It is great to have this data. It agrees with what Mikael said, and
also agrees with what I was thinking but unfortunately did not say
correctly. Strike what I said above, and change:

=> Yes, isolated large UDP packets with DF=0 must be dealt with.

to:

=> Yes, fragmented large UDP packets must be dealt with.

Thanks - Fred