Subject: [RRG] Perplexing PMTUD and packet length observations

From: Robin Whittle <[email-masked]>
Date: Tue, 12 Aug 2008 02:55:49 +1000

I expected that most hosts would send packets with DF=1, using RFC
1191 PMTUD, since it has been around since 1990 . . . and that it
would be considered antisocial to send any longish packets into the
Net with DF=0, saying the Network should do extra work if the
packets are too long for it.

In fact, some servers - including at least some Google servers, send
only DF=0 packets, and therefore never do any PMTUD.  They won't be
tempted by the client having a higher MSS than their own 1430, but
as long as the client's MSS is this or above, the server frequently
sends 1470 byte packets with DF=0.
 http://www.firstpr.com.au/ip/ivip/ipv4-bits/actual-packets.html#google-no-pmtud

I find this interesting:

  1 - I guess Google figure 1470 is a packet size which generally
      works in the core.

  2 - I guess they figure that if the client advertises an MSS and
      when 40 (IP and TCP headers) is added to that figure if the
      result is larger than any MTU limit between the client and
      Google's servers, then it is the client's problem.

If lots of hosts are sending long packets with DF=0, we need to cope
with then in any map-encap scheme.

We don't want to have to fragment these packets before
encapsulation, or after, but we have no way of telling the host to
send smaller ones.

Another perplexing discovery is that my web server in Texas does
ordinary TCP SYN handshakes with MSS values such as 1460 from both
sides and then quite merrily cranks up to packets of 7260 bytes and
longer.

http://www.firstpr.com.au/ip/ivip/ipv4-bits/actual-packets.html#jumbo1

I have no idea why this occurs, but at least it does it all with
DF=1.  These packets are received and acknowledged by clients who
are presumably on some kind of xDSL service.  So jumboframes are
alive and well in the core.

   - Robin

--
to unsubscribe send a message to [email-masked] with the
word 'unsubscribe' in a single line as the message text body.
archive: <http://psg.com/lists/rrg/> & ftp://psg.com/pub/lists/rrg

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 11 Aug 2008 23:39:46 +0200

On 11 aug 2008, at 18:55, Robin Whittle wrote:

> 1191 PMTUD, since it has been around since 1990 . . .

Most hosts do. It's enabled by default on Win, Mac, Linux and FreeBSD.  
Probably more.

> and that it
> Net with DF=0

Why?

> saying the Network should do extra work if the

Well, it's better than setting DF and ignoring the ICMP too bigs, like  
some people do (see recent thread on NANOG).

> sends 1470 byte packets with DF=0.

Hm, didn't know that, but yes, that's what I'm seeing too.

>  1 - I guess Google figure 1470 is a packet size which generally

Or they use a 30 byte encapsulation somewhere.

>      Google's servers, then it is the client's problem.

??? Why would advertising a large MSS be a problem? You send what the  
other advertises he/she can handle and obviously _they_ will be  
sending you what they can handle.

Should try with 1500 byte DF=1 packets just for kicks, though.

> with then in any map-encap scheme.

Like I've been saying for a long time, reducing the user-visible MTU  
of the internet is not an acceptable approach.

> sides and then quite merrily cranks up to packets of 7260 bytes and

Hm, could it be that this is happening between two hosts on the same  
subnet? I gather some systems ignore the MSS if the traffic is deemed  
"local".

BTW: if you haven't seen it before, have a look at this before it  
expires two weeks from now:

http://www.ietf.org/internet-drafts/draft-van-beijnum-multi-mtu-02.txt

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 11 Aug 2008 15:35:22 -0700

>> with then in any map-encap scheme.

Lots of hosts sending long packets with DF=0 does not
necessarily cause problems for ITRs/ETRs, if that's what
you mean. As long as the ITR configures a large-enough
MTU on its Internet-facing interface, then it won't be
stuck having to do a lot of fragmentation - but some
other router(s) further down the road might be. And,
the ETR will not be responsible for reassembling; the
final destination will.

>of the internet is not an acceptable approach.

SEAL fixes that. What unacceptable is a mechanism that
requires reliable and secure delivery of ICMPs from
anonymous nodes on the Internet.

Fred
[email-masked]

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 12 Aug 2008 09:40:01 +0200

> anonymous nodes on the Internet.

Well, nothing on the internet is reliable, and very few things are  
secure.

That being said, ICMP too big messages are ESSENTIAL and MUST NOT be  
filtered! Any other message from the IETF is unacceptable.

The first mistake was to invent the DF bit in the first place.

The second mistake is to suggest that the DF bit be set for ALL  
packets to do PMTUD in RFC 1191.

[omitting a number of mistakes]

I'm not sure if implicitly making IPv6 packets unfragmentable was a  
mistake, but relying on ICMP messages was.

In any event, all of these mistakes have been made and we aren't even  
close to cleaning up the mess with stuff like RFC 4821, so now we have  
to live with that, which means, among other things, taking very good  
care of ICMP too big messages.

</rant>

---

From: Robin Whittle <[email-masked]>
Date: Tue, 12 Aug 2008 20:01:47 +1000

Hi Iljitsch,

Before I reply to what you wrote, on the matter of my server sending
out jumboframe packets, with DF=1, and them being received correctly
by clients on DSL services:

I was able to reproduce the behavior and have documented it clearly
here:

http://www.firstpr.com.au/ip/ivip/ipv4-bits/actual-packets.html#2008-08-12

I just updated to a recent kernel 2.6.18-92.1.10.el5 and the
behavior continues. (This is Centos 5.1, so maybe we are seeing
something Red-Hat specific?)

My guess is the kernel is bundling the data of multiple full MSS
length TCP packets produced by httpd and emitting them as single
packets 2, 3, 4 etc. to 6 times as long.  I haven't yet found out
what aspect of the kernel (or httpd?) is doing this.  To me, it is
very surprising behavior - and violates the TCP protocol by sending
packets way longer than the MSS limit.

You mentioned some systems sending longer than MSS TCP packets to
hosts on the local network.  Can you recall what this feature might
be called?

These jumbo frames presumably reach the router at the ISP end of the
PPPoE DSL link, and that router does something I found surprising,
and again at odds with my understanding of what is righteous and
proper:  It splits up the data from this one long TCP packet and
sends it, (with sequentially numbered Identification fields) as
separate TCP packets to the destination.  These are not fragments -
they are standalone TCP packets.

It works fine for TCP - but would not for UDP or TCP with IPsec
Authentication Header.  I don't like my own server and DSL link
doing things I don't understand and can't find any documentation for.

You wrote:

>> I expected that most hosts would send packets with DF=1, using
>> RFC 1191 PMTUD, since it has been around since 1990 . . .
>
> Most hosts do. It's enabled by default on Win, Mac, Linux and
> FreeBSD. Probably more.

Yes.

>> and that it would be considered antisocial to send any longish
>> packets into the Net with DF=0
>
> Why?
>
>> saying the Network should do extra work if the packets are too
>> long for it.

When RFC 1191 has been around since 1990 and provides a civilised
way of sending large packets, without expecting some router in the
path to fragment packets and the rest of the routers to have to deal
with twice the number of packets.

> Well, it's better than setting DF and ignoring the ICMP too bigs,
> like some people do (see recent thread on NANOG).

That would be antisocial and stupid!

I couldn't easily find the thread at:

  http://www.merit.edu/mail.archives/nanog/

>> In fact, some servers - including at least some Google servers,
>> send only DF=0 packets, and therefore never do any PMTUD.  They
>> won't be tempted by the client having a higher MSS than their own
>> 1430, but as long as the client's MSS is this or above, the
>> server frequently sends 1470 byte packets with DF=0.
>

OK - my page has test cases.

> Or they use a 30 byte encapsulation somewhere.

Maybe so, but I guess they are not so irresponsible as to be
systematically sending out DF=0 packets which were causing
widespread fragmentation.

> ??? Why would advertising a large MSS be a problem? You send what
> the other advertises he/she can handle and obviously _they_ will
> be sending you what they can handle.

Yes, but what if, for some reason, there is a router in the path
with a smaller MTU than is generally seen by the client or by Google?

>> If lots of hosts are sending long packets with DF=0, we need to
>> cope with then in any map-encap scheme.
>
> Like I've been saying for a long time, reducing the user-visible
> MTU of the internet is not an acceptable approach.

I have figured out how to do ITR - ETR packet delivery without
encapsulation or address rewriting for IPv6:

  http://www.firstpr.com.au/ip/ivip/ivip6/

and yesterday figured out a somewhat similar scheme for IPv4.  I
will write them up as Internet drafts ASAP.  The both involve
changes to the use of header bits, and fairly minimal modifications
to core routers.  That makes these schemes much harder to deploy,
but there would be no PMTUD problems or lost bandwidth due to
encapsulation - which are both enormous benefits, including much
less complexity in ITRs and ETRs.

> http://www.ietf.org/internet-drafts/draft-van-beijnum-multi-mtu-02.txt

OK.  I was intrigued by the list of common MTU values you list.
Googling reveals the source:

http://www.ietf.org/mail-archive/web/ram/current/msg01854.html

in which you wrote:

> I've looked over some product literature and compiled the
> following list: 1508, 1530, 1536, 1546, 1998, 2000, 2018, 4464,
> 4470, 8092, 8192, 9000, 9176, 9180, 9216, 17976, 64000 and 65280.

I guess it makes sense to have mechanisms by which a host can
discover some MTU values for nearby routers and other hosts.

I am trying to do something different - find methods of delivering
packets for an Edge-Core Separation solution to the routing scaling
problem which doesn't upset conventional RFC 1191 PMTUD.

> ... ICMP too big messages are ESSENTIAL and MUST NOT be

I agree absolutely.

I think there is no workable alternative to RFC 1191 PMTUD.

RFC 4821 is so difficult to implement, since it involves the
packetisation layers of all UDP applications talking back and forth
with the operating system, and with the TCP packetisation layer, all
comparing notes giving and sharing information.

It would be a nightmare to debug, and involves flows of information
about specific hosts in ways which I guess are not normally done
inside an operating system.

How is the system to respond to packet loss - not caused by MTU
problems, but which is indistinguishable from loss caused by MTU
problems?  The estimate would have to be reduced - clobbering all
applications with shorter packets, just because a few were lost, or
their acknowledgement packets were lost.

RFC 4821 sounds too fluffy - and all too difficult - with only
marginal benefits accruing if all the apps do it well and the OS
integrates and shares the information well.

What is to stop a bad application from failing in a way it thinks is
an MTU problem, dragging all the other apps down to using a short
packet length?

> The first mistake was to invent the DF bit in the first place.

I guess you mean that all packets should always have been
non-fragmentable and that something like RFC 1191 should always have
been in existence.

> packets to do PMTUD in RFC 1191.

I don't understand your objection.

Removing fragmentation from the network is a really good aspect of
IPv6, I think.  Ideally, I think, all packets should be sent DF=1
and all applications should be ready to cope if their packets
(longer than some basic assumed minimum length) generate ICMP PTB
messages.  The trick is to be able to securely receive ICMP PTB
messages.  Without a nonce, this gets messy, and relies on the ICMP
code keeping records of recently sent packets, or at least the start
of them.  How recent?  How many bytes to store?  There seems to be
no hard answers.

It is difficult, but what else is there, since the MTU could change
at any time.  We can't be sending big probe packets every 30 seconds
in case the MTU drops.  We can't assume that lack of acknowledgement
is an MTU problem - it could be a temporary outage or destination
host going dead for a moment.  So it would be silly adjusting down
MTUs and trying to send smaller packets and smaller packets, until
when an ACK arrives we decide the problem must have been the PMTU
dropping!

The only reasonable solution seems to be send all packets DF=0 and
expect all routers to report PMTU troubles with a PTB message.
Networks which block PTB packets are doing themselves and anyone who
connects to them a grave disservice.

> I'm not sure if implicitly making IPv6 packets unfragmentable was

Do you suggest some other kind of message, or do you think PMTUD
should be done on the basis of positive acknowledgements alone, with
silent discarding of a too-big packet at whichever router can't
handle it?

> In any event, all of these mistakes have been made and we aren't
> even close to cleaning up the mess with stuff like RFC 4821,
> so now we have to live with that, which means, among other things,
> taking very good care of ICMP too big messages.

Google:   No results found for "RFC 4821 deployment".

Blocking ICMP PTB messages is a dumb thing to do.  I understand
people wanting to block ping or ping response packets - but I think
that PTBs are the best and probably the only practical method for
doing PMTUD.

 - Robin

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 12 Aug 2008 13:28:10 +0200

> I was able to reproduce the behavior and have documented it clearly

Ok, so this is not the same subnet, right? Note that if you feed  
tcpdump a few -v's you don't have to do as much header decoding in  
your head.

What it looks like to me is that you're actually tcpdumping rather  
than ipdumping: what you see is an initial two segment transmission  
but as a single packet. Could it be that you're tcpdumping some  
virtual interface rather than real packets on the wire? Can you  
capture ethernet headers? Maybe some work is being offloaded to the  
NIC? If not, I'd say that all of this is a bug in the linux networking  
code (which is weird to begin with) but I have no explanation about  
why you would be seeing normal size packets without fragmentation. I'm  
pretty sure ISPs wouldn't want to expend CPU cycles to do this on  
behalf of their hosted customers...

(BTW, I thought having a server 1600 km away was impressive...)

> hosts on the local network.  Can you recall what this feature might

No.

>  http://www.merit.edu/mail.archives/nanog/

Look for "microsoft".

> with a smaller MTU than is generally seen by the client or by Google?

MSS is end-to-end, you still need PMTUD or fragmentation.

> I think there is no workable alternative to RFC 1191 PMTUD.

What they should have done back then was create a mechanism that  
allows the receiver of fragments to tell the sender that the packet  
was fragmented and what the size of the largest fragment was. This  
would have been harder to deploy (changes on both ends) but more robust.

> RFC 4821 is so difficult to implement

Indeed. Still, it probably has to be done at some point, especially if  
we ever want to move away from 1500 as the internet's maximum packet  
size.

> non-fragmentable and that something like RFC 1191 should always have

No: if you have fragmentation anyway, there is no reason to have a  
source say it can't be done. It would arguably be useful for the  
destination to say that, but this isn't what DF does so before RFC  
1191 came along it was useless.

> I don't understand your objection.

Set it only for 10% of your packets and you still have connectivity  
when there is a black hole and the PMTUD works just fine.

> and all applications should be ready to cope

No. This is a layer 3 job, not a layer 7 job.

An interesting approach would be to simply truncate packets that are  
too big rather than fragment or drop them. A difference between the IP  
length field and the actual length of the packet indicates truncation.  
Transports would have to be changed to semi-ACK truncated data so the  
sender only retransmits a checksum over the semi-ACKed data after  
which a full ACK/NAK is possible. The semi-ACK also implicitly signals  
the maximum path MTU.

> expect all routers to report PMTU troubles with a PTB message.

You mean DF=1?

DF=0 is not a solution for IPv6...

> connects to them a grave disservice.

Yes, but they've been getting away with it so far because "everyone"  
supports a 1500-byte MTU. So now breaking _that_ assumption creates  
problems.

> silent discarding of a too-big packet at whichever router can't

With IPv6, it would have been possible to come up with a truncation  
approach or maybe something where routers write a maximum packet size  
in certain packets.

But now the only way forward is RFC 4821 etc while working hard to fix  
PMTUD black holes until 4821 is widely implemented.

> Google:   No results found for "RFC 4821 deployment".

Yeah, none for "RFC 791 deployment" either...

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 12 Aug 2008 06:27:43 -0700

>> and all applications should be ready to cope

As with declaring fragmentation "harmful", removing
in-the-network fragmentation in IPv6 was a black-and-white
decision point that doesn't match well with the real world.

In-the-network fragmentation is useful as long as the
endpoints have a way of knowing that it is happening.
In-the-network fragmentation provides a means for the
endpoints to sense that something is out of tune and
then make the necessary adjustments to tune it out.

Fred

---

From: Robin Whittle <[email-masked]>
Date: Tue, 12 Aug 2008 23:55:43 +1000

Hi Iljitsch,

> tcpdump a few -v's you don't have to do as much header decoding in

Please take a look at my page, which documents the situation
clearly, complete with the commands I give tcpdump:

> than ipdumping: what you see is an initial two segment
> transmission but as a single packet. Could it be that you're
> tcpdumping some virtual interface rather than real packets on the
> wire?

A complete packet dump, with lengths easily visible in a different
colour, is:

  http://www.firstpr.com.au/ip/ivip/ipv4-bits/jumbo-mss.ht << ml

     (Type in the "ml" at the end manually, I don't want links
      to this big file.)

This is the full hex dump of the packets - the TCP handshake, the
HTTP request, the data packets and the ACKs.  They all have full TCP
headers, checksums etc.

If you search for 7260 you will see the longest packet - carrying 5
times the MSS amount of data.  If you look at its Identification
field (3rd block of hex) it is 3d47.  In the next outgoing packet,
the Identification field is 5 more: 3d4c:

> Can you capture ethernet headers?

Yes.  Just add 't' to this URL:

http://www.firstpr.com.au/ip/ivip/ipv4-bits/with-ethernet-headers.tx

I just did this - it is a capture of a similar packet transfer to a
client here at home.  This shows an ACK packet and in response, just
13 microseconds later, a jumboframe going out.

    07:15:57.497378 IP 72.36.140.10.80 > 150.101.162.123.3941: .
                    18357:25417(7060) ack 634 win 6963
	0x0000:  0012 807c 117f 0015 609b 0c04 0800 4500
	0x0010:  1bbc 244c 4000 4006 ede0 4824 8c0a 9665
        ...
	0x1ba0:  af6b 6664 0dcc 168e a084 dc28 a81b efb6
	0x1bb0:  26ac 8aeb 4149 413c dba6 f5e8 0e1a 5d2d
	0x1bc0:  bc10 7fa0 5d33 0310 695c

The IPv4 packet is:

         Length = 7100 bytes
         ||||
    4500 1bbc 244c 4000 4006 ede0 4824 8c0a
                   |
                   DF=1

The TCP segment size is 7060 - exactly 5 times the lower of the two
MSS values - 1412.

This section:

shows the timing of outgoing packets with respect to the ACKs which
seem to prompt them.  Two of the packets are 8512 bytes.  This is a
TCP segment of 6 x 1412 = 8472.

> Maybe some work is being offloaded to the NIC?

That wouldn't fit with the complete IP header for the whole
jumboframe, or the ethernet packet dump.  I am sure this is a true
record of the physical packet leaving the machine.

> If not, I'd say that all of this is a bug in the linux networking
> code (which is weird to begin with)

I can't imagine it is a bug.  It is conceivable it is http doing
this, but the short turnaround time between each ACK arriving the
the large packet being sent out makes me think there is something in
the kernel which is bunching together the contents of TCP packets
created by httpd, and then having a go at firing them out to the
Net, with DF=1 - presumably being able to fire the same stuff out in
smaller, or even normal sized, TCP packets if it gets a PTB.

> but I have no explanation about why you would be seeing normal
> size packets without fragmentation. I'm pretty sure ISPs wouldn't
> want to expend CPU cycles to do this on behalf of their hosted
> customers...

I have no explanation for either of these things - the server
bundling together TCP data in flagrant violation of the RFCs as I
understand them, and (as best I can guess) the PPPoE router taking
it upon itself to recreate the individual RFC conformant TCP packets.

> (BTW, I thought having a server 1600 km away was impressive...)

Its 14,500km or so (9,000 miles) from Melbourne to Dallas Fort
Worth.  Editing text files on the server involves my keystroke going
through a planet thickness of quartz to the server, and the new
character coming back the same way, all in about 230ms.

>>  http://www.merit.edu/mail.archives/nanog/

Its not in the subject lines, and there is no search facility.

>>> ??? Why would advertising a large MSS be a problem? You send
>>> what the other advertises he/she can handle and obviously _they_
>>> will be sending you what they can handle.
>>

Yes - and with Google sending out large packets with DF=0, it is
expecting any hapless router in the middle, with a lower next hop
MTU than this length, to do a lot of work without complaint.

> allows the receiver of fragments to tell the sender that the
> packet was fragmented and what the size of the largest fragment
> was.
>
> This would have been harder to deploy (changes on both ends) but
> more robust.

OK - so packets would not be dropped, just chopped into two or more
fragments, which themselves might be fragmented too.

Then you rely on the destination host to tell the sender about the
fragmentation, rather than the router which fragments.

I think this is a lot of work for the router.

Better to drop the too-big packet and send a PTB.  That would be
faster and so bring forward the time when the sending host creates
packets which are suitable for the whole path.  RFC 1191 has the
router send the exact MTU, which is better than what I think you are
suggesting, since the destination host wouldn't be able to tell what
the MTU limit was which caused the fragmentation.  That would leave
the sending host no option but to use trial and error - sending more
not quite so long packets which would have a high chance of being
fragmentted too.

I think RFC 1191 is a better approach than what I understand of your
suggestion.

> Indeed. Still, it probably has to be done at some point,
> especially if we ever want to move away from 1500 as the
> internet's maximum packet size.

We should all stop burning fossil fuel at some point too.

I don't see a problem with the widespread use of 1500 byte MTU gear
being incompatible with RFC 1191.

It would be nice if the sending host had a better clue about the
outside world than the simple fact that its Ethernet link has an MTU
of 9k or so.  This is pretty dumb, and it might involve each session
sending a 9k packet towards the destination host, where some poor
1500 MTU next hop router goes "Not again . . . " and sends back a
PTB for the millionth time.

As long as most of the PTBs get back to the RFC 1191 compliant
sending host, I think it will work fine.

However, designing a map-encap system which does not completely
disrupt this in terms of MTU limits between the ITR and ETR is very
challenging.

>> non-fragmentable and that something like RFC 1191 should always
>> have  been in existence.
>

I can't understand why the PTB message was first defined without
also including the MTU value which the packet's length exceeded.

It seems like such a no-brainer - and the RFC states that the MTU
Discovery Working Group spents months reinventing what "was first
suggested by Geof Cooper, who in two short paragraphs set out all
the basic ideas".

> Set it only for 10% of your packets and you still have
> connectivity when there is a black hole and the PMTUD works just
> fine.

OK - so routers would fragment 90% of the packets and the PTB only
goes back when one of the 10% of packets has its DF flag set?

That just seems to slow down the sending host's response to the MTU
 situation.

I still like RFC 1191 better.  There's no fragmentation and the
sending host gets the fastest possible feedback that it needs to
send smaller packets.

That fragmented situation is less reliable than when the packets
sail straight through, so I think it is a good thing get the sending
host properly adapted to the PMTU ASAP.

>> Removing fragmentation from the network is a really good aspect
>> of IPv6, I think.  Ideally, I think, all packets should be sent
>> DF=1 and all applications should be ready to cope
>

I don't understand this.

> An interesting approach would be to simply truncate packets that
> are too big rather than fragment or drop them. A difference
> between the IP length field and the actual length of the packet
> indicates truncation.
>
> Transports would have to be changed to semi-ACK truncated data so
> the sender only retransmits a checksum over the semi-ACKed data
> after which a full ACK/NAK is possible.

I don't clearly understand this either - but it sounds messy.

> The semi-ACK also implicitly signals the maximum path MTU.

Yes, this would have the advantage that if there were a series of
narrower bottlenecks - 1400, 1300, 1170 - that in a single
round-trip time the sending host would know the full PMTU to the
destination host.  With RFC 1191, this would take three round trip
times.  Unfortunately, the surviving packet fragment isn't much use
to the destination host, so it still takes 1.5 RTTs to get the data
there.  Still, that is better than 3.5 RTTs with RFC 1191.

>> The only reasonable solution seems to be send all packets DF=0
>> and expect all routers to report PMTU troubles with a PTB
>> message.
>
> You mean DF=1?

Oops - Yes.

>> Networks which block PTB packets are doing themselves and anyone
>> who connects to them a grave disservice.
>
> Yes, but they've been getting away with it so far because
> "everyone" supports a 1500-byte MTU. So now breaking _that_
> assumption creates problems.

My understanding of this is that if all hosts have a next hop MTU of
1500, and the core has an MTU of 9000, then it is no problem if the
destination network blocks PTBs from leaving that network since no
host would be sending packets bigger than 1500 anyway.

But plenty of servers - probably most by now - have gigabit ethernet
 and so have a real PMTU for most of the core, and into quite a few
edge networks, of 9k or so.

When they send a packet to some edge network with 1500 MTU links,
which blocks the PTBs which should go back to the sending host, then
there is a black hole.  It gets messier with various hosts having
various next hop and nearby MTU limits in the ~1460 to 1500 range,
various host settings and various ICMP-blockig destination networks
with their own ~1460 to 1500 MTU limits.

I guess the majority of websites now can send jumboframes, like my
server can.  But my server offers an MSS of 1460 for a packet size
of 1500.  Then, if an ICMP-filtering edge network has a 1500 MTU,
there is no problem.  I am not sure where the MSS is configured.

But for reasons unknown, my server is trying its luck with DF=1
jumboframes way longer than 1500 or the MSS from the client.

I would think that this strategy would come unstuck with an
ICMP-filtering edge network with a 1500 MTU - unless this
"TCP bundling" facility also disabled itself in the absence of ACKs.

If this mysterious system did come unstuck due to networks blocking
PTBs, then I would get complaints that people couldn't access some
things (larger files) on my website - but there are no such complaints.

>>> I'm not sure if implicitly making IPv6 packets unfragmentable
>>> was mistake, but relying on ICMP messages was.
>
>> should be done on the basis of positive acknowledgements alone,
>> with silent discarding of a too-big packet at whichever router
>> can't handle it?
>
> With IPv6, it would have been possible to come up with a
> truncation approach or maybe something where routers write a
> maximum packet size in certain packets.
>
> But now the only way forward is RFC 4821 etc while working hard to
> fix PMTUD black holes until 4821 is widely implemented.
>

Touché - but where is the evidence of applications and operating
systems actually implementing RFC 4821?  Is there any site, any
working group or whatever where this is discussed?

 - Robin

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 12 Aug 2008 17:23:58 +0200

> I can't imagine it is a bug.

Well the Linux people do stuff that other people call bugs  
sometimes... IMO, limiting the amount of data but not the number of  
packets as per the correspondent's MSS is not an appropriate way to do  
TCP congestion control.

> It is conceivable it is http doing this,

Not really. HTTP needs to go through TCP so it has to be TCP.

However, Linux has a "sendfile" syscall that does what you imagine it  
would, so maybe it's that code that does this. You may want to try  
with a static file and a script that creates output that is large  
enough to see if that's the case.

> it upon itself to recreate the individual RFC conformant TCP packets.

Hm, I'd still like to see if the Linux box actually spits out  
jumboframes or not from another system. What do you have your MTU set  
to?

> Its not in the subject lines, and there is no search facility.

Hm, it wasn't so recent apparently, and ack on the quality of the  
archive:

http://readlist.com/lists/trapdoor.merit.edu/nanog/7/35484.html

> MTU than this length, to do a lot of work without complaint.

Such are the perils of implementing RFC 791.

Fragmentation CAN be done in hardware. Not sure how much gear does  
that. But you can always get around this by configuring all your stuff  
with the same MTU. (Your ISPs may not want to go along with that,  
though.)

> We should all stop burning fossil fuel at some point too.

The difference is that burning oil is cheaper than the alternative,  
while breaking connectivity and sending packets smaller than what the  
hardware is capable of isn't.

> of 9k or so.

There are two ways to accomplish this:

1. ask someone who knows
2. measure

MTU information isn't known so there's noone to ask (could be added to  
routing protocols, though, but apparently few people cared enough to  
do that), so the only alternative is to send packets and observe what  
comes back.

> That just seems to slow down the sending host's response to the MTU

Yes, that's what the RFC 1191 people keep saying.

Apparently there's a disconnect between spec writers and implementers  
on the one hand and people who have to debug connectivity problems on  
the other hand, with the clueless firewall admins living in a bubble  
disconnected from everything.

> sending host gets the fastest possible feedback that it needs to

I'll take reliable over fast.

> I don't understand this.

Application writers can't be trusted with the internals of the network.

> Unfortunately, the surviving packet fragment isn't much use

You can still use the data, except that you can't check its integrity  
because the checksum is now incorrect. So the semi-ACK asks the other  
side to send just the checksum over the data that was correctly  
received. Did I forget to explain this part?

> host would be sending packets bigger than 1500 anyway.

Your premise is invalid so the conclusion is meaningless.

Actually, it would be interesting to do some research into the MTU  
distribution across the internet.

> edge networks, of 9k or so.

Note that although the 9000-byte jumboframe capability is common,  
there are also very many implementations that use different sizes so  
it's impossible to standardize on anything, even if you could ignore  
the fact that the current internet expects 1500.

Also, because of the 802.3 spec, the jumboframe capability must be  
enabled administratively, and because of the IP-over-ethernet specs,  
all hosts on a subnet must use the same MTU, so basically deployment  
is impossible. (This is what my draft addresses.)

> there is a black hole.

You mean: a network that doesn't generate them in the outgoing  
direction?

In practice this won't be a problem because few people will connect to  
the internet with a 1500+ MTU and then not generate too bigs. Since  
routers generate them out of the box and ISPs usually don't have  
firewalls in the middle of their networks and don't like support  
calls, ISPs tend to generate them.

If you use an MTU bigger than the standard 1500, then you shoot  
yourself in the foot with ICMP filtering so you're not likely to do  
both. The trouble is mainly with using a smaller MTU: then the problem  
is caused by _other_ people not listening to _your_ too bigs and there  
is little that you can do.

> server can.

That doesn't mean that all the stuff in the middle can handle  
jumboframes. The core of the network generally can, but the stuff  
around the edges, like the cheap switches that connect dozens of  
servers like yours, are likely to only support small packet sizes,  
either 1500 or "mini jumbos" of 1500 - 2000 bytes.

> I am not sure where the MSS is configured.

It isn't. The MSS option is created from the MTU of the interface the  
destination address is reachable through.

> systems actually implementing RFC 4821?

I haven't seen any...

> Is there any site, any

Not sure if this is officially dead or not:

http://staff.psc.edu/mathis/MTU/

"Just remember: The glass is neither half full nor half empty, it is  
merely the wrong size."

The optimist says the glass is half full. The pessimist says it's half  
empty. The engineer says the glass is twice as big as it needs to be.