Subject: End System PMTUD behavior question

From: "Dunn, Jeffrey H." <[email-masked]>
Date: Wed, 21 Jan 2009 13:56:23 -0500

Colleagues,

We
have been performing some PMTUD tests and have found that different operating
systems handle PMTUD differently. Specifically, we found that the
“ping” application behaves in the following way when the PMTU is
set to 1280 and a 1500 octet ICMPv6 echo request is sent to that routed path.
The hosts we tested:

1.
Send one or two un-fragmented echo requests

2.
Receive the “Packet Too Big” ICMPv6 message from the
first interface with the 1280 octet MTU

3.
Begin to send echo requests in two fragments

The
following questions occurred to us:

1.
Should the hosts re-transmit the ICMPv6 echo request(s) in
fragments?

2.
If not, should the host that sent two echo requests have waited
for a response in support of PMTUD?

Any
assistance you can provide, especially commercial specification for shipping
product or other applications, would be most helpful. Many thanks in advance.

Best
Regards,

Jeffrey Dunn

Info Systems Eng., Lead

Center for Enterprise Modernization

MITRE Corp.

301-429-5377 (Office)

301-731-8862  (FAX)

---

From: Rémi Denis-Courmont <[email-masked]>
Date: Wed, 21 Jan 2009 21:33:27 +0200

> Colleagues,
>
> We have been performing some PMTUD tests and have found that different
> operating systems handle PMTUD differently. Specifically, we found that the
> "ping" application behaves in the following way when the PMTU is set to
> 1280 and a 1500 octet ICMPv6 echo request is sent to that routed path. The
> hosts we tested:

I'm afraid ping applications are not representative of what the operating 
system IP stacks do. Besides, there are no standards regarding the behavior 
of "ping" in case of Path MTU problems; it's really up to the implementor of 
each "ping" whether to report an error or to transparently fragment.

The behavior of real transport protocols, such as TCP, DCCP and SCTP is more 
interesting when it comes to evaluating operating systems and their IP 
stacks.

-- 
Rémi Denis-Courmont
http://www.remlab.net/

---

From: "Dunn, Jeffrey H." <[email-masked]>
Date: Wed, 21 Jan 2009 14:44:18 -0500

ersus TCP; however, I am looking for some concrete documentation or experiences.

Best Regards, 
  
Jeffrey Dunn 
Info Systems Eng., Lead 
MITRE Corporation.
(301) 448-6965 (mobile)

-----Original Message-----
From: Rémi Denis-Courmont [mailto:[email-masked]] 
Sent: Wednesday, January 21, 2009 2:33 PM
To: [email-masked]
Cc: Dunn, Jeffrey H.; [email-masked]; [email-masked]; Sherman, Kurt T.; Liou, Chern; [email-masked]; Huang, Frank; Grayeli, Parisa
Subject: Re: End System PMTUD behavior question

-- 
Rémi Denis-Courmont
http://www.remlab.net/

---

From: "Hemant Singh (shemant)" <[email-masked]>
Date: Wed, 21 Jan 2009 15:35:16 -0500

>  1. Should the hosts re-transmit the ICMPv6 echo request(s) in
fragments? 

If the ICMPv6 echo reqs were being used for PMTUD, the answer should be
a No.  The host just received an Indication of Too Big that also said
the PMTU is 1280, so the host should be sending any subsequent packets
keeping PMTU of 1280 in mind without have to fragment. Then my question
is, if ICMPv6 is not being used for PMTUD, then for what purpose is
ICMPv6 being used such that an ICMPv6 req of 1500 is needed when a req
of size 1280 can be sent?

>  2. If not, should the host that sent two echo requests have waited

PMTUD does recommend a host send a probe and wait for a timeout interval
before sending another probe. The RFCs recommend five times larger
timeout value that the normal timeout of the protocol/app being used to
probe.  So, say, the ICMPv6 timeout being used in your test is 2 secs,
then the host should wait for 10 secs for a reply.  See RFC 4821 for
timeout recommendations for how long to wait for a probe reply.  In your
test case, I have got to believe, the Too Big error reached the host in
10 secs of sending the ICMPv6 req - the first hop router is most likely
to catch the Too Big condition and respond to the probe.

Hemant

---

From: "Dunn, Jeffrey H." <[email-masked]>
Date: Wed, 21 Jan 2009 17:24:18 -0500

We would be happy to see any results you may have concerning application behavior during PMTUD. 

Best Regards, 
  
Jeffrey Dunn 
Info Systems Eng., Lead 
MITRE Corporation.
(301) 448-6965 (mobile)

-----Original Message-----
From: Thomas Peterson [mailto:[email-masked]] 
Sent: Wednesday, January 21, 2009 3:00 PM
To: Dunn, Jeffrey H.
Cc: Rémi Denis-Courmont; [email-masked]; Huang, Frank; Sherman, Kurt T.; Liou, Chern; [email-masked]; [email-masked]; [email-masked]; Grayeli, Parisa
Subject: Re: End System PMTUD behavior question

Hi Jeffrey,

This is Thomas Peterson from the InterOperability Laboratory at UNH.

We have done extensive testing in this area and would be happy to work  
with you off line to examine this scenario.

If you would like we can even set up your test topology in our lab.

Thanks,
Tom

On Jan 21, 2009, at 2:44 PM, Dunn, Jeffrey H. wrote:

> Rémi,
>
> I agree with you assertions concerning "ping" versus TCP; however, I  
> am looking for some concrete documentation or experiences.
>
> Best Regards,
>
> Jeffrey Dunn
> Info Systems Eng., Lead
> MITRE Corporation.
> (301) 448-6965 (mobile)
>
>
> Cc: Dunn, Jeffrey H.; [email-masked]; [email-masked];  
> Sherman, Kurt T.; Liou, Chern; [email-masked]; Huang,  
> Frank; Grayeli, Parisa
> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> [email-masked]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
> --------------------------------------------------------------------

---

From: "Dunn, Jeffrey H." <[email-masked]>
Date: Wed, 21 Jan 2009 17:35:46 -0500

I do not understand your answer to question 1.  How would the purpose of a user sending ICMPv6 echo requests bear on system behavior? Also, unless the operating system has a facility to pass the PMTU size to the application, e.g., the MMS_S value mentioned in RFC 1981, AND the application can make use of the information. There is no expectation that the application, e.g., DNS, or the layer above IPv6, e.g., UDP, can make use of the information in RFC 1981; therefore your assertion that the host should be sending packets within the PMTU is unreasonable.

Many thanks for pointing out RFC 4821 in answer to question 2. I had forgotten about that and, interestingly, it does not appear in the NIST IPv6 profile, which we are using to guide our testing. 

Best Regards, 
  
Jeffrey Dunn 
Info Systems Eng., Lead 
MITRE Corporation.
(301) 448-6965 (mobile)

-----Original Message-----
From: Hemant Singh (shemant) [mailto:[email-masked]] 
Sent: Wednesday, January 21, 2009 3:35 PM
To: Dunn, Jeffrey H.; [email-masked]; [email-masked]; 6man mailing list
Cc: Sherman, Kurt T.; Liou, Chern; [email-masked]; Huang, Frank; Grayeli, Parisa
Subject: RE: End System PMTUD behavior question

> The following questions occurred to us:

fragments? 

Hemant

---

From: Thomas Peterson <[email-masked]>
Date: Thu, 22 Jan 2009 15:38:03 -0500

On Jan 21, 2009, at 5:24 PM, Dunn, Jeffrey H. wrote:

> Thomas,
> Cc: Rémi Denis-Courmont; [email-masked]; Huang, Frank; Sherman, Kurt  
> T.; Liou, Chern; [email-masked]; [email-masked]; [email-masked] 
> ; Grayeli, Parisa

---

From: "Hemant Singh (shemant)" <[email-masked]>
Date: Thu, 22 Jan 2009 17:35:47 -0500

Both IPv4 and IPv6 standards recommend not to fragment.  So the first
impact on the system behavior for host receiving an PTB message is to
try and not fragment.  I have not paid attention to older RFC's.  I am
working off of RFC 4821 that recommends a host keep "conceptual"
variables in cache at the IP layer and all other layers of code on the
host may access this cache to get the current value of the MTU to be
used by host.  See following text from RFC 4821.

[Furthermore, the natural mechanism to 
share Path MTU information between concurrent or subsequent connections 
is a path information cache in the IP layer. The various 
Packetization Protocols need to have the means to access and update 
the shared cache in the IP layer. This memo describes PLPMTUD in 
terms of its primary subsystems without fully describing how they 
are assembled into a complete implementation.]

Also see section 5 of RFC 4821.

What UNH has tested for current host behavior and so have you is because
host are following older PMTU RFCs like 1981.  Also, I agree with you on
RFC 1981 that host may fragment if there is no easy means for the host's
layers to pass MTU information back and forth.  My 2 cents is for new
hosts to follow RFC 4821 and not fragment packets.

Thanks

Hemant

---

From: "Dunn, Jeffrey H." <[email-masked]>
Date: Thu, 22 Jan 2009 23:27:45 -0500

Could you elaborate on what you mean by "DNS servers just tell the kernel to fragment at network mtu for UDP/IPv6 and ensure that DF is off for UDP/IPv4."  What is the "network MTU?" Also, to which implementations of DNS server are you referring?

Best Regards, 
  
Jeffrey Dunn 
Info Systems Eng., Lead 
MITRE Corporation.
(301) 448-6965 (mobile)

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Thursday, January 22, 2009 8:01 PM
To: [email-masked]
Cc: [email-masked]; Dunn, Jeffrey H.; Huang, Frank; Sherman, Kurt T.; [email-masked]; Liou, Chern; [email-masked]; [email-masked]; [email-masked]; Grayeli, Parisa
Subject: Re: End System PMTUD behavior question 

In message <[email-masked]>, Pet
[email-masked] writes:
> Hemant,
> =20
> Forgive me if I've misunderstood, but it sounds like you're saying that =
> we should require protocols or applications above IP to always send data =
> in messages small enough to avoid IP fragemntation.
> =20
> I agree it makes sense for a higher layer to use the PMTU information in =
> IP's cache when it can. Expecting TCP to use the PMTU is perfectly =
> reasonable, as it's already chopping up a byte stream into packets. For =
> protocols which are already packetized, though, I think it's less =
> advantageous to burden them (or the application using them) with the =
> problem of fragmentation and reassembly, to avoid IP fragmentation.
> =20
> For example, if a user does a "ping -s 1500" to a destination whose PMTU =
> is 1280, the only way to avoid IP fragmentation is for the ping =
> application to split the data into multiple messages, or for IPCMPv6 to =
> do so. Either way, you have to introduce some way to identify them as =
> "ping fragments" and reassemble them in order. That will require changes =
> to the ICMPv6 protocol, I think. Furthermore, you're no longer really =
> doing a "ping 1500", but two pings of 1280 and 220 bytes, respectively.
> =20
> In the case of an application which sends records in single UDP frames, =
> to avoid fragmentation is must split its messages into MTU-sized =
> chuncks, and come up with a way at the destination to identify and =
> reassemble the chunks in order. This seems a bit unreasonable, given =
> that IPv6 has a perfectly good mechanism to do this already.
> =20

	For the record.  DNS servers just tell the kernel to fragment
	at network mtu for UDP/IPv6 and ensure that DF is off for
	UDP/IPv4.

	DNS clients don't usually generate packets big enough to
	be a issue.  If they do need to send a big (> 512) message
	they usually just switch straight to TCP to avoid having
	to probe the server to see how big a UDP message it will
	handle.

> So I think the behaviour observed by Thomas during his testing is =
> correct. I don't think ping or ICMPv6 should reduce the ICMP message =
> size to avoid IP fragmentation.
> =20
> Peter Hunt
> Software Engineer
> Nokia S&S.
> =20
> 
> ------_=_NextPart_001_01C97CEC.C9F8E0DD
> Content-Type: text/html;
> 	charset="iso-8859-1"
> Content-Transfer-Encoding: quoted-printable
> 
> <HTML dir=3Dltr><HEAD><TITLE>RE: End System PMTUD behavior =
> question</TITLE>=0A=
> <META http-equiv=3DContent-Type content=3D"text/html; charset=3Dunicode">=0A=
> <META content=3D"MSHTML 6.00.6001.18183" name=3DGENERATOR></HEAD>=0A=
> <BODY>=0A=
> <DIV id=3DidOWAReplyText26585 dir=3Dltr>=0A=
> <DIV dir=3Dltr><FONT face=3D"Courier New" color=3D#000000 =
> size=3D2>Hemant,</FONT></DIV>=0A=
> <DIV dir=3Dltr><FONT size=3D2></FONT>&nbsp;</DIV>=0A=
> <DIV dir=3Dltr><FONT face=3D"Courier New" size=3D2>Forgive me if I've =
> misunderstood, but it sounds like you're saying that we should require =
> protocols or applications above IP to always send data in messages small =
> enough to avoid IP fragemntation.</FONT></DIV>=0A=
> <DIV dir=3Dltr><FONT face=3D"Courier New" size=3D2></FONT>&nbsp;</DIV>=0A=
> <DIV dir=3Dltr><FONT face=3D"Courier New" size=3D2>I agree it makes =
> sense for a higher layer to use the PMTU information in IP's cache when =
> it can. Expecting TCP to use the PMTU is perfectly reasonable, as it's =
> already chopping up a byte stream into packets. For protocols which are =
> already packetized, though, I think it's less advantageous to burden =
> them (or the application using them) with the problem of fragmentation =
> and reassembly, to avoid IP fragmentation.</FONT></DIV>=0A=
> <DIV dir=3Dltr><FONT face=3D"Courier New" size=3D2></FONT>&nbsp;</DIV>=0A=
> <DIV dir=3Dltr><FONT face=3D"Courier New" size=3D2>For example, if a =
> user does a "ping -s 1500" to a destination whose PMTU is 1280, the only =
> way to avoid IP fragmentation is for the ping application to split the =
> data into multiple messages, or for IPCMPv6 to do so. Either way, you =
> have to introduce some way to identify them as "ping fragments" and =
> reassemble them in order. That will require changes to the ICMPv6 =
> protocol, I think.&nbsp;Furthermore,&nbsp;you're no longer really doing =
> a "ping 1500", but two pings of 1280 and 220 bytes, =
> respectively.</FONT></DIV>=0A=
> <DIV dir=3Dltr><FONT face=3D"Courier New" size=3D2></FONT>&nbsp;</DIV>=0A=
> <DIV dir=3Dltr><FONT face=3D"Courier New" size=3D2>In the case of an =
> application which sends records in single UDP frames, to avoid =
> fragmentation is must split its messages into MTU-sized chuncks, and =
> come up with a way at the destination to identify and reassemble the =
> chunks in order. This seems a bit unreasonable, given that IPv6 has a =
> perfectly good mechanism to do this already.</FONT></DIV>=0A=
> <DIV dir=3Dltr><FONT face=3D"Courier New" size=3D2></FONT>&nbsp;</DIV>=0A=
> <DIV dir=3Dltr><FONT face=3D"Courier New" size=3D2>So I think the =
> behaviour observed by Thomas during his testing is correct. I don't =
> think ping or ICMPv6 should reduce the ICMP message size to avoid IP =
> fragmentation.</FONT></DIV>=0A=
> <DIV dir=3Dltr><FONT face=3D"Courier New" size=3D2></FONT>&nbsp;</DIV>=0A=
> <DIV dir=3Dltr><FONT face=3D"Courier New" size=3D2>Peter =
> Hunt</FONT></DIV>=0A=
> <DIV dir=3Dltr><FONT face=3D"Courier New" size=3D2>Software =
> Engineer</FONT></DIV>=0A=
> <DIV dir=3Dltr><FONT face=3D"Courier New" size=3D2>Nokia =
> S&amp;S.</FONT></DIV>=0A=
> <DIV dir=3Dltr><FONT size=3D2></FONT>&nbsp;</DIV></DIV></BODY></HTML>
> ------_=_NextPart_001_01C97CEC.C9F8E0DD--
> 
> --===============0174434914==
> Content-Type: text/plain; charset="us-ascii"
> MIME-Version: 1.0
> Content-Transfer-Encoding: 7bit
> Content-Disposition: inline
> 
> --===============0174434914==--
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Pekka Savola <[email-masked]>
Date: Fri, 23 Jan 2009 08:03:18 +0200 (EET)

On Fri, 23 Jan 2009, [email-masked] wrote:
> For example, if a user does a "ping -s 1500" to a destination whose PMTU is 1280, the only way to avoid IP fragmentation is for the ping
> application to split the data into multiple messages, or for IPCMPv6 to do so. Either way, you have to introduce some way to identify them
> as "ping fragments" and reassemble them in order. That will require changes to the ICMPv6 protocol, I think. Furthermore, you're no longer
> really doing a "ping 1500", but two pings of 1280 and 220 bytes, respectively.

FWIW, what Remi said, different ping programs probably do this 
differently.  And good ones allow you to do exactly what you want 
(this is from Linux)

        -M hint
               Select Path MTU Discovery strategy.  hint may be either _do_ (prohibit fragmentation, even local one), _want_ (do PMTU
               discovery, fragment locally when packet size is large), or _dont_ (do not set DF flag).

When I use ping to figure out Path MTU issue, I usually have to run 
tcpdump on the side to be 100% sure how ping is actually behaving, 
because additionally, there's also PMTU caching on the local host. 
Some older versions also didn't support '-M do' properly for IPv6.

So there are quite a few things that could lead to non-deterministic 
behaviour.

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

---

From: "Dunn, Jeffrey H." <[email-masked]>
Date: Fri, 23 Jan 2009 07:06:03 -0500

Fair enough, it appears the short answer to which implementation you know of is BIND (I assume 9.x). With respect to IPV6_USE_MIN_MTU, I thought the value for this was 1280. If so, this simply circumvents PMTUD and uses the IPv6 minimum MTU.

Best Regards, 
  
Jeffrey Dunn 
Info Systems Eng., Lead 
MITRE Corporation.
(301) 448-6965 (mobile)

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Thursday, January 22, 2009 11:37 PM
To: Dunn, Jeffrey H.
Cc: [email-masked]; [email-masked]; Huang, Frank; Sherman, Kurt T.; [email-masked]; Liou, Chern; [email-masked]; [email-masked]; [email-masked]; Grayeli, Parisa
Subject: Re: End System PMTUD behavior question 

In message <[email-masked]>, "Dun
n, Jeffrey H." writes:
> Mark,
> 
> Could you elaborate on what you mean by "DNS servers just tell the kernel t=
> o fragment at network mtu for UDP/IPv6 and ensure that DF is off for UDP/IP=
> v4."  What is the "network MTU?" Also, to which implementations of DNS serv=
> er are you referring?

	Cut-and-pasted from BIND's bsd socket implementation.

	Mark

#ifdef IPV6_USE_MIN_MTU        /* RFC 3542, not too common yet*/
                /* use minimum MTU */
                if (sock->pf == AF_INET6) {
                        (void)setsockopt(sock->fd, IPPROTO_IPV6,
                                         IPV6_USE_MIN_MTU,
                                         (void *)&on, sizeof(on));
                }
#endif

#if defined(IP_MTU_DISCOVER) && defined(IP_PMTUDISC_DONT)
                /*
                 * Turn off Path MTU discovery on IPv4/UDP sockets.
                 */
                if (sock->pf == AF_INET) {
                        int action = IP_PMTUDISC_DONT;
                        (void)setsockopt(sock->fd, IPPROTO_IP, IP_MTU_DISCOVER,
                                         &action, sizeof(action));
                }
#endif
#if defined(IP_DONTFRAG)
                /*
                 * Turn off Path MTU discovery on IPv4/UDP sockets.
                 */
                if (sock->pf == AF_INET) {
                        int off = 0;
                        (void)setsockopt(sock->fd, IPPROTO_IP, IP_DONTFRAG,
                                         &off, sizeof(off));
                }
#endif

> Best Regards,=20
> =A0=20
> Jeffrey Dunn=20
> Info Systems Eng., Lead=20
> MITRE Corporation.
> (301) 448-6965 (mobile)

---

From: "Dunn, Jeffrey H." <[email-masked]>
Date: Fri, 23 Jan 2009 09:36:01 -0500

Good point.  We use Wireshark to capture the send/receive sequence on the host. In addition, we reboot the machine between tests to ensure the PMTU cache is cleared, since we have show that, once learned, the host will immediately send fragmented ICMPv6 echo requests within the PMTU size.

Best Regards, 
  
Jeffrey Dunn 
Info Systems Eng., Lead 
MITRE Corporation.
(301) 448-6965 (mobile)

-----Original Message-----
From: Pekka Savola [mailto:[email-masked]] 
Sent: Friday, January 23, 2009 1:03 AM
To: [email-masked]
Cc: [email-masked]; Dunn, Jeffrey H.; Huang, Frank; Sherman, Kurt T.; [email-masked]; Liou, Chern; [email-masked]; [email-masked]; [email-masked]; Grayeli, Parisa
Subject: RE: End System PMTUD behavior question

        -M hint

---

From: "Dunn, Jeffrey H." <[email-masked]>
Date: Fri, 23 Jan 2009 09:54:41 -0500

Many thanks for the information. Could you tell me the OS variants on the hosts and router? Also, do you have any tests involving applications?  I will send you an off-line e-mail to discuss possible additional testing.

Best Regards, 
  
Jeffrey Dunn 
Info Systems Eng., Lead 
MITRE Corporation.
(301) 448-6965 (mobile)

-----Original Message-----
From: Thomas Peterson [mailto:[email-masked]] 
Sent: Thursday, January 22, 2009 3:38 PM
To: Dunn, Jeffrey H.
Cc: Rémi Denis-Courmont; [email-masked]; Huang, Frank; Sherman, Kurt T.; Liou, Chern; [email-masked]; [email-masked]; [email-masked]; Grayeli, Parisa
Subject: Re: End System PMTUD behavior question

Hi Jeffrey,

I have attached a picture which shows one of the topologies we use for  
our PMTUD tests.

In this test case we transmit an Echo Request from REF-Host2 to TAR- 
Host1 with a packet size of 1500 bytes. REF-Host2 fragments the Echo  
Request it transmits. TAR-Host1 replies to this Echo Request with an  
Echo Reply to REF-Host2 with a size of 1500 bytes that is not  
fragmented. TAR-Router1 sends a Packet Too Big message in response as  
this Echo Reply is too large to forward onto Network 2.

In all of the cases we have seen TAR-Host1 does fragment future Echo  
Replies to REF-Host2, however, it does not retransmit any Echo Replies  
for Echo Requests received prior to receiving the Packet Too Big  
message from TAR-Router1.

Additionally from the tests we have performed in our lab if TAR-Host1  
were to send an Echo Request with a packet size of 1500 bytes TAR- 
Router1 would send a Packet Too Big message in response. In all cases  
we have seen TAR-Host1 would not re-transmit this Echo Request and  
this would be counted as packet loss in the ping command results.

If this does not ideally match your test scenario we'd be happy to  
work together off-line to replicate your scenario.

Thanks,
Tom

---

From: Thomas Peterson <[email-masked]>
Date: Fri, 23 Jan 2009 11:50:47 -0500

Here is a link to the list of all of the different hosts that we have  
performed this testing on:

http://www.iol.unh.edu/services/testing/ipv6/equipment.php

Currently we use the ping application to test PTMUD, however, we would  
be interested in performing addition tests  using various applications.

Thanks,
Tom

On Jan 23, 2009, at 9:54 AM, Dunn, Jeffrey H. wrote:

> Tom,
>
> Many thanks for the information. Could you tell me the OS variants  
> on the hosts and router? Also, do you have any tests involving  
> applications?  I will send you an off-line e-mail to discuss  
> possible additional testing.
>
> Best Regards,
>
> Jeffrey Dunn
> Info Systems Eng., Lead
> MITRE Corporation.
> (301) 448-6965 (mobile)
>
>