Subject: [v6ops] PMTUD issue discussion

From: "Fred Baker (fred)" <[email-masked]>
Date: Mon, 25 Aug 2014 17:20:27 +0000

http://datatracker.ietf.org/doc/draft-v6ops-pmtud-ecmp-problem
http://tools.ietf.org/html/draft-v6ops-pmtud-ecmp-problem
 "Close encounters of the ICMP type 2 kind (near misses with ICMPv6
 PTB)", Matt Byerly, Matt Hite, Joel Jaeggli, 2014-08-24,

As requested at IETF 90, Joel has edited and reposted his draft. There are two questions before the house:
 - do we want to make this a working group draft?
 - what do we want to do next?

Note that, by charter, what we are not permitted to do is change implementations or protocols; we are allowed to define operational procedure. That said, we *can* make recommendations to other working groups, asking them to change something.

So, for example, we might ask 6man to do something specific, or we might ask tcpm to do something specific. Something specific that we might ask tcpm to do would be to get operational experience with RFC 4821 and commit it back to open source, for example.

---

From: Joe Touch <[email-masked]>
Date: Mon, 25 Aug 2014 13:49:56 -0700

Hi, all,

Speaking from TCPM-land, I would observe the following:

- PMTUD already has many known problems, which is why PLMTUD is 
recommended instead

- the issue here appears to be a device that routes TCP and UDP packets 
based on a hash, but does not apply that hash to the ICMP messages
	that's clearly an oversight of those devices.
	ICMP feedback is a known part of the Internet architecture,
	and any device that demultiplexes packets based on transport
	info needs to similarly process ICMP messages

	that goes for NATs, load balancers, or anything else.

I'm not sure what would be added other than to say "we found this 
problem here too". It's a bug that ought to be fixed, but endpoints that 
intend to be robust already know not to rely on ICMP.

Joe

On 8/25/2014 10:20 AM, Fred Baker (fred) wrote:
> _______________________________________________
> v6ops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/v6ops
>

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 25 Aug 2014 20:56:54 +0000

> -----Original Message-----
> From: v6ops [mailto:[email-masked]] On Behalf Of Joe Touch
> Sent: Monday, August 25, 2014 1:50 PM
> To: Fred Baker (fred); IPv6 Ops WG
> Subject: Re: [v6ops] PMTUD issue discussion
> 
> Hi, all,
> 

+1, and that also reinforces Fred's point about becoming more proactive
in encouraging RFC4821.

Thanks - Fred
[email-masked]

> > Note that, by charter, what we are not permitted to do is change implementations or protocols; we
> are allowed to define operational procedure. That said, we *can* make recommendations to other working
> groups, asking them to change something.
> >
> > So, for example, we might ask 6man to do something specific, or we might ask tcpm to do something
> specific. Something specific that we might ask tcpm to do would be to get operational experience with
> RFC 4821 and commit it back to open source, for example.
> >

---

From: joel jaeggli <[email-masked]>
Date: Mon, 25 Aug 2014 14:13:05 -0700

> - PMTUD already has many known problems, which is why PLMTUD is

I agree, operationally however I'm trying not to break existing devices
attempting to connect to me, is the motivation for the note.

>     info needs to similarly process ICMP messages

If you use source / dest / flow label or even just source / dest you
have the same issue. e.g. using the transport header for hash
computation is not required to induce this.

>     that goes for NATs, load balancers, or anything else.

This requires that I not only be transport aware but be able to parse
into the payload. As noted, the data I would need can probably be found
at a fixed offset (modula extension headers) so in fact that  is
probably feasible.

> intend to be robust already know not to rely on ICMP.

I don't disagree with that sentiment.

>> Note that, by charter, what we are not permitted to do is change
>> implementations or protocols; we are allowed to define operational
>> procedure. That said, we *can* make recommendations to other working
>> So, for example, we might ask 6man to do something specific, or we
>> might ask tcpm to do something specific. Something specific that we
>> might ask tcpm to do would be to get operational experience with RFC
>> 4821 and commit it back to open source, for example.
>>

---

From: Andrew 👽 Yourtchenko <[email-masked]>
Date: Tue, 26 Aug 2014 11:50:01 +0200

Joel,

> attempting to connect to me, is the motivation for the note.

During the presentation in the v6ops session I made a comment about
sysctl net.ipv4.tcp_mtu_probing. Since then I had a chance to test it
- and indeed, it affects the behavior of TCP over IPv6 as well.

However, the value of 1 causes a large value of MSS to be used,
resulting in about 1.5 second hiccup. So, yeah it "works" but for
barely acceptable values of "works"

Setting the value to 2 causes the initial MSS to be 512, which
"mitigates" the problem and avoided the hiccups on a 100kb file I was
testing with - at the expense of almost 3x more packets of course.

I think it would be useful to see the discussion of this measure and
its applicability/tradeoffs in the draft.

--a

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 26 Aug 2014 14:59:32 +0000

Hi Andrew,

> From: v6ops [mailto:[email-masked]] On Behalf Of Andrew ?? Yourtchenko
> Sent: Tuesday, August 26, 2014 2:50 AM
> To: joel jaeggli
> Cc: IPv6 Ops WG

PTB message loss is also a concern for IPv6, and not just for IPv4.
 
> testing with - at the expense of almost 3x more packets of course.

I think that implementation was added to the linux kernel a long
time ago, and I am not sure if anyone is still maintaining it. Is
this a problem that should be reported to netdev?

Thanks - Fred

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Tue, 26 Aug 2014 15:13:51 +0000

> its applicability/tradeoffs in the draft.

It seems like it might have value to also test the interaction with various initial window settings, and look at its inclusion in other relevant OS’s including FreeBSD, Windows, and MacOSX. My reading of 4821 says it’s grand theory, but I’m not precisely sure how I would write the code. As an initial setting, for example, I could imagine sending an initial burst containing a 9K byte packet, a 1500 byte packet, and a 1280 byte packet in that order, and set the MSS to the the size of the first acknowledgment I receive. How does that relate to IW=10 (RFC 6928)?

---

From: Fernando Gont <[email-masked]>
Date: Tue, 26 Aug 2014 12:35:02 -0300

Hi, Fred,

On 08/26/2014 12:13 PM, Fred Baker (fred) wrote:
>> I think it would be useful to see the discussion of this measure 
>> and its applicability/tradeoffs in the draft.
> 
> It seems like it might have value to also test the interaction
> with various initial window settings, and look at its inclusion in
> other relevant OS’s including FreeBSD, Windows, and MacOSX.

FWIW, while RFC4821 can work without relying on PLPMTUD at all, some
OSes (such as Windows) only fall back to ICMP-less PMTUD as part of
"ICMP blackhole detecion" -- and they have been doing this for years
now...

> My reading of 4821 says it’s grand theory, but I’m not precisely
> sure how I would write the code. As an initial setting, for
> example, I could imagine sending an initial burst containing a 9K
> byte packet, a 1500 byte

Unlikely if the underlying link-layer technology is Ethernet (PMTUD
will never be assumed to be larger than your own MTU).

Additionally, I doubt you'd employ different packet
sizes/assumed_PMTUDs all at once.

Thanks,
-- 
Fernando Gont
SI6 Networks
e-mail: [email-masked]
PGP Fingerprint: 6666 31C6 D484 63B2 8FB1 E3C4 AE25 0D55 1D4E 7492

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 26 Aug 2014 15:42:51 +0000

Hi Fernando,

> From: v6ops [mailto:[email-masked]] On Behalf Of Fernando Gont
> Sent: Tuesday, August 26, 2014 8:35 AM
> To: Fred Baker (fred); Andrew 👽 Yourtchenko
> Cc: IPv6 Ops WG

I think the point is that PLPMTUD is not well supported in modern
OSes in the spirit in which it was specified in RFC4821. That
needs to change.

Thanks - Fred

---

From: Fernando Gont <[email-masked]>
Date: Tue, 26 Aug 2014 13:19:04 -0300

> OSes in the spirit in which it was specified in RFC4821. That

FWIW, I'm all in favor of RFC4821, since it improves robustness of PMTUD.

That said, as far as I can recall, using RFC4821 for blackhole detection
(when traditional PMTUD fails) is well within the spirit of RFC4821
(RFC4821 doesn't push ICMP-less as the only implementation strategy).

What I recall from discussing this with Linux developers circa 2005 is
that a complete ICMP-less PMTUD is rather undesirable, since it tends to
have longer convergence time when compared with "traditional PMTUD" ||
"Traditional PMTUD + RFC4821 for blachole detection".

Thanks,

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 26 Aug 2014 16:33:53 +0000

Hi Fernando,

> From: Fernando Gont [mailto:[email-masked]]
> Sent: Tuesday, August 26, 2014 9:19 AM
> To: Templin, Fred L; Fred Baker (fred); Andrew 👽 Yourtchenko
> Cc: IPv6 Ops WG

Accepting and processing PTBs as part of the PLPMTUD procedure is
desirable and speeds convergence. But, the mechanism still needs to
work when there are no PTBs.

So, the network should still do its best to return PTBs, and hosts
should still do their best to observe them.

Thanks - Fred

---

From: Andrew 👽 Yourtchenko <[email-masked]>
Date: Tue, 26 Aug 2014 18:41:53 +0200

Hi Fred,

> PTB message loss is also a concern for IPv6, and not just for IPv4.

Absolutely agree.

Just that at a first look one would see a bunch of sysctls that are
labeled with ".ipv4." and a bunch of sysctls that are labeled with
".ipv6." and possibly make a conclusion they affect just that
protocol.

> this a problem that should be reported to netdev?

I think it works the same way it would work in IPv4 - if the ICMP PTB
is lost, and the too large MSS was used, then there is a delay which
is less tolerable today for the interactive traffic (IMHO) than when
this code was introduced, hence my remark that "works" value is
relatively small by today's high standards.

But the 512-byte MSS, while preventing the delay, effectively triples
the PPS, so might cause problems in some of the setups if their gear
is "only" capable of 2x of the pre-change PPS value.

That was the reason to suggest that if this setting can be used to
solve the problem, there probably should be a discussion of the
tradeoffs in the draft.

I did not notice any wrong behavior - but, my testing was rather cursory.

--a

---

From: Fernando Gont <[email-masked]>
Date: Tue, 26 Aug 2014 13:41:56 -0300

> should still do their best to observe them.

Agreed. IIRC, this behavios has been implemented in Windows and OpenBSD
for 5+ years, though.

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 26 Aug 2014 17:06:20 +0000

Hi Fernando,

> Sent: Tuesday, August 26, 2014 9:42 AM

==> But, the mechanism still needs to work when there are no PTBs.

If the implementations are in fact already doing this well and
seeking to maximize the path MTU by progressively probing for
larger packet sizes without causing excessive packet duplication
then good. Otherwise, more work is needed.

Thanks - Fred

---

From: Ray Hunter <[email-masked]>
Date: Sat, 30 Aug 2014 19:50:04 +0200

Fred Baker (fred) wrote:
> On Aug 26, 2014, at 2:50 AM, Andrew 👽 Yourtchenko<[email-masked]>  wrote:
>

Whilst reading your discussion, I had a "what if" brainwave.

Paths are learned via routing. What if Path MTU was also learned via 
routing, rather than being probed for by individual end hosts?

Then I checked: EIGRP discovers the set of 6 metrics (bandwidth, load, 
delay, reliability, minimum MTU, hop count)

MTU is never used in the routing decision AFAICS, but couldn't the local 
router be programmed to generate ICMP PTB messages (as a proxy for 
downstream routers) with a link-local source address to each host if a 
path MTU is going to be exceeded, plus a hint of the actual PMTU 
discovered by routing?

Thus avoiding ICMP PTB message black holes?And avoiding the PLMTUD 
discovery process and extensive probing/ ramp up for every session?
And also avoiding trust issues (as hosts presumably trust their local 
on-link default router)?

Obviously for this to work internet wide, BGP would also need to carry 
an AS-PMTU attribute per AS path.

-- 
Regards,
RayH

---

From: Matthew Petach <[email-masked]>
Date: Sat, 30 Aug 2014 22:00:39 -0700

On Sat, Aug 30, 2014 at 10:50 AM, Ray Hunter <[email-masked]> wrote:

>> It seems like it might have value to also test the interaction with
>> various initial window settings, and look at its inclusion in other
>> relevant OS’s including FreeBSD, Windows, and MacOSX. My reading of 4821
>> says it’s grand theory, but I’m not precisely sure how I would write the
>> code. As an initial setting, for example, I could imagine sending an
>> initial burst containing a 9K byte packet, a 1500 byte packet, and a 1280
>> byte packet in that order, and set the MSS to the the size of the first
>> acknowledgment I receive. How does that relate to IW=10 (RFC 6928)?
>>
>
> downstream routers) with a link-local source address to each host if a path
> MTU is going to be exceeded, plus a hint of the actual PMTU discovered by
> routing?
>
> Obviously for this to work internet wide, BGP would also need to carry an
> AS-PMTU attribute per AS path.
>

This seems to imply you think there's a single
MTU for an entire prefix?  Or would this entail
deaggregating prefixes down to the point where
an accurate MTU could be associated with
each more specific?

It's a interesting idea, but highly, highly
impractical without potentially blowing
up the routing table size.

Matt

---

From: Ray Hunter <[email-masked]>
Date: Sun, 31 Aug 2014 08:36:26 +0200

> Matthew Petach <mailto:[email-masked]>
> 31 August 2014 07:00
>
>
>
> On Sat, Aug 30, 2014 at 10:50 AM, Ray Hunter <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Fred Baker (fred) wrote:
>
>         On Aug 26, 2014, at 2:50 AM, Andrew 👽
>         Yourtchenko<[email-masked] <mailto:[email-masked]>> 
>         wrote:
>
>         with various initial window settings, and look at its
>         inclusion in other relevant OS’s including FreeBSD, Windows,
>         and MacOSX. My reading of 4821 says it’s grand theory, but I’m
>         not precisely sure how I would write the code. As an initial
>         setting, for example, I could imagine sending an initial burst
>         containing a 9K byte packet, a 1500 byte packet, and a 1280
>         byte packet in that order, and set the MSS to the the size of
>         the first acknowledgment I receive. How does that relate to
>         IW=10 (RFC 6928)?
>
>
>     Paths are learned via routing. What if Path MTU was also learned
>     via routing, rather than being probed for by individual end hosts?
>
>     Then I checked: EIGRP discovers the set of 6 metrics (bandwidth,
>     load, delay, reliability, minimum MTU, hop count)
>
>     MTU is never used in the routing decision AFAICS, but couldn't the
>     local router be programmed to generate ICMP PTB messages (as a
>     proxy for downstream routers) with a link-local source address to
>     each host if a path MTU is going to be exceeded, plus a hint of
>     the actual PMTU discovered by routing?
>
>     And also avoiding trust issues (as hosts presumably trust their
>     local on-link default router)?
>
>     Obviously for this to work internet wide, BGP would also need to
>     carry an AS-PMTU attribute per AS path.
>
>
> ------------------------------------------------------------------------
Not necessarily.

What might help is a realistic and early estimate of the minimum MTU 
we're likely going to encounter on the path,.
even if this is just used as a hint for the PLMTUD learning process that 
is communicated from the local router to local hosts.

MTU is configured on routers. Routers determine the path (based on 
routes). Routes are based on prefixes.
So there is certainly a relationship between MTU and prefixes.

Your concern seems to be how good that correlation is.

IMVHO I believe we could generate a single minimum MTU per path (based 
on prefixes) that is a pretty good first approximation (whilst avoiding 
blowing up the routing table).

In EIGRP, a lot of AS'es don't even use summary routes. So in this case 
there's a very good correlation between MTU, path, and prefix, at least 
for traffic within the AS.
So I could see this working quite well within an individual AS, like a 
corporate network.

In BGP4, the path is a set of AS'es to be traversed.

The minimum MTU of an individual AS would be the minimum MTU of all 
individual paths encountered when traversing that AS.
e.g. the minimum of all MTUs learned in the IGP, or a static manually 
configured number in the AS border router.

The minimum MTU of a BGP path would be the minimum of the individual AS 
MTUs.

I can see this being potentially problematic, but is it meaningfully 
better than a single global constant (based on a minimum of the whole 
Internet)?

Or is a search for the path MTU for every host for every session already 
a good enough solution?

-- 
Regards,
RayH