Subject: [tcpm] PLPMTUD for all protocols

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 28 Mar 2018 10:43:40 +0200 (CEST)

Hi,

In trying to advocate for all protocols implementing PLPMTUD (RFC4821) and 
shipping with it default on, I received pushback when I suggested/asked 
why modern TCP stacks don't come with PMTU blackhole detect turned on.

Relevant reading is draft-bonica-intarea-frag-fragile-01, where PMTU 
blackhole is one part of this problem space.

I tried turning on PMTU blackhole in Linux quickly in a case I had, and it 
worked around the PMTU blackhole we had for a certain deployment scenario 
(which couldn't be fixed unless the NMS could reconfigure the device in 
question, which it couldn't because PMTU blackhole existed).

People suggested to me that the current TCP stack implementations are 
broken when it comes to PMTU blackhole detection and workaround, so that's 
why it's not on.

What's wrong with it, and how do we fix it?

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Tom Jones <[email-masked]>
Date: Wed, 28 Mar 2018 10:03:33 +0100

> What's wrong with it, and how do we fix it?

Related discussion on the freebsd-net mailing list.
https://lists.freebsd.org/pipermail/freebsd-net/2018-March/049916.html

As I understand there are issues with FreeBSD increasing again after
detecting a blackhole. There is not currently enough state kept to
regrow the mtu. 

- [tj]

---

From: William Herrin <[email-masked]>
Date: Wed, 28 Mar 2018 11:26:11 -0400

On Wed, Mar 28, 2018 at 4:43 AM, Mikael Abrahamsson <[email-masked]> wrote:
> shipping with it default on, I received pushback when I suggested/asked why
> modern TCP stacks don't come with PMTU blackhole detect turned on.

Hi Mikael,

My knee-jerk response reading the RFC is that the up-probe process
seems very brittle with layer violations all over the place.

> What's wrong with it, and how do we fix it?

Don't attempt a layer-3 general purpose solution that boils down to a
large set of dirty hacks? Spitballing here, but in TCP you could
create an option for the remote to respond with the largest received
fragment size when a fragmented packet is received and if the option
is negotiated then send the up-probes with the DF bit cleared. If the
remote doesn't support it, shrink the MSS but don't try to grow it
again. If the remote does respond with the largest fragment size then
you get the exact path MTU as a result of one probe without disrupting
the communication flow.

Sacrilege to suggest sending anything without the DF bit set, but if
the router's return has a black hole, maybe the answer is to let the
router communicate forward instead.

Regards,
Bill Herrin

-- 
William Herrin ................ [email-masked]  [email-masked]
Dirtside Systems ......... Web: <http://www.dirtside.com/>

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 28 Mar 2018 17:51:34 +0200 (CEST)

On Wed, 28 Mar 2018, William Herrin wrote:

> Sacrilege to suggest sending anything without the DF bit set, but if the 
> router's return has a black hole, maybe the answer is to let the router 
> communicate forward instead.

1. IPv6 doesn't have a DF bit that can be on or off, it's always 
implicitly on.
2. Fragmentation doesn't usually happen if there is PMTUD blackhole.

So I have no idea how to come to a best conclusion regarding the tradeoff 
between "doesn't work at all" and "doesn't work optimally in some cases". 
Generally, I'd be happy with a blackhole detect that falls back to low MTU 
(576 for IPv4, 1280 for IPv6) whenever there seems to be a real blackhole, 
and be very cautious in going back up again in MSS.

Right now I would like things to be more resilient to complete failure 
than what TCP is today without blackhole detect.

---

From: "Maxim Proshin [GMAIL]" <[email-masked]>
Date: Thu, 29 Mar 2018 12:45:05 +0300

Hi,

What is a status of classical PMTUD in Linux TCP? Is it enabled by default?
I guess it's enabled by default but TCP implementation actually relies on
IP for that.

BR, Maxim

On Wed, Mar 28, 2018 at 6:51 PM, Mikael Abrahamsson <[email-masked]>
wrote:

> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
>

-- 
BR, Maxim

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 29 Mar 2018 15:10:09 +0200 (CEST)

> I guess it's enabled by default but TCP implementation actually relies on

Classic PMTUD (as in getting back need-to-frag/Packet-too-big ICMP 
messages) is something TCP relies on by standard, and this is default on.

The problem is when an ICMP error message isn't generated or doesn't reach 
the intended recipient, that's when blackhole occurs, and what I want to 
solve.

---

From: Joe Touch <[email-masked]>
Date: Thu, 29 Mar 2018 06:32:22 -0700

> On Mar 28, 2018, at 8:51 AM, Mikael Abrahamsson <[email-masked]> wrote:
> 
> So I have no idea how to come to a best conclusion regarding the tradeoff between "doesn't work at all" and "doesn't work optimally in some cases".

If making things more optimal in some cases causes some things to fail, you have chosen poorly.

The Internet’s use of one common interchange message is founded on the idea that “being able to talk to everyone someone”is more important than “being able to talk to anyone more efficiently”.

I.e Metcalfe’s law of utility is more important than performance.

Joe

---

From: William Herrin <[email-masked]>
Date: Thu, 29 Mar 2018 11:02:55 -0400

> If making things more optimal in some cases causes some things to fail, you have chosen poorly.

Hi Joe,

Status quo is "doesn't work at all," the PMTUD black hole problem.
PMTUD is arguably a design defect in IP: one of few cases in which the
end to end principle is abandoned, requiring the intermediate router
to communicate with an endpoint or else end to end communication
fails. It's also arguable that every instance of a PMTUD black hole is
a consequence of an operator configuration error.

PLPMTUD is proposed as "doesn't work optimally in some cases." More to
the point: suffers in common packet loss scenarios. Everybody takes a
performance penalty to overcome common operator configuration errors.

Alternatives to PLPMTUD include configuring the local TCP stack's MSS
to the IPv6 minimum MTU (1280 bytes). This pretty much eliminates the
PMTUD black hole problem by eliminating the need for detection.
There's a negligible performance hit but a slightly increased
consumption of bandwidth in the normal case. I've used this on my
servers for years and the bottom line is that it works.

It's also fair to wonder why router and firewall vendors don't offer
configurations designed to mitigate the PMTUD black hole problem.
Firewall vendors could warn on attempts to block ICMP or require
explicit overrides to block ICMP destination unreachable messages.
Router vendors could offer a configuration option to issue ICMP
destination unreachables from the a configured public IP address
rather than from the interface's (sometimes private) IP address. They
don't do these things.

In principle the IETF could even designate a standard public IP
address to be used when routers that have no public IP addresses need
to issue a a fragmentation needed message. After a couple equipment
cycles the PMTUD black hole problem would presumably fall back to
1990s levels which weren't so bad.

On Wed, Mar 28, 2018 at 11:51 AM, Mikael Abrahamsson <[email-masked]> wrote:
> 1. IPv6 doesn't have a DF bit that can be on or off, it's always implicitly
> on.

Thanks Mikael, I didn't realize that.

> 2. Fragmentation doesn't usually happen if there is PMTUD blackhole.

Do we have data on this? Is it a general case issue where routers
refuse to fragment packets even if the DF bit is clear? We expect the
router to forward a small enough unfragmented packet onward. I can't
think of a reason that the PMTUD black holes and routers failing to
fragment IPv4 packets where DF is clear would be linked issues.

Regards,
Bill Herrin

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 29 Mar 2018 17:31:30 +0200 (CEST)

> a consequence of an operator configuration error.

Some are, some are not.

> destination unreachables from the a configured public IP address rather 
> than from the interface's (sometimes private) IP address. They don't do 
> these things.

These are some reasons for PMTUD blackhole, but far from all. Some are 
caused by anycast configurations where the ICMP packet simply doesn't make 
it to the node that needs it. Load balancers doing the wrong thing.

> In principle the IETF could even designate a standard public IP address 
> to be used when routers that have no public IP addresses need to issue a 
> a fragmentation needed message. After a couple equipment cycles the 
> PMTUD black hole problem would presumably fall back to 1990s levels 
> which weren't so bad.

My take on this is that operators should do the right thing (make PMTUD 
work), and protocols should handle when things go wrong.

I have very little problem with TCP using an extremely stupid algorithm as 
in "I'll use PMTUD, but if it doesn't work I'm going to try 576/1280, and 
if that works I sometimes go back to PMTUD value to see if things have 
improved". This still means it'll work, somewhat.

---

From: William Herrin <[email-masked]>
Date: Thu, 29 Mar 2018 12:58:07 -0400

On Thu, Mar 29, 2018 at 11:31 AM, Mikael Abrahamsson <[email-masked]> wrote:
>> configurations designed to mitigate the PMTUD black hole problem. Firewall
>> vendors could warn on attempts to block ICMP or require explicit overrides
>> to block ICMP destination unreachable messages. Router vendors could offer a
>> configuration option to issue ICMP destination unreachables from the a
>> configured public IP address rather than from the interface's (sometimes
>> private) IP address. They don't do these things.
>

Hi Mikael,

Load balancers doing the wrong thing seems to me like some combination
of implementation error and configuration error.

How would PLPMTUD detect a path MTU black hole for DNS response
packets from an anycasted node? It's fire and forget; the server is
not expecting another packet from the querent.

Where something more complicated is done with anycast such as TCP
without suitable modifications (e.g.
https://bill.herrin.us/network/anycasttcp.html) then they've already
accepted that a portion of their prospective users will be
unreachable. I have no sympathy when folks decide to intentionally
implement breakage because it will, statistically, only be broken some
of the time.

You make an interesting point here about icmp unreachables in the TCP
anycast scenario that I haven't accounted for in that paper. I'll have
to think that through.

> work), and protocols should handle when things go wrong.

Operators "should" have the tools which:

A) make it practical to make PMTUD work in all commonly deployed
architectures and

B) do a respectable job preventing "design induced operator error"
with respect to PMTUD.

Currently shipping routers and firewalls mostly fail both criteria.

> in "I'll use PMTUD, but if it doesn't work I'm going to try 576/1280, and if
> that works I sometimes go back to PMTUD value to see if things have

I have no problem with falling back after a smart enough delay/retry
cycle concludes a very high probability a PMTUD black hole. I have
little problem with those connections suffering the performance hit of
small packets for the duration. I have a bigger problem if routine
packet loss or a brief path outage gets commonly misdetected as a
PMTUD black hole. I worry most that allowing the algorithm to the hunt
for the MTU will justify allowing a higher false positive rate for
black hole detection so that everybody suffers a performance hit even
when there are no black holes. There are no obvious ways to implement
that hunt which don't harm performance. I'd rather see it fall back
and stay back with an OS tunable for how confident it has to be of a
black hole before it falls back.

Regards,
Bill Herrin

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 29 Mar 2018 19:49:20 +0200

So, whats the summary ?

Write rfc4821bis with recommendations for an additional,
simplified procedure dropping to 1280/576 and use most simple upspeeding 
option ? Because 10 years have shown full rfc4821 is not widely supported
in host stack because its too complex ?

Another, potentially complementary option is to do an rfc3542bis
or some small "highly reliable socket" BCP recommendation for
apps requring highest reliability to use 1280/576 MTU through appropriate
socket API calls. Could be done n TAPS ?

In any case, it would be good to document the broken cases Mikael
mentions as well as the difficulties getting complex host stack
workarounds implemented broadly - otherwise it will be hard to persuade
the MTU/performance part of the community of any such recommendations.

Cheers
    toerless

-- 
---
[email-masked]

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 29 Mar 2018 22:41:06 +0200 (CEST)

On Thu, 29 Mar 2018, William Herrin wrote:

> How would PLPMTUD detect a path MTU black hole for DNS response packets 
> from an anycasted node? It's fire and forget; the server is not 
> expecting another packet from the querent.

Then the resolver needs to ask again, but make sure that the response is 
small enough. This is up to the protocol implementer to figure out. If the 
protocol implementer wants stateless unicast to work as a deployment 
scenario, then they need to design the protocol for PLPMTUD to work.

> packet loss or a brief path outage gets commonly misdetected as a PMTUD 
> black hole. I worry most that allowing the algorithm to the hunt for the 
> MTU will justify allowing a higher false positive rate for black hole 
> detection so that everybody suffers a performance hit even when there 
> are no black holes. There are no obvious ways to implement that hunt 
> which don't harm performance. I'd rather see it fall back and stay back 
> with an OS tunable for how confident it has to be of a black hole before 
> it falls back.

I am perfectly fine with the PMTUD blackhole algorithm kicking in after 5 
seconds of complete loss and TCP going back to slow start. I do not care 
much if performance is really bad when there is a PMTUD blackhole. What I 
want to avoid is that things don't work at all. Abysmal performance is 
better than things not working at all (which is often the case right now).

Also, if PMTUD blackhole detect kicks in, I'd like to be able to diagnose 
it with counters or logs from the TCP stack, telling me that this is going 
on.

---

From: [email-masked]
Date: Thu, 29 Mar 2018 16:51:19 -0400

On 2018-03-29 11:02, William Herrin wrote:
>>> So I have no idea how to come to a best conclusion regarding the 
>>> tradeoff between "doesn't work at all" and "doesn't work optimally in 
>>> some cases".
>> 

Agreed. I was speaking more to the general principle - we need to find 
something that helps work as much of the time as possible *first*, and 
worry about optimizing later (or perhaps not as much at all).

> fails.

Even E2E it wouldn't necessarily work. There are two issues at play 
here, both of which work against PMTUD:

1) in-band vs. out-of-band signaling
Relying on ICMPs to fix an issue with TCP might not work because the two 
protocols don't "fate share" (e.g., it's possible to block one and not 
the other)

2) active vs. passive feedback
PMTUD works only with active feedback (the presence of feedback 
signals).
PLPMTUD allows for passive feedback (i.e., the lack of feedback ensures 
recovery).

The fact that PMTUD also relies on participation of parties other than 
the receiving endpoint makes things worse, but it isn't the only 
problem.

I agree that problems with PLPMTUD implementation should be documented 
and hopefully fixed - either by revising the spec or fixing the 
implementations. It would be nice if we could start to rely on it rather 
than suffering from the consequences of PMTUD breakage.

And I don't care much *why* ICMPs are blocked, dropped, or never reach 
the source - i.e., whether this is operator misconfiguration, deliberate 
configuration, problems with NAT translation, or problems with tunnel 
ingress relay back to the source. The result is the same, and I don't 
think we can expect any of that to get better.

Joe

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 30 Mar 2018 07:38:31 +0200 (CEST)

> think we can expect any of that to get better.

Agreed. Things go wrong, and protocols need to handle this fact. I don't 
really care how they handle it, and I'm ok with performance being 
severely reduced in this "limp mode" (the equivalent of a internal 
combustion engine mode where it still runs but at a severely reduced 
performance level, just so you can safely get yourself to a mechanic).

Once we have this limp mode, then people can think of how to optimize 
this, if they so want. There are smart people here, I've already seen 
proposals that look like they are worthwhile pursuing, but I would really 
like to get in place a very straight forward solution for the "this will 
not work at all"-case, where in TCPs case it seems to me that this is 5-8 
seconds of total loss, back in slow-start, nothing is working. We can then 
have probing back to (larger) PMTUD packet sizes after some time. Frankly, 
I don't much mind if this TCP session goes back to this painfully bad 
state every 30 seconds to re-probe the PMTUD value, at least it'll not be 
completely broken.

I just want this default-on in all major TCP stacks, in the next few 
years. The simpler, the better and the higher chance that it'll get 
adopted, right?

---

From: Joe Touch <[email-masked]>
Date: Thu, 29 Mar 2018 22:46:14 -0700

On 3/29/2018 10:38 PM, Mikael Abrahamsson wrote:
> ...Frankly, I don't much mind if this TCP session goes back to this
> painfully bad state every 30 seconds to re-probe the PMTUD value, at
> least it'll not be completely broken.
>

It would be useful to be careful. I'd rather see a TCP that runs just
fine (but maybe not as fast as you think it should) with 400 byte
packets than one that keeps bashing its head against a rock trying to
find the "best" larger packet size, failing, and "limping" along that
way continually.

Joe

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 30 Mar 2018 08:20:01 +0200 (CEST)

> find the "best" larger packet size, failing, and "limping" along that 

I have no fundamental objections to this. The only question I guess needs 
to be answered is if there is an 10 second 100% packet loss outage 
situation, will all my TCP sessions now be in limp mode after that?

Because I imagine that if TCP now went into slow-start and that 576/1280 
byte packet happened to make it through when the network started working 
again, would the implementation deduce that the outage was caused by PMTUD 
blackhole, and keep itself in limp mode?

---

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 30 Mar 2018 08:23:25 +0100

On 30/03/2018, 07:20, Mikael Abrahamsson wrote:
> I have no fundamental objections to this. The only question I guess 
> needs to be answered is if there is an 10 second 100% packet loss 
> outage situation, will all my TCP sessions now be in limp mode after 
> that?
>
> Because I imagine that if TCP now went into slow-start and that 
> 576/1280 byte packet happened to make it through when the network 
> started working again, would the implementation deduce that the outage 
> was caused by PMTUD blackhole, and keep itself in limp mode?
>
Just to be sure, the email thread was entitled PLPMTUD, and yet it talks 
about PMTU blakcholes.

PL-PMTUD doesn't rely on ICMP messages to detect the PMTU. Sure, a 
transport using PLPMTUD can use these messages as a PART of the input to 
finding a PMTU, but it can also ignore them (e.g., when it can not 
verify that they apply to the node, or discovers that the  reported MTU 
is not appropriate). If you're adding PLPMTUD to TCP you need to work 
out how to handle probing using TCP segments and also whether/how to 
utlise the ICMP messages - this does not need to lead to a "limp" mode, 
as in ICMP black-hole detection.

Gorry

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 30 Mar 2018 10:04:38 +0200 (CEST)

> Just to be sure, the email thread was entitled PLPMTUD, and yet it talks 

I believe these are closely related. A PMTU blackhole detection algorithm, 
needs to do some PLPMTUD. It can be extremely stupid and simple, and just 
try a few packet sizes. Falling back to IPv4 68 byte minimum MTU would 
probably be too stupid.

> as in ICMP black-hole detection.

Agreed. So I want SOMETHING that works around PMTU blackholes, and I want 
people to deploy it. The smarter and advanced the algorithm, the less 
chance people will deploy it (it seems, since it's default off 
everywhere).

The fundamental problem I want to solve is PMTU blackhole. I want to solve 
this by doing PLPMTUD, and I want to discuss how to get this into a 
default-on state on basically everything shipped, and perhaps even into 
older versions that receive updates.

---

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 30 Mar 2018 09:17:15 +0100

On 30/03/2018, 09:04, Mikael Abrahamsson wrote:
>> Just to be sure, the email thread was entitled PLPMTUD, and yet it 
>> talks about PMTU blakcholes.
>
> I believe these are closely related. A PMTU blackhole detection 
> algorithm, needs to do some PLPMTUD. It can be extremely stupid and 
> simple, and just try a few packet sizes. Falling back to IPv4 68 byte 
> minimum MTU would probably be too stupid.
>
Yes.  I am all for exploring a simple set of packet sizes  and 
considering different algorithms to probe!
>> transport using PLPMTUD can use these messages as a PART of the input 
>> to finding a PMTU, but it can also ignore them (e.g., when it can not 
>> verify that they apply to the node, or discovers that the reported 
>> MTU is not appropriate). If you're adding PLPMTUD to TCP you need to 
>> work out how to handle probing using TCP segments and also 
>> whether/how to utlise the ICMP messages - this does not need to lead 
>> to a "limp" mode, as in ICMP black-hole detection.
>
> Agreed. So I want SOMETHING that works around PMTU blackholes, and I 
> want people to deploy it. The smarter and advanced the algorithm, the 
> less chance people will deploy it (it seems, since it's default off 
> everywhere).
>
Then I agree.
> The fundamental problem I want to solve is PMTU blackhole. I want to 
> solve this by doing PLPMTUD, and I want to discuss how to get this 
> into a default-on state on basically everything shipped, and perhaps 
> even into older versions that receive updates.
>
Indeed. I think once I (?) understand the algorithm components it should 
be possible to provide simple as well as sophisticated combinations - 
depending onthe  goal, and I do hope that at least onw will be simple 
and easy to implement in TCP. We could also explore whether a simplistic 
padding approach in Datagram PLPMTUD also is easier to add to a TCP 
implementation.

Gorry

---

From: William Herrin <[email-masked]>
Date: Fri, 30 Mar 2018 08:44:46 -0400

> So, whats the summary ?

Howdy,

My thoughts run this direction:

1. Recommend that router implementations "SHOULD" provide the option
to generate ICMPv4 type 3 code 4 (fragmentation needed) messages and
ICMPv6 type 2 (packet too big) messages from an address different than
the source interface address. They Dan't today, creating lots of cases
where the operator has to sacrifice PMTUD.

2. Recommend that firewall implementations "SHOULD NOT" filter ICMPv4
type 3 code 4 (fragmentation needed) messages or ICMPv6 type 2 (packet
too big) messages unless explicitly configured to do so. In other
words, the instruction to filter all ICMP will not result in dropping
fragmentation needed messages but the explicit instruction to drop
fragmentation needed messages specifically will. Today's firewall
tools make it too easy to accidentally kill PMTUD.

Let's maximize the chance of PMTUD working as designed and then build
something optimized for the residual cases where it still doesn't.

> option ?

That is #3. And I'd like to be smart about fallback at the expense of
efficiency in the black hole case. Send the 1280 packet. If it works,
send another full size packet to confirm it still fails before
accepting 1280 as the path mtu. If the 1280 and 576 packets don't work
either, don't try the fallback again: it's not path mtu. Maybe the RFC
is already smart about this; I didn't read closely enough.

4. Go back to the ground truths and reconsider whether we have
satisfactory information at layer 3 to make reliable decisions about
path mtu. Because of the fate sharing issue, I think PMTUD wants to be
a layer 4 problem implemented independently in each layer 4 protocol
with layer 3 offering hints from ICMP when available. Maybe this is
something that should be kicked over to the IRTF for investigation.

Regards,
Bill Herrin

---

From: Toerless Eckert <[email-masked]>
Date: Fri, 30 Mar 2018 16:04:57 +0200

William,

Thanks for making suggestions how the network can behave
better, no idea why i am starting to give up on that idea *sigh*

This part of the discussion would probably be better on [email-masked]

more inline.

> where the operator has to sacrifice PMTUD.

rfc1918/ULA on interfaces plus RPF later on ?
SP edge filtering of IF-address ranges ?
Any other known systematic reasons ?

Would be good to document systematic reasons to make sure suggested
workarounds can fix them.

The main problem iwth such a command is to make operators not forget
applying it. Or even having an address that would allow for the packet
to be passed back.

> tools make it too easy to accidentally kill PMTUD.

I am a lot more pessimistic of any form of IETF recommendation to
impact firewalls positively. At its core, most IETF design does not
even acknowledge the need for firewalls and has not designed network
services for their presence. I would also bet there are possible
attack vectors with ICMP fragmentation needed messages. And what else
does a fireall operator need to break something...

We can try of course. I think these days the way how you would be
achieve something like that is to define some YANG model that
contains the equivalent of such commands - You can make such a YANG
model standards track. CLI recommendations at best are BCP.

> something that should be kicked over to the IRTF for investigation.

Sure, you can bring the problem to PANRG.

The main issue is that cool new solutions will not fix broken old
equipment, so most efforts we seem to spend in IETF are about
workarounds for that broken old equipment (points 1..3).

And every time you try to
do something new & better, you get hammered down by IETF with
"oh, this will take a decade to see any adoption, go away". 

-- 
---
[email-masked]

---

From: Toerless Eckert <[email-masked]>
Date: Fri, 30 Mar 2018 16:10:11 +0200

Michael, question:

On Fri, Mar 30, 2018 at 10:04:38AM +0200, Mikael Abrahamsson wrote:
> want people to deploy it. The smarter and advanced the algorithm,
> the less chance people will deploy it (it seems, since it's default
> off everywhere).

Is it a problem of enabling it because its not on per default, or
is the more complex algorithm actually missing in the source code
of important OS ? I had the impression from the thread it was the latter ?!

Toerless

-- 
---
[email-masked]

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 30 Mar 2018 16:13:28 +0200 (CEST)

On Fri, 30 Mar 2018, Toerless Eckert wrote:

> Is it a problem of enabling it because its not on per default, or is the 
> more complex algorithm actually missing in the source code of important 
> OS ? I had the impression from the thread it was the latter ?!

When I am asking why it's not on by default, people claim it has 
drawbacks, that's why it's not on by default.

So to get SOMETHING on by default, I'm now trying to convince people to at 
least do PMTUD blackhole detection and workaround in the completely 
pathological case of "ok, I've had 5-10 seconds of outage, I'm in 
slow-start, let's see if a smaller packet can make it through".

If people can't agree that this kind of feature can be default-on, I don't 
know what could.

---

From: Joe Touch <[email-masked]>
Date: Fri, 30 Mar 2018 07:17:42 -0700

> On Mar 30, 2018, at 1:04 AM, Mikael Abrahamsson <[email-masked]> wrote:
> 
>> Just to be sure, the email thread was entitled PLPMTUD, and yet it talks about PMTU blakcholes.
> 
> I believe these are closely related. A PMTU blackhole detection algorithm, needs to do some PLPMTUD. 

IMO, this is looking at the problem the wrong way.

PMTUD and blackhole detection tries to find out what doesn’t work.

PLPMTUD tries to find out what does work. It can just as easily find other issues, such as conditions where large packets take paths with longer delay or higher loss (depending on how the feedback algorithm is written).

Basically, the probing system should be trying to find out the place where the connection has better goodput. That includes considering not only whether a packet gets there, but also with what latency, loss, and (if the TCP implementation is sensitive to this) reordering.

This is ultimately a kind of a Kalman filter with many inputs that need to control a few outputs - packet size, transmission rate, number of outstanding snack’d packets, and (possibly too) packet repeats.

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 30 Mar 2018 07:23:04 -0700

> On Mar 30, 2018, at 7:10 AM, Toerless Eckert <[email-masked]> wrote:
> 

AFAICT, there are other significant issues:

- NATs
- tunnels (esp. encrypted ones)

In both cases, an ICMP that arrives might not be able to be translated into the appropriate upstream indication - even if the device WANTED to participate.

Ultimately, getting info direct from the network as to what doesn’t work is always problematic. It can be useful - when available - for debugging, but IMO should never be the basis of normal operations.

There are two solutions to MTU discovery:
- for IPv4, use positive E2E feedback (PLPMTUD), but also let it control anything that helps goodput (i.e., PLPMTUD isn’t separate from congestion control or any other feedback)
- use IPv6 and go home (we shouldn’t be optimizing PMTUD for IPv6 - we should just be sticking to IPv6 MTU requirements

Joe

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 30 Mar 2018 16:28:12 +0200 (CEST)

On Fri, 30 Mar 2018, Joe Touch wrote:

> - use IPv6 and go home (we shouldn’t be optimizing PMTUD for IPv6 - we 
> should just be sticking to IPv6 MTU requirements

In the longer term, I want to be able to deploy jumbo frames and run 9k 
access network including customer LAN. This means that a stupid MTU1280 
fallback isn't optimal, even though I'll gladly take that if the 
alternative is no PMTU blackhole detect at all.

---

From: Joe Touch <[email-masked]>
Date: Fri, 30 Mar 2018 07:34:23 -0700

> On Mar 30, 2018, at 7:28 AM, Mikael Abrahamsson <[email-masked]> wrote:
> 
> In the longer term, I want to be able to deploy jumbo frames and run 9k access network including customer LAN. This means that a stupid MTU1280 fallback isn't optimal, even though I'll gladly take that if the alternative is no PMTU blackhole detect at all.

That “stupid” default MTU is the basis of all IPv6 performance comparison, IMO. 

Yes, it *might* be useful to run faster, but a system that supports 1280 isn’t a “fallback” - it’s the upper bound until you know otherwise. Sure, it might be useful to be able to know otherwise, but don’t expect everyone to engineer and deploy network devices that actively participate in your desire to optimize.

Again, this is why it’s important to fix this E2E. But it’s also why it’s important to fix this as a *system* of parameters, not just a single one. Do you really care whether your protocol uses 9K packets, or that it goes faster? Would you push to use a larger packet size of it increased loss or delay?

That’s why we need to look at this as a whole and not try to fix each parameter determination in isolation.

Joe

---

From: Toerless Eckert <[email-masked]>
Date: Fri, 30 Mar 2018 16:40:17 +0200

So, what would happen if TCP applications that do not need
max performance would default to 1280 ? And only those that need
would go through the trouble of higher MTU negotiation. Maybe
not even during startup but after some seconds (aka: only longer
lived flows, therefore also reducing the number of flows trying to
do PMTUD that could result in ICMP replies) ?

Network gear at places with high utilization is typically scoped
to perform @ linerate for 128 byte average or even smaller. I
can't see how this would therefore hurt useful equipment badly.
I may be totally wrong on this though, but mileage of "useful"
varies widely ;-)

Might even create another interesting throughput distinction
between "default is fine" and "high throughput" flow,
although i totally can not guess the outcome. Eg: Everything else
being equal, a congestion point has a bunch of TCP flows with 1280
and another bunch with 8192. Q: whats going to be their
bandwidth share (with default queuing or one of our AQM schemes on
the congestion queue) ?

Cheers
    Toerless

On Fri, Mar 30, 2018 at 04:13:28PM +0200, Mikael Abrahamsson wrote:
> So to get SOMETHING on by default, I'm now trying to convince people
> to at least do PMTUD blackhole detection and workaround in the
> completely pathological case of "ok, I've had 5-10 seconds of
> outage, I'm in slow-start, let's see if a smaller packet can make it
> through".
> 
> If people can't agree that this kind of feature can be default-on, I
> don't know what could.
> 
> -- 

-- 
---
[email-masked]

---

From: Joe Touch <[email-masked]>
Date: Fri, 30 Mar 2018 08:00:45 -0700

> On Mar 30, 2018, at 7:40 AM, Toerless Eckert <[email-masked]> wrote:
> 
> 
> max performance would default to 1280 ?

For IPv6, nothing should happen.

> And only those that need
> would go through the trouble of higher MTU negotiation.

Which, FWIW, would be fairly rare - yes, bulk transfers between datacenter might care, but in many other cases, it wouldn’t come up.

> Maybe

Sure...

> I may be totally wrong on this though, but mileage of "useful"

Going faster for larger packets is typically an end system optimization, not a network path one.

> bandwidth share (with default queuing or one of our AQM schemes on

ABC should help with that, but yes, there’s also the issue of the “granularity” of congestion control. It helps if the send window is typically composed of around 10-20 “units of congestion feedback”. This is why it actually can help some flows to use smaller packets - esp. over low speed or congested paths.

Again, this should all go into something that would look a lot like a Kalman filter, IMO.

Joe

---

From: William Herrin <[email-masked]>
Date: Fri, 30 Mar 2018 11:21:26 -0400

On Fri, Mar 30, 2018 at 10:04 AM, Toerless Eckert <[email-masked]> wrote:
>> the source interface address.
>

Hi Toerless,

These are the most common ones AFAIK.

> Any other known systematic reasons ?

"Don't ping me bro." ACL drops ICMP except to administrative addresses.

"Can't get there from here." No route to the source available to the
objecting router.

MPLS. Objecting router doesn't have the protocol wrapped inside the
oversize MPLS frame.

> applying it. Or even having an address that would allow for the packet

Start with making it possible for operators to apply it. If we need
more, ask IANA to assign an IP address dedicated to fragmentation
needed / packet too big messages and define the conditions in which
routers should select that standard address by default instead of the
interface local address. Perhaps when the interface is configured with
an RFC1918 address but the source of the offending packet is not an
RFC1918 address. Then adjust reverse path filtering to pass that
standard address. Maybe some delay so that passing the standard
address becomes a must for RPF implementations immediately but the
default substitution of the standard address should not be enabled by
default in implementations for a few years.

Tunables all around so that sufficiently savvy (or sufficiently
foolish) operators can force the behavior.

> I would also bet there are possible
> attack vectors with ICMP fragmentation needed messages.

One obvious attack vector is to block them. ;)

I do take your point though.

Regards,
Bill