Subject: [pmtud] Re: [dccp] PMTU issues

From: Eddie Kohler <[email-masked]>
Date: Fri, 14 Nov 2003 08:01:10 -0800

Hi Tom,

> 1)  Start the connection by sending some probe packets.  Note that this
> means getting feedback that they were delivered before going on with the
> real work.

Yes, but this could take just 1 RTT to set up -- send 4 packets back to
back with a range of sizes and wait to see what gets acked.

> 2)  Never send packets bigger than some size that's likely to be OK, but is
> probably less than the PMTU that DCCP says.  And be prepared to adjust down
> from that in the few situations where it's necessary.  It's probably pretty
> difficult to come up with this recommended "safe" PMTU.

On IPv4 networks, at least, there is a third option:

3) Send without DF and let the network fragment.  Perhaps the receiver, in
   an acknowledgement, could send an option back saying "This arrived
   fragmented".  Problem: IPv6 compatibility.

The new PMTU draft would suggest a fourth:

4) Start smaller and gradually increase your size until you experience loss.
   Use a pattern such as 

   small, small, small, small, large, small, small, small, small, ...

   Wait until the "large" has been acknowledged to increase your PMTU.  We
   could engineer it so that this probing happened independent of the
   application (for example, DCCP could send an Ack with a large payload
   area, which by definition will be ignored), so that app loss was not a
   problem.  Problem: This assumes the app can be told at any time that it
   should increase its MTU.  It may be difficult for applications to change
   their packet sizes dynamically.

Michael Welzl suggests a fifth, which would require router changes:

5) IP options for checking the PMTU.

Eddie

_______________________________________________
pmtud mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/pmtud

---

From: Michael Welzl <[email-masked]>
Date: 14 Nov 2003 17:08:36 +0100

Hi,

> 5) IP options for checking the PMTU.

Just to avoid confusions (god knows I've had my share of "why do
you want to change all routers in the world?" discussions...):

I am _NOT_ suggesting this as another option (i.e. instead
of this or that PMTU method). It is meant as a supplement:
you do it before you do PMTUD, and if you get something with
this, you use it as an upper limit for PMTUD, so with some
luck, you have no loss.

And I'm not suggesting to do this all the time and under
all circumstances. It could help with tunnels, and it is
an optional scheme that, e.g., applications on top of UDP
or DCCP _MAY_ want to use. Clearly, it is experimental.

Cheers,
Michael

---

From: [email-masked] (Jun-ichiro itojun Hagino)
Date: Sat, 15 Nov 2003 01:13:20 +0900 (JST)

> 5) IP options for checking the PMTU.

	from our experience with VIP (mobility solution proposed by Fumio
	Teraoka before mobile-ip4 is standardized), many of the routers in the
	wild drops any packet with IP option.  so i don't think it is
	deployable.

itojun

---

From: Michael Welzl <[email-masked]>
Date: 14 Nov 2003 17:24:47 +0100

> 	wild drops any packet with IP option.  so i don't think it is

I would agree if this was meant as a _REPLACEMENT_ for PMTUD.
But since it is an additional experimental feature that could
be used by applications that desire reduced packet loss and
can live with having a packet delayed, the situation is different.

We did measurements in 2001 and 2003 where we successfully
traversed some 72000 and 80000 different router addresses
using the "NOP" option, respectively - this suggests that
a lot of routers don't drop packets that carry KNOWN options.
Rather, I believe that they tend to drop packets with
UNKNOWN options, but that is a general problem when you
introduce something new.

To me, it's good enough for such experimental usage.

Cheers,
Michael

---

From: [email-masked] (Jun-ichiro itojun Hagino)
Date: Sat, 15 Nov 2003 01:38:17 +0900 (JST)

> UNKNOWN options, but that is a general problem when you

	i may not have been clear enough:
	the drops were not because of new option, but because of the presense
	of option (ip_hl != 5).  the most critical one was that a major router
	which connects US and JP drops any packet with IP option, so we could
	not use VIP, which is mobility protocol, to roam between US/JP.

itojun

---

From: Fred Templin <[email-masked]>
Date: Fri, 14 Nov 2003 09:54:47 -0800

I hate to say it, but frankly I think this whole PMTU business is
a bunch of hooey. We have RFCs 1191 and 1981 as the service for
packetization layers that require network level feedback, and those
packetization layers can happily continue doing what they've been
doing for the past decade or so.

But, new packetization layers that take the example of PLPMTUD
require no feedback from the network and so they should have a
means by which to turn the network layer feedback off. This should
eventually dampen the noise from the unnecessary ICMP's as the
new packetization layers supplant the old.

Anyway, that's my story and I'm sticking to it.

Fred
[email-masked]

---

From: "Michael Welzl" <[email-masked]>
Date: Fri, 14 Nov 2003 22:08:47 +0100

Dear all,

The obvious reason for Fred's opinion is the lack of proof that shows
the difference between the various schemes (or maybe, such proof
would show that it IS totally irrelevant what we do in practice?).

I would like to have the student who did the IP Option measurements
examine the efficiency / advantages / disadvantages of PLPMTUD vs.
1191-PMTUD with and without my option idea ( you guessed it   :)   )
as the next step, but I wonder how to do this ... one thought is to start
with an examination of the Linux 1191 code because it really depends
on the implementations.

Any suggestions?

Cheers,
Michael

----- Original Message ----- 
From: "Fred Templin" <[email-masked]>
To: "Jun-ichiro itojun Hagino" <[email-masked]>
Cc: <[email-masked]>; <[email-masked]>; <[email-masked]>;
<[email-masked]>; <[email-masked]>; <[email-masked]>
Sent: Friday, November 14, 2003 6:54 PM
Subject: Re: [pmtud] Re: [dccp] PMTU issues

---

From: Fred Templin <[email-masked]>
Date: Fri, 14 Nov 2003 13:52:47 -0800

>1191-PMTUD with and without my option idea ( you guessed it   :)   )

See the introduction of my document for disadvantages of any scheme
that relies on untrusted network elements to behave correctly.  Such
schemes include both 1191 and 1063-style options. Current document
is at:

  www.geocities.com/osprey67/tunnelmtu-06.txt

>with an examination of the Linux 1191 code because it really depends

OK, here's one. When we have some assurance that the packetization
layer is doing something akin to PLPMTUD, then it is desireable to
disable the network mechanism that generates ICMP packet too big
and/or frag needed messages (perhaps only the latter). In IPv4, the
100% certain way to do this is to NOT set the DF bit in the IPv4 header.

So, for IPv6-in-IPv4 tunnels, if we send packets identified as being
part of a flow that is doing PLPMTUD with the DF bit NOT set
in the encapsulating IPv4 header then we can ask that the decapsulator
drop any packets larger than 1280 bytes that it observes to have been
fragmented by the IPv4 network. The packetization layer doing the
PLPMTUD will notice the packet as being dropped (i.e., w/o having
rec'd an ICMP indication) and will not increase its current estimate
of the Path MTU.

Only question - how can we know that the decapsulator is
going to drop the frag's?

Fred
[email-masked]

>> >>> Teraoka before mobile-ip4 is standardized), many of the routers in
>>    
>>
>the
>  
>

---

From: Fred Templin <[email-masked]>
Date: Fri, 14 Nov 2003 17:56:45 -0800

BTW, one last parting thought on this subject (and then I'll
shut up) is that we have perhaps an opportunity to specify
the following good thing:

  A packetization layer should set an ECN codepoint in
  the packets it sends IFF it is also doing Packetization
  Layer Path MTU Discovery (PLPMTUD) and is not
  expecting to get ICMP's back from the network in
  response to too-large packets being dropped.

I have already implied this in my document, which should
hit the I-D repository soon. See:

But, perhaps this needs to be spelled out in a more general-use
type of document, e.g., a per-hop behavoir (PHB) document for
RFC 3168?

Thanks - Fred
[email-masked]

---

From: Eddie Kohler <[email-masked]>
Date: Fri, 14 Nov 2003 19:04:56 -0800

>   response to too-large packets being dropped.

Why overload the ECN bit this way?  ECN should be used to indicate
end-to-end congestion control compliance.  In fact RFC 3168 requires that:
"... the transport protocol must be capable of reacting appropriately to
the receipt of CE packets." THat's independent of MTU discovery, or at
least should be pointed out.

Eddie

---

From: Fred Templin <[email-masked]>
Date: Mon, 17 Nov 2003 12:12:10 -0800

Eddie,

A transport protocol that is smart enough to react to the receipt of
CE packets should also be smart enough to figure out the size of the
largest piece of data that can traverse the network without incurring
fragmentation. Such a "smart transport" should be able to figure this
out with *no* feedback from unreliable and untrustworthy ICMP
messages from the network if the example from Packetization
Layer Path MTU Discovery (PLPMTUD) is taken.

So, if the ICMP's are unneeded, I believe it is desireable to have
a way to turn them off. This would serve the beneficial purpose
of reducing the overall congestion in the Internet as deployment
of transports that don't need the ICMPs ramps up

Thanks - Fred
[email-masked]

---

From: Eddie Kohler <[email-masked]>
Date: Mon, 17 Nov 2003 17:39:19 -0800

Hi Fred,

* PLPMTUD is useful.
* Designing PMTUD so that it works in the absence of ICMP feedback seems
  necessary.

BUT

* Suitable ICMP feedback hints might significantly improve the performance
  of a transport protocol.
* We can program our transports to react to ICMP as a hint -- i.e., not
  trust it, but use it to optimize performance.
* So ICMP should not be "needed", but it might, and probably would, be quite
  helpful in some cases.
* For instance, not all packetization layers have as easy a time as TCP
  with packet size changes.  The smooth ramp-up suggested in PLPMTUD may
  require intervention from the application for example.  For good
  performance, these applications may apply PMTUD in unexpected ways --
  they might start large, for example.  ICMP feedback would really help
  them.
* ICMP is not a significant cause of Internet congestion and need not ever
  become one (mark it less-than-best-effort).

I still think your overloading of ECN capable as "PLPMTUD capable, don't
send ICMP" is not necessary, a bad idea, and will not fly.

Eddie

---

From: Michael Welzl <[email-masked]>
Date: 18 Nov 2003 08:38:03 +0100

... I agree 100% with Eddie on these two (1. "we should
get rid of ICMP feedback in the long run", 2. "combine PMTUD
with ECN") issues.

Cheers,
Michael

-- 
Michael Welzl <[email-masked]>
University of Innsbruck

---

From: Fred Templin <[email-masked]>
Date: Tue, 18 Nov 2003 15:47:52 -0800

Eddie,

Your opinion is from the perspective of a modern-day expert
technologist with specific insight into the problem space, and
I have a great deal of respect for that. My opinion brings an
historical perspective that I believe also bears consideration:

I was very close to "ground-zero" when the current RFC-1191
style Path MTU discovery method took shape. In 1988-1990, I
was heading up the team that developed ULTRIX support for
Digital's initial FDDI product offering. The initial offering
included an FDDI wiring concentrator (i.e., a fancy hub), two
host adapters, and an Ethernet-to-FDDI L2 bridge.

This latter product presented the interesting problem of how to
support path MTU discovery in the presence of bridged media
with diverse MTUs. The oft-cited problem case was the "dumbell
configuration" in which two FDDI rings were on either side of
an Ethernet pipe - in this example, how could hosts A and B
on FDDI rings know whether an Ethernet was on the path?

Some interesting alternatives were kicked around, including
marking L2 packet headers with a bit to indicate that an Ethernet
was traversed. But, a vocal minority at that time insisted that the
only way to go was to either have the bridge support IPv4
fragmentation, or have it send "fragmentation needed"
messages when dropping a packet with the DF bit set.

The vocal minority got their way, but the chosen alternatives
disqualified the opportunity to support transparent L2 bridging
between media with diverse MTUs. Some made valiant attempts
to market "brouter" or "transluscent bridge" products, but the
vocal minority was easily able to shoot these down with counter-
marketing and the advent of L3 routing (i.e., "smart networks")
took flight.

Since then, we have seen the boom and subsequent bust of the
Internet revolution. It would be a far stretch to say that this all
came as result of the path MTU discovery decisions, but I do
believe it fair to say that a harmful precedent was set: folks
started to believe they could engineer the network with no
regard for the End-to-End Principle.

So, where does this leave us today? We have an unreliable,
untrustworthy (and, frankly, noisy) network-based mechanism
that only works when forwarding nodes get involved with
inspecting IPv4 packet headers. We'd like to be able to hook
a dumb 9KB MTU Gbps Ethernet hub to a dumb 1500B MTU
10/100 Ethernet hub and enable the larger MTUs, but we can't
because we blew the option out of the water back in the good
old days and we have based so many of our architectural
decisions on that precedent since.

Now, with the emergence of techniques like PLPMTUD, we
have the opportunity for healing by allowing new packetization
layers, APIs, etc to gracefully supplant the old network-based
mechanism. I envision an Internet restored to the End-to-End
Principles, with new opportunities for innovation enabled by
seamless support for L2 media with diverse MTUs.

So, in response to a network that keeps endlessly screaming:

  "Packet Too Big!"
  "Fragment Needed!"

 I say:

  "Turn Off The Noise!"

Fred
[email-masked]

---

From: Eddie Kohler <[email-masked]>
Date: Tue, 18 Nov 2003 16:50:37 -0800

> believe it fair to say that a harmful precedent was set:  ...

> inspecting IPv4 packet headers.  ...

Well, I wouldn't want to be responsible for any further destruction of the
Internet revolution, but most of this seems besides the point.  Old-style
PMTUD, which *depended* on the delivery of ICMPs, is a far cry from
new-style PMTUD, which *can benefit* from the delivery of ICMPs.  The
problem was the PMTU algorithm, not the idea of ICMPs.  And the network
isn't full of useless ICMPs.

Eddie

---

From: Fred Templin <[email-masked]>
Date: Tue, 18 Nov 2003 17:29:05 -0800

>isn't full of useless ICMPs.

Nonetheless, I believe we should keep options open and investigate
ways to support L2 devices that don't send ICMPs and don't support
IP fragmentation without relegating them to second-class citizens.
PLPMTUD gives us one such tool, but does it *really need* the ICMPs?

We got this wrong on the first iteration some 15 years ago, and if there
is now an opportunity to set things right we should at least give it serious
consideration and make the right informed decisions.

Thanks - Fred
[email-masked]

---

From: Michael Welzl <[email-masked]>
Date: 19 Nov 2003 08:18:16 +0100

Fred,

Let us reconsider the end-to-end arguments:
They are NOT about keeping anything and everything out
of the network, not placing any helpful additional mechanisms
inside the network, and not putting any complexity inside the
network.

Routing algorithms, for instance, can be arbitrarily complex,
and still, it is all very much in line with the end-to-end
arguments. There is also nothing wrong with network mechanisms
that help end systems (see "network transparency" in the
"e2e and active networks" paper: lower layers should provide
"...usable means for effective sharing of resources and
resolution of resource conflicts."); the idea is not to place
any kind of application level functionality inside the
network.

So, yes: since only the application knows the packet size it's
going to send, it's a good idea to leave it up to the application
to determine the MTU. I believe that you're right about this -
this became evident when I read the update of your tunnel draft,
which has become much shorter and simpler.

Not relying on ICMP packets is indeed a big step - but
utilizing ICMP packets as an optional enhancement is something
entirely different. In particular, the difference is that
ISPs can find out that it was a bad idea (too many ICMP's, too
much work for routers, ...) AFTERWARDS and simply turn it
off if they want to. So there's really not much harm done.

Cheers,
Michael

---

From: Michael Welzl <[email-masked]>
Date: 19 Nov 2003 08:54:11 +0100

... on second thought, there may be a security issue with these
ICMP messages:

what happens if I, as an evil attacker, send ICMP "fragmentation
needed" messages with a REALLY small MTU value to a host?

Actually, what happens when I do this nowadays?! Is this
a common DoS attack? How can it be prevented?

Maybe I'm missing something here...

Cheers,
Michael

---

From: Eddie Kohler <[email-masked]>
Date: Wed, 19 Nov 2003 06:13:32 -0800

Hi Michael,

(1) You can't send a Too Big message unless you have the original packet
    headers;
(2) That's exactly why we need to be quite careful about any algorithm for
    reacting to ICMPs.  We need to treat them as "hints" -- hints that
    might be wrong.

Eddie

---

From: Fred Templin <[email-masked]>
Date: Wed, 19 Nov 2003 13:49:09 -0800

Michael,

No, I don't think you're missing anything. In fact, a compromised
man-in-the-middle box could get even more diabolical and use
"packet too big" messages to cause the source to count down by
small increments, e.g.:

  src                            middlebox
  ***                           *********
  2000byte pkt -->  
                                    <--  pkt too big (MTU=1999)
  1999byte pkt -->
                                     <-- pkt too big (MTU=1998)
  1998byte pkt -->
                                     <-- pkt too big (MTU=1997)
         ...                                           ....

Sure, a smart packetization layer could easily recognize the
strategy and refuse to take the bait. But, there is not even any
guarantee the the initial "pkt too big" would include a legitimate
value - it could well be that the legitimate path MTU to the
destination was 9K!

If you trust *only* what you hear from the other endpoint then
the decision is binary - either the big packet got there or it didn't.

Fred
[email-masked]

Michael Welzl wrote:

>.. on second thought, there may be a security issue with these
>ICMP messages:
>

---

From: Michael Welzl <[email-masked]>
Date: 20 Nov 2003 08:07:11 +0100

Hi,

I see - still I think that using those messages as a *hint*
may be useful (faster convergence etc.). And, as you state,
a smart packetization layer could detect such problems.

I wonder if we should collect these thoughts and write a
separate draft for the PMTUD group called something like
"PMTUD security issues and recommendations" ?

That might be quite a useful document...

Matt+Matt - what do you think?

Cheers,
Michael

---

From: Fred Templin <[email-masked]>
Date: Thu, 20 Nov 2003 09:36:04 -0800

What aspects of the example I offered provided anything
that could serve as an even halfway-useful hint? Each end
every one of the ICMPs in the example were nothing more
than dis-information!

Fred
[email-masked]

---

From: Michael Richardson <[email-masked]>
Date: Wed, 19 Nov 2003 18:13:12 -0500

>>>>> "Michael" == Michael Welzl <[email-masked]> writes:
    Michael> ... on second thought, there may be a security issue with these
    Michael> ICMP messages:

    Michael> what happens if I, as an evil attacker, send ICMP "fragmentation
    Michael> needed" messages with a REALLY small MTU value to a host?

  1) Implementations are supposed to take it as a hint. 
  2) there is a set of "reasonable" values.
  3) 576 is the minimum for IPv4.

]       ON HUMILITY: to err is human. To moo, bovine.           |  firewalls  [
]   Michael Richardson,    Xelerance Corporation, Ottawa, ON    |net architect[
] [email-masked]      http://www.sandelman.ottawa.on.ca/mcr/ |device driver[
] panic("Just another Debian GNU/Linux using, kernel hacking, security guy"); [

---

From: "Michael Welzl" <[email-masked]>
Date: Thu, 20 Nov 2003 23:07:56 +0100

Hi,

I meant "take ICMP fragmentation needed messages as a hint", not
"take exactly the messages in your diagram as a hint"   :)

What inspired me to say we should collect these thoughts is this
comment of yours:

>>strategy and refuse to take the bait

... I don't know any document describing how a "smart packetization
layer" should exactly behave in this aspect.

It should not really be a separate document ... but maybe an extended
security considerations section as a contribution to the PMTUD group
work?

Cheers,
Michael

---

From: Fred Templin <[email-masked]>
Date: Thu, 20 Nov 2003 14:42:21 -0800

Yes, but on further consideration I think the "smartest" packetization
layer would be the one that simply ignores all ICMPs. That way, there
is no opportunity for confusion by dis-information.

(I believe the behavior would be very easy to document as well! :^} )

Fred
[email-masked]

Michael Welzl wrote:

>>>>>>>>Well, I wouldn't want to be responsible for any further destruction
>>>>>>>>                
>>>>>>>>
>of the
>  
>
>>>>>>>>Internet revolution, but most of this seems besides the point.
>>>>>>>>                
>>>>>>>>
>Old-style
>  
>
>>>>>>>>new-style PMTUD, which *can benefit* from the delivery of ICMPs.
>>>>>>>>                
>>>>>>>>
>The
>  
>
>>>>>>>>problem was the PMTU algorithm, not the idea of ICMPs.  And the
>>>>>>>>                
>>>>>>>>
>network
>  
>
>>>>>>>We got this wrong on the first iteration some 15 years ago, and if
>>>>>>>              
>>>>>>>
>there
>  
>
>>>>>>>is now an opportunity to set things right we should at least give it
>>>>>>>              
>>>>>>>
>serious
>  
>

---

From: "Michael Welzl" <[email-masked]>
Date: Fri, 21 Nov 2003 00:00:22 +0100

> layer would be the one that simply ignores all ICMPs. That way, there

I know    :)    ... and I still disagree. Let's leave it at that   :)

Cheers,
Michael