Subject: [Int-area] Tunneling/MTU BCP, was: Re: Mucking with IP ID

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 2 Aug 2007 13:57:26 +0200

[to our little ad hoc group AND internet area list]

On 2-aug-2007, at 1:38, Templin, Fred L wrote:

> Instead, I have proposed a BCP approach on the int-area
> mailing list. Here's what I said there:

> + Focusing only on the ULP, can we say that 1500 bytes is
> + the nominal MTU of today's Internet, and therefore all
> + attached devices SHOULD configure a minMRU of 1500?

We can say that, but is such a statement helpful? I can't remember  
ever seeing someone reduce their MTU below 1500 when they are  
physically and logically capable of receiving 1500 byte packets.

> Then,
> + can we also say that applications that send packets larger
> + than 1500 bytes (with DF=1) are RECOMMENDED to use RFC4821?

RFC 4821 should be recommended whenever path MTU discovery is used,  
which is pretty much in all implementations. PMTUD black holes are  
often the result of MTUs lower than 1500 in the path so limiting this  
recommendation to 1500+ interfaces makes little sense.

> + Finally, could we also say that tunnel decapsulators SHOULD
> + configure a minMRU of 2048 to account for encapsulations
> + that might extend a 1500 byte ULP out to the IEEE 802.as
> + maximum frame size?

Not sure what you mean here. Obviously people should be prepared to  
receive the largest possible packets supported by the hardware  
configuration in use. That means 1500 bytes for ethernet, or whatever  
jumboframe size is in effect. Trying to receive more than the  
hardware will do is an exercise in futility...

However, we CAN recommend that people make their tunnel MRUs equal to  
the size of the largest physical interface MRU - encapsulation size,  
even if the tunnel MTU is smaller than that. But I would recommend  
making the tunnel MTU equal to that MRU rather than use the IPv6-in- 
IPv4 tunneling practice of a 1280 byte MTU to accommodate multiple  
levels of encapsulation.

_______________________________________________
Int-area mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/int-area

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 2 Aug 2007 14:56:38 +0200

On 2-aug-2007, at 13:57, Iljitsch van Beijnum wrote:

[...]

Looks I wasn't done yet... Sorry for the bandwidth use, I should  
really optimize my Maximum Message Unit.

Thinking about tunneling/MTU/PMTUD:

There is a recommendation floating around that suggests setting the  
outer tunnel header DF bit to the value of the inner header DF bit.  
The trouble with this is that it only works if:

A. RFC 4821 is used, which is generally something the tunnel operator  
can't know (also note that RFC 4821 capability is per transport, so  
even if TCP has it other transports are likely to still fail on PMTUD  
black holes), or
B. Packet too big messages generated for the outer tunnel header are  
translated into packet too big messages relating to the inner tunnel  
header, something that I don't think can be done in all cases with  
IPv4 because not enough of the original packet is returned

So for IPv4, I don't think copying the DF bit makes sense. This  
leaves two options:

1. Do PMTUD for the tunnel
2. Don't do PMTUD for the tunnel

In the first case, the outer header DF bit should be set and the  
tunnel source should dynamically adjust the tunnel MTU based on too  
big messages that come back for tunneled packets. Then, any large  
packets with the DF bit set will be dropped and a too big is sent  
back to the source. Large packets with the DF bit cleared will have  
to be fragmented. (Yes, you can always try to send back a too big for  
these too, but if they're ignored you'll have to implement logic to  
rate limit the too bigs.) So you do have fragmentation overhead on  
the tunnel ingress point but no reassembly overhead at the tunnel  
egress point.

In the second case, the tunnel MTU will still probably be something  
like 1480 so incoming packets with DF set will be dropped and a too  
big sent back, but an administrator could choose to set the tunnel  
MTU to 1500 (or bigger) and accept the fact that all large packets  
will be fragmented at the tunnel ingress point and reassembled at the  
tunnel egress point. Since the DF bit is cleared, any hops with a  
reduced MTU will lead to fragmentation in the middle with reassembly  
at the tunnel egress point.

Then we have IPv6. Since DF is implied here, translating too bigs  
when the outer header is IPv6 is possible, because ICMPv6 returns  
enough of the original packet. But since it's impossible to carry a  
minimum maximum sized IPv6 packet of 1280 bytes through a tunnel  
running over a link with the minimum MTU of 1280 bytes, it's  
extremely important to set link MTUs higher than 1280 bytes wherever  
possible, or black holes that can't be recovered from will be the  
result.

An interesting question in this regard is what the maximum packet  
size should be for applications/transports that can't adjust their  
packet size dynamically and/or don't implement RFC 4821. For IPv6,  
the obvious answer is 1280, but maybe this is a bit too conservative  
for a general recommendation. I know that 1450 is often recommended  
for use with streaming video, but in practice this becomes 1478 with  
IP and RTP headers. (1498 for IPv6...) For IPv4, this leaves enough  
room for an extra IP header (20 bytes) OR a PPPoE header (8 bytes),  
but not for both or for GRE encapsulation (at least 24 bytes).

Some choices and the extra headers they allow for:

1492: PPPoE
1480: PPPoE / IPv4
1476: PPPoE / IPv4 / IPv4 + GRE
1472: PPPoE + IPv4 / IPv4 + GRE
1460: PPPoE + IPv4 / IPv4 + GRE / 2 x IPv4 / IPv6
1452: PPPoE + 2 x IPv4 / 2 x IPv4 + GRE / PPPoE + IPv6

---

From: Joe Touch <[email-masked]>
Date: Thu, 02 Aug 2007 06:18:18 -0700

> [to our little ad hoc group AND internet area list]
...
> jumboframe size is in effect. Trying to receive more than the hardware
> will do is an exercise in futility...

RFC1122 already requires that they be able to receive and reassemble to
the size of the largest packet of their directly attached interfaces.

> the size of the largest physical interface MRU - encapsulation size,

Which physical interface? If directly attached, then that's already
required by 1122, since a tunnel endpoint acts as a host (it sources
and/or sinks packets). If some other interface, e.g., along the path of
a tunnel, that cannot be required any more than for any other host -
that is the purpose of PMTUD to discover anyway. I doubt we can or
should require PMTUD for tunnel hosts, since tunnel encapsulation
protocols are not necessarily amenable to RFC4821.

> making the tunnel MTU equal to that MRU rather than use the IPv6-in-IPv4
> tunneling practice of a 1280 byte MTU to accommodate multiple levels of
> encapsulation.

Any guess of a specific size is likely to be incorrect. Some tunneling
mechanisms use quite a bit of space (e.g., IPsec tunnel mode), and the
number of levels of encapsulation that might occur - especially as we
move forward with more VPN and overlays - will be hard to predict.

Joe

-- 
----------------------------------------------------------------------
Joe Touch                Sr. Network Engineer, USAF TSAT Space Segment
               Postel Center Director & Research Assoc. Prof., USC/ISI

---

From: Joe Touch <[email-masked]>
Date: Thu, 02 Aug 2007 06:30:20 -0700

Hi, all,

Iljitsch van Beijnum wrote:
> Looks I wasn't done yet... Sorry for the bandwidth use, I should really
> optimize my Maximum Message Unit.
> 
> outer tunnel header DF bit to the value of the inner header DF bit. The
> trouble with this is that it only works if:
> 
> can't know (also note that RFC 4821 capability is per transport, so even
> if TCP has it other transports are likely to still fail on PMTUD black
> holes), or
> header, something that I don't think can be done in all cases with IPv4
> because not enough of the original packet is returned
> 
> So for IPv4, I don't think copying the DF bit makes sense. This leaves
> two options:
> 

How can a tunnel do PMTUD if it is:
	a) unidirectional
	b) uses IP or IPsec as encapsulation (4821 describes methods
	for connection-oriented protocols, e.g., TCP and SCTP)

Overall, conventional PMTUD (via ICMP response) SHOULD be supported at
tunnel endpoints, but even if it is, there's no reason to expect that
tunnels will receive ICMPs any more than endpoints. (i.e., they tend to
be dropped by firewalls, etc.)

4821-PMTUD cannot be supported at most tunnel endpoints, AFAICT.

IMO,

tunnel endpoints MUST either:

	1. set outer DF=0 and allow fragmentation (including at the
	tunnel source

	2. set outer DF=1 when their payload fits, or
	drop the payload (when it doesn't) and send ICMPs back at
	that time

Receipt of a too-big at the tunnel source should not be expected to be
translated to be sent to the original packet's source; it MAY be if
there is enough header to do so. The primary benefit of receiving such
messages is for subsequent packets; the tunnel source would decrease its
MTU, and then **other** packets from that source (or any other source)
would correct the actions above (#1 would make smaller fragments, #2
would generate ICMPs back to the source).

These rules apply equally to IPv4 and IPv6; in neither case should
tunnels fragment the encapsulated packet, IMO. Tunnels act like links,
and tunnel endpoints act like hosts - to the tunneled packet. Tunnels
should be transparent (and thus undetectable) to the packets that they
transit.

> An interesting question in this regard is what the maximum packet size
> should be for applications/transports that can't adjust their packet
> size dynamically and/or don't implement RFC 4821. For IPv6, the obvious
> answer is 1280, but maybe this is a bit too conservative for a general
> recommendation. I know that 1450 is often recommended for use with
> streaming video, but in practice this becomes 1478 with IP and RTP
> headers. (1498 for IPv6...) For IPv4, this leaves enough room for an
> extra IP header (20 bytes) OR a PPPoE header (8 bytes), but not for both
> or for GRE encapsulation (at least 24 bytes).
> 

There are many other cases - notably IPsec tunnels, which consume even
more bytes. Tunnel endpoints may employ header compression which may
somewhat compensate for size inflation too. IMO, it's not useful to
guess these sizes or expected layerings, as the use of layered VPNs and
overlays is likely to increase over time.

Joe

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 2 Aug 2007 16:26:11 +0200

On 2-aug-2007, at 15:30, Joe Touch wrote:

>> leaves

> 	for connection-oriented protocols, e.g., TCP and SCTP)

Same as elsewhere: send packets with DF bit set, observe ICMP too bigs.

> be dropped by firewalls, etc.)

Well, that consideration factors into the decision to either 1. do  
PMTUD, or 2. don't do PMTUD. I'm not sure what people use tunnels  
for. When I have occasion to use them, obviously the too bigs aren't  
filtered.

> 4821-PMTUD cannot be supported at most tunnel endpoints, AFAICT.

Right, it works at the transport layer.

> 	1. set outer DF=0 and allow fragmentation (including at the

So far so good...

> 	2. set outer DF=1 when their payload fits,

...but this makes no sense at all. The whole point of PMTUD is to  
_find_ _out_ whether stuff fits. You can't know that in advance.

> translated to be sent to the original packet's source;

Not for IPv4. For IPv6, that would be a valid choice but handling  
this in the same way as IPv4 would also be fine.

> The primary benefit of receiving such

Right. Note that TCP tends to send out two packets at a time, so with  
this in effect the first packet will trigger PMTUD in the tunnel, but  
by then, the second packet is also on its way, so both packets will  
be lost and TCP will probably stall for some time. Then when the  
third packet comes, the sending host finally sees the too big.

> tunnels fragment the encapsulated packet, IMO.

Why not?

Fragmentation needs to happen in certain cases with IPv4. The only  
choice is who is going to reassemble.

> overlays is likely to increase over time.

I'm aware that there is a race going on to see who can be the first  
to implement 1500 bytes of overhead per packet. Obviously whatever  
maximum packet size above 68 bytes a sender of a packet chooses,  
there will be some configuration that can't carry packets of that  
size. And since datagram based applications can't arbitrarily reduce  
their packet size, there will always be _some_ fragmentation. (Or  
black holes if people prevent fragmentation from working properly.)  
Reducing packet sizes a few percent for applications / transports  
that require a one time packet size choice seems like a good idea to  
avoid triggering these issues unnecessarily.

---

From: Joe Touch <[email-masked]>
Date: Thu, 02 Aug 2007 07:35:36 -0700

...
> ...but this makes no sense at all. The whole point of PMTUD is to _find_
> _out_ whether stuff fits. You can't know that in advance.

Fits in what you currently think the path MTU is (I should have been
more specific).

> Not for IPv4. For IPv6, that would be a valid choice but handling this
> in the same way as IPv4 would also be fine.

It's a valid choice, but should not be EXPECTED.

> this in effect the first packet will trigger PMTUD in the tunnel, but by
> then, the second packet is also on its way, so both packets will be lost
> and TCP will probably stall for some time. Then when the third packet
> comes, the sending host finally sees the too big.

Yes. Transport protocols will react poorly to this - once. Presumably
other connections will not experience this problem.

> choice is who is going to reassemble.

I like treating IPv4 and IPv6 similarly. Tunnels should not put undue
burden on endpoints. Since a tunnel destination MUST exist (to
decapsulate), it ought to be saddled with the work of reassembly, rather
than dropping it on the endpoint.

> I'm aware that there is a race going on to see who can be the first to
> implement 1500 bytes of overhead per packet. Obviously whatever maximum
> packet size above 68 bytes a sender of a packet chooses, there will be
> some configuration that can't carry packets of that size. And since
> datagram based applications can't arbitrarily reduce their packet size,
> there will always be _some_ fragmentation. (Or black holes if people
> prevent fragmentation from working properly.) Reducing packet sizes a
> few percent for applications / transports that require a one time packet
> size choice seems like a good idea to avoid triggering these issues
> unnecessarily.

Sure. Let's pick one we won't have to move too often, though. That means
a few layers of possible IPsec, e.g., 1300 or 1200. That's close enough
to 1500 for efficiency.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 2 Aug 2007 10:16:53 -0700

Iljitsch,

> -----Original Message-----
> From: Iljitsch van Beijnum [mailto:[email-masked]] 
> Sent: Thursday, August 02, 2007 4:57 AM
> To: Templin, Fred L
> Cc: Joe Touch; Joe Macker; Fred Baker; Gorry Fairhurst; 
> Stephen Casner; Internet Area
> Subject: Tunneling/MTU BCP, was: Re: Mucking with IP ID
> 

I believe we can say that the nominal MTU of today's Internet
is 1500 bytes, and yes I think that statement is helpful.
 
> recommendation to 1500+ interfaces makes little sense.

I never said the recommendation was limited to 1500+ interfaces.
I said that applications (and/or transports) that send 1501+
packets with DF=1 are RECOMMENDED to use RFC4821, but this is
*not* to say that other applications/transports are not. 
 
> configuration in use.

IMHO, it is perfectly reasonable for MRU to be larger than
the largest possible packet supported by the underlying
hardware. Think about larger-than-MTU-sized packets sent
with DF=0 and/or fragmented by the sending host itself. 

> That means 1500 bytes for ethernet, or 
> whatever  

Disagree; it may be beneficial for MRU to be larger than
MTU and perhaps even much larger. Else, how can we support
applications that have no way of reducing the size of the
packets they send to fit MTU?

> IPv4 tunneling practice of a 1280 byte MTU to accommodate multiple  

The nominal MTU in todays Internet is 1500 bytes, but that is
NOT to say that all devices on the Internet configure a 1500
byte MTU because that isn't true. IEEE 802.as may begin to push
common device MTUs up to 2048 bytes, so it is not unreasonable
that reassemblers will soon need to be able to reassemble larger
than 1500 even if senders limit the size of the packets they
send to the MTU of their outgoing interface.

Fred
[email-masked]

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 3 Aug 2007 23:57:10 +0200

[CCing int area again]

On 3-aug-2007, at 2:13, Templin, Fred L wrote:

>> I don't think "unless RFC4821" is reasonable either; perhaps "unless
>> they make measures to react to silent failure, by RFC4821, or
>> some other
>> means, e.g., application timeout and retry with other MTUs".

> I like the spirit of your text, but the problem I have with
> it is that it doesn't say how the application can discern loss
> due to an MTU restriction from loss due to, e.g., congestion.

I strongly disagree here. PMTUD isn't something that should happen in  
applications. It's easy to get it wrong, and applications tend to  
stick around for a LONG time.

Applications that use UDP or other transports that can't implement  
RFC 4821 should clear the DF bit in IPv4 so that fragmentation can  
happen as needed. Anything else will cause breakage.

> For *-in-IPv4 tunnels, the choke point will come in routers in
> the middle of the network that have to support decapsulating
> tunnel endpoints. They may need to support reassembly for *many*
> such tunnels, and so they need to place a conservative and
> convenient upper bound on MRU - say, 2KB.

I think I get why we've been disagreeing on MRU issues: it seems what  
you mean when you say "MRU" is "maximum size of packets that can be  
reassembled". The MRU is generally dictated by hardware limitations  
and as such not all that interesting. But the maximum packet size  
that a tunnel endpoint is willing to reassemble is a very important  
choice. Limiting this to 1500 bytes or some such will probably come  
back to haunt us as jumboframe capability becomes more common. (Note  
that the tunnel egress device may not support jumboframes for other  
interfaces, but if the tunnel ingress point does, it's likely that  
large packets will come in as fragments.)

>> The IEEE isn't the only arbiter of link specs, but if you're
>> picking an
>> IEEE spec, you need to pick 1500, which means you probably want
>> something like 1200 for the reasons you cite (reasonable levels of
>> recursion).

> 1200 is too small for IPv6-in-(foo)-in-IPv4 tunnels; IPv6
> will refuse to run over any link that can't do at least 1280.

1200 or 1280 is way too conservative. Trying to reserve room for  
IPsec tunnels is a no-win game IMO, and IPsec users don't care about  
performance anyway. For anything else ~ 1450 as a "shouldn't be  
subject to too much fragmentation" packet size used by UDP apps  
should be workable.

>> Because it's not a BCP; it's an update to a standards-track
>> doc, not an
>> operational recommendation (which is what I think of BCPs as being).
>> Perhaps 2003bis should encompass 6-in-6, etc. as well.

> The 2KB MRU is an operational recommendation, so maybe that
> should go as a 1pg BCP? In terms of the tunnel encapsulation
> requirements, why couldn't we just write up a brief spec and
> say: "This document updates RFC2003 (et al)"? (Then again,
> maybe we could get by with just tucking the 2KB MRU into
> the spec also...)

Note that EDNS0 often advertises a 4096 byte packet size. If we  
aren't going to allow a full 9000 byte "standard" jumboframe or  
larger, I'd shoot for around 4.5 kilobytes. If you're going to  
fragment anyway you may as well get to send packets of at least a  
somewhat decent size, 2048 is hardly enough to bother with. BTW, on  
FreeBSD and MacOS X:

 > sysctl -a | grep dgram
net.local.dgram.maxdgram: 2048
net.local.dgram.recvspace: 4096
net.inet.udp.maxdgram: 9216
net.inet.raw.maxdgram: 8192

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 3 Aug 2007 15:27:23 -0700

Iljitsch,

See below: 

> Sent: Friday, August 03, 2007 2:57 PM
> To: Templin, Fred L
> Subject: Re: Mucking with IP ID
> 
> [CCing int area again]
> 
> I strongly disagree here. PMTUD isn't something that should 
> happen in applications. It's easy to get it wrong, and
> applications tend to stick around for a LONG time.

I believe all Joe was saying was that an application that can
discern MTU-related loss (with or without explicit indications
from the network), and has a way to reduce the size of the
messages it sends, should probably do so.  
 
> happen as needed. Anything else will cause breakage.

Any application can clear DF in the packets it sends and
hope that the final destination has a large enough buffer to
reassemble any fragmentation that occurs in the network. But,
it would be well advised to try to ascertain the reassembly
buffer size first - unless it has some way of knowing or
guessing this in advance.

> large packets will come in as fragments.)

By: "MRU", I mean the same thing as for "EMTU_R" (Effective MTU
to Receive) per ([RFC1122], Section 3.3.2).

> subject to too much fragmentation" packet size used by UDP apps  

For *-in-IPv4 tunnels, if we have EMTU_R >= 2KB and Maximum Tunnel
Fragment Size (MTFS) = 1500B, then the tunnel MTU can be as large
as the largest IPv4 packet (the only limitation being the 16bit
IPv4 length field). 

> somewhat decent size, 2048 is hardly enough to bother with.

The 2048 isn't an MTU; it is only the minimum MRU (or, EMTU_R)
for reassembling tunneled packet fragments, which could in
turn be reassembled into a quite large packet at the final
destination (assuming the final destination configures an
EMTU_R >> 2KB). 

> BTW, on FreeBSD and MacOS X:
>
>  > sysctl -a | grep dgram

Good to know, but I don't think that relates to this discussion?

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Sat, 04 Aug 2007 14:55:24 -0700

> messages it sends, should probably do so.  

Yes, but I agree with Iljitsch's point as well. This doesn't address
legacy apps, and apps want transparent networking services.

> guessing this in advance.

It currently does: 576 for IPv4. That doesn't stop some UDP apps from
exceeding that - but that's incorrect app behavior. There ought to be a
way to find out what the reassy size is - though I don't recall one
(short of SNMP).

Joe

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sun, 5 Aug 2007 18:26:20 +0200

> messages it sends, should probably do so.

I know what he said; I disagree. As a rule, application developers  
have very little understanding about network issues, and as a second  
rule, they _think_ they know a lot more than they do. The potential  
for harm here is many times larger than the potential for good.

> reassemble any fragmentation that occurs in the network.

The latter is not a consideration: regardless of whether the DF bit  
is set to cleared, if packets are larger than what can be reassembled  
AND fragmentation is required, no communication is possible.

> But,

There are only two sources of this knowledge: talk to the other side  
(as in DNS EDNS0) or the value is listed in the relevant RFC. No  
guessing PLEASE. This is important: if you guess, you will guess  
wrong a significant part of the time. If you can't know that you're  
going to do it right, at least do it wrong consistently (i.e., fixed  
value in an RFC) so operators can work around it.

> to Receive) per ([RFC1122], Section 3.3.2).

To me, that's not what "MRU" means. Also note that the same 18-year- 
old RFC tells us that a hardcoded limit is bad here.

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 6 Aug 2007 08:30:48 -0700

Iljitsch, 

> old RFC tells us that a hardcoded limit is bad here.

OK; I will start using the term "EMTU_R" instead of "MRU" from
now on so that there is no confusion. (I believe the term "MRU"
came from one of the PPP specs and there they say that MRU can
be greater than MTU.) About a hardcoded limit, [RFC1122] says:

         "DISCUSSION:
              A fixed EMTU_R limit should not be built into the code
              because some application layer protocols require EMTU_R
              values larger than 576."

To me, this says that a *max* EMTU_R should not be set but
that it is perfectly OK to specify a *min* EMTU_R. [RFC791]
says only that EMTU_R must be 576 bytes or larger for IPv4;
all I am saying is: "if you configure a tunnel interface
over IPv4, set its EMTU_R to 2KB or larger".

Fred