Subject: Teredo MTU logic

From: [email-masked] (Arnaud Ebalard)
Date: Mon, 24 Sep 2007 10:57:00 +0200

Hi,

Last week I spent some time to have [IPsec-protected] Mobile IPv6
traffic be routed over a Teredo interface (everything on Linux using
miredo, umip and the IPsec stack but this is not system specific).

Basically, Teredo interfaces have a default MTU of 1280 bytes. This
prevents their _direct_ use as the underlying device for another
IPv6-in-IPv6 tunnel interface. To clarify things, I expected to be able
to have that kind of traffic on the wire:

      IPv4 / UDP / IPv6 / ESP / IPv6 / Data ...
 
       20     8     40    50    40      

By setting up a Teredo relay at the entry of the site for that specific
kind of traffic, I have the insurance that the traffic from the Mobile
Nodes to their Home Agent will not use another relay (all the traffic
goes to HAs inside the site). This means an higher MTU can be used on
the clients without problem. Keeping it low (1400, i.e. 1430 bytes on
the wire) avoids IPv4 PMTUD related issues. 

The question is quite simple: what is the expected way to have Teredo
clients get a higher MTU than the default one?  

Even if Teredo servers act as routers for their clients (RS/RA
exchange), I don't expect them to advertise an higher MTU value in
general. In fact, because the IPv4 PMTU on the path between a client and
a relay is specific to the pair (client, relay), this won't happen in
general. At the moment, my conclusion is that you have 2 solutions to
solve the issue: 

1) setting up a relay at the edge of the site with a higher MTU value
   and statically configure clients
2) setting up a relay at the edge of the site _and_ a server that the
   clients will use to acquire an higher MTU from. 

I don't like 2) because it consumes a public IPv4 address and increase
the attack surface. In all cases, setting up a relay is almost required
to improve efficiency (i.e. having a direct routing path).

Comments are welcome.

Regards,

a+

---

From: Rémi Denis-Courmont <[email-masked]>
Date: Thu, 27 Sep 2007 22:15:15 +0300

Le Monday 24 September 2007 11:57:00 Arnaud Ebalard, vous avez écrit :
> At the moment, my conclusion is that you have 2 solutions to
> solve the issue:
>

After some more not-so-long thinking, what about:
3) get rough consensus that there are no paths on the IPv4 Internet with a MTU 
larger or equal to 1308 bytes (the current "outer" Teredo MTU), but smaller 
than 1348 bytes (or any slightly bigger value)? Then, it is safe to update 
everything to a bigger inner MTU of 1320 bytes (or more); it will not cause 
any extra on-path fragmentation. In fact, it will even diminish the overhead.

So my naive question of the day is:
Does anyone have any practical case for an IPv4 path MTU within [1308;1348[ ?

-- 
Rémi Denis-Courmont
http://www.remlab.net/

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 27 Sep 2007 21:45:43 +0200

> Does anyone have any practical case for an IPv4 path MTU within  

Can't think of any but lack of imagination never stopped breakage...

I'm starting to think we may need an MTU BoF or MTU wg. There are  
just so many issues sourrounding packet sizes and fragmentation, and  
they're dealt with in a rather fragmented way currently.

---

From: [email-masked] (Arnaud Ebalard)
Date: Thu, 27 Sep 2007 23:15:23 +0200

Hi,

Rémi Denis-Courmont <[email-masked]> writes:

> Le Monday 24 September 2007 11:57:00 Arnaud Ebalard, vous avez écrit:

What would be the precise drawbacks of setting the MTU for Teredo to
1472 (1500 - teredo overhead) or something just below that (to deal with
common other encapsulation on the path, which will most of the time
appear at both ends, like with PPPoE or similar things)? 

If the bet is fair, there would only be a limited amount of
fragmentation while still providing clients with a better initial MTU
(more efficient and with more potential). 

I won't say that what i described in my previous email (Tunnel mode
IPsec over Teredo) is the worst case scenario in term of MTU usage by a
client but it's a good starting point of the kind of MTU everyone would
be happy to consider as available on the IPv4 Internet. Let's call that
value A. Just for the discussion, let's set it to 1408 (1280 + 60 + 40 +
20 + 8,    60 is some reasonable value for ESP).

Then, taking the problem in the reverse direction, by temporarily
considering an initial MTU of 1500 for the path and removing the kind of
encapsulation that the network can "consume" (PPPoE, IPsec in tunnel
mode (IPv4 for outer header), ..., ?) we could get a value B. Again, for
the discussion, with both PPPoE and IPsec in tunnel mode, we can
probably consider 1412 (1500 - 60 - 20 - 8) as an initial value.

Now, it seems to me that considering sth in [A,B] is simply making a
reasonable bet that this value of IPv4 MTU should work on all pathes in
the IPv4 Internet and with all kinds of encapsulations at the
client. Here, something in [1408, 1412]. 

This silly idea gets me to a default MTU for Teredo interfaces in [1380,
1384]. I know it looks like cooking but it seems a fair bet and it
provides in default cases almost a 10% boost of TCP MSS value for Teredo
clients without obvious drawbacks.

Comments welcome. Errors correction too.

> Does anyone have any practical case for an IPv4 path MTU within

or [1308, 1410[ ? ;-)

Regards,

a+

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 27 Sep 2007 15:42:38 -0700

> clients without obvious drawbacks.

Why not just set the Teredo interface MTU to (64KB-ENCAPS)?
Then, take special care to make sure the small packets get
through but let the big packets take care of themselves.

Fred
[email-masked]

---

From: Christian Huitema <[email-masked]>
Date: Thu, 27 Sep 2007 16:36:09 -0700

> through but let the big packets take care of themselves.

The problem is UDP fragmentation. By nature, it is unreliable: the more fragments, the more chances to lose one of them, thus causing the loss of the entire packet. This is aggravated by NAT behavior: some NAT will try to reassemble the packets, but others will just pass the 1st fragment and drop the next ones.

That would not be too much of a problem if TCP implementations could do MTU detection by just reacting to packet losses. But today, many TCP implementations don't. The MTU discovery is triggered by ICMP "packet too big" messages. These messages are not generated if a UDP fragment is lost.

-- Christian Huitema

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 27 Sep 2007 17:04:18 -0700

> -----Original Message-----
> From: Christian Huitema [mailto:[email-masked]] 
> Sent: Thursday, September 27, 2007 4:36 PM
> To: Templin, Fred L; Arnaud Ebalard; Rémi Denis-Courmont
> Cc: [email-masked]
> Subject: RE: The IPv4 Internet MTU
> 
> The problem is UDP fragmentation.

I guess you mean IPv4 fragmentation? Even so, there is
a lot we can do within the no-man's-land of the Teredo
interface after we get a packet from IPv6 and before we
give it over to IPv4. So, why not just give IPv6 an
optimistic MTU estimate then deal with reality from
within?

> By nature, it is 
> unreliable: the more fragments, the more chances to lose one 
> of them, thus causing the loss of the entire packet. This is 
> aggravated by NAT behavior: some NAT will try to reassemble 
> the packets, but others will just pass the 1st fragment and 
> drop the next ones.

That doesn't sound too good. Maybe we should just forget
about IPv4 fragmentation altogether and fall back to IPv6
fragmentation instead?
 
> That would not be too much of a problem if TCP 
> implementations could do MTU detection by just reacting to 
> packet losses. But today, many TCP implementations don't. The 
> MTU discovery is triggered by ICMP "packet too big" messages. 
> These messages are not generated if a UDP fragment is lost.

Well, that's more or less what I was driving at in terms
of taking care of the smalls and letting the bigs take
care of themselves. If we had the tunnel endpoint send
specially-padded probes to the tunnel far-end, then it
would know the largest packet it could admit into the
tunnel before either fragmenting or dropping and sending
back a PTB. But, probing for sizes larger than 1500+
would be wasteful if the original source was already
doing its own MTU detection anyway.

BTW, how big did you say those bubbles were that you 
are blowing? (Or, maybe it was the echo requests...)

Fred

---

From: james woodyatt <[email-masked]>
Date: Thu, 27 Sep 2007 17:06:34 -0700

On Sep 27, 2007, at 16:36, Christian Huitema wrote:
>
> That would not be too much of a problem if TCP implementations  
> could do MTU detection by just reacting to packet losses. But  
> today, many TCP implementations don't. The MTU discovery is  
> triggered by ICMP "packet too big" messages. These messages are not  
> generated if a UDP fragment is lost.

These messages are also sometimes not generated as a matter of  
deliberate policy.  I am aware of several popular PPPoE network  
operators who have deliberately chosen for their access concentrators  
to be IPv6 path MTU discovery black holes for reasons that have never  
been disclosed to me.  The software distributions they bundle with  
their service have elaborate mechanisms for configuring the MTU on  
the PPPoE interface properly, despite exchanging MTU options in the  
PPP negotiation with the PPPoE access concentrators that require  
MTU=1500, which can't work over ethernet.  There is no good reason  
for continuing to do this that I can see.

The result of these black holes being a permanent fixture in the IPv4  
Internet is that NAT gateways typically clamp the MSS field during  
TCP stateful connection tracking to prevent path MTU discovery  
brokenness from interfering with applications.  (Apple's AirPort base  
stations in PPPoE mode clamps the MSS on TCP streams to 1400 bytes.   
Larger values have proven perilous in the past.)

If I were more impolite, then I would do more than merely suggest  
that some people view the brokenness of IPv4 path MTU discovery as a  
feature rather than a bug.  I'm learning to be more polite.

--
james woodyatt <[email-masked]>
member of technical staff, communications engineering

---

From: Rémi Denis-Courmont <[email-masked]>
Date: Fri, 28 Sep 2007 08:14:17 +0200

On Thu, 27 Sep 2007 16:36:09 -0700, Christian Huitema

> The problem is UDP fragmentation. By nature, it is unreliable: the more

> fragments, the more chances to lose one of them, thus causing the loss of

> the entire packet. This is aggravated by NAT behavior: some NAT will try

to

> reassemble the packets, but others will just pass the 1st fragment and

drop

> the next ones.

Maybe Fred was hinting that Teredo could have defined its

fragmentation layer between UDP and IPv6. But while this would

work-around on-path fragmentation it would bring a naughty congestion

control issue - one simply cannot send 40 datagrams of 1500 bytes each

with no pacing. And we don't want to have to do congestion control,

right?

-- 

Rémi Denis-Courmont

http://www.remlab.net/

---

From: [email-masked] (Arnaud Ebalard)
Date: Fri, 28 Sep 2007 09:07:12 +0200

Hi Fred, Hi *,

> optimistic MTU estimate then deal with reality from

IMHO, the optimistic Teredo interface MTU is what I provided in my
previous post (sth in [1380, 1384]): 

1) it should prevent obvious fragmentation in all cases by taking into
   account reasonable encapsulation that can happen on the path
2) it also leave on the side the wrong assumption that you'll either get
   a message from the network or have your packets fragmented by it if
   it happens to be too big. There are people that simply prevent that
   to happen in their networks and there are vendors that are unable to
   implement those kind of things properly (don't ask for name, i would
   reply ;-) ). 

Reality is that just like you perform MSS clamping when you are aware
that the return path to clients in your site is less than what they
advertise, setting the interface MTU from the value one would consider
a common PMTU (the one clamped value is computed from) will simply have
the advantage of ensuring that all L4 protocols work with it, not only
TCP.

Using (64KB-ENCAPS) is _interesting_ but IMHO unfeasible directly. You
have the insurance that almost all your packets will be IPv4-fragmented
to more or less 1500 bytes. Full packets will become more than 40
fragments. I consider that packets don't get lost or killed that much in
the core _when_ they fit the PMTU: packet loss occurs at both ends 
because of some bad connectivity (unreliable medium). If the path is
reliable _and_ the communication takes place between teredo clients
directly, you have the benefit of IPv4 fragmentation (no header
cost).

Now, in all those cases, this will not work (or not work well):

1) bad medium (unreliable path): fragmentation will amplify the problem
   and there is pretty much nothing to do to prevent that except lower
   the MTU.
2) IPv4 MTU is higher than PMTU: using the reasonable value i proposed
   for IPv4 packets generated by the teredo interface coudl solve that
   point. 
3) Packets are for a native IPv6 client: if the proposed reasonable MTU
   value allow packets to hit the relay in all cases (expected), this
   creates an interesting situation because it would be able to warn the
   Teredo interface of the emitter that its packets are too big for the
   output path (ICMPv6 Packet Too Big). The positive aspect is that
   with the insurance that the mechanism is implemented by all Teredo
   relays (1 or 2 implementations at the moment ?), this removes IPv4
   internet issues from the table. Only drawback is the required work by
   the relay (regarding packet reconstruction, i.e. first 1280 bytes
   for the citation) which imply higher memory consumption. Rémi, what
   do you think?
4) Packets are for Teredo client: chances are high that the client's NAT
   will try to reassemble the packet instead of letting fragments go
   through. Well educated NAT GW will send an IPv4 ICMP to the client,
   that could be used to change the size of IPv6 packets for that
   peer. Rémi, is it feasible?

To sum up, a default MTU of (64Kb-ENCAPS) for the Teredo interface, even
with the use of IPv4 packets to carry everything with a reasonable MTU 
([1408,1412]) still requires many assumptions. I don't think it would
fly ;-)

Cheers,

a+

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 28 Sep 2007 16:11:23 -0700

Arnaud,

Thanks for the interesting note, but I am still holding
out optimism for a (64KB-ENCAPS) IPv6 MTU for the teredo
interface. I'd ask for bigger if I could, but we have
the 16bit IPv4 length field as a limiting factor. (Next
question is how we get it to do jumbograms...)

Thanks - Fred

> From: Arnaud Ebalard [mailto:[email-masked]] 
> Sent: Friday, September 28, 2007 12:07 AM
> To: Templin, Fred L
> Cc: Christian Huitema; Rémi Denis-Courmont; [email-masked]
> Subject: Re: The IPv4 Internet MTU
> 
> Hi Fred, Hi *,
>