Subject: [BEHAVE] DF bit handling

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 23 Dec 2009 16:38:27 +0100

In Hiroshima, the chairs put the question of whether we should work around path MTU discovery issues in a translater to the room. I think this question was largely misunderstood, and many of the people who hummed also did so from a place of pedanticness, rather than sticking to Jon Postel's robustness principle. Also, the question shouldn't have been asked this way because in order to answer it, it's necessary to know what the cost of the workaround is.

So I want to revisit this issue. Please respond with technical arguments.

It's legal for IPv4 hosts to run PMTUD and have a path MTU < 1280. And according to RFC 2460:

   It is strongly recommended that IPv6 nodes implement Path MTU
   Discovery [RFC-1981], in order to discover and take advantage of path
   MTUs greater than 1280 octets.  However, a minimal IPv6
   implementation (e.g., in a boot ROM) may simply restrict itself to
   sending packets no larger than 1280 octets, and omit implementation
   of Path MTU Discovery.

Now IPv6 hosts will generally implement PMTUD, but operators may disable it by using an MTU of 1280 and then filtering incoming packet too big messages. That is certainly not breaking RFC 2460 as much as filtering those incoming too big messages while still sending IPv4 packets with DF=1 that many IPv4 networks do, even ones that know the RFCs like Microsoft with www.microsoft.com. More importantly, MTU=1280 and too big filtering doesn't break anything in IPv6.

So an IPv4 practice that is ok within IPv4, and an IPv6 practice that is ok within IPv6. Now put a translator between them, and a black hole is born if no changes to the currently defined method of translation is done: the IPv6 host sends a 1280-byte packet, IPv4 host says "too big, try 576" and the IPv6 host never sees that message and keeps sending those 1280-byte packets.

Solution: simply set the DF bit to 0 for all IPv6 packets <= 1280 bytes that are translated to IPv4. This doesn't get in the way if the too bigs aren't filtered, because the fragment header is still there in that case, but it does solve the PMTUD blackhole issue with absolutely no disadvantages. Two lines of code, that's all.

Then the handling of IPv4 packets with DF=0 that are translated to IPv6.

The xlate draft currently says to add a fragment header if the packet is smaller than 1280 bytes (and DF=0). That fragment header on unfragmented packets is there for the benefit of a translator that may want to translate back into IPv4 that can then set DF=0. If DF is always set to 0 for packets < 1280, then the code to detect this condition is unnecessary, as are the cycles to add the fragment header and 64 bits of network capacity are saved. Also, many IPv6 packet filters play fast and loose and don't recognize a packet as TCP or UDP if there is a fragment header in between. So not having this unnecessary fragment header also increases robustness.

Last but not least, mandating that a translator always fragment packets with DF=0 > 1280 very strange, and can end up being very suboptimal. Because there can't be any PMTUD for these packets, they need some additional handling. Always fragmenting them is one solution, but it has the downside that packets are fragmented even if they could have been carried to the destination without problem because the path MTU is large enough. The alternative is for the translator to simply send the packet intact initially, and thereby implicitly run PTMUD. If there is then a too big message, obviously this can't be forwarded to the original source because with DF=0 that host will not expect such a message and not reduce its packet size. So in this case the translator would have to fragment subsequent packets.

Now this seems like an imposition, but:

1. The state is per destination address, not per flow
2. There is only actual state for destination addresses with a reduced path MTU
3. Hosts running PMTUD must do this, too, and seem to be able to without issues

Practical tradeoffs could happen in both directions: a stateless translator implemented on a router platform with fast hardware-assisted processing but relatively constrained state storage facilities serving 6to4 (or similar) hosts all using a 1280-byte MTU would probably want to simply always fragment. A stateful translator serving hosts running native IPv6 over ethernet (which would normally have a 1500-byte MTU) may not see many PMTUD hits and/or not be concerned with the small increase in state.

So our specifications should allow for both solutions.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 23 Dec 2009 23:28:36 +0100

> Authoritative IPv6 DNS name servers send fragment their DNS/UDP
> responses at 1280 because they don't want to have to deal with PMTU
> to millions of machines.

Ah, I didn't know that.

> Authoritative IPv4 DNS nameserver turn off PMTU discovery for DNS/UDP.

In IPv4 you have to do PMTUD for UDP in the application, which I don't think any application does.

> We don't want a middle box turning it on underneath us.

What I was talking about was the translator "terminating" the IPv6 PMTUD in this case. So the too big messages aren't relayed to the original source, so the original source remains unaware of all of this.

One thing I forgot to mention in my original message: video streaming over UDP typically happens with packets just under 1500 bytes. If those happen to be in the 1473 - 1480 byte range then the fragment header can push the packet size after translation just over the edge and require fragmentation that wouldn't be necessary without the fragment header. People might choose 1476 or 1480 because that's the ethernet MTU with either GRE or IP-in-IP encapsulation. I seem to remember that 147x was common, but can't find a reference right now.

---

From: Mark Andrews <[email-masked]>
Date: Thu, 24 Dec 2009 09:11:45 +1100

In message <[email-masked]>, Iljitsch van Beijn
um writes:
> In Hiroshima, the chairs put the question of whether we should work around pa
> th MTU discovery issues in a translater to the room. I think this question wa
> s largely misunderstood, and many of the people who hummed also did so from a
>  place of pedanticness, rather than sticking to Jon Postel's robustness princ
> iple. Also, the question shouldn't have been asked this way because in order 
> to answer it, it's necessary to know what the cost of the workaround is.
> 
> Now IPv6 hosts will generally implement PMTUD, but operators may disable it b
> y using an MTU of 1280 and then filtering incoming packet too big messages. T
> hat is certainly not breaking RFC 2460 as much as filtering those incoming to
> o big messages while still sending IPv4 packets with DF=1 that many IPv4 netw
> orks do, even ones that know the RFCs like Microsoft with www.microsoft.com. 
> More importantly, MTU=1280 and too big filtering doesn't break anything in IP
> v6.
> 
> So an IPv4 practice that is ok within IPv4, and an IPv6 practice that is ok w
> ithin IPv6. Now put a translator between them, and a black hole is born if no
>  changes to the currently defined method of translation is done: the IPv6 hos
> t sends a 1280-byte packet, IPv4 host says "too big, try 576" and the IPv6 ho
> st never sees that message and keeps sending those 1280-byte packets.
> 
> Solution: simply set the DF bit to 0 for all IPv6 packets <= 1280 bytes that 
> are translated to IPv4. This doesn't get in the way if the too bigs aren't fi
> ltered, because the fragment header is still there in that case, but it does 
> solve the PMTUD blackhole issue with absolutely no disadvantages. Two lines o
> f code, that's all.
> 
> The xlate draft currently says to add a fragment header if the packet is smal
> ler than 1280 bytes (and DF=0). That fragment header on unfragmented packets 
> is there for the benefit of a translator that may want to translate back into
>  IPv4 that can then set DF=0. If DF is always set to 0 for packets < 1280, th
> en the code to detect this condition is unnecessary, as are the cycles to add
>  the fragment header and 64 bits of network capacity are saved. Also, many IP
> v6 packet filters play fast and loose and don't recognize a packet as TCP or 
> UDP if there is a fragment header in between. So not having this unnecessary 
> fragment header also increases robustness.
> 
> Last but not least, mandating that a translator always fragment packets with 
> DF=0 > 1280 very strange, and can end up being very suboptimal. Because there
>  can't be any PMTUD for these packets, they need some additional handling. Al
> ways fragmenting them is one solution, but it has the downside that packets a
> re fragmented even if they could have been carried to the destination without
>  problem because the path MTU is large enough. The alternative is for the tra
> nslator to simply send the packet intact initially, and thereby implicitly ru
> n PTMUD. If there is then a too big message, obviously this can't be forwarde
> d to the original source because with DF=0 that host will not expect such a m
> essage and not reduce its packet size. So in this case the translator would h
> ave to fragment subsequent packets.
> 

> Practical tradeoffs could happen in both directions: a stateless translator i
> mplemented on a router platform with fast hardware-assisted processing but re
> latively constrained state storage facilities serving 6to4 (or similar) hosts
>  all using a 1280-byte MTU would probably want to simply always fragment. A s
> tateful translator serving hosts running native IPv6 over ethernet (which wou
> ld normally have a 1500-byte MTU) may not see many PMTUD hits and/or not be c
> oncerned with the small increase in state.
> 
> _______________________________________________
> Behave mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/behave
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 24 Dec 2009 00:04:17 +0100

On 23 dec 2009, at 23:50, Mark Andrews wrote:

>> In IPv4 you have to do PMTUD for UDP in the application, which I don't =
>> think any application does.

> There are kernels which turn it on PMTUD for *everything* in IPv4 despite
> the fact the it was explicitly advised against.

Note that sending packets with DF=1 is not the same thing as performing PMTUD.

I once talked to a root server from behind a 296 MTU and I never got any packets because they set DF to 1. If they had actually been running PMTUD I would have received the retransmissions.

>> What I was talking about was the translator "terminating" the IPv6 PMTUD =
>> in this case. So the too big messages aren't relayed to the original =
>> source, so the original source remains unaware of all of this.

> But you still loose the original UDP response.

Only the first packet within the time that the PMTU is cached.

And as we learned in another thread recently, it is a profoundly bad idea to run a DNS resolver through NAT64 so authoritative servers don't have anything to worry about in any event.

Note that I'm not saying that the translator doing "proxy PMTUD" is always the best option. It's just that fragmenting all packets with DF=0 is also not always the best option.

> With DNSSEC DNS/UDP packet are often 2-3K range.  I've got a change
> request to raise the maximum UDP packet we support to 8K from the
> 4K we currently support.

O, I thought that was already a configuration setting in BIND.

Iljitsch

---

From: Mark Andrews <[email-masked]>
Date: Thu, 24 Dec 2009 09:50:24 +1100

In message <[email-masked]>, Iljitsch van Beijn
um writes:
> >> 3. Hosts running PMTUD must do this, too, and seem to be able to =
> without issu
> >> es
> 

> source, so the original source remains unaware of all of this.

But you still loose the original UDP response.  There are already
too many middle boxes that do the wrong things with DNS/UDP today.
We don't need yet another one.

Mark

> One thing I forgot to mention in my original message: video streaming =
> over UDP typically happens with packets just under 1500 bytes. If those =
> happen to be in the 1473 - 1480 byte range then the fragment header can =
> push the packet size after translation just over the edge and require =
> fragmentation that wouldn't be necessary without the fragment header. =
> People might choose 1476 or 1480 because that's the ethernet MTU with =
> either GRE or IP-in-IP encapsulation. I seem to remember that 147x was =
> common, but can't find a reference right now.=

---

From: Mark Andrews <[email-masked]>
Date: Thu, 24 Dec 2009 10:29:32 +1100

In message <[email-masked]>, Iljitsch van Beijn
um writes:
> >> In IPv4 you have to do PMTUD for UDP in the application, which I =
> don't =3D
> > There are kernels which turn it on PMTUD for *everything* in IPv4 =
> despite
> Note that sending packets with DF=3D1 is not the same thing as =
> performing PMTUD.
> 
> I once talked to a root server from behind a 296 MTU and I never got any =
> packets because they set DF to 1. If they had actually been running =
> PMTUD I would have received the retransmissions.

Which require that the ICMP messages got back to them and required
that the kernel is keeping a per host table.

> >> What I was talking about was the translator "terminating" the IPv6 =
> PMTUD =3D
> =3D

And DNS clients ask with EDNS@4096, then on timeout ask with EDNS@512,
which then often triggers fallback to TCP as there is not enough
space in 512 bytes to fit the response which fits quite well into
4096 bytes.

> And as we learned in another thread recently, it is a profoundly bad =
> idea to run a DNS resolver through NAT64 so authoritative servers don't =
> have anything to worry about in any event.

There is nothing wrong with providing authoritative service through
a NAT64.

> Note that I'm not saying that the translator doing "proxy PMTUD" is =
> always the best option. It's just that fragmenting all packets with DF=3D0=
>  is also not always the best option.

It's a router where the effective next hop MTU is 1280.  If the
IPv4 packet says fragmentment me if needed then fragment the packet.
 
> O, I thought that was already a configuration setting in BIND.

Named support EDNS between 512-4096.  Also one shouldn't have to
tweak settings in named because of stupid middleware.
 
> Iljitsch=