Subject: [BEHAVE] The fragmentation issue

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 21 May 2009 16:08:05 +0200

Hi,

As I promised in the call yesterday, I wrote down the issues with the  
fragmentation handling specified in RFC 2765 (SIIT) and RFC 2460  
(IPv6). Basically, doing it the way that is specified there allows for  
path MTU discovery black holes that would otherwise not occur. A  
number of choices to address this issue in the stateful and stateless  
case are listed, we need to get rough consensus on which one we want  
to move forward with.

The text is a bit longer than I intended (about 3 pages, the draft is  
5) so I put it in a draft rather than an email message.

Iljitsch

Begin forwarded message:

> Filename:	 draft-van-beijnum-behave-frag64
> Revision:	 00
> Title:		 IPv6-to-IPv4 translation fragmentation issue
> Creation_date:	 2009-05-21
> WG ID:		 Independent Submission
> Number_of_pages: 5

> Abstract:
> This draft outlines a way to handle IPv4 MTUs smaller than 1280 bytes
> and a way to handle the identification field in fragmentation that
> are different from the ones specified in [RFC2460] for the purposes
> of discussion in the BEHAVE working group.

http://www.ietf.org/internet-drafts/draft-van-beijnum-behave-frag64-00.txt

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 21 May 2009 08:03:51 -0700

Iljitsch,

All of these alternatives have strong a dependence
on even getting back ICMPv4 messages from the IPv4
side at all. I believe there is a growing consensus
that we can't depend on that in all circumstances.

AFAICT, the most effective option to avoid black
holing and corruption is to locate the translator
as near as possible to the IPv4 host; preferably
co-resident on the IPv4 host if that can be arranged.

Fred
[email-masked]

> -----Original Message-----
> From: Iljitsch van Beijnum [mailto:[email-masked]]
> Sent: Thursday, May 21, 2009 7:08 AM
> To: Behave WG
> Subject: [BEHAVE] The fragmentation issue
> 
> Hi,
> 
http://www.ietf.org/internet-drafts/draft-van-beijnum-behave-frag64-00.t
xt
> _______________________________________________
> Behave mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/behave

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 21 May 2009 17:09:21 +0200

On 21 mei 2009, at 17:03, Templin, Fred L wrote:

> side at all.

Sure, but that's normal IPv4 behavior. What we're trying to do here is  
translate IPv4 to IPv6, not fix IPv4.

> I believe there is a growing consensus

Doesn't look like that to me, because people are still sending packets  
with DF=1.

One solution would be of course to send all packets on the v4 side  
with DF=0.

> co-resident on the IPv4 host if that can be arranged.

So if I want to go to ftp.dell.com which only has IPv4 and I only have  
IPv6, how exactly am I going to do that?

What you're saying is basically that everyone should adopt IPv6. While  
I agree with that it doesn't solve the problem at hand, which is how  
to communicate with people who _don't_ run IPv6.

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 21 May 2009 08:24:58 -0700

> to communicate with people who _don't_ run IPv6.

Have them put a translator in *their* network; then, do
what ever you have to do to reach their translator from
*your* network via IPv6 - even if that means tunneling.

Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 22 May 2009 15:45:04 -0700

Iljitsch - here are some comments for your document.

Fred

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sat, 23 May 2009 11:33:19 +0200

In response to an
ICMP PTB with an MTU less than 1280,
preferred IPv6 node implementations
may take it upon themselves to only
produce IPv6 packets smaller than 1280
bytes so that the translator need not
generate IPv4 fragments.

Do we know how many implementations do this rather than stick to 1280  
but now with a fragment header?

For stateful, there is
additional uniqueness material, including
(sport, dport, saddr, daddr).

No. The port numbers aren't present in fragments except the first one.  
So if I use the same ID value for a packet with ports 1, 2 and a  
packet with ports 3, 4 then the second fragment of the first packet  
may be reassembled with the first fragment of the second.

multi-fragment IPv6 packets should have
DF=1in the IPv4 header of each
translated fragment.

I don't like this, as there is no way this can occur in IPv4. This has  
the potential to trigger previously unknown bugs. We REALLY don't want  
to have to deploy ugly hacks because our ugly hack (NAT64) doesn't work.

Disagree because:
1) loss of packet too big messages would
be an identical black hole condition to the
pure IPv4 or pure IPv6 case, and 2) a n
IPv6 node’s failure to react to the packet
too big messages would be a violation of
RFC2460.

No. Although discouraged, it is allowed for an IPv6 host to not run  
PMTUD if it doesn't send packets larger than 1280 bytes. So someone  
who doesn't like ICMP messages could set the MTU option in RAs to 1280  
and filter incoming "too big" messages. That wouldn't create any  
problems for IPv6 communication. However, when you start translating  
and send translated packets to IPv4 destinations with an MTU smaller  
than 1260, you have a black hole because the IPv6 host will not reduce  
its packet size or include the fragment header. I guess you could say  
"but people create those black holes in the pure IPv4 case too by  
setting DF=1 and filtering "too big" messages" but that doesn't make  
it a valid choice that we can mandate in an RFC.

You may argue that the host SHOULD react to the "too big" messages  
despite the fact it doesn't want to run PMTUD, but that doesn't fix  
anything: RFC 2460 allows for this so we must address the situation.

I can't seem to make sense of the suggested added text for option 1.  
So please answer the following questions:

- if a packet 1280 bytes or larger is translated to IPv4, set DF to 1?

- if a packet 1280 bytes or smaller is translated to IPv4:

- without a fragment header, set DF to 0?

- with a fragment header more fragments = 0, offset = 0, set DF to 1?

- with a fragment header, more fragments = 1 or offset > 0, set DF to 0?

If the above is correct, it solves the black hole case without  
creating IPv4 fragments with DF = 1 so this does what we want but I  
would still prefer option 3 because it's simpler.

Also, if you take the identification handling from option 1 that is  
still very broken in the stateful case and somewhat broken in the  
stateless case.

You also say something about "fragment the packet locally". That's not  
how things work in IPv4. As long as the translator has the same MTU on  
both sides, it never has to fragment IPv4 packets because the  
translation never increases them beyond the MTU size so they can  
always be sent whole. Of course when the destination is sitting behind  
a link with a reduced MTU, then a router along the way will have to  
fragment. However, that doesn't involve the translator. The only thing  
it has to do is set DF to 0.

Exposure to
SUSTAINED IPv4 fragmentation at high
data rates is dangerous no matter how
carefully the IP_ID is crafted; too
dangerous to allow for a long-term
solution. This would seem to be much
more of a concern for translation than for
tunneling.

In particular, tunnels that allow IPv4
fragmentation are likely to occur between
endpoints over paths with robust links
that will almost always set larger MTUs.
With translation, on the other hand, IPv4
end systems may be deeply embedded
within sites that have high-speed links
with degenerate MTUs such that
sustained IPv4 fragmentation at high data
rates may be unavoidable.

Near-term IPv6 migration of sites and
end systems containing IPv4 servers is
therefore strongly encouraged (where
dual-stacked IPv6/IPv4 services may
be maintained for an extended period).

If IPv4 networks don't use MTUs smaller than 1280 bytes and they won't  
have trouble.

Note that the sustained stream of fragments issue is much less  
problematic in the translator case compared to the tunnel case,  
because presumably, the setup won't be such that ALL traffic for a  
given host goes through the same translator, while it's very possible  
that all traffic for a given host goes through the same tunnel. Also,  
the use of a tunnel makes an MTU reduction unavoidable, while having  
an MTU < 1280 (1260, really) is orthogonal from the use of a  
translator. And if the host uses an MSS that reflects the reduced MTU  
(or the MSS is adjusted somewhere in the path) then the issue doesn't  
come up for TCP. For UDP, it would be quite extraordinary for an IPv4  
host to receive a large amount of fragmented UDP traffic. Basically  
that only happens with applications like video streaming, but then  
usually not in the client-to-server direction.

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 26 May 2009 11:31:36 -0700

Iljitsch,

Much of your discussion was too nebulous for me to follow,
so I'll just say one thing for now. Including a frag hdr
with (MF=0; Offset=0) is the IPv6 host's way of telling
the translator to set DF=0; everything else is DF=1.

Fred

> Sent: Saturday, May 23, 2009 2:33 AM
> To: Templin, Fred L
> Cc: Behave WG
> Subject: Re: [BEHAVE] The fragmentation issue
> 
> In response to an
> IPv6 node's failure to react to the packet

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 26 May 2009 22:54:56 +0200

> the translator to set DF=0; everything else is DF=1.

I explained in the draft and my emails how this introduces PMTUD black  
holes, so I don't think this is a valid choice.

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 27 May 2009 14:09:41 -0700

Iljitsch,

> Sent: Tuesday, May 26, 2009 1:55 PM
> To: Templin, Fred L
> Cc: Behave WG

Let's look at the RFC2460, Section 5 text:

   "It is strongly recommended that IPv6 nodes implement Path MTU
   Discovery [RFC-1981], in order to discover and take advantage of path
   MTUs greater than 1280 octets.  However, a minimal IPv6
   implementation (e.g., in a boot ROM) may simply restrict itself to
   sending packets no larger than 1280 octets, and omit implementation
   of Path MTU Discovery."

All this says is that nodes that restrict themselves to
sending packets no larger than 1280 need not follow the
MUST/SHOULD/MAY specifications of RFC1981 - it does *not*
say that the nodes will never get an ICMP PTB. Indeed,
two paragraphs later it says:

   "In response to an IPv6 packet that is sent to an IPv4 destination
   (i.e., a packet that undergoes translation from IPv6 to IPv4), the
   originating IPv6 node may receive an ICMP Packet Too Big message
   reporting a Next-Hop MTU less than 1280.  In that case, the IPv6 node
   is not required to reduce the size of subsequent packets to less than
   1280, but must include a Fragment header in those packets so that the
   IPv6-to-IPv4 translating router can obtain a suitable Identification
   value to use in resulting IPv4 fragments.  Note that this means the
   payload may have to be reduced to 1232 octets (1280 minus 40 for the
   IPv6 header and 8 for the Fragment header), and smaller still if
   additional extension headers are used."

In other words, originating IPv6 nodes can receive the
PTBs *even if they restrict themselves to sending packets
no larger than 1280*. This is because the translator has 
no way of knowing whether the originating node is restricting
itself once and for all, or whether it is only sending small
packets for the time being and will begin sending larger
packets at a later time. In both cases, the originating node
needs to receive the PTBs, i.e., as it says in the spec.
  
Fred

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 27 May 2009 23:53:12 +0200

On 27 mei 2009, at 23:09, Templin, Fred L wrote:

> say that the nodes will never get an ICMP PTB.

So what you're saying is that even if a host doesn't implement PMTUD  
it should still include the fragment header after receiving a "too  
big". I agree that's a reasonable way to interpret RFC 2460, but I  
doubt someone who feels the need to omit PMTUD from their  
implementation will do it that way, because in this case it's still  
necessary to have state for each destination address in order to know  
whether to include the fragment header or not. This is exactly what  
you would want to avoid by not implementing PMTUD.

However, all of that is moot in the case where the _operator_ decides  
to turn off PMTUD by filtering "too big" messages and advertising a  
1280-byte MTU on all subnets. Don't forget that today there are  
networks that filter all ICMP but still set DF = 1. So I'm sure the  
same thing will happen with IPv6. At least with using a 1280-byte MTU  
that doesn't cause black holes for IPv6 destinations, but in the RFC  
2460/2765 case or the one you suggest this _would_ create PMTUD black  
holes with IPv6-IPv4 translation.

Is there any particular reason why you want to send <= 1280 byte  
packets with DF = 1? Simply setting DF to 0 for all packets equal to  
or smaller than 1280 bytes removes the the potential black hole. If  
the "too bigs" are propagated intact this means the IPv6 hosts can  
still send packets with a fragment header if they want.

(Although I see no point in that in the stateful case as the ID must  
be rewritten by the translator anyway.)

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 27 May 2009 15:28:04 -0700

> Sent: Wednesday, May 27, 2009 2:53 PM
> To: Templin, Fred L
> Cc: Behave WG

The RFC4861 destination cache is for more than just
tracking the path MTU, and AFAICT it is soft state.
The amount of state needed for each destination is
just 1-bit - set to 1 if a frag header needs to be
included and set to 0 otherwise.

> holes with IPv6-IPv4 translation.

Correct that 1280s will not black-hole for IPv6 destinations.
But for IPv4 destinations, the situation is no different
whether the source is IPv6 and going through a translator
or IPv4 and going through native IPv4 - if the ICMPs are
lost, even 1280's can black hole.

Plus, let's face it - IPv4 destinations are underprivileged
in relation to IPv6 ones. For example, the source has no way
of knowing whether an IPv4 destination can even reassemble
a packet as large as 1280 (whereas IPv6 destinations are
guaranteed to reassemble at least 1500). Similarly, if there
is any chance that 1280 packets will fragment the source
needs to limit the number of packets it sends so as to
avoid RFC4963 issues. If the fragmentation occurs in the
network, the source has no way of knowing this (i.e., as
far as the source is concerned, if it does not insert a
frag header then DF=1.)

> packets with DF = 1?

Several:
  - to avoid RFC4963 issues
  - to allow IPv6 nodes that prefer to reduce the size of
    the packets they send to discover an MTU smaller than
    1280
  - to allow IPv6 nodes to know when they have to insert
    a fragment header and limit their data rates

> Simply setting DF to 0 for all packets equal to

If DF=0, there will be no PTBs coming through the
translator. Go-fast hosts run the risk of loss and/or
data corruption if they don't realize their packets
are being fragmented by the network.

Also a matter for concern is the detrimental effect
on slow-path router processing when sustained IPv4
fragmentation is taking place in the network.

Fred

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 28 May 2009 10:24:36 +0200

On 28 mei 2009, at 0:28, Templin, Fred L wrote:

>> I doubt someone who feels the need to omit PMTUD from their

> included and set to 0 otherwise.

Right. But I'm not convinced all non-PMTUD implementations will apply  
this logic.

> a packet as large as 1280

IPv4 hosts that can't reassemble 1280-byte packets have bigger  
problems...

> Similarly, if there
> avoid  issues. If the fragmentation occurs in the

Well, this is a leap of faith inherent to the IPv6-to-IPv4  
translation: with IPv6 you have 32 bits worth of identification so you  
can send some 16 Mpps before you run into trouble, with the 16-bit  
identifcation in IPv4 it's only 0.25 kpps, assuming that packets live  
for ~ 255 seconds. This of course gets much better if you set your  
reassembly timer to something that reflects today's realities, such as  
100 ms.

But under what circumstances would an IPv6 client send large amounts  
of fragments to an IPv4 server?

>  - to avoid RFC4963 issues

There is no evidence that the fragment header helps avoid these issues.

>    the packets they send to discover an MTU smaller than

If those hosts are configured with an MTU larger than 1280 they will  
almost certainly send > 1280 packets that will trigger PMTUD anyway,  
so the chances of a missed PMTUD opportunity are very small even when  
sending <= 1280 packets with DF = 0.

>    a fragment header and limit their data rates

If we don't rewrite the reported MTU to 1280 then the frag header will  
still be inserted after a "too big" triggered by a > 1280 packet.

I don't think we can reasonably expect applications to limit their  
sending rate because the < 1280 PMTU indicates the presence of a  
NAT64. This requires that the application is able to control the  
sending rate in the first place and some serious API magic to detect  
the condition, just in case you're behind a NAT64 and the other side  
you're talking to is misguided enough to use an MTU <= 1280. Even  
though not illegal, using such a small MTU in IPv4 is completely  
unnecessary and AFAIK only ever done over dial-up links or links of  
similar speed.

> fragmentation is taking place in the network.

That's the choice you make when you use a <= 1280 MTU.

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 28 May 2009 05:34:48 -0700

> Sent: Thursday, May 28, 2009 1:25 AM
> To: Templin, Fred L
> Cc: Behave WG

I'm just going by what the RFC says - and it says to
be on the lookout for PTBs reporting MTUs smaller than
1280.

> IPv4 hosts that can't reassemble 1280-byte packets have bigger

But they don't violate RFC1122.

> of fragments to an IPv4 server?

Under circumstances that cannot be forseen if we are to have
IPv4-only servers and NAT64s as a long-term deployment
strategy.

That's one reason why IPv4 servers would be well served to
also arrange to be accessible via IPv6 - and that does not
necessarily mean dual stacked servers.

> There is no evidence that the fragment header helps avoid these
issues.

No; inserting the fragment header in itself does not
avoid RFC4963 issues. But, the knowledge that it must
insert the fragment header lets the host know that it
needs to avoid RFC4963 issues.

> sending <= 1280 packets with DF = 0.

I don't think it is possible for us to unambiguously
assume this. For instance, USE_MIN_MTU is a socket
option available to individual applications and can
restrict packet sizes even if the underlying interfaces
configure a larger MTU.

> still be inserted after a "too big" triggered by a > 1280 packet.

I didn't get that...

> you're talking to is misguided enough to use an MTU <= 1280.

Sure. That's why reducing the size of the packets it
sends may be the preferred option for many hosts.
Especially when they know they are talking to an
IPv4 host, e.g., when the IPv6 address is formed
from a WKP.

> Even

There is RFC2734, but I'm not sure it is possible to
enumerate all examples because IPv4 can run over
pretty much everything.

> That's the choice you make when you use a <= 1280 MTU.

No; that's the choice you make when you don't reduce
the size of the packets you send when there might be
a packet shredder in the path.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 29 May 2009 10:25:06 +0200

> But they don't violate RFC1122.

Well, RFC 1122 is from 1989, predating the fast ethernet standard by 6  
years. These days I can buy a _computer_ that supports fast ethernet  
for less than what a 10 Mbps ethernet card cost 20 years ago. So the  
mere fact that a configuration isn't specifically disallowed by RFC  
1122 doesn't mean that it's a reasonable thing to do today.

> IPv4-only servers and NAT64s as a long-term deployment

IF it's a long term deployment strategy (I disagree with that, but  
deployments don't always follow the initially set out strategy) then  
part of that strategy would be to rid the IPv4 internet of < 1260  
MTUs. People should upgrade to IPv6 to begin with. If they don't do  
that, we can accommodate them with a NAT64. We can even avoid breakage  
if they use unreasably small MTUs (to reiterate: there is no valid  
reason to use an MTU smaller than 1260 with IPv4 for bandwidths higher  
than 50 kbps [RFC3150]) but protecting against yet unknown  
circumstances is a bridge too far.

> also arrange to be accessible via IPv6

At least we can agree on that.

> restrict packet sizes even if the underlying interfaces

I'm more interested in avoiding black holes and keeping the mechanism  
simple than optimizing for use of an uncommon socket option along with  
an unreasonably small remote MTU.

> I didn't get that...

What I would like to do is set DF = 0 for packets smaller than 1280  
translated to IPv4 and DF = 1 for packets larger than 1280. Since we  
need to create a new ID value in the stateful case anyway and we don't  
need the fragment header to make the DF 0/1 decision in this case, I  
would then simply rewrite the MTU in the "too big" message to 1280  
during translation if it reported an MTU smaller than than on the IPv4  
side.

So: an IPv4 host with MTU=576 sends a "too big", translator turns this  
into "too big, use 1280", the IPv6 host will start sending 1280-byte  
packets, the translator translates those to IPv4 with DF = 0, packets  
are fragmented by a router somewhere between the translator and the  
IPv4 host.

So that's what I would like. But what I suggested above was to change  
the rewriting of the "too big" message, and then the following would  
happen, depending on the host implementation:

Case A: IPv4 host with MTU=576 sends a "too big", translator turns  
this into "too big, use 596", the IPv6 host will start sending 596- 
byte packets, the translator translates those to IPv4 with DF = 0,  
packets are NOT fragmented because on the IPv4 side, they're 576 bytes.

Case B: IPv4 host with MTU=576 sends a "too big", translator turns  
this into "too big, use 596", the IPv6 host will start sending 1280- 
byte packets with a fragment header, the translator translates those  
to IPv4 with DF = 0, packets are fragmented by a router somewhere  
between the translator and the IPv4 host.

(Note that if the host announces a 536 TCP MSS all of this is moot for  
TCP, but if the host can handle a normal MTU and there is a 576 MTU  
link in the middle we get fragments for TCP, which is bad news re high  
speed/ID issues. However, this should be rare in practice because  
people use MSS clamping to be able to talk to people who filter ICMP.)

> There is RFC2734

That's IPv4 over firewire (RFC 3146 is IPv6 over firewire). This  
specification is a bit nebulous, but upon careful reading you will see  
that is specifies that IP packets are fragmented at the link layer and  
then reassembled at the other end so that IP packets larger than 512  
bytes can be supported even when running at 100 Mbps (firewire link  
MTU depends on the operating speed). This is also why I never had any  
trouble communicating between my FW400 and FW800 Macs even though the  
former had a ~2k MTU and the latter a ~4k one. (However, for over a  
year now Apple has had a bug where IPv6 packets larger than 2k  
received over firewire are silently ignored.)

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 29 May 2009 07:05:14 -0700

Iljitsch,

It is good that we agree on IPv6 deployment. Also, I don't
know how common it is today but NAT rewriting of the IP ID
is suggested in the Bellovin paper and for all I know may
be enabled in shipping products (are there any references
on this?).

The setting of DF is where we still disagree - having the
NAT64 decide that on its own is asking it to know the mind
of the host, and may result in behavior that is not at all
what the host intended.

Fred

> Sent: Friday, May 29, 2009 1:25 AM
> To: Templin, Fred L
> Cc: Behave WG

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 29 May 2009 18:34:58 +0200

> of the host, and may result in behavior that is not at all

If the IPv4 host has that infamous 576 MTU and the IPv6 host reduces  
its packets to 576 after the too big, then everything is ok, right?

(Although there could be a corner case where afteer that 576 link  
there is a 296 link...)