Subject: Re: Fragmentation-related security issues

From: RJ Atkinson <[email-masked]>
Date: Mon, 2 Jan 2012 19:21:45 -0500

Earlier, Fernando Gont wrote, in part:
> ... my question was about whether the "atomic fragments"
> that were found in the wild were the result of translators,
> or of IPv6 networks that "violate" the standard and
> do not support an MTU of >= 1280.

I have been told of real-world IPv6 deployments
where the link MTU is smaller than 1280, 
while the limited link bandwidth makes adding a new 
segmentation protocol layer impractical.  I believe 
these account for some of the 576 byte or 512 byte
advertised Link MTU sizes in IPv6 PMTU "Too Big" 
messages that are seen in the real world.

I also know of real-world IPv6 deployments with
translators.  I expect translators to become 
much more common soon.

As far as I am aware, neither of the above would
produce PMTU "Too Big" messages with an advertised
link MTU very much under 500 bytes.  

IMHO, the 1280 byte number is far too big for a
minimum IPv6 MTU.  I've held the view that much 
above 512 bytes is problematic on RF links for 
MANY years now, as the Minutes from long ago
illustrate (search for "MTU"):

<ftp://ftp.ietf.org/ietf-online-proceedings/94dec/area.and.wg.reports/ipng/ipngwg/ipngwg-minutes-94dec.txt>

<ftp://ftp.ietf.org/ietf-online-proceedings/94jul/area.and.wg.reports/ipng/sipp/sipp-minutes-94jul.txt>

Ran

---

From: Philip Homburg <[email-masked]>
Date: Tue, 03 Jan 2012 10:31:47 +0100

>messages that are seen in the real world.

So if you run TCP over such a link then you an get extra overhead of more than
60 octets (IPv6 header + TCP header) compared to a link with an MTU of 1280.
For UDP the amount is about the same (IPv6, fragment header, UDP header).

And you are saying that that sort of overhead doesn't warrant finding an unused
bit somewhere to signal an adaptation layer?

---

From: RJ Atkinson <[email-masked]>
Date: Tue, 3 Jan 2012 07:32:17 -0500

On 03  Jan 2012, at 04:31 , Philip Homburg wrote:
> So if you run TCP over such a link then you an get
> extra overhead of more than 60 octets (IPv6 header + TCP header)
> compared to a link with an MTU of 1280.  For UDP
> the amount is about the same (IPv6, fragment header, UDP header).
> 
> And you are saying that that sort of overhead
> doesn't warrant finding an unused bit somewhere
> to signal an adaptation layer?

Of course an adaptation layer has various costs 
beyond a single bit, but please also consider the 
case where there are no unused bits at hand.

I'm saying the folks responsible for such links 
tell me they have looked into it in detail, tell me
they looked at the whole problem including all
of the system impacts and costs, and tell me that 
they concluded that it was MUCH better to continue 
to use IPv6 without adaptation layer overhead.
Reports say their current approach is operationally
very workable (i.e. with the IPv6 Fragment Header) 
and that multiple vendors' products work well 
in their deployment.

I'll note that RFC-1883, Section 5 specified a
minimum IPv6 Link MTU of 576 octets.  If that
number had not later been changed, then we would
not be having this discussion.

The choice of minimum IPv6 Link MTU is necessarily
somewhat arbitrary.  Different numbers are optimal
for different deployments.  It is merely sad that
the IETF chose such a large number.

The other approach I've heard about being taken
for working over small MTU links is simply to 
perform (one-way) IPv6->IPv4 protocol translation, 
since IPv4 works natively on links that small.  
It seems sad that the IETF decision is pushing 
people towards a protocol translation approach.

I can't blame the operations folks responsible
for such deployments; they are simply trying to move
their bits reliably and cost-effectively.

Yours,

Ran

---

From: Philip Homburg <[email-masked]>
Date: Tue, 03 Jan 2012 23:57:23 +0100

>case where there are no unused bits at hand.

Well, Van Jacobson made his header compression work with SLIP. And everybody
was happy.

>people towards a protocol translation approach.

What I see in IPv4 PMTU and also IPv6 PMTU is:
- it just doesn't work reliably.
- for IPv4 MSS clamping is widespread to make TCP work. and UDP just gets 
  fragmented.
- For IPv6, given that the host has to do the fragmentation, a big minimum
  MTU is required. Otherwise, too much stuff will end up being fragmented at
  576.

My guess is, that anybody who is running a links at less than 1280 is just
going to get a lot of trouble. That applies both to the links you mentioned
and any IPv6-to-IPv4 translators.

Most IPv4 links are bigger than 1280. So, those translators will seem to work
in most cases. IPv4 links smaller than 1280 will just see failures that
are quite hard to debug.

---

From: RJ Atkinson <[email-masked]>
Date: Wed, 4 Jan 2012 06:36:25 -0500

On 03  Jan 2012, at 17:57 , Philip Homburg wrote:
> - For IPv6, given that the host has to do the fragmentation,
>   a big minimum MTU is required. Otherwise, too much stuff
>   will end up being fragmented at 576.

There is no evidence for the claim above.  When the minimum 
Link MTU for IPv6 *was* set to 576, that did not happen.

Instead, most traffic was Ethernet MTU without fragmentation,
and the next most common packet size was (Ethernet MTU -
overhead for DSL encapsulation/tunnelling).  Packets at
576 were seen -- but very very infrequently -- so they
were NOT a particular burden on the end systems.

> My guess is, that anybody who is running a links at less
> than 1280 is just going to get a lot of trouble. That applies
> both to the links you mentioned and any IPv6-to-IPv4 translators.

Clearly it is more complex to run an IPv6 link below 1280 today
than it was when IPv6 specifications supported a 576 byte MTU.
Folks with such links tell me that it mostly works today.

IPv6-IPv4 translators aren't going away.  There are lots
of other reasons they get deployed -- including transition
and interoperability.

> in most cases. IPv4 links smaller than 1280 will just see failures that

Actually, IPv4 links with 576 byte MTUs aren't problematic.
They have worked well for ~30 years now.

RF links aren't going away, and they aren't all being
replaced by IEEE 802.11 or IEEE 802.16 either. 

Cheers,

Ran

---

From: Philip Homburg <[email-masked]>
Date: Wed, 04 Jan 2012 13:17:22 +0100

>Link MTU for IPv6 *was* set to 576, that did not happen.

RFC-2460 is from 1998. You are talking about the IPv6 network before 1998?
And that resembles todays IPv6 internet in what way?

>were NOT a particular burden on the end systems.

Except that DNS(SEC) packets will be fragmented at the mimimum MTU
(see for example the discussion in
http://tools.ietf.org/html/draft-andrews-dnsext-udp-fragmentation-00)

Setting the minimum MTU to 576 has to potential to cause of damage to DNSSEC.

>Folks with such links tell me that it mostly works today.

That's the problem with PMTU in general: it mostly works.

>of other reasons they get deployed -- including transition

Most IPv4 links are bigger than 1280. Those that aren't will cause suffering.

>They have worked well for ~30 years now.

Except when they do. When I set the link MTU of my WAN link link to 576,
VoIP stops working (over IPv4). I could either try to track down a technical
person to get them to fix the problem, or I just raise the MTU.

And that's how it goes in lots of cases.

>replaced by IEEE 802.11 or IEEE 802.16 either. 

They have had since 1998 to find a fix for the 1280 mimimum MTU problem. It
is not my problem.

---

From: RJ Atkinson <[email-masked]>
Date: Wed, 4 Jan 2012 08:46:31 -0500

On 04  Jan 2012, at 07:17 , Philip Homburg wrote:
> RFC-2460 is from 1998. You are talking about the IPv6 network
> before 1998?  And that resembles todays IPv6 internet in what way?

The network layer is largely the same.  Routing 
is largely the same, except that table sizes
always seem to increase.  Certainly transport
protocols are largely the same.  We now have SCTP,
although it seems not yet widely deployed.

I very much hope DNSsec will become much more 
widely deployed.  I have spent a fair amount of
energy over the past ~15 years helping to ensure 
that [other folks'] DNSsec-related R&D would be 
funded. 

The bulk of the Internet continues to use IPv4 today, 
and probably will for the next decade.  The IPv4 
specifications and widely deployed IPv4 
implementations both support a 576 byte Link MTU.

So DNSsec will need to work over 576 byte links 
just to be deployable in the bulk of the deployed
Internet.  That might be awkward, or sub-optimal, 
but it is not a recent development.

> When I set the link MTU of my WAN link to 576, 
> VoIP stops working (over IPv4).

Curious.  I know of several deployments of VoIP
over small MTU links.  They work fine using standard
off-the-shelf IETF protocols for VoIP (e.g. SIP, RTP).
So it isn't a protocol problem.

> They have had since 1998 to find a fix for the
> 1280 mimimum MTU problem. It is not my problem.

The Link MTU minimum size is the IETF's problem, 
because the IETF is trying to support the whole 
globe's networking needs.  IPv4 does so today.
It would be sad if IPv6 could not do so.

Cheers,

Ran

---

From: Philip Homburg <[email-masked]>
Date: Wed, 04 Jan 2012 16:24:57 +0100

>although it seems not yet widely deployed.

Yes, but PMTU failures are not a protocol issue. Is it is an operational issue.
So when the IPv6 network is just a bunch of techies who are connected by
tunnels, you expect PMTU to sort of work. 

By the time the internet is big is enough that some routers just send ICMPs
with link local source and nobody notices, then PMTU starts to break down.

>but it is not a recent development.

I'm sure you know about the small difference between IPv4 and IPv6 when it
comes to fragmentation. For IPv4, if a DNS server need to send a, say, 1000 
octet reply then it can just send it. If necessary, routers on the way will
fragment the reply. Your 576 octet link will have no effect on me.

On the other hand, for IPv6, a DNS server will have to fragment at the lowest
common denominator. So making the minimum link MTU 576, will cause a lot more
IPv6 fragments then you would get for IPv4. And makes IPv6 quite a bit worse 
than IPv4.

If you follow this to the logical conclusion, then with the IPv6-IPv4
translators, a DNS server has to add a fragmentation header to every DNS reply,
even the small ones.

Wasting an enormous amount of processing power in the years to come.

>So it isn't a protocol problem.

It is not a protocol problem. My guess is that this is some kind of firewall
that drops fragmented packets. But can't be sure.

>It would be sad if IPv6 could not do so.

Well, you can always tunnel IPv6 over IPv4. Problem solved :-)

---

From: RJ Atkinson <[email-masked]>
Date: Wed, 4 Jan 2012 13:47:24 -0500

On 04  Jan 2012, at 10:24 , Philip Homburg wrote:
> Yes, but PMTU failures are not a protocol issue.
> Is it is an operational issue.  So when the IPv6
> network is just a bunch of techies who are connected by

I didn't expect it to work, given the history of PMTU
in the deployed Internet.  When it does work, PMTU is 
wonderful.  It has not been reliable for IPv4 either.

> By the time the internet is big is enough that some
> routers just send ICMPs with link local source and
> nobody notices, then PMTU starts to break down.

We have long standing experience with PMTU.  It hasn't
been completely reliable for IPv4.  It is no surprise 
that it is not completely reliable for IPv6 either.

> I'm sure you know about the small difference between IPv4 and IPv6
> when it comes to fragmentation. For IPv4, if a DNS server need
> to send a, say, 1000 octet reply then it can just send it.
> If necessary, routers on the way will fragment the reply.

Actually, IPv4 routers generally won't fragment the reply
for too-big IPv4 packets -- not even if the DF bit allows
them to do so.  Most deployed IPv4 transit routers 
disabled all router fragmentation of IPv4 packets 
years ago.

My understanding is that existing DNS servers often
choose to send less-than-MTU sized DNS replies
in part because of issues with IPv4 PMTU.  So, again,
this situation is not new with IPv6.

> On the other hand, for IPv6, a DNS server will have
> to fragment at the lowest common denominator. So making
> the minimum link MTU 576, will cause a lot more IPv6
> fragments then you would get for IPv4. And makes IPv6
> quite a bit worse  than IPv4.

That does not sound identical with other folks' analysis
about DNS.  For example, that isn't identical with what
Mark Andrews has said.

> If you follow this to the logical conclusion, then with the
> IPv6-IPv4 translators, a DNS server has to add a fragmentation
> header to every DNS reply, even the small ones.

That does seem consistent with what other folks concerned
about DNS have already noted.

> Well, you can always tunnel IPv6 over IPv4. Problem solved :-)

If there were infinite bandwidth, it would be.  Sadly,
RF links with smaller MTUs generally are also relatively
low data rate -- mostly visibly lower data rate than
Ethernet.

Yours,

Ran

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 4 Jan 2012 13:53:36 -0800

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of RJ Atkinson
> Sent: Wednesday, January 04, 2012 10:47 AM
> To: [email-masked]
> Subject: Re: Fragmentation-related security issues 
> 
> 

"Most deployed IPv4 transit routers disabled all router
fragmentation of IPv4 packets years ago." Are you sure
about that? Because, I have had at least one person from
a major router vendor tell me that router fragmentation
is well supported in their products.

Disabling router fragmentation for DF=0 packets seems
risky; too many things could break.

Fred
[email-masked]
 
> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> [email-masked]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
> --------------------------------------------------------------------
>

---

From: RJ Atkinson <[email-masked]>
Date: Wed, 4 Jan 2012 17:44:01 -0500

> is well supported in their products.

I have no doubt their product possesses the capability.
That is subtly different from folks responsible for
configuring routers disabling that capability in 
particular routers of a particular deployment.

I was referring to the latter, not the former.
My apologies for being unclear.

Yours,

Ran

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 4 Jan 2012 15:11:28 -0800

Hi Ran, 

> Sent: Wednesday, January 04, 2012 2:44 PM
> To: [email-masked]

This can only be done at the peril of black-holing
DF=0 packets that are too large. Meaning, either the
responsible folks have a high degree of certainty that
their core links will pass the packets w/o loss due to
an MTU restriction, or they just don't care that large
packets are silently lost. So, it has to be the former
since folks that just don't care don't stay in business.

This current "pool pah" of FUD reminds me of the one
that transpired at the 2002 v6ops interim meeting when
the subject of tunnel MTU got started. Others may
choose to repeat that, but I've personally seen enough
of it for one lifetime.

Thanks - Fred

---

From: Shane Amante <[email-masked]>
Date: Wed, 4 Jan 2012 16:59:30 -0700

Fred, Ran,

> since folks that just don't care don't stay in business.

Just to shed some current operational reality to this thread.  There are providers who are using 6PE as the means to transport IPv6 packets over a theoretically IPv4-only core/Backbone[1].  One of the chief design principles of 6PE is that those "IPv4-only" Core LSR's only know of IPv4 routes (and, IPv4 routing) and do not have any knowledge/understanding of IPv6.  As such, if/when a Core LSR in a 6PE Backbone receives a IPv6/MPLS packet, i.e.: due to Hop-Limit expired or, in this case, the Core LSR considers it too big to label-switch out the outgoing interface ... the "simple", but ugly, answer is for the Core LSR to drop the IPv6/MPLS packet on the floor.  Fortunately, most providers should be clued in to this "limitation" of 6PE and (should) have Core links set to use [very] large Jumbo frames, thus working around the issue of needing Core LSR's to generate PTB's back to the source.  One potential workaround to the Hop-Limit issue is to use "TTL-hiding" on LSP's, but that tends to be frowned on by Operations folks and customers who rightfully want to be able to use traceroute and cut+paste output that shows where along a path something in broken ...

In short, 6PE *was* a decent stop-gap technology, but should be put out to pasture sooner rather than later.  Although dual-stack on Core devices is one answer, another one would be support for MPLS-signaling protocols (LDP & RSVP) that use native, IPv6 for Transport, which has kind of been on the back-burner -- although some of us have been trying to push it's development more and more.

-shane

[1] Reasons which were likely valid as of just a few years ago, but are no longer so.