Subject: Review of draft-gont-6man-deprecate-atomfrag-generation-01

From: Hannes Frederic Sowa <[email-masked]>
Date: Tue, 16 Sep 2014 11:04:31 +0200

Hi,

Fernando asked me to review this draft. Here it goes:

> 3.  Denial of Service (DoS) attack vector
> 
>    Let us assume that Host A is communicating with Server B, and that,
>    as a result of the widespread filtering of IPv6 packets with
>    extension headers (including fragmentation)
>    [I-D.gont-v6ops-ipv6-ehs-in-real-world], some intermediate node
>    filters fragments between Host A and Server B.  If an attacker sends
>    a forged ICMPv6 "Packet Too Big" (PTB) error message to server B,
>    reporting a Next-Hop MTU smaller than 1280, this will trigger the
>    generation of IPv6 atomic fragments from that moment on (as required
>    by [RFC2460]).  When server B starts sending IPv6 atomic fragments
> 
> 
> 
> Gont, et al.            Expires February 28, 2015               [Page 3]
> 
> Internet-Draft  Deprecate Generation of IPv6 Atomic Frags    August 2014
> 
> 
>    (in response to the received ICMPv6 PTB), these packets will be
>    dropped, since we previously noted that packets with IPv6 EHs were
>    being dropped between Host A and Server B.  Thus, this situation will
>    result in a Denial of Service (DoS) scenario.
> 
>    Another possible scenario is that in which two BGP peers are
>    employing IPv6 transport, and they implement ACLs to drop IPv6
>    fragments (to avoid control-plane attacks).  If the aforementioned
>    BGP peers drop IPv6 fragments but still honor received ICMPv6 Packet
>    Too Big error messages, an attacker could easily attack the peering
>    session by simply sending an ICMPv6 PTB message with a reported MTU
>    smaller than 1280 bytes.  Once the attack packet has been fired, it
>    will be the aforementioned routers themselves the ones dropping their
>    own traffic.
> 
>    The aforementioned attack vector is exacerbated by the following
>    factors:
> 
>    o  The attacker does not need to forge the IPv6 Source Address of his
>       attack packets.  Hence, deployment of simple BCP38 filters will
>       not help as a counter-measure.
>
>    o  Only the IPv6 addresses of the IPv6 packet embedded in the ICMPv6
>       payload need to be forged.  While one could envision filtering
>       devices enforcing BCP38-style filters on the ICMPv6 payload, the
>       use of extension (by the attacker) could make this difficult, if
>       at all possible.

On FreeBSD and Linux you must hit an open socket to get the ICMP PtB
error to be accepted. E.g. FreeBSD does not deal with ICMP PtB messages
at all on UDP and on both, FreeBSD and Linux, the payload of the ICMP
message will also be checked to the TCP-window. So the unkown variables
for an attacker would be the connection quadruple and the sequence
numbers. Hitting non-connected UDP on Linux sockets seems pretty easy,
but TCP might not be as easy.

>    o  Many implementations fail to perform validation checks on the
>       received ICMPv6 error messages, as recommended in Section 5.2 of
>       [RFC4443] and documented in [RFC5927].  It should be noted that in
>       some cases, such as when an ICMPv6 error message has (supposedly)
>       been elicited by a connection-less transport protocol (or some
>       other connection-less protocol being encapsulated in IPv6), it may
>       be virtually impossible to perform validation checks on the
>       received ICMPv6 error messages.  And, because of IPv6 extension
>       headers, the ICMPv6 payload might not even contain any useful
>       information on which to perform validation checks.

Ack, those would be the above mentioned security checks.

>    o  Upon receipt of one of the aforementioned ICMPv6 "Packet Too Big"
>       error messages, the Destination Cache [RFC4861] is usually updated
>       to reflect that any subsequent packets to such destination should
>       include a Fragment Header.  This means that a single ICMPv6
>       "Packet Too Big" error message might affect multiple communication
>       instances (e.g., TCP connections) with such destination.

Correct for Linux at least.

> 4.  Additional Considerations
> 
>    Besides the security assessment provided in Section 3, it is
>    interesting to evaluate the pros and cons of having an IPv6-to-IPv4
>    translating router rely on the generation of IPv6 atomic fragments.
> 
>    Relying on the generation of IPv6 atomic fragments implies a reliance
>    on:
> 
>    1.  ICMPv6 packets arriving from the translator to the IPv6 node
> 
>    2.  The ability of the nodes receiving ICMPv6 PTB messages reporting
>        an MTU smaller than 1280 bytes to actually produce atomic
>        fragments
> 
>    3.  Support for IPv6 fragmentation on the IPv6 side of the translator
> 
>    Unfortunately,
> 
>    o  There exists a fair share of evidence of ICMPv6 Packet Too Big
>       messages being dropped on the public Internet (for instance, that
>       is one of the reasons for which PLPMTUD [RFC4821] was produced).
>       Therefore, relying on such messages being successfully delivered
>       will affect the robustness of the protocol that relies on them.
> 
>    o  A number of IPv6 implementations have been known to fail to
>       generate IPv6 atomic fragments in response to ICMPv6 PTB messages
>       reporting an MTU smaller than 1280 bytes (see Appendix A for a
>       small survey).  Additionally, results included in Section 6 of
>       [RFC6145] note that 57% of the tested web servers failed to
>       produce IPv6 atomic fragments in response to ICMPv6 PTB messages
>       reporting an MTU smaller than 1280 bytes.  Thus, any protocol
>       relying on IPv6 atomic fragment generation for proper functioning
>       will have interoperability problems with the aforementioned IPv6
>       stacks.
> 
>    o  IPv6 atomic fragment generation represents a case in which
>       fragmented traffic is produced where otherwise it would not be
>       needed.  Since there is widespread filtering of IPv6 fragments in
> 
> 
> 
> Gont, et al.            Expires February 28, 2015               [Page 5]
> 
> Internet-Draft  Deprecate Generation of IPv6 Atomic Frags    August 2014
> 
> 
>       the public Internet [I-D.gont-v6ops-ipv6-ehs-in-real-world], this
>       would mean that the (unnecessary) use of IPv6 fragmentation might
>       result, unnecessarily, in a Denial of Service situation even in
>       legitimate cases.
> 
>    Finally, we note that SIIT essentially employs the Fragment Header of
>    IPv6 atomic fragments to signal the translator how to set the DF bit
>    of IPv4 datagrams (the DF bit is cleared when the IPv6 packet
>    contains a Fragment Header, and is otherwise set to 1 when the IPv6
>    packet does not contain an IPv6 Fragment Header).  Additionally, the
>    translator will employ the low-order 16-bits of the IPv6 Fragment
>    Identification for setting the IPv4 Fragment Identification.  At
>    least in theory, this is expected to reduce the Fragment ID collision
>    rate in the following specific scenario:
> 
>    1.  An IPv6 node communicates with an IPv4 node (through SIIT)
> 
>    2.  The IPv4 node is located behind an IPv4 link with an MTU < 1260
> 
>    3.  ECMP routing [RFC2992] with more than one translator are employed
>        for e.g., redundancy purposes
> 
>    In such a scenario, if each translator were to select the IPv4
>    Fragment Identification on its own (rather than selecting the IPv4
>    Fragment ID from the low-order 16-bits of the Fragment Identification
>    of atomic fragments), this could possibly lead to IPv4 Fragment ID
>    collisions.  However, since a number of implementations set IPv6
>    Fragment ID according to the output of a Pseudo-Random Number
>    Generator (PRNG) (see Appendix B of
>    [I-D.ietf-6man-predictable-fragment-id]) and the translator only
>    employs the low-order 16-bits of such value, it is very unlikely that
>    relying on the Fragment ID of the IPv6 atomic fragment will result in
>    a reduced Fragment ID collision rate (when compared to the case where
>    the translator selects each IPv4 Fragment ID on its own).

OT:
This is very well true! On Linux still we use per-host increasing
randomized IDs, but at least since the paper "Counting Packets Sent
Between Arbitrary Internet Hosts" by Jeffrey Knockel and Jedidiah
Crandall showed up, we also seed in a bit of randomness. Also multiple
IP addresses now share one IP-state "bucket".

I still think that for generating fragmentation ids using just an
increasing counter and encrypting it with a very small block cipher
(e.g. RC5) is the way to go, but very slow.

>    Finally, we note that [RFC6145] is currently the only "consumer" of
>    IPv6 atomic fragments, and it correctly and diligently notes (in
>    Section 6) the possible interoperability problems of relying on IPv6
>    atomic fragments, proposing as a workaround something very similar to
>    what we propose in Section 6.  We believe that, by making the more
>    robust behavior the default behavior of the "IP/ICMP Translation
>    Algorithm", robustness is improved, and the corresponding code is
>    simplified.
> 
> 
> 
> 
> 
> 
> 
> 
> Gont, et al.            Expires February 28, 2015               [Page 6]
> 
> Internet-Draft  Deprecate Generation of IPv6 Atomic Frags    August 2014
> 
> 
> 5.  Updating RFC2460
> 
>    The following text from Section 5 of [RFC2460]:
> 
>       "In response to an IPv6 packet that is sent to an IPv4 destination
>       (i.e., a packet that undergoes translation from IPv6 to IPv4), the
>       originating IPv6 node may receive an ICMP Packet Too Big message
>       reporting a Next-Hop MTU less than 1280.  In that case, the IPv6
>       node is not required to reduce the size of subsequent packets to
>       less than 1280, but must include a Fragment header in those
>       packets so that the IPv6-to-IPv4 translating router can obtain a
>       suitable Identification value to use in resulting IPv4 fragments.
>       Note that this means the payload may have to be reduced to 1232
>       octets (1280 minus 40 for the IPv6 header and 8 for the Fragment
>       header), and smaller still if additional extension headers are
>       used."
> 
>    is formally replaced with:
> 
>       "An IPv6 node that receives an ICMPv6 Packet Too Big error message
>       that reports a Next-Hop MTU smaller than 1280 bytes (the minimum
>       IPv6 MTU) MUST NOT include a Fragment header in subsequent packets
>       sent to the corresponding destination.  That is, IPv6 nodes MUST
>       NOT generate IPv6 atomic fragments."

This is the same as ignoring the packet? Should we still notify any
errors without updating path MTUs?

I am not sure because user space might have cared about IPv6 < 1280
errors before and they very well might be listening for them.

So in an operating system we just don't update the "destination cache"
anymore but we don't treat it as an illegal ICMP error?

> Appendix A.  Small Survey of OSes that Fail to Produce IPv6 Atomic
>              Fragments
> 
>    [This section will probably be removed from this document before it
>    is published as an RFC].
> 
>    This section includes a non-exhaustive list of operating systems that
>    *fail* to produce IPv6 atomic fragments.  It is based on the results
>    published in [RFC6946] and [Morbitzer].
> 
>    The following Operating Systems fail to generate IPv6 atomic
>    fragments in response to ICMPv6 PTB messages that report an MTU
>    smaller than 1280 bytes:
> 
>    o  FreeBSD 8.0
> 
>    o  Linux kernel 2.6.32
> 
>    o  Linux kernel 3.2
> 

I am a bit unsure but hadn't yet time to test this. We definitely have
the logic to generate atomic fragments in those mentioned linux
versions.

Maybe the injection of the ICMP errors didn't work? At least Linux I can
recheck very soon.

Greetings,
Hannes

---

From: Hannes Frederic Sowa <[email-masked]>
Date: Tue, 16 Sep 2014 20:14:16 +0200

> Maybe the injection of the ICMP errors didn't work? At least Linux I can

This following packetdrill script might be helpful for testing:

/* check current mtu with connected socket */
0.000 `tcpdump -n -i tun1 &`

0.000 socket(..., SOCK_DGRAM, IPPROTO_UDP) = 3
0.000 connect(3, ..., ...) = 0
0.000 getsockopt(3, IPPROTO_IP, IP_MTU, [1500], [4]) = 0
0.000 close(3) = 0

/* inject frag_needed to unconnected socket */
2.000 socket(..., SOCK_DGRAM, IPPROTO_UDP) = 3
2.000 sendto(3, ..., 1500, 0, ..., ...) = 1500
2.000 > udp (1500)
/* this simulates a forged icmp packet which tries to lower the mtu */
2.100 < [udp (1000)] icmp packet_too_big mtu 1200

2.5 sendto(3, ..., 1000, 0, ..., ...) = 1000
2.5 > frag | udp (1000)
3.000 close(3) = 0

4.000 socket(..., SOCK_DGRAM, IPPROTO_UDP) = 3
4.000 connect(3, ..., ...) = 0
4.000 getsockopt(3, IPPROTO_IP, IP_MTU, [1280], [4]) = 0
4.000 close(3) = 0

# /home/hsowa/bin/packetdrill -v --ip_version=ipv6 --mtu=1500 /home/hsowa/mtu.pdrill 
tcpdump: WARNING: tun1: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun1, link-type RAW (Raw IP), capture size 65535 bytes
outbound sniffed packet:  2.000293 udp (1500)
20:13:25.687362 IP6 fd3d:fa7b:d17d::1.42626 > 2001:db8::1.webcache: UDP, length 1500
inbound injected packet:  2.100098 icmpv6
20:13:25.787266 IP6 2001:db8::1 > fd3d:fa7b:d17d::1: ICMP6, packet too big, mtu 1200, length 56
20:13:26.187208 IP6 fd3d:fa7b:d17d::1 > 2001:db8::1: frag (0|1008) 42626 > webcache: UDP, length 1000

This is on an almost current 3.16.2 kernel.

Bye,
Hannes

---

From: Fernando Gont <[email-masked]>
Date: Tue, 16 Sep 2014 17:29:13 -0300

Hi, Hannes,

Thanks so much for the review! Please find my comments in-line...

On 09/16/2014 06:04 AM, Hannes Frederic Sowa wrote:
> message will also be checked to the TCP-window.

What if the TCP header is not there? Is the ICMPv6 dropped?

(Actually, I'll try to check that with a current Linux kernel..)

> So the unkown variables

Does an ICMPv6 PTB targetted at a UDP socket span all sockets with that
Destination Address? i.e., could I use UDP (which enforces fewer checks)
to attack TCP flows?

[....]
> Crandall showed up, we also seed in a bit of randomness.

We had sent a report to some Linux folks regarding issues with
predictable Frag IDs arounf 2010 or 2011. At the time, it was changed
from a global counter to a per-dest counter (initialized to a random
value, IIRC).

Has anything changed since then? e.g., what do you mean by "seed in a
bit of randomness).

> Also multiple

That's so that the per-destination counter is less deterministic, or what?

> (e.g. RC5) is the way to go, but very slow.

Take a look at the hash-based approach in
[I-D.ietf-6man-predictable-fragment-id]. You probably get the same
thing... but faster...

> errors without updating path MTUs?

The original advice was to just drop the ICMPv6 PTB. However, it was
later noted that still notifying the errors (without updating the Path
MTU) would be better. -- And I'd say I'd agree. e.g., ythat could result
in a more specific error message if eventually e.g. the TCP connection
times out.

> errors before and they very well might be listening for them.

Any ideas what they might be using those ICMPv6 PTB for? -- The only
consumer for ICMPv6 PTBs<1280 is SIIT... so one would expect that any
other app would just drop those PTBs if received....

> anymore but we don't treat it as an illegal ICMP error?

Exactly.

> the logic to generate atomic fragments in those mentioned linux

Since there are tools to do that
(<http://www.si6networks.com/tools/ipv6toolkit>), I'll try to check that
for TCP, UDP, and ICMPv6.

Thanks!

Best regards,
-- 
Fernando Gont
SI6 Networks
e-mail: [email-masked]
PGP Fingerprint: 6666 31C6 D484 63B2 8FB1 E3C4 AE25 0D55 1D4E 7492

---

From: Fernando Gont <[email-masked]>
Date: Wed, 17 Sep 2014 02:12:26 -0300

Hi, Hannes,

On 09/16/2014 05:56 PM, Hannes Frederic Sowa wrote:
> Yes, you can.

So... if there's a socket with the same remote address, the TCP SEQ
check can be bypassed by performing the attack with ICMPv6 messages
embedding UDP packets. (just double-checking)

> So next outgoing packet will already use new mtu information.
> 
> Kernel before 3.14 also used this MTU value for the forwarding path but
> did not look at dst_allfrag attribute (which is the function which
> checks if atomic fragments should be generated).

"Forwarding path" meaning "if the box is operating as a router for this
packet"?

[...]
> The above paper showed that it is possible to infer from the
> fragmentation ids if two independent machines on the internet are
> communicating with each other:
> 
> https://www.cs.unm.edu/~crandall/foci2014.pdf

I'll check the paper.

> The tree we kept the per-destination in became a bottleneck: The new
> method uses less memory and does faster lookup. A side effect is also
> that the per-destination counter are less deterministic.
> 
> The new hash to store the counters also keeps a timestamps to later
> check when the last fragmentation id was used.
> 
> In case we didn't communicate with a host for some time we add a random
> value clamped to (current time - last timestamp).

Cool. Thanks for the clarification.

> I think you're referring to 6.3? I'll study that, thanks!

Section 5.3. -- As far as I recall, Linux uses/used the same sort of
algorithm for TCP port randomization -- with the exception that you
employ a global counter rather than counter[G(src IP, Dst Pref, secret2)]

> I also only think those would be SIIT applications sitting on top of
> Linux. I am unsure if they can make use of those notifications any more,

Why?

> but I don't think they will upgrade right away. If we push the changes
> to linux stable, all currently supported kernel will ignore PTB<1280
> right away, that's why I ask.

Certainly the idea is that the kernel ignores PTB<1280. If it's also
possible to still notify the errors to the upper layer, that'd be
perfect. Thus e.g. TCP can report a more specific error message other
than "timeout" and, if there's any upper-layer app that has not yet been
patched to *not* rely on ICMPv6 PTBs, it can still do its job.

Thoughts?

> Merci! I also attached a packetdrill snippet in the other mail, which
> you could use in FreeBSD (IIRC) and Linux to check the stacks behavior.

I'll report back asap.

Thanks so much!

Best regards,

---

From: Hannes Frederic Sowa <[email-masked]>
Date: Tue, 16 Sep 2014 22:56:30 +0200

Hi Fernando,

> What if the TCP header is not there? Is the ICMPv6 dropped?

Yes, it is. All actual ICMP error notifications and updates always
happen in the context of the socket.

Linux supports multiple routing tables chained with rules, so we need
the socket to also find the correct routing table. If we don't find the
socket, we don't have anything in our hands to redo the exact same
lookup, sometimes this depend on socket values. Also important for
namespaces, soon.

Yes, you can.

Every socket caches its routing lookup. If the routing lookup gets
modified by an icmp error, we increase a serial number. Next time a
socket tries to reuse its cached routing lookup, it will notice the old
serial number, destroy the old lookup and redo it.

Bye,
Hannes

---

From: Ray Hunter <[email-masked]>
Date: Wed, 17 Sep 2014 09:01:07 +0200

> Destination Address? i.e., could I use UDP (which enforces fewer checks)

I think the basic question is "what do the Linux kernel developers 
consider as a 'flow'?"

RFC 1192 and RFC 4821 are deliberately vague.

You are correct, Fernando: if the flow is simply cached based on 
<destination address> then you could use UDP to attack TCP.

But even if a "flow" is identified as the duple <destination address, 
transport protocol> then you could also potentially target an attack 
against NFS or FCIP over TCP by targeting an HTTP or other seemingly 
innocuous loopback process listening on another interface on the device, 
that may connected to a completely different network.

if a "flow" was identified as the triple <destination address, transport 
protocol, outbound interface> or<destination address, transport 
protocol, sourceaddress> that attack would be tougher.

However, if the "flow" was <destination address, sourceaddress, 
transport protocol, source port, destination port> you'd be far less 
likely to be able to cache the learned MTU to be used on other sessions, 
meaning PMTUD effectively becomes PLPMTUD.

Unless the developers used something like <destination address, 
sourceaddress, transport protocol, "passive open port"> and remembered 
which party initiated the flow via a passive open (server) and which an 
active open (client).

>> I am not sure because user space might have cared about IPv6<  1280

-- 
Regards,
RayH

---

From: Hannes Frederic Sowa <[email-masked]>
Date: Wed, 17 Sep 2014 11:52:12 +0200

Hi, Fernando,

> embedding UDP packets. (just double-checking)

That's correct.

Exactly.

It would be great if this could be clarified. IMHO path mtus should
never be used in forwarding paths. There was a rather long discussion
going on on netdev@ and I couldn't cite RFCs that much, wich would
helped a lot (I know there are special cases, so people can still switch
to the old mode if necessary).

> employ a global counter rather than counter[G(src IP, Dst Pref, secret2)]

Hmm. The TCP port randomization is mostly steered by the hash table.
First try is a hash, after that the layout of the hash table determines
the free port.

> > Linux. I am unsure if they can make use of those notifications any more,

I am not sure at all - just asking. Never had to do anything with SIIT.

> patched to *not* rely on ICMPv6 PTBs, it can still do its job.

We certainly can do that.

Bye,
Hannes