Subject: Re: [v6ops] Conclusion (was Re: CVE-2016-1409: IPv6 Neighbor Discovery Crafted Packet Denial of Service Vulnerability)

From: Hemant Singh <[email-masked]>
Date: Sun, 21 Aug 2016 18:54:09 -0400

Not quite.  I am specifically comparing ND to ARP which has its own
ethertype while ND does not.   I am not proposing a new ethertype for
crafted packets in attack vectors.   Obviously there was no firewall in
many deployments where this crafted ND packet went through or there was a
firewall which did not catch the crafted packet.  The new ethertype for ND
and another uses.  It can be used to to filter packets for a DAD or ND
Proxy implementation.

Hemant

On Sun, Aug 21, 2016 at 3:03 PM, Philip Homburg <[email-masked]>
wrote:

>
>
> In the past, a device that filters out traffic that hosts should not
> receive was called a 'firewall'.
>
> I guess it would be nice if every different type of a packet that a
> firewall needs to recognize would get a different ether type, but I'm sure
> that that's not going to happen.
>
> I think it is safe to assume that a firewall can scan TCP and UDP traffic
> at wire speed, so it doesn't seem unreasonable to assume that the same can
> be done for ND traffic.
>
>
>

---

From: Mark Smith <[email-masked]>
Date: Mon, 22 Aug 2016 09:40:48 +1000

On 22 August 2016 at 08:54, Hemant Singh <[email-masked]> wrote:
> crafted packets in attack vectors.   Obviously there was no firewall in many
> deployments where this crafted ND packet went through or there was a
> and another uses.  It can be used to to filter packets for a DAD or ND Proxy
> implementation.
>

Sort of seems to be a lot of effort to mitigate what is fundamentally
a problem of not following very clear processing requirements in
RFC4861 by the target node:

"A node MUST silently discard any received Neighbor Solicitation
   messages that do not satisfy all of the following validity checks:

      - The IP Hop Limit field has a value of 255, i.e., the packet
        could not possibly have been forwarded by a router."

Writing more specifications to mitigate against people not completely
following existing specifications only creates more new specifications
they may not follow completely either.

Does anyone have any insights into how multiple implementations might
have missed or perhaps intentionally decided not to perform this
validity check? It seems a bit unusual that two different commonly
used implementations didn't do it (I just checked the Linux and
FreeBSD source and they do perform the check.)

In cases such as this, perhaps implementation checklists (perhaps as
an RFC appendix) could assist either the implementer or a code
reviewer in ensuring they've implemented things completely.

Regards,
Mark.

> _______________________________________________
> v6ops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/v6ops
>

---

From: Hemant Singh <[email-masked]>
Date: Sun, 21 Aug 2016 22:04:57 -0400

Mark,

It's been already discussed why some nodes missed it and I have also
pointed out that RFC4861 spells out the requirement clearly.   Thanks for
the reminder again.

The reason some nodes missed it because the nodes are high-end routers
forwarding traffic at high rate such as 200 Gbps.  Such a router's data
plane has to perform DPI (Deep Packet Inspection) to catch this crafted ND
packet and the router's data plane incurs the DPI cost on every forwarded
packet.  This is why some routers do not perform the check and forwarded
the packet instead of dropping the packet.

Another error happened that an end node receiving the packets did not drop
it but decided to issue an address resolution, and thus the cpu the packets
are punted to, got inundated.  The end host should have checked the Hop
Limit and dropped the packet.

Regards,

Hemant

---

From: Mark Smith <[email-masked]>
Date: Mon, 22 Aug 2016 15:32:19 +1000

On 22 August 2016 at 12:04, Hemant Singh <[email-masked]> wrote:
> Mark,
>
> It's been already discussed why some nodes missed it and I have also pointed
> out that RFC4861 spells out the requirement clearly.   Thanks for the
> reminder again.
>
> packet.  This is why some routers do not perform the check and forwarded the
> packet instead of dropping the packet.
>

So this sort of issue has also popped up recently with forwarding of
packets with link-local source and GUA destination addresses off-link
- routers are purely forwarding based on destination address for
performance reasons.

It has also popped up with the HBH header - RFC2460bis is loosening
the HBH processing requirement for routers, again for performance
reasons.

Forgetting for the moment about how Source Address Dependent Routing
might change this whole space, perhaps we need to loosen the
requirement that these sorts of checks must be performed at each and
every hop.

Instead, view them as checks to be performed either by the first hop
router or by the last hop router where appropriate (e.g., how we view
BCP38 checks), and allow them to be disabled on trusted router to
router hops for performance, at the possible cost of occasionally
forwarding packets that shouldn't be if they're originated within the
network somehow (e.g., router control plane bug). (I've sort of been
thinking this way a bit already, which is why I was thinking the
HL==255 check is only to be performed by the receiving host, and the
use of the term 'node' should have really been 'host' in RFC4861).

Ideally it would be good if there was a way to automate whether to
enable or them or not e.g., between a pair of routers, the link is
known to be a true point-to-point link, or the set of neighbor cache
entries exactly match the routing protocol neighbors, or all of the
neighbors in the neighbor cache have the Router flag set. (or, as they
seem to be being disabled by default now, it could be as simple as
enabling them if there is at least one neighbor cache entry that
doesn't have the Router flag set).

> Limit and dropped the packet.

So in summary, the HL=255 check actually wasn't being performed in two
different places, and if it had been being performed in one of them,
it attack wouldn't have succeeded:

- intermediate routers weren't performing it

- the ND NS target hosts (control plane of the router implementations)
weren't performing it either

Regards,
Mark.

---

From: [email-masked]
Date: Mon, 22 Aug 2016 10:35:47 +0200

Mark,

> reviewer in ensuring they've implemented things completely.

The issue isn't about the hop limit validation check not being done (it is), it is about _where_ it is done.
A packet is determined to be for the router itself, the hardware forwarding plane then needs to decide if it should process the packet itself, punt the packet to software or drop it. There might be orders of magnitude difference in pps capability between software and hardware.

Is there anything we can learn from this?
We have to consider in protocol design how expensive it is to determine if a forus() packet should be dropped or not.
Looking deep into a packet to determine it is an ND packet and then do the validity checks in hardware might not be optimal.
Perhaps ND packets should have been limited to using link-local addresses only.

In-band signalling is fraught with dangers.

Cheers,
Ole

---

From: Mark Smith <[email-masked]>
Date: Mon, 22 Aug 2016 20:22:01 +1000

On 22 Aug 2016 7:49 PM, "Philip Homburg" <[email-masked]> wrote:
> I'm confused. As far as I know, routers are not supposed to look at the
HL field,
> other than to decrement it and drop the packet when it reaches zero.
>
> Af for a receiving 'host', as far as I know, 'host' refers to a node that
is not
> router. Router are also involved in ND. So the proper requirement is that
any
> node that is the target of an ND (as opposed to just forwarding it) should
> perform the HL check.
>

I've also been confused until I found that section 7.2.3, later than the
HL==255 check section (7.1.1), is where the ND NS DA match check is
performed. (If those sections were in opposite order it would be a bit more
obvious what is happening).

So yes, only the ND NS target performs the HL==255 check.

> Given that processing a ND packet on a router already takes it out of the
> fast path, adding the HL check shouldn't be a big deal.
>
> As long as the router is just forwarding the ND packet, it doesn't have
to care
> so it can use the fast path just fine.
>

So I think what has happened here is that the attack packets have been
manufactured to be sent off-link, and the receiving implementations, when
they were the target of the attack ND NS packet, didn't do the HL==255
check.

> (Obviously it makes sense for firewalls to drop ND packets, but firewalls
are
> supposed to do deep packet inspection anyhow)
>
>

Relating to a comment Ole made, I've wondered why ND didn't exclusively
user link-locals for a while, but never got around to thinking about it. I
think if it did, then nodes would have to maintain a binding between each
non-LL address and the LL address to use to test it e.g. perform NUD.
Probably possible, but more complex.

Regards,
Mark.

---

From: Tore Anderson <[email-masked]>
Date: Mon, 22 Aug 2016 14:33:02 +0200

* [email-masked]

> The issue isn't about the hop limit validation check not being done
> (it is)

Are you certain that it is? Both Cisco CSCuz66542 and Juniper JSA10749
describe how the crafted ND packets will be *processed* by the control
plane - not dropped.

Furthermore, both advisories go on to describe how successful
exploitation might cause the device to stop handling *IPv6* traffic
specifically - as opposed to impacting *all* traffic.

While the advisories don't go into detail, this suggests to me that
successful exploitation of the issue is dependent on the crafted
packets actually being able to manipulate/overwhelm the IPv6 neighbour
caches. That couldn't happen if the hop limit validation was done and
the packet was dropped - even though this was done in software, by the
control plane itself.

> A packet is determined to be for the router itself, the hardware
> forwarding plane then needs to decide if it should process the packet
> itself, punt the packet to software or drop it. There might be orders
> of magnitude difference in pps capability between software and
> hardware.

Sure. If you're able to bombard the control plane with packets,
bypassing any hardware ACL, you're most likely going to be successful
in taking down the box.

However such an attack will not only impact IPv6 (it will offline
everything), nor is there any reason to single out ICMPv6 ND packets as
particularly problematic.

That is: if you're able to saturate the link bandwith between the
control plane and the forwarding plane and/or generate enough PPS to
overwhelm the control plane's CPU (even though it's only going to drop
the packets), then it doesn't really matter what the packets contain.

So I think there's more to this story than a simple volumetric DoS
attack against the control plane that's not being stopped by the
hardware forwarding plane.

Tore

---

From: Gert Doering <[email-masked]>
Date: Mon, 22 Aug 2016 14:39:50 +0200

Hi,

> Perhaps ND packets should have been limited to using link-local addresses only.

Indeed...  (that's the conclusion I reached after building an ACL that
permits only valid combinations of link-local, unspecified and GUA addresses
for each external interface a router is connected to).

ARP is not so bad, in retrospective.  Can't be routed, can't be ACLed by
too-strict IP ACLs, doesn't use multicast, ...

Gert Doering
        -- NetMaster
-- 
have you enabled IPv6 on something today...?

SpaceNet AG                        Vorstand: Sebastian v. Bomhard
Joseph-Dollinger-Bogen 14          Aufsichtsratsvors.: A. Grundner-Culemann
D-80807 Muenchen                   HRB: 136055 (AG Muenchen)
Tel: +49 (0)89/32356-444           USt-IdNr.: DE813185279

---

From: [email-masked]
Date: Mon, 22 Aug 2016 15:35:59 +0200

Tore,

> the packet was dropped - even though this was done in software, by the

you could explain that behaviour differently.
the consequence of a massive stream of ND packets are that they will be heavily throttled between hardware and software forwarding planes. that will also impact legitimate ND packets, meaning that whatever entries there are in the ND cache or the replies from address resolution will not make it back to the ND process.

cheers,
Ole

---

From: Tore Anderson <[email-masked]>
Date: Mon, 22 Aug 2016 17:10:46 +0200

* [email-masked]

> > Are you certain that it is? Both Cisco CSCuz66542 and Juniper
> > JSA10749 describe how the crafted ND packets will be *processed* by
> > the control plane - not dropped.
> > 
> > packets actually being able to manipulate/overwhelm the IPv6
> > neighbour caches. That couldn't happen if the hop limit validation
> > was done and the packet was dropped - even though this was done in
> > software, by the control plane itself.  
> 
> the consequence of a massive stream of ND packets are that they will
> be heavily throttled between hardware and software forwarding planes.
> that will also impact legitimate ND packets, meaning that whatever
> entries there are in the ND cache or the replies from address
> resolution will not make it back to the ND process.

Sure, but that'd just be a variation on the simple volumetric DoS,
nothing that would be unique to ND.

That is: if you rate-limit FOO packets, and someone floods you with FOO
packets, then legitimate FOO packets will be dropped and thus the
operation of the FOO protocol will be impacted. (Replace FOO with ND,
BGP, ARP, OSPF, etc.)

This is common knowledge and does not justify security advisories being
published for each possible value of FOO. Yet now we have one for ND.
So I'm not really convinced that there isn't more to this, i.e., that
the crafted ND packets in question are successfully tricking the
control plane into doing actual work/processing (beyond dropping them).

You didn't answer my question, by the way.

Tore

---

From: [email-masked]
Date: Mon, 22 Aug 2016 19:39:42 +0200

Tore,

> nothing that would be unique to ND.

I believe that's largely correct.

> control plane into doing actual work/processing (beyond dropping them).

Correct. I suppose with ND there is a relatively simple workaround that can be hardcoded. One which may not apply to any generic protocol.

> You didn't answer my question, by the way.

if I'm certain? yes, I'm quite certain. with the reservation that I'm not particularly familiar with XR code.

Cheers,
Ole

---

From: Fernando Gont <[email-masked]>
Date: Tue, 23 Aug 2016 08:10:21 +0200

On 08/22/2016 10:35 AM, [email-masked] wrote:
[....]
>> 
>> Sort of seems to be a lot of effort to mitigate what is
>> fundamentally a problem of not following very clear processing
>> requirements in RFC4861 by the target node:
>> 
>> - The IP Hop Limit field has a value of 255, i.e., the packet could
>> not possibly have been forwarded by a router."
>> 
>> Writing more specifications to mitigate against people not
>> completely following existing specifications only creates more new
>> specifications they may not follow completely either.
>> 
>> Does anyone have any insights into how multiple implementations
>> might have missed or perhaps intentionally decided not to perform
>> this validity check? It seems a bit unusual that two different
>> commonly used implementations didn't do it (I just checked the
>> Linux and FreeBSD source and they do perform the check.)
>> 
>> In cases such as this, perhaps implementation checklists (perhaps
>> as an RFC appendix) could assist either the implementer or a code 
> (it is),

I would expect this to be the case. However, folks seem to have
mentioned that this wasn't really the case?

> it is about _where_ it is done. A packet is determined to be for the
> router itself, the hardware forwarding plane then needs to decide if
> it should process the packet itself, punt the packet to software or
> drop it. There might be orders of magnitude difference in pps
> capability between software and hardware.
> 
> Is there anything we can learn from this? We have to consider in
> protocol design how expensive it is to determine if a forus() packet
> should be dropped or not. Looking deep into a packet to determine it
> is an ND packet and then do the validity checks in hardware might not
> be optimal. Perhaps ND packets should have been limited to using
> link-local addresses only.

If "deep" == "parsing the ICMPv6 header to figure out type/code and tell
the packet type", then... isn't this essentially the same thing with
EHs? That's kind of what all the debate about EHs is about...

(FWIW, I do agree with all your statements above)

Cheers,
-- 
Fernando Gont
SI6 Networks
e-mail: [email-masked]
PGP Fingerprint: 6666 31C6 D484 63B2 8FB1 E3C4 AE25 0D55 1D4E 7492

---

From: Tore Anderson <[email-masked]>
Date: Tue, 23 Aug 2016 10:37:15 +0200

* [email-masked]

> I believe that's largely correct.

Ack. It seems the HL check is indeed being performed by JUNOS at least
(12.3 and 13.3, MX and EX platforms). I tested this by artificially
lowering the HL of the ICMPv6 packets generated by a directly attached
Linux host, like so:

$ ip6tables -t mangle -I OUTPUT -p icmpv6 -m hl --hl-eq 255 -j HL --hl-set 254

As one would expect with a compliant ND implementation, the host lost
IPv6 connectivity as neighbour resolution no longer worked.

So if all there is to this vulnerability is a simple volumetric DoS
attack on the control plane I really wonder what all the fuss is about.
Most operators are fully aware that *any* traffic from untrusted
sources that is being allowed to reach the control plane is a potential
DoS vector. Shockingly enough, IPv6 ND is no exception - news at 11.

Tore

---

From: Gert Doering <[email-masked]>
Date: Tue, 23 Aug 2016 23:08:14 +0200

Hi,

> Most operators are fully aware that *any* traffic from untrusted
                 ^^^^^^^^^^^^^^^
Your optimism on the state of the Internet is amazing :-) - you should
be reading IXP mailing lists for a reality check on the amazingly low
level of understanding out there.

> DoS vector. Shockingly enough, IPv6 ND is no exception - news at 11.

Seriously - I blatantly assumed that IOS XR LPTS would "do the right thing
here", namely, protect me from Joe Random's ND packets sent all over the
Internet to my boxes.  Now, if a peer at the local IXP turns out to be
malicous or gets hacket, yes, this is an obvious attack angle - but
"random packets from the Internet drowning out legitimate on-link
IPv6 ND packets"?

Well, for me, this came as a surprise.

(Arguably, the hardware might not be able to look at TTL - but if it's
smart enough to have a queue for IPv6 ND packets in the first place, why
is it not smart enough to program proper source address filters there?)

Gert Doering
        -- NetMaster

---

From: Greg Daley <[email-masked]>
Date: Wed, 24 Aug 2016 02:13:17 +0000

Hi Gert et al,

While there's not an explicit vulnerability in the protocol which wasn't considered previously, some of the points raised indicate it is not obvious to operators how to provide protection from ND messages (amongst other ICMP) ingressing or egressing your network.

Is it worth providing some guidance steps?

There's been some specific work done with ND-Shield (draft-gont-opsec-ipv6-nd-shield-00) and ND Problems (RFC6583), as well as filtering rules for ICMPv6 (RFC4890)

There is an explicit assumption in RFC 4890 S4.3.3 that the end device will make the right decision and drop packets based on protocol behaviour (i.e. pass ND traffic only with correct hop limits).  I would recommend a change to this behaviour to explicitly control traffic passing a network border.

This would allow systems with bugs and are hard to patch (e.g. IoT) to operate within a safer environment, even if they don't implement full ND checks.

If so, my guess is that sounds more like an operational WG approach rather than 6man.

Sincerely, 

Greg Daley
Solutions Architect
Logicalis Australia Pty Ltd
e [email-masked]
t +61 3 8532 4042
m +61 401 772 770

> -----Original Message-----
> From: ipv6 [mailto:[email-masked]] On Behalf Of Gert Doering
> Sent: Wednesday, 24 August 2016 7:08 AM
> To: Tore Anderson
> Cc: 6man WG; [email-masked]; v6ops list; [email-masked]
> Subject: Re: [v6ops] Conclusion (was Re: CVE-2016-1409: IPv6 Neighbor
> Discovery Crafted Packet Denial of Service Vulnerability)
> 
> Hi,
> 
> Your optimism on the state of the Internet is amazing :-) - you should be
> reading IXP mailing lists for a reality check on the amazingly low level of
> understanding out there.
> 
> > sources that is being allowed to reach the control plane is a
> > potential DoS vector. Shockingly enough, IPv6 ND is no exception - news at
> 11.
> 
> malicous or gets hacket, yes, this is an obvious attack angle - but "random
> packets from the Internet drowning out legitimate on-link
> IPv6 ND packets"?
> 
> (Arguably, the hardware might not be able to look at TTL - but if it's smart
> enough to have a queue for IPv6 ND packets in the first place, why is it not
> smart enough to program proper source address filters there?)
> 
> Gert Doering
>         -- NetMaster
> --
> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> [email-masked]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
> --------------------------------------------------------------------