Subject: [v6ops] draft-palet-v6ops-464xlat-opt-cdn-caches concerns, including a potential major security vulnerability

From: Erik Nygren <[email-masked]>
Date: Mon, 12 Aug 2019 10:29:55 -0400

Some comments on draft-palet-v6ops-464xlat-opt-cdn-caches from the
perspectives of having a $dayjob at a CDN....
This comes down to three general concerns, plus a major security concern:

1) Will this do more harm than good?  (ie, creating brokenness which
motivates content to switch back to IPv4-only)
2) Stateless EAMT just won't work due to the typical rate of changes of
A/AAAA pairings in the DNS
3) The IPv4:IPv6 associations may not be adequately 1:1 and client behavior
may be hard to anticipate
4) Poisoning risks/vulnerability  (a major security consideration, which
may make the DNS approach not viable)

TL;DR is that the security issue (an attacker being able to poison the EAMT
cache, which can
also happen non-maliciously as well) may mean the DNS-based approach is far
too dangerous to deploy.

For #1  (the risk/concern factor)...

I worry really worry that while the DNS association option is a clever
hack, that it will turn out to be highly fragile and problematic in
reality.  It only takes some corner-cases that are missed or one bad
implementation for it to do much more harm than good.  (I wrote this even
before I realized the security issue listed below.)

One of the biggest challenges I'm already seeing on getting video content
dual-stacked in some parts of the world is consumer electronics with poor
IPv6 implementations interacting badly with buggy CPE/ISP/home-router
configurations.  Even if IPv6 is generally faster (and may be more
available if a good happy eyeballs implementation is used), it only takes a
relatively small number of subscribers with some Smart TV + ISP
configuration that breaks in the face of dual-stack content to cause a
content provider to get enough calls to switch the content back to
IPv4-only.  I've seen this happen in at least both the Nordics and East
Asia.  I worry that this optimization feature could introduce an entirely
new class of problems like this if poorly implemented.  If the only
recourse content providers have is to switch their content back to being
IPv4-only (as that *may* make the difference between content appearing
broken to IPv4 users and between it appearing to be working), then this
could hinder IPv6 usage much more than it helps.

One approach that might help this would be to have hostname "allowed"
lists, allowing a ISPs to turn this on for select content.  That could
still be problematic if the "allowed" list of hostnames overlaps on the
same IPs as other content. It might at least provide for a way for ISPs to
use this feature with their own STB+content combinations, such as for their
own "walled garden" use-cases where they control both the IPv4-only devices
and the content being delivered.  (Getting robust IPv6 support onto the
devices would obviate the need for this, however.)  Another possibility
might be some opt-in mechanism?  (eg, if an additional DNS record is looked
up that indicates that this translation is supported?)

For #2  (the stateless/stateful factor)...

The A/AAAA IPs returned in DNS can be very dynamic.  For example, many CDNs
have DNS TTLs in the 20 second to 300 second range.  At least for Akamai,
the assignments for which addresses are returned for A and AAAA lookups are
done independently.  As such, frequent churn in associations should be
expected.

If the churn only impact new TCP connections then this part might work some
(much?) of the time (minus other caveats below).
However if the translation is stateless then this is going to be breaking
TCP connections all the time and will make this solution unusable.
It sounds like the current EAM Table (EAMT) lookups are stateless?
This means that each time an association is removed or changed that it will
break all of the long-lived connections using that association.  That seems
likely to result in lots of timeouts and rebuffering  (ie, as streams will
start playing and then will fail when the association changes or disappears
and the connection breaks).

I think a requirement for this solution to work would be for the EAMT table
lookups to result in stateful NAT entries on the home router.    (ie, the
lookups are just used to create stateful NAT entries for the 4-tuple for
the duration of that session/connection.)

For #3 (IPv4:IPv6 associations may not be adequately 1:1 and client
behavior may be hard to anticipate)
This issue has a longer subset of concerns/issues.  The doc seems to touch
on a bunch of them, but I worry if there are lots of lurking corner-cases,
each one of which could cause problems.  Without a good way to fix or
disable bad implementations, each one could be a major problem:

* Not all clients are good at obeying DNS TTLs.  Some are really bad.  (eg,
Java will sometimes just do a single DNS lookup and use it for the duration
of the process lifetime.  Many major browsers also will use a connection
for the duration of its lifetime even if the TTL has changed.  In the
best-case (with stateful translation) this just results in more IPv4 due to
less cache entries.  In a bad case (without stateful translation) it could
mean that connections are pretty much guaranteed to break when they pass
their TTLs and the EAMT entries time out but the clients keep using them.

* The aliasing issue (where the IPv4:IPv6 associations between hostnames
are far from 1:1) is real and will result in brokenness.  The doc has a
bunch of heuristics that help. but given how clients often don't obey TTLs
there may not be enough information available to detect and fix enough of
these cases.  It seems like there are lots of possibilities for timing and
race conditions for cache entries.   As an example, I know of one case
where IPv4 is shared by lots of hostnames and SNI is used to determine
which TLS certificate to hand out, but on the IPv6 side the address
determines which TLS certificate to return.  This or similar may be fairly
common as IPv4 is much more address-constrained.  The result is that a bad
association will return in clints seeing the wrong certificate.  There are
lots of similar but more subtler corner-cases here.

* Related to the previous, it will be tricky to figure out how to handle a
set of IPv4 addresses from A records and sets of IPv6 addresses from AAAA
records.  It will be important to make sure to have some permutations of
the association to avoid hotspotting.

* The aliasing issue may result in effectively dual-stacking some IPv4-only
content which would also be a serious problem.  (This is related to #4
below as well.)

For #4 (major security risk / vulnerability):

It seems like a malicious actor can cause one client (eg, a browser
visiting web sites) to create lots of bad associations by returning A/AAAA
associations unrelated to them.  If done for IPv4-only content and in a
consistent manner, this seems like it can be used to hijack this content.
This vulnerability may be a death-nail for this DNS approach, at least
without a hostname allowlist.

* Take an IPv4-only site "www.example.com" which returns an IPv4 address
"192.0.2.2" (with a fairly short DNS TTL)
* Malicious attacker tricks a client in the same network to resolve "
hostile.example.net" (eg, as a single-pixel ad) which has
{"2001:db8::2:2","192.0.2.2"} but with long DNS TTLs.  The attacker doesn't
need to have any relationship to the IPv4 address to perform this attack.
* Now clients (even dual-stack clients) trying to visit "192.0.2.2" will
have their traffic sent to the attacker-controlled IPv6 address
("2001:db8::2:2"), which could be a middlebox for surveillance, or
something to try and phish the user, or even a DDoS target.

This also potentially is a problem when the same IPv4 address has a mixture
of dual-stack and IPv4-only hostnames pointing to it.
Using DNS to remove EAMT entries might help a little here (eg, if IPv4-only
hostnames with NODATA are found for AAAA lookups), but given that this is
also a security vulnerability and given other issues (eg, widely differing
DNS client behaviors and TTLs) it seems highly likely that this is very
exploitable.

Best regards, Erik