Subject: Re: [DNSOP] Favor: Weigh in on draft-ietf-ipsecme-split-dns?

From: Scott Morizot <[email-masked]>
Date: Sat, 1 Dec 2018 09:09:12 -0600

I guess I'll speak up as someone who has been managing the DNS/DNSSEC
design and implementation of a large organization with a complex set of DNS
requirements (operational and security-related) since we began the process
of signing our zones in 2011. We have universal DNSSEC validation in place
across our enterprise recursive layers and almost all zones, internal and
external, are DNSSEC signed. I forget the exact count without checking, but
we have 100+ 3rd level and lower internal zones. Outside public placeholder
zones required for proper delegation and chain of trust at the appropriate
level where internal/external divisions are made (typically the 3rd level)
we do not normally maintain differing versions of external/internal zones,
but we have a few explicitly defined for that purpose because there is a
legitimate requirement for differing internal/external resolution. Those
needs are always segregated into zones defined for that purpose to simplify
ongoing administration. (We never want to be in a situation where normal,
routine record updates require changes in multiple versions of a zone. That
would be an administrative nightmare.) We employed manual trust anchors at
different phases of our implementation, but now have very few in place. One
is to establish trust for an internal zone for which we do not control the
administration of the parent zone. Off the top of my head, I believe that's
the only significant one left. Maintain trust anchors was unpleasant and
always viewed as a transitional mechanism, not something we wanted to do on
a lasting basis.

On Fri, Nov 30, 2018 at 11:22 AM Paul Wouters <[email-masked]> wrote:

> On Fri, 30 Nov 2018, Ted Lemon wrote:
> That means your public DNS zone must be signed for your internal DNS
> zone to be signed? Otherwise you cannot have this signed delegation?
> That would mean you cannot run DNSSEC internally before you run it
> externally.
>
>
Outside placing trust anchors on every device performing validation, that's
true. Establishing trust in a zone when the parent zone is insecure is a
pain. I guess DLV could still be used. I considered it at one point. I
forget why I decided not to go that route. It was too many years ago. Then
again, from the signing side, I'm hard-pressed to imagine a reason to start
signing internal zones without signing their external parents except in the
situation where you do not control the public parent zone. Normally you
want to start at the top of the hierarchy and work your way down.

> >   When you look things up in that zone outside the firewall, you get
> NXDOMAIN for everything but the SOA on the zone, which returns an old
> serial number.   Inside the firewall, they get answers, which
> > are signed, and which validate.   There's no need for a special trust
> anchor here.
>
> This scheme seems to require both inside and outside zones are signed
> with the same key, or as Mark pointed out, both internal and external
> zones need to share their DS records and keep these in sync. As these
> are usually different organisations/groups/vendors/services, that is
> an actual management problem.

Or you can do what we did and place DS records for both the internal and
external KSKs (using public placeholder versions of the internal child
zones just to establish the delegation point) in the parent zone. That
establishes chain of trust whichever version of the zone is resolved. That
works whether the public version is just a placeholder with nothing else of
significance in it or if it is a zone where both the external and internal
records matter and are used by the appropriate source for the query, but
differ.

> >   There are two ways to approach this.   One is to assume that the
> validator never checks the SOA on the zone.   This is almost certainly the
> case for nearly any use case.   In that case, you just
> > run the internal and external name servers with the same ZSK, and have a
> delegation above it.   You don't worry about zone serial numbers, because
> they don't affect validation.   When you're inside the VPN,
> > you get answers for the internal zone; when you're outside the vpn, you
> get answers for the outside.   Both validate, because the DS record(s) are
> referring to the same ZSK(s).
>
> coordinating shared ZSK's is even harder then requiring sharing DS
> records! ZSKs roll every month, and a lot of software auto-generates
> and performs the roll without any humans involved. It seems extremely
> fragile to need to coordinate ZSKs between different organisations,
> be ready for the same algorithm rollovers, etc etc.
>
>
Yes, absolutely. I wouldn't want to try to coordinate sharing signing keys
(KSK or ZSK) across different zones under any circumstance. I know some
places that do it with KSKs because the product they use supports it as
long as both versions of the zone are signed on the same device, but we
don't even sign internal and external zones in the same place. Or maintain
the master data in the same place for that matter.

I don't really have any thoughts on the draft itself, but wanted to add my
own real world experience with managing DNSSEC trust. On the validation
side, we also don't allow any device in the enterprise network, including
internal nameservers, to send or receive DNS queries to the Internet.
Period. Only our perimeter recursive, DNS caches can do that and those will
only accept queries from authorized enterprise recursive, caching
nameservers. And we have protections, including different DNS blacklists,
at each layer. So we have a lot of control over what devices connected to
our network can and cannot do and they do not have unrestricted ability to
query Internet DNS even through our multiple layers. We also use the cli
version of dnsviz a lot to ensure we understand what different points on
the recursive side of our enterprise DNS infrastructure are seeing.

Scott

---

From: Paul Wouters <[email-masked]>
Date: Sat, 1 Dec 2018 10:49:42 -0500 (EST)

On Sat, 1 Dec 2018, Scott Morizot wrote:

> I guess I'll speak up as someone who has been managing the DNS/DNSSEC design and implementation of a large
> organization with a complex set of DNS requirements

Thanks for the write up! It is always good to hear actual enterprise
deployments.

> We employed manual trust anchors at different phases of our implementation, but now have
> very few in place. One is to establish trust for an internal zone for which we do not control the administration of
> the parent zone.

So the interesting question here is, how is this one zone securely
resolved when you are connected to the network via a VPN? Or is this
zone not available for VPN users? And if this draft could make it
available, would you? Or, do your VPN users not use split-tunnel,
and connect to an inside DNS server once connected?

> Or you can do what we did and place DS records for both the internal and external KSKs (using public placeholder
> versions of the internal child zones just to establish the delegation point) in the parent zone. That establishes
> chain of trust whichever version of the zone is resolved. That works whether the public version is just a
> placeholder with nothing else of significance in it or if it is a zone where both the external and internal records
> matter and are used by the appropriate source for the query, but differ.

That is a good solution as well. Although I would be a little worried that
a compromise of the public view could trick people into establishing the
"internal" zones in the external view to trick people into logging on
and giving out their password. But that is surely far down the list of
easy attacks to run.

> I don't really have any thoughts on the draft itself

That is too bad, because I'd be very curious how you handle DNSSEC with
your VPN clients. Not many large organizations run DNSSEC on both
internal and external views. So I am curious how you handle DNSSEC
for your (split?) VPN clients. But I understand if you cannot share
more data about that in a public forum.

> , but wanted to add my own real world experience with managing
> DNSSEC trust. On the validation side, we also don't allow any device in the enterprise network, including internal
> nameservers, to send or receive DNS queries to the Internet. Period. Only our perimeter recursive, DNS caches can do
> that and those will only accept queries from authorized enterprise recursive, caching nameservers. And we have
> protections, including different DNS blacklists, at each layer. So we have a lot of control over what devices
> connected to our network can and cannot do and they do not have unrestricted ability to query Internet DNS even
> through our multiple layers. We also use the cli version of dnsviz a lot to ensure we understand what different
> points on the recursive side of our enterprise DNS infrastructure are seeing.

So I guess the interesting thing here is the device that is in
split-tunnel mode. It would not receive the protection of your
DNS infrastructure for external lookups, so if you allow BYOD
to connect to your network, it would be bypassing those DNS
security checks for non-organisation DNS queries, which would
put the BYOD under greater risk. Perhaps you do not allow
split-tunnel or BYOD for those reasons :)

Thanks for your input,

Paul

---

From: Benjamin Kaduk <[email-masked]>
Date: Sun, 2 Dec 2018 19:12:26 -0600

> Thanks for the write up! It is always good to hear actual enterprise

I will second this "thank you" -- it was very helpful to me, since this is
not an area that I am normally exposed to in my day-to-day work.

-Benjamin

---

From: Paul Wouters <[email-masked]>
Date: Sun, 30 Dec 2018 14:17:36 -0500 (EST)

On Tue, 27 Nov 2018, Joe Abley wrote:

> On 27 Nov 2018, at 10:02, Tony Finch <[email-masked]> wrote:
>
>> Really, I don't think this is (just) a VPN issue: the question of how to
>> install DNSSEC trust anchors for private zones has not been solved for the
>> general case, so a VPN-only patch seems premature and unhelpfully specific
>> to me.

As currently no one is able to do internal DNSSEC zones and VPNs, I
cannot agree that is is premature or unhelpfully specific. It is a real
solution to a real problem. Regardless of any generic solution, you end
up with the same kind of considerations we reached here. A whitelist
of zones that is securely provisioned and not dynamically updated, and
flexibility to roll your internal trust anchor without locking out all
your [VPN] clients from accessing internal resources.

> To your point, Paul, I did see the text in the applicability and security considerations sections about ignoring the INTERNAL_DNSSEC_TA payloads in the case of non-split-tunnel situations. I understand the restriction to use in split tunnel scenarios, and the requirement to implement a whitelist, but I don't think either of those are particularly strong (and the consequences of ignoring or subverting them are more serious than I think the document gives credit for).

When installing provisioning profiles, eg Apple tells you things like
"this party can monitor everything you do, are you sure you want to do
this". I don't think there is anything more you can do than that.
Although if there is, clients can apply those additional security
measures as local policy. For instance, a phone or laptop that is
under Enterprise Management could forbid DNSSEC_TA installations
outside of their signed profiles. Or a phone could decide DNSSEC_TA
installations will only ever be accepted when under Enterprise
management control. This draft does not forbid any of these kind of
decisions.

> For the split tunnel restriction, my pedantic objection is that all VPNs whose payload and scaffolding uses a single protocol (e.g. an IPv4 tunnel endpoint to build a tunnel through which to send IPv4) is a split tunnel, since the tunnel endpoint needs necessarily not to be encapsulated by the VPN client. Local networks are also frequently excluded in non-split configurations, too. Where is the strict definition of "split tunnel"? This is a real question; you know more about tunnel semantics than I do.
>
> My arguably more reasonable observation is that any non-enterprise, Netflix-gaming VPN service that wanted to be a bad actor could simply provide a split tunnel service (according to the definition I was looking for) described as something different, but covering enough of the IPv4 address space that the end-user had no real way of noticing, at which point the INTERNAL_DNSSEC_TA payload could presumably be accepted by the client.

Yes they can. Netflix could send you a provisioning profile that says it
wants to MITM gmail.com. You could install that because you trust
Netflix. Trust is part of a company's success. Let's hope trustworthy
companies won't screw around like this. Note also these companies
already can trick people into install Root CA's, so this is basically
a minor variant of the power they already have to subvert endusers.

Sure, people are tricked in installing Random Inc things. They will
click okay on any warning. Those people are beyond help. Yes we
could not allow enterprises to have DNSSEC (which is basically what
you say when you do not allow them to rollover keys indepedently of
the VPN configuration) but that would be a tragedy of the commons.

> In a strictly enterprise scenario it's not uncommon for the client configuration to be handled by the IT department as part of normal IT asset management. If the IT department is configuring the VPN client, is it really a stretch to imagine that they couldn't install a local validator and trust anchors at the same time? That would solve the .CORP and RFC1918-reverse cases for those end-users without having to worry about nefarious use of these proposed new payloads for end-users whose clients essentially have no informed administrator. If you expect administrators to maintain an accurate whitelist, surely they could just configure TAs and forward zones themselves. If there's no centralised mechanism to update that kind of configuration in a friction-free way to accommodate TA rolls then presumably the devices concerned should all be considered compromised and banned from connecting to the network anyway.

VPN users cannot always update their provisioning on the spot, without
being on the VPN already. VPN users often don't use the VPN for months,
and only use it when traveling for example. Like on a dedicated travel
phone. Your suggestion of hardcoding TA keys is imply impractical and
won't fly in real operational deployments. It is extremely fragile. Having
restrictions on the whitelist was already a compromise. Originally,
the draft let IKE update the white list too. We (reluctantly) decided
that we didn't want one compromised VPN device to be able to install
malicious DNSSEC_TA's. So we limited the whitelist update to a (hopefully
more secure) provisioning system.

> One more thought; in a possible future where it's applications like browsers that do validation and not an OS-provided stub resolver library, this proposal leaves a gap between the VPN client (which might receive new trust anchor information) and those applications (which need signalling to know that such new trust anchor information exists). That seems worth mentioning, even if it's out of scope to provide a solution.

If browsers create that painful world of overriding the OS, they will
need to solve that. That is very far out of scope for this VPN draft,
and we cannot stall this document for the multi-years discussion on
DoH and DOT impact on overriding random DNS trees at whim, by users
being ticked into configuring certain DoH/DOT servers. The problem is
similar in nature, but totally orthogonal.

> As is probably obvious I don't particularly like this proposal, but I also acknowledge that I'm perhaps not the target audience and I'm missing context that would make it seem more useful. If it goes ahead, maybe some of the observations here could help.

I think DNSOP needs to realise that DNS VPN configuration is going to
happen. With DNSSEC it is going to happen. We need to come up with a
solutation that seems least scary while still being deployable. Simply
saying no won't change the draft. If the draft doesn't change or is
blocked by this, non-standard or draft solutions will be used leading
to the same undesired outcome plus vague standards.

I still believe the current draft describes the most strict, while still
being usable, solution. I'm all up for smarter ideas from smarter people,
but we have already sat down with a bunch of security people back at
IETF 100, including the ADs and IPsecME chairs, and couldn't come up
with something better than the current compromise proposal.

Note that formally, the DISCUSS that triggered this was marked as
resolved by Warren. I'm happy to discuss this for a bit longer if
we are trying to build something that might be better. But we have
done Early Code Points already so at some point we have to stop
discussing and publish with the rough consensus.

Paul