Subject: [dane] DANE - Tales and Lessons Learned

From: Patrick Ben Koetter <[email-masked]>
Date: Fri, 26 Jun 2015 00:50:54 +0200

Greetings,

I gave a presentation called "One year of DANE - Tales and Lessons Learned" at
the 34th M³AA3WG General Meeting in Dublin a few weeks ago. Barry (Leiba)
asked me in an offlist mail to share the slides with you on this list and also
write about the experience we've made during the last 1,5 years.

# Slides
Here's a quick rundown to give an idea what my presentation was about:

The first part is technical. It starts off naming the issues traditional
opportunistic TLS faces today. Then they show an approach/plan to solve the
named issues only to present DANE as a solution next. The slides then show and
discuss current use cases for DANE (with a strong focus on email) - HTTPS,
SMTP, OPENPGPKEYS, SMIMEA.

The second part deals with adoption, possible markets and reasons why the one
or the other TLD has a better starting position to adopt DANE. It also lists
things we have learned during the last 1,5 years working with DANE or get to
hear often when we talk with people interested to DANE-enable their
infrastructure.

Download the slides at https://sys4.de/download/dane-maawg.pdf.

# Experience
Barry also suggested I summarize our experience with DANE. Here's a list of
things we've learned or I believe to have understood (sudden moments of clarity
are a great thing [tm]):

DANE is server and client
    DANE may be used by clients and offered to others. This is obvious once
    you've become aware of that, but people seem to focus at the "offering
    DANE part" above average. And they seem to fade out the client side.
    
    When they realize the server side isn't low hanging fruit they seem to get
    stuck. Shifting their focus to the client side helps. That's the low
    hanging fruit (assuming you have DANE capable clients at hand).

Using DANE in a client is easy
    You need a DNSSEC-capable resolver and DANE capable client software. It's
    simple, compared to offering DANE, because these componets are simplier to
    control and problems stay your own. Failures have comparatively limited
    effect. Postmasters change settings within their well-known application.
    They don't need to deal with a DNS server, which they might not know as
    well or need to talk a hostmaster to do something for them (they don't
    feel comfortable with).
    
DNSSEC-enabling your infrastructure requires careful thinking
    DNSSEC is about security. The right way [tm] to do DNSSEC on a host is to
    equip it with a local DNSSEC-capable resolver. If you want to add
    fallback resolvers (resolv.conf) *all* of them need to be DNSSEC-capable
    or you might end up with a (bad,insecure,...) DNS reply that would have
    been suppressed by a DNSSEC aware resolver.

Careful if your run your own, internal TLD
    Local TLDs are not DNSSEC signed by default. DNSSEC-aware resolvers will
    suppress answers unless you DNSSEC-enable the internal ZONE or configure
    the resolvers to ignore the DNSSEC chain of trust for that particular
    domain. Test before or you will risc needless service outage.

Check your firewalls
    TLSA, SMIMEA and OPENPGPKEYS queries to DNS servers end up in TCP answers.
    Your firewall must permit TCP on 53. (Most people will probably have that 
    in place if they also serve DKIM. If not this might be the answer to these
    strange DNS problems... ;)
    Check also that your firewall doesn't filter RRs. It might drop requests
    for TLSA and all the other new RRs. It works from inside, but fails for
    outside requests.

Offering DANE RRs is all about DNSSEC
    Offering DANE requires a DNSSEC-enabled DNS server. Everything after
    DNSSEC-enabling the DNS server is comparatively easy.

DNSSEC-enabling is the hardest part
    People are scared to DNSSEC-enable their domain(s). They have to upgrade
    their applications. They have to clean up/refactor/get a better
    understanding of their service. (You're up and running in a few minutes
    and once that works you don't spend time with DNS anymore - unless you're
    a large player or do DNS for business).
    
Registrars don't invest in DNSSEC
    Offering DNSSEC means to invest in security. It requires to invest in a
    race to the bottom market where cheaper seems to be more important than
    more secure. The market interest for DNSSEC isn't great enough at the
    moment to invest into DNSSEC. (In .nl the goverment subsidizes
    DNSSEC-enabled domains. Surveys indicate 50% of .nl is DNSSEC-enabled!)

DNSSEC-capable registrars are hard to find
    At least in .de its hard to find a registrar offering DNSSEC. One of the
    question we get to hear all the time is if we could recommend a list of
    DNSSEC registrars in Germany. To me this seems directly related to the
    little invest we see in DNSSEC on the registrars side - a chicken egg
    dilemma.

Monitor your DNSSEC domain
    Keep an eye on your zones signature TTL. If it expires DNSSEC-capable
    resolvers will suppress replies. They will ignore your domain until you've
    renewed the signature. Monitor your DNSSEC domain. Monitor TTLs and TLSA
    pinned certificate enddates (see next lesson).

RR rollover requires more careful handling
    If you need to roll in a new TLSA (or any other DANE related record) make
    you do it in advance. Leave the old one and add the new one. Wait two TTL
    cycles. Give clients out there time to revisit and learn the new RR. Only
    then remove the old one.

    If you forget to add the new RR you risk receiving no mail from DANE
    enabled clients. They check the RR and if you publish an old TLSA, but use
    a new cert they will not send. That's just how DANE should work. ;)

    Automate the rollover and make sure the rollover works reliably *before*
    you upload your zones signing key to your registrar and publicly
    DNSSEC-able your domain.

DNSSEC obstructs DANE
    People who critice DANE usually criticise it because of DNSSEC. They say
    DNSSEC hasn't been worked on for too long time. The standard should be
    updated. They say it doesn't provide the level of security it could and
    should for something as important as DANE to build upon. From their point
    of view DANE security is directly linked to the level of security DNSSEC
    provides. I'm no DNS expert to tell how much substance there is in their
    criticism. If there is updating DNSSEC standards might help to break the
    ice.

There's no market for DANE at the moment
    Everybody seems to wait for everybody else to start using/offering
    DNSSEC/DANE. Those who DANE enable their clients find only few DANE
    enabled servers to communicate with. Those who could DNSSEC-enable their
    domains and could start serving DANE-related RRs don't see the benefit to
    go through all the required work.

    We - heise.de (Germany's most popular IT magazine and portal), BSI
    (Federal Office for Information Security), DENIC (Germanys .de TLD
    registry) and sys4 - spent about half a year to prepare a DNSSEC-Day.

    DNSSEC-Day will take place next week Tuesday (June, 30th) between 2 p.m
    and 6 p.m. (CEST). It will be a (German language only) video livestream
    (see: http://www.heise.de/netze/dnssec-day) where all four parties will
    discuss topics related to DNSSEC and DANE. Articles and screencasts
    complement the broadcast.

DANE is for machine to machine communication
    As it is there's almost no support for DANE functionality in end user
    applications. If there is, the code is not part of the core, but an
    add-on. It should reside in the core for security reasons. The/some
    reasons why there's no core support in end user clients at the moment have
    been discussed on this list.
    For the time being DANE remains limited to machine to machine
    communication.

That's it for the moment. I hope this summary is useful and we will see wide
adoption of DANE soon.

p@rick

-- 
[*] sys4 AG
 
https://sys4.de, +49 (89) 30 90 46 64
Franziskanerstraße 15, 81669 München
 
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein

---

From: Viktor Dukhovni <[email-masked]>
Date: Fri, 26 Jun 2015 05:53:13 +0000

On Fri, Jun 26, 2015 at 12:50:54AM +0200, Patrick Ben Koetter wrote:

> # Experience
>
> Barry also suggested I summarize our experience with DANE. Here's a list
> of things we've learned or I believe to have understood (sudden moments
> of clarity are a great thing [tm]):

Thanks.

My experience behind is based primarily on tracking the implementations
and deployments of others, rather than doing my own.

The main take-away for me has been:

      Individuals deploying DANE in small scale personal domains
      to make a statement of support for on-line privacy, often do
      so as a point-in-time exercise.

      A year or more later, when the decide to replace their
      certificate and/or private key, they often neglect to update
      their TLSA RRset, and end-up with TLSA records that don't
      match reality, until my monitoring catches up with them and
      they fix the problem.

      We need to find ways to embed reminders to do key rotation
      right into configuration files, README files in directories
      with keys and certificates, perhaps even the certificates
      themselves.  Also into any monitoring tools that report
      "imminent" certificate "expiration", which need to remind
      users about any associated TLSA RRs if any also exist.  This
      needs to be emphasized in product documentation, we must
      shout it from the rooftops!

      Server operators really need to take to heart the content of
      section 8 of the "dane-ops" draft.

Substantially fewer early adopters had outright DNSSEC zone signature
problems, than failures to update TLSA records.  I think this is
in part due to the fact that zone re-signing is easily automated,
and failures are immediately felt, so are remediated quickly.

So while DNSSEC is difficult to deploy, most domains continue to
work reliably after initial deployment.

There are some domains that have *only* deployed DNSSEC (e.g. much
of the ".gov" TLD), and have not yet done anything with DANE, but
which have broken firewalls that block requests for "unexpected" DNS
query RRtypes.

    https://tools.ietf.org/html/draft-andrews-dns-no-response-issue-07

This causes some grief for DANE early adopters, but fortunately
the domains with working DANE already substantially (more than 10:1)
outnumber the domains with broken nameservers.  Just a handful of
sites fixing nameservers or firewalls would lower this number by
another factor of 10.  This is IMHO not a long-term concern.

Finally, some domains publish TLSA RRs, but install software that
selectively enables TLS support only for clients that pass some
sort of "good behaviour" test (e.g. SMTP grey-listing).  This creates
a catch-22, in which DANE clients can never pass the test because
the server never offers TLS, and so always fails authentication.
This is rather rare, and should not be a significant problem in
for most domains.

I am hoping that once one or more of the largest domains enables
DANE for SMTP, there will be much greater visibility of problems
for any administrator who misconfigures his system, resulting in
immediately noticeable mail delay.  This should act to substantially
reduce the number of sites having problems (for any length of time),
and I hope administrators will learn to avoid the above and similar
problems.

On the software implementation side, I continue to see flawed
attempts at DANE verification code.  Generally, usage DANE-EE(3)
is implemented correctly, while the verification logic for usage
DANE-TA(2), PKIX-TA(0) and PKIX-EE(1) is often rather insufficient
to the task.

    Implementors often don't take the time to understand that the "heap"
    of wire certificates from the remote peer don't necessarily form
    any sort of "chain", or even if a chain can be formed from the leaf
    to some ultimate issuer, that chain may well not include any of
    the certificates in the peer's "heap" that matched the TLSA RRset.

    Finding some certificate on the wire that matches the TLSA
    RRset is far from sufficient unless that is the leaf certificate,
    and the usage is DANE-EE(3).

    Then we have the usual failures to perform hostname checks (all
    usages other than DANE-EE(3)), or to support the additional
    reference identifiers required in the SMTP and SRV drafts.

    Bottom line, without a major investment of time, and great
    care, even reasonably experienced programmers are only able to
    write correct validation for DANE-EE(3) and no other usages.

Had the DANE WG in crafting RFC 6698 given much more weight to
programmer fallibility, we might have had just usage DANE-EE(3) to
contend with (and it likely would have been good enough for everyone,
despite the fact that DANE-TA(2) has some operational advantages
for large deployments).  Reaching rough consensus on such a
"radically" minimal model was likely not possible, but we're likely
to deal with security advisories for DANE-enabled products that
are due to the complexity of the other options.

As of today, I am tracking 1450 domains with DANE TLSA records
published for at least one of the best preference MX hosts,  Most
have all MX hosts covered by TLSA RRs.  

-- 
	Viktor.

---

From: Mark Andrews <[email-masked]>
Date: Fri, 26 Jun 2015 16:26:04 +1000

>     https://tools.ietf.org/html/draft-andrews-dns-no-response-issue-07

There are TLD operators that break lookups based on the query type.
See: http://ednscomp.isc.org/compliance/tld-typereport.txt

The "CDS=refused" are, we believe, due to a bug in BIND 9.8.8/9.9.6/9.10.1
which is fixed in BIND 9.9.7/9.10.2.  BIND 9.8.8/9.9.6/9.10.1 add
master file support for CDS.

4049.   [bug]           CDS and CDNSKEY had the wrong attributes. [RT #38491]

Firewalls are a pain in the backside for DNS.  Do anything sightly different
but still with well defined server behaviour specified and you will see a
firewall drop it.  http://ednscomp.isc.org/compliance/summary.html

Mark

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: "Olle E. Johansson" <[email-masked]>
Date: Fri, 26 Jun 2015 09:23:39 +0200

Thank you! Great summary, I learned a lot.

/Olle

On 26 Jun 2015, at 00:50, Patrick Ben Koetter <[email-masked]> wrote:

> _______________________________________________
> dane mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dane

---

From: "Wiley, Glen" <[email-masked]>
Date: Tue, 30 Jun 2015 17:11:53 +0000

I was there and felt that was an excellent presentation.  Thanks for
taking the time to bring that to us.
-- 
Glen Wiley

Principal Engineer
Verisign, Inc.
(571) 230-7917

http://vbsdcon.com

A5E5 E373 3C75 5B3E 2E24
6A0F DC65 2354 9946 C63A

On 6/25/15, 6:50 PM, "Patrick Ben Koetter" <[email-masked]> wrote:

>of