Subject: Re: irtf.org DNSSEC signatures (partly) expired

From: Glen <[email-masked]>
Date: Sat, 7 Nov 2015 13:09:37 -0800

Dear Mr. Dukhovni:

I'll open a trouble ticket with Afilias; however, for the moment, I
have re-signed all the files locally, and done a serial number
increment, and pushed them to Afilias.  I will watch to see if that
clears it.

Also, please let me remind everyone on the list that the reporting
address for things of this type is [email-masked].

Thanks,
Glen
Glen Barney
IT Director
AMS (IETF Secretariat)

On Sat, Nov 7, 2015 at 12:54 PM, Viktor Dukhovni <[email-masked]> wrote:
> It looks like master -> slave DNS updates are failing, only the master
> nameserver has unexpired signatures:
>
>     http://dnsviz.net/d/irtf.org/dnssec/
>
> However, all the nameservers report the same SOA serial as the master:
>
>     $ dig -t ns +noall +ans +nocl +nottl irtf.org @ns0.amsl.com.
>     irtf.org.           NS      ns0.amsl.com.
>     irtf.org.           NS      ns1.ams1.afilias-nst.info.
>     irtf.org.           NS      ns1.hkg1.afilias-nst.info.
>     irtf.org.           NS      ns1.mia1.afilias-nst.info.
>     irtf.org.           NS      ns1.sea1.afilias-nst.info.
>     irtf.org.           NS      ns1.yyz1.afilias-nst.info.
>
>     $ dig -t soa +noall +ans +nocl +nottl irtf.org @ns0.amsl.com.
>     irtf.org.           SOA     ns0.amsl.com. glen.amsl.com. 1200000226 1800 1800 604800 1800
>
>     $ while read ns; do dig -t soa +noall +ans +nocl +nottl irtf.org @$ns; done <<-EOF
>         ns1.ams1.afilias-nst.info.
>         ns1.hkg1.afilias-nst.info.
>         ns1.mia1.afilias-nst.info.
>         ns1.sea1.afilias-nst.info.
>         ns1.yyz1.afilias-nst.info.
>         EOF
>     irtf.org.           SOA     ns0.amsl.com. glen.amsl.com. 1200000226 1800 1800 604800 1800
>     irtf.org.           SOA     ns0.amsl.com. glen.amsl.com. 1200000226 1800 1800 604800 1800
>     irtf.org.           SOA     ns0.amsl.com. glen.amsl.com. 1200000226 1800 1800 604800 1800
>     irtf.org.           SOA     ns0.amsl.com. glen.amsl.com. 1200000226 1800 1800 604800 1800
>     irtf.org.           SOA     ns0.amsl.com. glen.amsl.com. 1200000226 1800 1800 604800 1800
>
> So perhaps the master zone resigning is no longer updating the SOA
> record.  In any case, DNS resolution for irtf.org is mostly down.
>
> --
>         Viktor.
>

---

From: Glen <[email-masked]>
Date: Sat, 7 Nov 2015 13:29:48 -0800

Dear IETF List -

I apologize for the noise and confusion.  Mr Dukhovni did not address
his email to the IETF list; however, he did send his email with a
manually-configured Reply-to: header set to the list.  I should have
noticed that when replying.

At any rate, DNS for the IRTF is not down, it does appear to be
functioning correctly, and other test sites confirm it.  The site
referenced in Mr. Dukhovni's email, dataviz.net, appeared to have been
caching old results.

I'll be sure I check for any "customized" headers before replying to
any future trouble reports I receive.

Thanks,
Glen
Glen Barney
IT Director
AMS (IETF Secretariat)

---

From: Viktor Dukhovni <[email-masked]>
Date: Sat, 7 Nov 2015 21:31:59 +0000

> increment, and pushed them to Afilias.  I will watch to see if that

Looks better now:

I highly recommend automated monitoring of RRSIG lifetimes of at
least the core zone apex records: DNSKEY, NS, SOA and MX across
all the nameservers, master and slaves.  I'm doing it for my own
zones, to avoid any surprises with failure of replication or refresh
of signatures on the master.  The watched kettle has not boiled yet.

> address for things of this type is [email-masked].

I'll squirrel that away.

-- 
	Viktor.

---

From: Viktor Dukhovni <[email-masked]>
Date: Sat, 7 Nov 2015 21:58:39 +0000

> referenced in Mr. Dukhovni's email, dataviz.net, appeared to have been

For the record, the dnsviz.net results were quite fresh, and all
the nameservers except the master were returing "bogus" results
with expired signatures.  With 5 out of 6 nameservers in that state,
most DNS lookups were failing for any validating stub or recursive
nameservers.

Dnsviz links and associated timestamps for the outage are:

    http://dnsviz.net/d/irtf.org/VjpVPA/dnssec/		2015-11-04 18:58:04 UTC
    http://dnsviz.net/d/irtf.org/VjqI6g/dnssec/		2015-11-04 22:38:34 UTC 
    http://dnsviz.net/d/irtf.org/Vjr5Wg/dnssec/		2015-11-05 06:38:18 UTC
    http://dnsviz.net/d/irtf.org/VjtpxA/dnssec/		2015-11-05 14:37:56 UTC
    http://dnsviz.net/d/irtf.org/VjvaWw/dnssec/		2015-11-05 22:38:19 UTC
    http://dnsviz.net/d/irtf.org/VjxKwA/dnssec/		2015-11-06 06:37:52 UTC
    http://dnsviz.net/d/irtf.org/Vjy7Yg/dnssec/		2015-11-06 14:38:26 UTC
    http://dnsviz.net/d/irtf.org/Vj0rxA/dnssec/		2015-11-06 22:37:56 UTC
    http://dnsviz.net/d/irtf.org/Vj2cYQ/dnssec/		2015-11-07 06:38:25 UTC
    http://dnsviz.net/d/irtf.org/Vj4MxQ/dnssec/		2015-11-07 14:37:57 UTC
    http://dnsviz.net/d/irtf.org/Vj5ihg/dnssec/		2015-11-07 20:43:50 UTC

with only the master nameserver showing valid signatures at those times.

After the zone refresh:

    http://dnsviz.net/d/irtf.org/Vj5rsA/dnssec/

the timestamp is "2015-11-07 21:22:56 UTC" with all nameservers
showing valid signatures.

If we look back just before the outage then all is well at:

    http://dnsviz.net/d/irtf.org/VjnfOA/dnssec/		2015-11-04 10:34:32 UTC

then the only hint of trouble is a possibly transient problem
fetching the DNSKEY RRset from the master.

Another 8 hours before that:

    http://dnsviz.net/d/irtf.org/VjlvmA/dnssec/		2015-11-04 02:38:16 UTC

all looks well. Though both then and now a 1 year signature validity
feels a bit too long to me.  And with re-signing so infrequent, it
is difficult to ensure that it works correctly.

-- 
	Viktor.

---

From: Randy Bush <[email-masked]>
Date: Sun, 08 Nov 2015 07:31:57 +0900

> any future trouble reports I receive.

my mta is configured to strip reply-to:  i recommend the practice.

randy

---

From: Glen <[email-masked]>
Date: Sat, 7 Nov 2015 16:20:42 -0800

Randy -

> my mta is configured to strip reply-to:  i recommend the practice.

Thank you for this.

Sage advice, as always.

Best regards,
Glen

---

From: Tony Finch <[email-masked]>
Date: Mon, 9 Nov 2015 10:23:01 +0000

Viktor Dukhovni <[email-masked]> wrote:
> all the nameservers, master and slaves.

Another thing you can do is get the re-signing schedule to match the
rfresh timer. E.g. in BIND the default sig-validity-interval of 30 days
replaces signatures when they have 7.5 days left, which works nicely with
an expiry timer of 1 week. Secondary servers should then expire the zone
before they go bogus.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Northwest Fitzroy, Sole: Southwesterly 5 to 7, increasing gale 8 at times.
Rough or very rough. Rain or drizzle at times. Moderate or poor, occasionally
good.