Subject: IRTF DNSSEC

From: Glen <[email-masked]>
Date: Sat, 10 Nov 2018 11:30:30 -0800

All -

I apologize for not properly quoting threads, I am traveling to
vacation today, and am in haste.

It's been brought to my attention that DNSSEC for IRTF isn't working
quite as well as it should.

I have generated new signatures for all zones, and manually signaled
all of the Afilias servers (who provide backup DNS for the IETF).   I
have observed that those servers have pulled zone transfers from us,
and would expect that this will update and resolve DNSSEC issues
within the next 10-15 minutes (it usually takes about that long to get
through all of Afilias' networks.)

I will check again at my next stop to ensure that we have resolution.

Thank you,
Glen
--
Glen Barney
IT Director
AMS (IETF Secretariat)

---

From: Viktor Dukhovni <[email-masked]>
Date: Sat, 10 Nov 2018 17:08:29 -0500

> vacation today, and am in haste.

Thanks for making the time to resolve this.

> I will check again at my next stop to ensure that we have resolution.

It looks much better now, and my resolver is seeing the expected
valid answers from all servers, however DNSViz is reporting some
packet drops with queries to the primary "ns0.amsl.com":

    http://dnsviz.net/d/irtf.org/dnssec/

    $ host ns0.amsl.com
    ns0.amsl.com has address 4.31.198.40
    ns0.amsl.com has IPv6 address 2001:1900:3001:11::28

When I manually query the primary with "dig", all looks well, but
when I run my copy of the "dnsviz" CLI, I also see the same packet
drops.  Looking at PCAP files with "tshark", it appears that some
sort of stringent rate-limiter may be in place, since queries
identical to the ones sent by "dig" go unanswered.

The main difference seems to be that "dnsviz" send multiple queries
in quick succession, and then waits for multiple answers.  It then
retransmits the outstanding queries eventually with various changes
to the EDNS buffer size, EDNS DO flag, ...

Perhaps the packet drops are intentional, but if not, they may be
something to investigate.

For the IPv4 address, for a total of 60 queries sent, only 5 answers
came back.  Four of the five responses were delayed by more than a
full second.

See below for a trimmed to bare essentials "tshark" decode of the
traffic.

-- 
	Viktor.

User Datagram Protocol, Src Port: 25264, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0x3392
    Queries
        irtf.org: type NS, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 48720, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0x197c
    Queries
        irtf.org: type A, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 25264, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0x3392
        [Expert Info (Warning/Protocol): DNS query retransmission. Original request in frame 3]
    Queries
        irtf.org: type NS, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 48720, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0x197c
        [Expert Info (Warning/Protocol): DNS query retransmission. Original request in frame 4]
    Queries
        irtf.org: type A, class IN
    Additional records
        <Root>: type OPT
	    UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 53, Dst Port: 25264
Domain Name System (response)
    Transaction ID: 0x3392
    Flags: 0x8400 Standard query response, No error
    Answer RRs: 7
    Authority RRs: 0
    Additional RRs: 3
    Queries
        irtf.org: type NS, class IN
    Answers
        irtf.org: type NS, class IN, ns ns0.amsl.com
        irtf.org: type NS, class IN, ns ns1.ams1.afilias-nst.info
        irtf.org: type NS, class IN, ns ns1.hkg1.afilias-nst.info
        irtf.org: type NS, class IN, ns ns1.sea1.afilias-nst.info
        irtf.org: type NS, class IN, ns ns1.yyz1.afilias-nst.info
        irtf.org: type NS, class IN, ns ns1.mia1.afilias-nst.info
        irtf.org: type RRSIG, class IN
            Name: irtf.org
            Type Covered: NS (authoritative Name Server) (2)
            Algorithm: RSA/SHA1 (5)
            Labels: 2
            Signature Expiration: Nov 10, 2019 14:21:42.000000000 EST
            Signature Inception: Nov 10, 2018 13:22:25.000000000 EST
            Key Tag: 46380
            Signer's name: irtf.org
    Additional records
        ns0.amsl.com: type A, class IN, addr 4.31.198.40
        ns0.amsl.com: type AAAA, class IN, addr 2001:1900:3001:11::28
        <Root>: type OPT
            UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs
    [Request In: 3]
    [Time: 1.072754000 seconds]

User Datagram Protocol, Src Port: 48720, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0x197c
        [Expert Info (Warning/Protocol): DNS query retransmission. Original request in frame 4]
    Queries
        irtf.org: type A, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 53, Dst Port: 48720
Domain Name System (response)
    Transaction ID: 0x197c
    Flags: 0x8400 Standard query response, No error
    Answer RRs: 2
    Authority RRs: 7
    Additional RRs: 3
    Queries
        irtf.org: type A, class IN
            Name: irtf.org
    Answers
        irtf.org: type A, class IN, addr 4.31.198.44
        irtf.org: type RRSIG, class IN
            Name: irtf.org
            Type Covered: A (Host Address) (1)
            Algorithm: RSA/SHA1 (5)
            Labels: 2
            Signature Expiration: Nov 10, 2019 14:21:55.000000000 EST
            Signature Inception: Nov 10, 2018 13:22:25.000000000 EST
            Key Tag: 46380
            Signer's name: irtf.org
    Authoritative nameservers
        irtf.org: type NS, class IN, ns ns0.amsl.com
        irtf.org: type NS, class IN, ns ns1.yyz1.afilias-nst.info
        irtf.org: type NS, class IN, ns ns1.ams1.afilias-nst.info
        irtf.org: type NS, class IN, ns ns1.sea1.afilias-nst.info
        irtf.org: type NS, class IN, ns ns1.hkg1.afilias-nst.info
        irtf.org: type NS, class IN, ns ns1.mia1.afilias-nst.info
        irtf.org: type RRSIG, class IN
            Name: irtf.org
            Type Covered: NS (authoritative Name Server) (2)
            Algorithm: RSA/SHA1 (5)
            Labels: 2
            Signature Expiration: Nov 10, 2019 14:21:42.000000000 EST
            Signature Inception: Nov 10, 2018 13:22:25.000000000 EST
            Key Tag: 46380
            Signer's name: irtf.org
    Additional records
        ns0.amsl.com: type A, class IN, addr 4.31.198.40
        ns0.amsl.com: type AAAA, class IN, addr 2001:1900:3001:11::28
        <Root>: type OPT
            UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs
    [Request In: 4]
    [Time: 2.125640000 seconds]

User Datagram Protocol, Src Port: 16337, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xe5a9
    Queries
        irtf.org: type MX, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 59476, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0x389a
    Queries
        phfauoewzk.irtf.org: type A, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 8626, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xc8e2
    Queries
        irtf.org: type TXT, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 10371, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xd05b
    Queries
        irtf.org: type CNAME, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 19796, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xa093
    Queries
        irtf.org: type SOA, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 42679, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0x7ee6
    Queries
        irtf.org: type MX, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 512
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 64611, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xdb59
    Queries
        irtf.org: type AAAA, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 35834, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xa44c
    Queries
        irtf.org: type DNSKEY, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 63520, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0x9809
    Questions: 1
    Queries
        irtf.org: type DNSKEY, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 512
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 16337, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xe5a9
        [Expert Info (Warning/Protocol): DNS query retransmission. Original request in frame 10]
    Queries
        irtf.org: type MX, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 59476, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0x389a
        [Expert Info (Warning/Protocol): DNS query retransmission. Original request in frame 11]
    Queries
        phfauoewzk.irtf.org: type A, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 8626, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xc8e2
        [Expert Info (Warning/Protocol): DNS query retransmission. Original request in frame 12]
    Queries
        irtf.org: type TXT, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 10371, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xd05b
        [Expert Info (Warning/Protocol): DNS query retransmission. Original request in frame 13]
    Queries
        irtf.org: type CNAME, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 19796, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xa093
        [Expert Info (Warning/Protocol): DNS query retransmission. Original request in frame 14]
    Queries
        irtf.org: type SOA, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 42679, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0x7ee6
        [Expert Info (Warning/Protocol): DNS query retransmission. Original request in frame 15]
    Queries
        irtf.org: type MX, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 512
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 64611, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xdb59
        [Expert Info (Warning/Protocol): DNS query retransmission. Original request in frame 16]
    Queries
        irtf.org: type AAAA, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 35834, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xa44c
        [Expert Info (Warning/Protocol): DNS query retransmission. Original request in frame 17]
    Queries
        irtf.org: type DNSKEY, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 63520, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0x9809
        [Expert Info (Warning/Protocol): DNS query retransmission. Original request in frame 18]
    Queries
        irtf.org: type DNSKEY, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 512
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 53, Dst Port: 59476
Domain Name System (response)
    Transaction ID: 0x389a
    Flags: 0x8403 Standard query response, No such name
    Answer RRs: 0
    Authority RRs: 4
    Additional RRs: 1
    Queries
        phfauoewzk.irtf.org: type A, class IN
    Authoritative nameservers
        irtf.org: type SOA, class IN, mname ns0.amsl.com
        irtf.org: type RRSIG, class IN
            Type Covered: SOA (Start Of a zone of Authority) (6)
            Algorithm: RSA/SHA1 (5)
            Labels: 2
            Signature Expiration: Nov 10, 2019 14:21:25.000000000 EST
            Signature Inception: Nov 10, 2018 13:22:25.000000000 EST
            Key Tag: 46380
            Signer's name: irtf.org
        irtf.org: type NSEC, class IN, next domain name www.irtf.org
            RR type in bit map: A (Host Address)
            RR type in bit map: NS (authoritative Name Server)
            RR type in bit map: SOA (Start Of a zone of Authority)
            RR type in bit map: MX (Mail eXchange)
            RR type in bit map: TXT (Text strings)
            RR type in bit map: AAAA (IPv6 Address)
            RR type in bit map: RRSIG
            RR type in bit map: NSEC
            RR type in bit map: DNSKEY
            RR type in bit map: SPF
        irtf.org: type RRSIG, class IN
            Type Covered: NSEC (47)
            Algorithm: RSA/SHA1 (5)
            Labels: 2
            Signature Expiration: Nov 10, 2019 14:19:52.000000000 EST
            Signature Inception: Nov 10, 2018 13:22:25.000000000 EST
            Key Tag: 46380
            Signer's name: irtf.org
    Additional records
        <Root>: type OPT
            UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs
    [Request In: 11]
    [Time: 1.073841000 seconds]

User Datagram Protocol, Src Port: 16337, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xe5a9
        [Expert Info (Warning/Protocol): DNS query retransmission. Original request in frame 10]
    Queries
        irtf.org: type MX, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 8626, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xc8e2
        [Expert Info (Warning/Protocol): DNS query retransmission. Original request in frame 12]
    Queries
        irtf.org: type TXT, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 10371, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xd05b
        [Expert Info (Warning/Protocol): DNS query retransmission. Original request in frame 13]
    Queries
        irtf.org: type CNAME, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 19796, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xa093
        [Expert Info (Warning/Protocol): DNS query retransmission. Original request in frame 14]
    Queries
        irtf.org: type SOA, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 42679, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0x7ee6
        [Expert Info (Warning/Protocol): DNS query retransmission. Original request in frame 15]
    Queries
        irtf.org: type MX, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 512
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 64611, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xdb59
        [Expert Info (Warning/Protocol): DNS query retransmission. Original request in frame 16]
    Queries
        irtf.org: type AAAA, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 35834, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xa44c
        [Expert Info (Warning/Protocol): DNS query retransmission. Original request in frame 17]
    Queries
        irtf.org: type DNSKEY, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 63520, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0x9809
        [Expert Info (Warning/Protocol): DNS query retransmission. Original request in frame 18]
    Queries
        irtf.org: type DNSKEY, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 512
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 16337, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xe5a9
        [Expert Info (Warning/Protocol): DNS query retransmission. Original request in frame 10]
    Queries
        irtf.org: type MX, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 8626, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xc8e2
        [Expert Info (Warning/Protocol): DNS query retransmission. Original request in frame 12]
    Queries
        irtf.org: type TXT, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 10371, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xd05b
        [Expert Info (Warning/Protocol): DNS query retransmission. Original request in frame 13]
    Queries
        irtf.org: type CNAME, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 19796, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xa093
        [Expert Info (Warning/Protocol): DNS query retransmission. Original request in frame 14]
    Queries
        irtf.org: type SOA, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 42679, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0x7ee6
        [Expert Info (Warning/Protocol): DNS query retransmission. Original request in frame 15]
    Queries
        irtf.org: type MX, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 512
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 64611, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xdb59
        [Expert Info (Warning/Protocol): DNS query retransmission. Original request in frame 16]
    Queries
        irtf.org: type AAAA, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 35834, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xa44c
        [Expert Info (Warning/Protocol): DNS query retransmission. Original request in frame 17]
    Queries
        irtf.org: type DNSKEY, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 63520, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0x9809
        [Expert Info (Warning/Protocol): DNS query retransmission. Original request in frame 18]
    Queries
        irtf.org: type DNSKEY, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 512
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 10072, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xe5a9
    Queries
        irtf.org: type MX, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 512
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 39292, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xc8e2
    Queries
        irtf.org: type TXT, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 512
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 57345, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xd05b
    Queries
        irtf.org: type CNAME, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 512
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 12474, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xa093
    Queries
        irtf.org: type SOA, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 512
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 46232, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xdb59
    Queries
        irtf.org: type AAAA, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 512
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 18594, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xa44c
    Queries
        irtf.org: type DNSKEY, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 512
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 10072, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xe5a9
        [Expert Info (Warning/Protocol): DNS query retransmission. Original request in frame 45]
    Queries
        irtf.org: type MX, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 512
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 39292, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xc8e2
        [Expert Info (Warning/Protocol): DNS query retransmission. Original request in frame 46]
    Queries
        irtf.org: type TXT, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 512
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 57345, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xd05b
        [Expert Info (Warning/Protocol): DNS query retransmission. Original request in frame 47]
    Queries
        irtf.org: type CNAME, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 512
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 12474, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xa093
        [Expert Info (Warning/Protocol): DNS query retransmission. Original request in frame 48]
    Queries
        irtf.org: type SOA, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 512
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 46232, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xdb59
        [Expert Info (Warning/Protocol): DNS query retransmission. Original request in frame 49]
    Queries
        irtf.org: type AAAA, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 512
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 18594, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xa44c
        [Expert Info (Warning/Protocol): DNS query retransmission. Original request in frame 50]
    Queries
        irtf.org: type DNSKEY, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 512
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs

User Datagram Protocol, Src Port: 53, Dst Port: 39292
Domain Name System (response)
    Transaction ID: 0xc8e2
    Flags: 0x8600 Standard query response, No error
        .... ..1. .... .... = Truncated: Message is truncated
    Answer RRs: 0
    Authority RRs: 0
    Additional RRs: 1
    Queries
        irtf.org: type TXT, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 4096
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs
    [Request In: 46]
    [Time: 1.098694000 seconds]

User Datagram Protocol, Src Port: 35541, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xe5a9
    Queries
        irtf.org: type MX, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 512
                0... .... .... .... = DO bit: Cannot handle DNSSEC security RRs

User Datagram Protocol, Src Port: 43840, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xd05b
    Queries
        irtf.org: type CNAME, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 512
                0... .... .... .... = DO bit: Cannot handle DNSSEC security RRs

User Datagram Protocol, Src Port: 37575, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xa093
    Queries
        irtf.org: type SOA, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 512
                0... .... .... .... = DO bit: Cannot handle DNSSEC security RRs

User Datagram Protocol, Src Port: 26621, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xdb59
    Queries
        irtf.org: type AAAA, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 512
                0... .... .... .... = DO bit: Cannot handle DNSSEC security RRs

User Datagram Protocol, Src Port: 39145, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xa44c
    Queries
        irtf.org: type DNSKEY, class IN
    Additional records
        <Root>: type OPT
            UDP payload size: 512
                0... .... .... .... = DO bit: Cannot handle DNSSEC security RRs

User Datagram Protocol, Src Port: 53, Dst Port: 43840
Domain Name System (response)
    Transaction ID: 0xd05b
    Flags: 0x8400 Standard query response, No error
    Answer RRs: 0
    Authority RRs: 1
    Additional RRs: 1
    Queries
        irtf.org: type CNAME, class IN
    Authoritative nameservers
        irtf.org: type SOA, class IN, mname ns0.amsl.com
    Additional records
        <Root>: type OPT
            UDP payload size: 4096
                0... .... .... .... = DO bit: Cannot handle DNSSEC security RRs
    [Request In: 59]
    [Time: 0.066312000 seconds]

User Datagram Protocol, Src Port: 47146, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xe5a9
    Queries
        irtf.org: type MX, class IN

User Datagram Protocol, Src Port: 60574, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xa093
    Queries
        irtf.org: type SOA, class IN

User Datagram Protocol, Src Port: 57051, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xdb59
    Queries
        irtf.org: type AAAA, class IN

User Datagram Protocol, Src Port: 51020, Dst Port: 53
Domain Name System (query)
    Transaction ID: 0xa44c
    Queries
        irtf.org: type DNSKEY, class IN

---

From: Glen <[email-masked]>
Date: Sat, 10 Nov 2018 15:01:57 -0800

> Thanks for making the time to resolve this.

No worries at all.

> ...
> ...

That is correct, we have rate limiting in place on our primary name
server, to help protect against some prior DDOS attacks against it.
(I'm always amazed at how much time other people have to craft and
launch attacks like this.  I *never* have that kind of time.  *sigh*)

Abridging (so to speak) a few other comments I've seen, now that I've
had time to browse the archive...

> Only these two get it right:
> That's ns0.amsl.com (the primary?)

Well, at least we got it right.  Many years ago, the IETF decided that
they wanted to have Afilias provide slave service to our primary -
Afilias continues to do an excellent job at it - but they have made us
aware that we need to change to some newer nameservers in the
registry, which we are trying to do.  Getting control of that process
is taking more time than we'd like, but it is in progress.

> And who is responsible for and gets to fix it?

Ultimately, I am.  I don't have the bandwidth to participate in this
list; sadly, this list is the place where everyone loves to talk about
issues.  I have a staff of engineers - they can be reached at
[email-masked].  But, ultimately, if someone feels like they're
not getting satisfaction, I am the "buck stops here" person.

> Perhaps the folks at AMSL and their ietf.org liaisons can be persuaded to adopt a more reliable operating model

Despite my responsibility to fix things, I lack the commensurate
direct authority to make operational model changes.  The Tools Team,
ultimately, makes those decisions.  However, I have put in place new
instructions to increase the frequency of manual handling, and the
scope of automatic verification, with my engineers.  More than anyone
else, I *certainly* do NOT want things to fail.

> the deliberate unnecessary complexity of the system

+1 to deliberate.  Many IETF systems are complex, this is the result
of many changes by many interested participants over time.  I'm not
sure I can make a judgement about "unnecessary" but many of the IETF's
systems do have many moving parts, and as new technologies and
"exciting toys" come out, those tend to get added to the mix,
increasing complexity further.  At the same time, it is very difficult
to take old things out, so we are left with an ever-increasing level
of complexity (and work and maintenance needed to handle these
things.)

> and the failure to maintain it is interesting...
> This reveals the questionable robustness of the operation and maintenance functions

Since we (AMS) don't control all of the moving parts in this case, I
can't respond to that directly, and I have no desire to get into a
throwing-under-the-bus situation here.  I will simply say that this
system, like others, is complex, it needs some updates, we are working
on getting access to the things we need in order to apply those
updates and, once done, I will hope that it never breaks again.

But, sometimes things do break despite all best efforts.  And when
something does break, I'll do what is needed to fix it again.  That's
why I'm here.

I trust that answers everyone's questions at this stage.  I *hope*
that I will be on vacation for the next two weeks, but there are
people both at AMS and in the Tools Team/community who know how to
reach me if it becomes necessary.

And if anyone wants to talk about "how the IETF does things," I would
invite and encourage you to participate in the Tools Team.  They're
doing a lot of work on a lot of things (including, hopefully, removing
some old things that need to go!) and can use all the support from the
rest of us that they can get.   If you haven't thought about it
before, but would like to help, reach out to Russ, Robert and Henrik,
and they'll be happy to work with you.

Glen
--
Glen Barney
IT Director
AMS (IETF Secretariat)

---

From: Mukund Sivaraman <[email-masked]>
Date: Sun, 11 Nov 2018 11:57:22 +0700

> came back.  Four of the five responses were delayed by more than a

digging around, I see curious behavior.

dig -4 +tcp @ns0.amsl.com consistently reports query time of upwards of
2s from a host in hetzner.de and upwards of 3s from hotel in Bangkok.

dig -4 @ns0.amsl.com (udp) reports query time of ~165ms from a host in
hetzner.de and ~230ms from hotel in Bangkok.

The response size for this query was 285 bytes. The TCP lookups take
much longer (note that this is the dig query time output and IIRC does
not include the handshake and @NS lookup times).

		Mukund

---

From: Viktor Dukhovni <[email-masked]>
Date: Sun, 11 Nov 2018 00:42:46 -0500

> On Nov 10, 2018, at 11:57 PM, Mukund Sivaraman <[email-masked]> wrote:
> 

FWIW, given the apparently rather tight rate limits, if it were up to me, I'd
just operate the primary as a hidden master, with a sole purpose of providing
AXFR service to the slave servers.  It is perhaps better to not list it at all,
than list a server that's going to be "aggressively" dropping packets.

However, the operative word is "apparently"!  It is quite possible that only
"DNSViz" and various synthetic probes run into issues, and that "normal"
resolvers are getting perfectly adequate service.  Since I don't know how
the rate limits are implemented, and what mix of queries the domain gets, ...
the first paragraph could be substantially off-base.

Glen mentioned that there'll be new attention on monitoring, and perhaps
more frequent, and less error-prone re-signing.  Even if manual, I hope
the process involves just a single step of running a robustly designed
and well-tested script that consistently performs all the requisite tasks.

Good luck the ietf-tools and AMSL teams, I hope this outage will help to
improve things going forward.  I'll stop kibitzing now.

-- 
	Viktor.

---

From: Glen <[email-masked]>
Date: Sat, 10 Nov 2018 22:23:01 -0800

> than list a server that's going to be "aggressively" dropping packets.

I agree.  As soon as we get our access issues resolved, I'll remove
NS0 when I update the other servers.

> resolvers are getting perfectly adequate service.

That's the hope.

> and well-tested script that consistently performs all the requisite tasks.

Definitely.

> Good luck the ietf-tools and AMSL teams

Thank you,

Glen
--
Glen Barney
IT Director
AMS (IETF Secretariat)