Subject: [DNSOP] Review of draft-ietf-dnsop-serve-stale-02.txt

From: Mukund Sivaraman <[email-masked]>
Date: Sat, 3 Nov 2018 15:12:28 +0700

I've reviewed older revisions of the draft and still +1 the idea. It
would be useful practically in today's world where temporary DDoS
attacks inundate authorities.

Review comments on this revision of the draft:

>    This document proposes that the definition of the TTL be explicitly
>    expanded to allow for expired data to be used in the exceptional
>    circumstance that a recursive resolver is unable to refresh the
>    information.  It is predicated on the observation that authoritative
>    server unavailability can cause outages even when the underlying data
>    those servers would return is typically unchanged.

While reading this, I wonder whether the last sentence was meant to be
written as: "It is predicated on the observation that zone data returned
by authoritative nameservers during a resolver refresh would typically
be unchanged."

>    We describe a method below for this use of stale data, balancing the
>    competing needs of resiliency and freshness.  While this is intended
>    to be immediately useful to the installed base of DNS software, we
>    also propose an [RFC6891] EDNS option for enhanced signalling around
>    the use of stale data by implementations that understand it.

> 2.  Notes to readers

>    [ RFC Editor, please remove this section before publication!
>    Readers: This is conversational text to describe what we've done, and
>    will be removed, please don't bother sending editorial nits. :-) ]

>    Due to circumstances, the authors of this document got sidetracked,
>    and we lost focus.  We are now reviving it, and are trying to address
>    and incorporate comments.  There has also been more deployment and
>    implementation recently, and so this document is now more describing
>    what is known to work instead of simply proposing a concept.

>    Open questions:

>    o  The EDNS option that we propose for debugging is relatively full-
>       featured to identify which RRsets are stale.  It could be
>       simplified to just indicate that an answer is stale, or even
>       removed entirely in favour of an Extended Error result that
>       indicates staleness.

>    o  What TTL value to recommend be set in stale answers returned by
>       recursive resolvers.

> 3.  Terminology

>    The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
>    "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
>    "OPTIONAL" in this document are to be interpreted as described in
>    [RFC2119] when, and only when, they appear in all capitals, as shown
>    here.

RFC 8174 has updated RFC 2119. Please check this draft vs. what's in RFC
8174 and update the above paragraph.

>    For a comprehensive treatment of DNS terms, please see [RFC7719].

> 4.  Background

>    There are a number of reasons why an authoritative server may become
>    unreachable, including Denial of Service (DoS) attacks, network
>    issues, and so on.  If the recursive server is unable to contact the
>    authoritative servers for a name but still has relevant data that has

s/for a name/for a query/

>    aged past its TTL, that information can still be useful for
>    generating an answer under the metaphorical assumption that, "stale

Remove this comma ---------------------------------------------^^^

>    bread is better than no bread."

>    [RFC1035] Section 3.2.1 says that the TTL "specifies the time
>    interval that the resource record may be cached before the source of
>    the information should again be consulted", and Section 4.1.3 further
>    says the TTL, "specifies the time interval (in seconds) that the
>    resource record may be cached before it should be discarded."

>    A natural English interpretation of these remarks would seem to be
>    clear enough that records past their TTL expiration must not be used,
>    However, [RFC1035] predates the more rigorous terminology of

Either make comma after "used" a full-stop, or lowercase the H in
"However".

>    [RFC2119] which softened the interpretation of "may" and "should".

>    [RFC2181] aimed to provide "the precise definition of the Time to
>    Live" but in Section 8 was mostly concerned with the numeric range of

s/ but in Section 8/, but in Section 8,/

>    values and the possibility that very large values should be capped.
>    (It also has the curious suggestion that a value in the range
>    2147483648 to 4294967295 should be treated as zero.)  It closes that
>    section by noting, "The TTL specifies a maximum time to live, not a
>    mandatory time to live."  This is again not [RFC2119]-normative
>    language, but does convey the natural language connotation that data
>    becomes unusable past TTL expiry.

>    Several major recursive resolver operators currently use stale data
>    for answers in some way, including Akamai (via both Nomimum and
>    Xerocole), Knot, OpenDNS, and Unbound.

You can now add BIND to this list. :)

>    Apple can also use stale data as part of the Happy Eyeballs
>    algorithms in mDNSResponder.  The collective operational experience
>    is that it provides significant benefit with minimal downside.

> 5.  Standards Action

>    The definition of TTL in [RFC1035] Sections 3.2.1 and 4.1.3 is
>    amended to read:

>    TTL  a 32 bit unsigned integer number of seconds in the range 0 -
>       2147483647 that specifies the time interval that the resource
>       record MAY be cached before the source of the information MUST
>       again be consulted.  Zero values are interpreted to mean that the
>       RR can only be used for the transaction in progress, and should

>       not be cached.  Values with the high order bit set SHOULD be
>       capped at no more than 2147483647.  If the authority for the data
>       is unavailable when attempting to refresh the data past the given
>       interval, the record MAY be used as though it is unexpired.

>    [ Discussion point: capping values with the high order bit as being
>    max positive, rather than 0, is a change from [RFC2181].  Also, we
>    could use this opportunity to recommend a much more sane maximum
>    value like 604800 seconds, one week, instead of the literal maximum
>    of 68 years. ]

It is a curious thing why it was decided that values with the high order
bit set are not clamped but set to zero. Possibly because it can be
thought that such high values are bogus and assumed to be made in error,
and so a resolver should attempt to re-query such records instead of
caching them for a very long time. OTOH, one can think the same of a
TTL=2147483647 answer too. :P

> 6.  EDNS Option

>    While the basic behaviour of this answer-of-last-resort can be
>    achieved with changes only to resolvers, explicit signalling about
>    the use of stale data can be done with an EDNS [RFC6891] option.
>    This option can be included from a stub or forwarding resolver to a
>    recursive resolver, explicitly signalling that it does not want stale
>    answers, or for learning that stale data was in use.  It is expected
>    that this could be useful for debugging.

>    [ NOTE: This section will be fleshed out a bit more thoroughly if
>    there is interest in pursuing the option.  Here are two potential
>    options that could be used, one more fully-featured to indicate which
>    RRsets are stale and one much more simple to indicate that stale data
>    is present.  These are proposed as mutually exclusive; the final
>    document will have one or zero such options.  We're especially
>    soliciting feedback on this from the working group. ]

Was an option 3 explored where stale answers are only returned if the
stub/forwarder signals explicitly that stale answers can be returned?
In that case, the default behavior (with option missing in query) would
be RFC 1035 behavior.

> 6.1.  Option Format Proposal 1

>    The option is structured as follows:

>                  +0 (MSB)                        +1 (LSB)
>       +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
>    0: |                         OPTION-CODE                       |
>       +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
>    2: |                        OPTION-LENGTH                      |
>       +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
>    4: |                     STALE-RRSET-INDEX 1                   |
>       +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
>    6: |                                                           |
>    8: |                         TTL-EXPIRY 1                      |
>       +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
>       :  ... additional STALE-RRSET-INDEX / TTL-EXPIRY pairs ...  :
>       +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+

>    OPTION-CODE:  2 octets per [RFC6891].  For Serve-Stale the code is
>       TBD by IANA.

>    OPTION-LENGTH:  2 octets per [RFC6891].  Contains the length of the
>       payload following OPTION-LENGTH, in octets.

>    STALE-RRSET-INDEX:  Two octets as a signed integer, indicating the
>       first RRSet in the message which is beyond its TTL, with RRSet
>       counting starting at 1 and spanning message sections.

>    TTL-EXPIRY:  Four octets as an unsigned integer, representing the
>       number of seconds that have passed since the TTL for the RRset
>       expired.

> 6.2.  Option Usage

>    Software making a DNS request can signal that it understands Serve-
>    Stale by including the option with one STALE-RRSET-INDEX initialized
>    to any negative value and TTY-EXPIRY initialized to 0.  The index is
>    set to a negative value to detect mere reflection of the option by
>    responders that don't really understand it.

>    If the request is made to a recursive resolver which used any stale
>    RRsets in its reply, it then fills in the corresponding indices and
>    staleness values.  If no records are stale, STALE-RRSET-INDEX and
>    TTL-EXPIRY are set to 0.

>    If the request is made to an authoritative nameserver, it can use the
>    option in the reply to indicate how the resolver should treat the
>    records in the reply if they are unable to be refreshed later.  A
>    default for all RRsets in the message is established by setting the
>    first STALE-RRSET-INDEX to 0, with optional additional STALE-RRSET-
>    INDEX values overriding the default.  A TTL-EXPIRY value of 0 means
>    to never serve the RRset as stale, while non-zero values represent
>    the maximum amount of time it can be used before it MUST be evicted.
>    [ Does anyone really want to do this?  It adds more state into
>    resolvers.  Is the idea only for purists, or is there a practical
>    application? ]

>    No facility is made for a client of a resolver to signal that it
>    doesn't want stale answers, because if a client has knowledge of
>    Serve-Stale as an option, it also has enough knowledge to just ignore
>    any records that come back stale.  [ There is admittedly the issue
>    that the client might just want to wait out the whole attempted
>    resolution, which there's currently no way to indicate.  The absolute
>    value of STALE-RRSET-INDEX could be taken as a timer the requester is
>    willing to wait for an answer, but that's kind of gross overloading
>    it like that.  Shame to burn another field on that, but on the other

>    hand it might also be nice if a client could always signal its
>    impatience level - "I must have an answer within 900 milliseconds!" ]

This option seems to me too complicated to generate, parse and make use
of. RRs are re-ordered very late during message rendering in some DNS
implementations, and updating this syntax in the EDNS option just looks
too painful. It does not appear parsing (by resolvers) will be easy
either, and whether this fine granularity in determining staleness is
generally useful.

> 6.3.  Option Format Proposal 2

>    The option is structured as follows:

>                 +0 (MSB)                        +1 (LSB)
>       +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
>    0: |                         OPTION-CODE                       |
>       +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
>    2: |                        OPTION-LENGTH                      |
>       +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
>    4: | D | U | S |             RESERVED                          |
>       +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+

>    OPTION-CODE:  2 octets per [RFC6891].  For Serve-Stale the code is
>       TBD by IANA.

>    OPTION-LENGTH:  2 octets per [RFC6891].  Contains the length of the
>       payload following OPTION-LENGTH, in octets.

>    D Flag:  If set, the client explicitly does NOT want stale answers.
>       If clear, the client would like an indication of whether any data
>       in the response is stale.

The D flag seems unnecessary. Just the presence of the EDNS option in
query from the client should serve to indicate to a server that the
client explicitly does not want stale answers.

>    U Flag:  This indicates that the server understands the Serve-Stale
>       EDNS option, and more information is communicated via the S flag.
>       It exists to get around the issue of some authorative servers
>       simply echoing back ENDS options it does not understand.

What should it be set to by the client?

>    S Flag:  If set, this indicates that the response contains stale
>       data.  If clear, no data in the response has reached its TTL
>       expiry.

What should it be set to by the client?

>    RESERVED:  Reserved for future use.  Should be set to zero on send
>       and ignored on receipt.

> 7.  Example Method

>    There is conceivably more than one way a recursive resolver could
>    responsibly implement this resiliency feature while still respecting
>    the intent of the TTL as a signal for when data is to be refreshed.

>    In this example method three notable timers drive considerations for
>    the use of stale data, as follows:

>    o  A client response timer, which is the maximum amount of time a
>       recursive resolver should allow between the receipt of a
>       resolution request and sending its response.

>    o  A query resolution timer, which caps the total amount of time a
>       recursive resolver spends processing the query.

>    o  A maximum stale timer, which caps the amount of time that records
>       will be kept past their expiration.

>    Recursive resolvers already have the second timer; the first and
>    third timers are new concepts for this mechanism.

>    When a request is received by the recursive resolver, it SHOULD start
>    the client response timer.  This timer is used to avoid client
>    timeouts.  It SHOULD be configurable, with a recommended value of 1.8
>    seconds.

FWIW, to other reviewers, applications on Linux (including e.g.,
Firefox) use the stub resolver and that times out after waiting for 5
seconds. dig also has a default per-query timeout of 5 seconds (though
it retries the query up to 2 more times if it didn't receive a response,
but each capped to 5 seconds).

In a nutshell, anything above or equal to 5 seconds for the second timer
(which is inclusive of the first timer) will cause lookup failures.

>    The resolver then checks its cache for an unexpired answer.  If it
>    finds none and the Recursion Desired flag is not set in the request,
>    it SHOULD immediately return the response without consulting the
>    cache for expired records.

>    If iterative lookups will be done, it SHOULD start the query
>    resolution timer.  This timer bounds the work done by the resolver,
>    and is commonly around 10 to 30 seconds.

See above for practical timeout issues.

>    If the answer has not been completely determined by the time the
>    client response timer has elapsed, the resolver SHOULD then check its
>    cache to see whether there is expired data that would satisfy the
>    request.  If so, it adds that data to the response message and SHOULD
>    set the TTL of each expired record in the message to 30 seconds.  The
>    response is then sent to the client while the resolver continues its
>    attempt to refresh the data.  30 second was chosen because
>    historically 0 second TTLs have been problematic for some
>    implementations, and similarly very short TTLs could lead to
>    congestive collapse as TTL-respecting clients rapidly try to refresh.
>    30 seconds not only sidesteps those potential problems with no
>    practical negative consequence, it would also rate limit further
>    queries from any client that is honoring the TTL, such as a
>    forwarding resolver. [ NOTE: we're looking for further working group
>    feedback on this value. ]

Would it be better to limit fetches by the resolver for 30 seconds,
while still returning TTL=0 answers? It is expected that a
stub/forwarder knows how to handle TTL=0 answers today. A stub would not
cache it and query for the same if required again, but isn't that
desired for a stale answer, if it doesn't create any further fetches on
the resolver side? Can the resolver return a TTL=0 answer if the client
response timeout is reached, and further not repeat that upstream fetch
for another 30 seconds if it receives such a query from a client?

>    The maximum stale timer is used for cache management and is
>    independent of the query resolution process.  This timer is
>    conceptually different from the maximum cache TTL that exists in many
>    resolvers, the latter being a clamp on the value of TTLs as received
>    from authoritative servers.  The maximum stale timer SHOULD be

>    configurable, and defines the length of time after a record expires
>    that it SHOULD be retained in the cache.  The suggested value is 7
>    days, which gives time to notice the resolution problem and for human
>    intervention to fix it.

>    This same basic technique MAY be used to handle stale data associated
>    with delegations.  If authoritative server addresses are not able to
>    be refreshed, resolution can possibly still be successful if the
>    authoritative servers themselves are still up.

> 8.  Implementation Caveats

>    Answers from authoritative servers that have a DNS Response Code of
>    either 0 (NOERROR) or 3 (NXDOMAIN) MUST be considered to have
>    refreshed the data at the resolver.  In particular, this means that
>    this method is not meant to protect against operator error at the
>    authoritative server that turns a name that is intended to be valid
>    into one that is non-existent, because there is no way for a resolver
>    to know intent.

>    Resolution is given a chance to succeed before stale data is used to
>    adhere to the original intent of the design of the DNS.  This
>    mechanism is only intended to add robustness to failures, and to be
>    enabled all the time.  If stale data were used immediately and then a
>    cache refresh attempted after the client response has been sent, the
>    resolver would frequently be sending data that it would have had no
>    trouble refreshing.  As modern resolvers use techniques like pre-
>    fetching and request coalescing for efficiency, it is not necessary
>    that every client request needs to trigger a new lookup flow in the
>    presence of stale data, but rather than a good-faith effort have been
>    recently made to refresh the stale data before it is delivered to any
>    client.  The recommended period between attempting refreshes is 30
>    seconds.

Do all implementations mentioned earlier supporting the idea of this
draft attempt to refresh stale data before serving it? Does this draft
prescribe if resolvers SHOULD/MUST do so? Because the two approaches
result in quite different behaviors.

>    It is important to continue the resolution attempt after the stale
>    response has been sent, until the query resolution timeout, because
>    some pathological resolutions can take many seconds to succeed as
>    they cope with unavailable servers, bad networks, and other problems.
>    Stopping the resolution attempt when the response with expired data
>    has been sent would mean that answers in these pathological cases
>    would never be refreshed.

I think some implementations of this draft do not implement the client
response timer, and so waiting for the query resolution timer (which may
be a large duration) may result in application getaddrinfo() timeouts.

>    Canonical Name (CNAME) records mingled in the expired cache with
>    other records at the same owner name can cause surprising results.
>    This was observed with an initial implementation in BIND when a
>    hostname changed from having an IPv4 Address (A) record to a CNAME.
>    The version of BIND being used did not evict other types in the cache
>    when a CNAME was received, which in normal operations is not a

>    significant issue.  However, after both records expired and the
>    authorities became unavailable, the fallback to stale answers
>    returned the older A instead of the newer CNAME.

>    [ This probably applies to other occluding types, so more thought
>    should be given to the overall issue. ]

>    Keeping records around after their normal expiration will of course
>    cause caches to grow larger than if records were removed at their
>    TTL.  Specific guidance on managing cache sizes is outside the scope
>    of this document.  Some areas for consideration include whether to
>    track the popularity of names in client requests versus evicting by
>    maximum age, and whether to provide a feature for manually flushing
>    only stale records.

> 9.  Implementation Status

>    [RFC Editor: per RFC 6982 this section should be removed prior to
>    publication.]

>    The algorithm described in the Section 7 section was originally
>    implemented as a patch to BIND 9.7.0.  It has been in production on
>    Akamai's production network since 2011, and effectively smoothed over
>    transient failures and longer outages that would have resulted in
>    major incidents.  The patch was contributed to Internet Systems
>    Consortium and the functionality is now available in BIND 9.12 via
>    the options stale-answer-enable, stale-answer-ttl, and max-stale-ttl.

I'm not sure BIND 9.12 implements exactly the behavior described in
Section 7. As far as I last know, it does not.

>    Unbound has a similar feature for serving stale answers, but it works
>    in a very different way by returning whatever cached answer it has
>    before trying to refresh expired records.  This is unfortunately not
>    faithful to the ideal that data past expiry should attempt to be
>    refreshed before being served.

>    Knot Resolver has an demo module here: https://knot-
>    resolver.readthedocs.io/en/stable/modules.html#serve-stale

>    Details of Apple's implementation are not currently known.

>    In the research paper "When the Dike Breaks: Dissecting DNS Defenses
>    During DDoS" [DikeBreaks], the authors detected some use of stale
>    answers by resolvers when authorities came under attack.  Their
>    research results suggest that more widespread adoption of the
>    technique would significantly improve resiliency for the large number
>    of requests that fail or experience abnormally long resolution times
>    during an attack.

> 10.  Security Considerations

>    The most obvious security issue is the increased likelihood of DNSSEC
>    validation failures when using stale data because signatures could be
>    returned outside their validity period.  This would only be an issue
>    if the authoritative servers are unreachable, the only time the
>    techniques in this document are used, and thus does not introduce a
>    new failure in place of what would have otherwise been success.

>    Additionally, bad actors have been known to use DNS caches to keep
>    records alive even after their authorities have gone away.  This
>    potentially makes that easier, although without introducing a new
>    risk.

> 11.  Privacy Considerations

>    This document does not add any practical new privacy issues.

> 12.  NAT Considerations

>    The method described here is not affected by the use of NAT devices.

> 13.  IANA Considerations

>    This document contains no actions for IANA.  This section will be
>    removed during conversion into an RFC by the RFC editor.

It appears that a new EDNS option is being proposed, so this section
will have to be updated.

> 14.  Acknowledgements

>    The authors wish to thank Matti Klock, Mukund Sivaraman, Jean Roy,
>    and Jason Moreau for initial review.  Feedback from Robert Edmonds,
>    Giovane Moura, Davey Song, and Ralf Weber has also been incorporated.

> 15.  References

> 15.1.  Normative References

>    [RFC1034]  Mockapetris, P., "Domain names - concepts and facilities",
>               STD 13, RFC 1034, DOI 10.17487/RFC1034, November 1987,
>               <https://www.rfc-editor.org/info/rfc1034>.

>    [RFC1035]  Mockapetris, P., "Domain names - implementation and
>               specification", STD 13, RFC 1035, DOI 10.17487/RFC1035,
>               November 1987, <https://www.rfc-editor.org/info/rfc1035>.

> Lawrence, et al.         Expires April 17, 2019                [Page 11]
> 
> Internet-Draft               DNS Serve Stale                October 2018

>    [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
>               Requirement Levels", BCP 14, RFC 2119,
>               DOI 10.17487/RFC2119, March 1997,
>               <https://www.rfc-editor.org/info/rfc2119>.

>    [RFC2181]  Elz, R. and R. Bush, "Clarifications to the DNS
>               Specification", RFC 2181, DOI 10.17487/RFC2181, July 1997,
>               <https://www.rfc-editor.org/info/rfc2181>.

>    [RFC6891]  Damas, J., Graff, M., and P. Vixie, "Extension Mechanisms
>               for DNS (EDNS(0))", STD 75, RFC 6891,
>               DOI 10.17487/RFC6891, April 2013,
>               <https://www.rfc-editor.org/info/rfc6891>.

> 15.2.  Informative References

>    [DikeBreaks]
>               Moura, G., Heidemann, J., Mueller, M., Schmidt, R., and M.
>               Davids, "When the Dike Breaks: Dissecting DNS Defenses
>               During DDos", ACM 2018 Internet Measurement Conference,
>               DOI 10.1145/3278532.3278534, October 2018,
>               <https://www.isi.edu/~johnh/PAPERS/Moura18b.pdf>.

>    [RFC7719]  Hoffman, P., Sullivan, A., and K. Fujiwara, "DNS
>               Terminology", RFC 7719, DOI 10.17487/RFC7719, December
>               2015, <https://www.rfc-editor.org/info/rfc7719>.

> Authors' Addresses

>    David C Lawrence
>    Oracle + Dyn

>    Email: [email-masked]

>    Warren Kumari
>    Google
>    1600 Amphitheatre Parkway
>    Mountain View  CA 94043
>    USA

>    Email: [email-masked]

>    Puneet Sood
>    Google

>    Email: [email-masked]

> Lawrence, et al.         Expires April 17, 2019                [Page 12]

		Mukund

---

From: Mukund Sivaraman <[email-masked]>
Date: Sat, 3 Nov 2018 15:18:15 +0700

> client explicitly does not want stale answers.

I withdraw this comment. It appears that the client can also find out
whether an option is stale by sending the option.

		Mukund

---

From: Dave Lawrence <[email-masked]>
Date: Mon, 5 Nov 2018 01:37:25 -0500

Thanks very much for the review, Mukund!  Puneet has already
incorporated the editorial feedback into the GitHub copy.

Mukund Sivaraman writes:
>>  "It is predicated on the observation that authoritative server
>>   unavailability can cause outages even when the underlying data 
>>   those servers would return is typically unchanged."
> 

I agree that it's a reasonable rephrasing, and I'm wondering if you
see that there's a practical difference.  Not that I'm particularly
wed to the original text, just wondering if I'm missing something
either in grammar or semantics that makes the rewrite superior.

> s/for a name/for a query/

Puneet made this change, but I'll observe that since names are how
authoritative servers get delegated, I think "servers for a name" is an
acceptable, natural way to refer to the process.

> It is a curious thing why it was decided that [TTL] values with the
> high order bit set are not clamped but set to zero. Possibly because
> it can be thought that such high values are bogus and assumed to be
> made in error, and so a resolver should attempt to re-query such
> records instead of caching them for a very long time. OTOH, one can
> think the same of a TTL=2147483647 answer too. :P

Yeah, I don't know the reasoning and haven't searched the dnsext
archives to see if there was discussion on it back in the mid-90s when
the treat-it-as-zero clarification was decided.  Neither zero not 2^31
seem like good ideas really, and the issue will be in the presentation
to dnsop today.

> either, and whether this fine granularity in determining staleness is

I do agree that it is somewhat more complicated, but I'm not sure
about "too complicated".  My thinking when I first offered it is that
if I'm using an option for diagnostic purposes, I want explicit
information returned.  So for something like "dig +any +edns-stale
example.com" (or whatever) when I'm debugging, I can count off the
indices well enough.

I would not expect automated systems that receive the option to really
care much about specifically what RRSets were stale, if they were
concerned about staleness at all.  And if they do care ... they can
count off the indices well enough too.

Here's another fun idea, to specifically identify the relevant parts
of the message:  name compression pointers!  Okay, okay ... yeah it
makes me feel a little skeevy too.  

On the other hand you could also iterate over each name/type with
either recursion disabled or the other EDNS option, so there's that
way of dealing with diagnostics too.

> while still returning TTL=0 answers?

It's an interesting though, but besides my general wariness of TTL 0
records I do note that this would mean keeping more state in the
resolver than it currently has to keep, and when I think about how I'd
implement that in BIND at least there's a fair bit of complexity there.

> result in quite different behaviors.

To the best of my knowledge, no, though hopefully I've missed news on
an update to Unbound.  My understanding about how Unbound does it's
feature is that it's basically "shoot first and ask questions later".
That is, I think it'll use any data from the cache first, stale or
not, before trying the resolution.   That's part of the reason that I
wrote explicitly in the draft about honoring the intent of the TTL and
using stale data only in unusual circumstances.

> be a large duration) may result in application getaddrinfo() timeouts.

That would circumvent pretty much the whole intent to add resiliency
for clients waiting for answers.

---

From: Tony Finch <[email-masked]>
Date: Mon, 5 Nov 2018 18:45:25 +0000

A few notes following the presentation and discussion earlier today
(unrelated to Mukund's comments - I'm just stealing a suitable thread)

Re. the EDNS options, if you go for a 1 bit version it should apply only
to the answer section. The only time this will be ambiguous is when there
are CNAME/DNAME chains present.

I was rather disconcerted by the 1 week default serve-stale limit in
BIND's implementation. It seems to me that the value should be tuned to
match typical outage lengths. A day seems to me to be much more reasonable
than a week, though for my servers I have chosen an hour.

Part of the reason I like serve-stale is that I think it will make outages
easier to triage for my IT support colleagues. Network connectivity
problems often look like DNS problems to even fairly knowledgable people.
If the DNS continues to provide answers when the network is a bit broken
then the investigation is more likely to head in the right direction
sooner. (My logic for choosing an hour is that if things are broken for
longer than that then it clearly isn't my fault any more!)

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
disperse power, foster diversity, and nurture creativity

---

From: Paul Vixie <[email-masked]>
Date: Mon, 05 Nov 2018 10:51:04 -0800

because of deliberate reconfiguration or takedown, i'll hope that 
serve-stale offers authority operators (both apex and parent) a 
signalling pattern that says, "actually, i want this dead, NOW."

---

From: Tony Finch <[email-masked]>
Date: Mon, 5 Nov 2018 18:56:50 +0000

Paul Vixie <[email-masked]> wrote:

> because of deliberate reconfiguration or takedown, i'll hope that serve-stale
> offers authority operators (both apex and parent) a signalling pattern that
> says, "actually, i want this dead, NOW."

Thanks for making that point for me, I did a remarkably bad job of
transcribing my notes and that was one of the arguments against a 1 week
serve-stale timeout that I missed :-)

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
East Sole, Lundy, Fastnet, Irish Sea: Northwesterly at first in Sole and
Fastnet, otherwise southerly or southeasterly 5, increasing 6 or 7,
occasionally gale 8 later. Slight or moderate until later in Lundy and Irish
Sea, otherwise moderate or rough. Rain or showers. Good, occasionally poor.

---

From: Bob Harold <[email-masked]>
Date: Mon, 5 Nov 2018 14:39:10 -0500

> signalling pattern that says, "actually, i want this dead, NOW."

Good point.  I think that would mean that if using all the NS records in
the cache fail to get a good response, then the resolver should check the
parent domain to see if the NS records have changed or have been removed.
(answers or NXDOMAIN being a good response in this case, REFUSED or LAME or
timeout being bad responses)

Would that work?   Should that be in the draft?

-- 
Bob Harold

---

From: Paul Wouters <[email-masked]>
Date: Tue, 6 Nov 2018 00:08:03 -0500 (EST)

On Mon, 5 Nov 2018, Paul Vixie wrote:

> Subject: Re: [DNSOP] Review of draft-ietf-dnsop-serve-stale-02.txt
> 

That sword cuts both ways. It would be useful to have a Human Rights
section in the document that discusses this issue.

Perhaps the HRPC WG can assist us here,

Paul

---

From: Paul Wouters <[email-masked]>
Date: Tue, 6 Nov 2018 00:12:44 -0500 (EST)

On Mon, 5 Nov 2018, Bob Harold wrote:

> Good point.  I think that would mean that if using all the NS records in the cache fail to get a good response, then the resolver should check
> the parent domain to see if the NS records have changed or have been removed.
> (answers or NXDOMAIN being a good response in this case, REFUSED or LAME or timeout being bad responses)
> 

Something along those lines should be added. But this particular
approach might be too simple. What if the parent is also under
DDOS attack? When can/should you look at the parent's parent (eg
think Public Suffix boundaries)

Paul

---

From: Paul Vixie <[email-masked]>
Date: Mon, 05 Nov 2018 22:11:00 -0800

Paul Wouters wrote:
>> On Mon, Nov 5, 2018 at 1:51 PM Paul Vixie <[email-masked]>
>> wrote: because of deliberate reconfiguration or takedown, i'll hope
>> that serve-stale offers authority operators (both apex and parent)
>> a signalling pattern that says, "actually, i want this dead, NOW."
>>
>>
>> Good point.  I think that would mean that if using all the NS
>> records in the cache fail to get a good response, then the resolver
>> should check the parent domain to see if the NS records have
>> changed or have been removed. (answers or NXDOMAIN being a good
>> response in this case, REFUSED or LAME or timeout being bad
>> responses)
>>

something like it should be, yes.

> approach might be too simple. What if the parent is also under DDOS
> attack? When can/should you look at the parent's parent (eg think
> Public Suffix boundaries)

eight years ago we said it like this:

> 2. Delegation Revalidation Upon NS RRSet Expiry
>
>    2.1. Because the delegating NS RRset at the bottom of the parent zone
>    and the apex NS RRset in the child zone are unsynchronized, the TTL
>    of the parent's delegating NS RRset is meaningless. A child zone's
>    apex NS RRset is authoritative and thus has a higher cache
>    credibility than the parent's delegating NS RRset, so, the NS RRset
>    "below the cut" immediately replaces the parent's delegating NS RRset
>    in cache when an iterative caching DNS resolver crosses a zone cut.
>
>    2.2. The lowest TTL found in a parent zone's delegating NS RRset
>    should be stored in the cache and used to trigger delegation
>    revalidation as follows.  Whenever a cached RRset is being considered
>    for use in a response, the cache should be walked upward toward the
>    root, looking for expired delegations. At the first expired
>    delegation encountered while walking upward toward the root,
>    revalidation should be triggered, putting the processing of dependent
>    queries on hold until validation is complete.
>
>    2.3. To revalidate a delegation, the iterative caching DNS resolver
>    will forward the query that triggered revalidation to the nameservers
>    at the closest enclosing zone cut above the revalidation point. While
>    searching for these nameservers, additional revalidations may occur,
>    perhaps placing an entire chain of dependent queries on hold,
>    unwinding in downward order as revalidations closer to the root must
>    be complete before revalidations further from the root can begin.
>
>    2.4. If a delegation can be revalidated at the same node, then the
>    old apex NS RRset should be deleted from cache and then the new
>    delegating NS RRset should be stored in cache. The minimum TTL from
>    the new delegating NS RRset should also be stored in cache to
>    facilitate future revalidations. This order of operations ensures
>    that the RRset credibility rules do not prevent the new delegating NS
>    RRset from entering the cache. It is expected that the child's apex
>    NS RRset will rapidly replace the parent's delegating NS RRset as
>    soon as iteration restarts after the revalidation event.
>
>    2.5. If the new delegating NS RRset cannot be found (RCODE=NXDOMAIN)
>    or if there is a new zone cut at some different level of the
>    hierarchy (insertion or deletion of a delegation point above the
>    revalidation point) or if the new RRset shares no nameserver names in
>    common with the old one (indicating some kind of redelegation, which
>    is rare) then the cache should be purged of all names and RRsets at
>    or below the revalidation point. This facilitates redelegation or
>    revocation of a zone by a parent zone administrator, and also
>    conserves cache storage by deleting unreachable data.
>
>    2.6. To make the timing of a revalidation event unpredictable from
>    the point of view of a potential cache-spoof attacker, the parent's
>    delegating NS RRset TTL should be reduced by a random fraction of its
>    value before being stored for use in revalidation activities.

so what you'd do is revive this part of resimprove and then refer to it 
by saying "before considering the extended publication of stale data, 
first revalidate as in [RESIMPROVE], as if the delegating NS RRset TTL 
had expired".

(https://www.ietf.org/archive/id/draft-vixie-dnsext-resimprove-00.txt)

note that vernon schryver and i implemented this in a from-scratch rdns 
server called "robodns" in the early 2000's, and it worked really well.

-- 
P Vixie