Subject: [radext] Fwd: Mail reguarding draft-ietf-radext-dynamic-discovery

From: Stefan Winter <[email-masked]>
Date: Fri, 28 Jun 2013 17:41:51 +0200

Hi,

forwarding more comments from an implementer of dynamic discovery which
I had received over the draft- address. Shared with permission.

Stefan

-------- Original Message --------
Subject: 	Mail reguarding draft-ietf-radext-dynamic-discovery
Resent-Date: 	Tue, 18 Jun 2013 18:21:09 +0200 (CEST)
Resent-From: 	[email-masked]
Resent-To: 	[email-masked], [email-masked]
Date: 	Tue, 18 Jun 2013 16:20:54 +0000
From: 	Brian Julin <[email-masked]>
To: 	[email-masked]
<[email-masked]>

Greetings,

After starting an implementation of draft-ietf-radext-dynamic-discovery
for the FreeRADIUS project, the following are my comments on areas that
I found problematic:

Section 2.3.3 leaves a lot up to implementors when deciding what
constitutes "name resolution returns with error" as there are many
flavors of "error" in DNS.  Also, what to do when RRs are found
that match a search's service/protocol tags but which are deemed
invalid for other reasons, such as unknown NAPTR flags or obviously
unusable owner labels, DNSSec policy, or limitations on local resources
exceeded due to an abusively byzantine NAPTR tree.  When, exactly, the
default SRV target is to be used will vary from implementation to
implementation unless this is nailed down in the draft.

The draft might want to explicitly prohibit the fallback A record
lookup specified in rfc2782 (the final else clause in the "Usage
Rules" section.)

RFC 6613 section 2.2. places restrictions on the use (or not) of TLS
and/or TCP on certain RADIUS assigned port numbers.  How to deal with
the conflict that arises when a NAPTR specifies RadSec as a protocol
and then the SRV points to port 1812 rather than 2083, for example,
is not clear and would bear mention in the draft.

Section 2 lacks a fully explicit definition for RFC 3958 section
3.1.2.  Further, section 2.3.4 is open to multiple interpretations,
and RFC 3403 Section 3 leaves certain details to inference:

1) Section 2.3.5. mandates the termination of the discovery
    process after three seconds, with fallback to static configuration.
    If said static configuration is to reject the request, however,
    in many environments the client will automatically retry some
    seconds or minutes later.  It is therefore useful to continue
    the algorithm in the hopes that a positive result may be obtained
    before that time, allowing follow-up requests from the client
    to succeed promptly.  This might be noted as acceptable behavior, as
    long as requests that wait for 3 seconds are always routed through
    the static configuration for as long as no positive result is
    available.

2) It is not clear as to how to proceed if TTLs, either positive or
    negative, expire during the execution of the algorithm due to DNS
    delays; Section 2.3.4 only specifies that they apply when considering
    the re-use of results for additional user sessions.  RFC3403 Section 3
    does specify that records that are "relied upon" will cause a restart
    of the algorithm if their TTL has expired during a "backtrack", but
    says nothing of what to do during the descent.  Also, it is not
    explicitly stated in RFC3403 Section 3 that a negative result which
    has caused a previous skip of a branch of the NAPTR tree is "relied
    upon", so that may be worth explicit mention.

3) The concerns in 2) may also be mitigated substantially if the draft
    were to put normative limits on how small NAPTR and SRV TTL values
    may be in compliant DDNS database entries.  Allowing small TTLs in
    these records leaves open the possibility that "relied upon" RRs will
    expire before negative results return from DNS servers, the algorithm
    will backtrack, and if those negative results also have a short TTL or
    are not well cached, this process could repeat itself for as long as
    the discovery algorithm is allowed to run.  The draft would do well to
    permit compliant implementations to apply a reasonable minimum TTL
    value to all NAPTR and SRV records received, something on the order of
    60 seconds, and to advise database creators to keep NAPTR and SRV
    record TTLs even much higher than that.  One source of advice on this
    matter is an expired draft, draft-ietf-speermint-srv-naptr-use.

Thank you,

Brian S. Julin

P.S. If the authors would like to keep tabs on the progress of the
abovementioned implementation, it is currently residing on github:

https://github.com/skids/freeradius-server/tree/ddds

---

From: Stefan Winter <[email-masked]>
Date: Fri, 28 Jun 2013 17:42:17 +0200

Here's my initial reply, with proposed changes to the draft.

-------- Original Message --------
Subject: Re: Mail reguarding draft-ietf-radext-dynamic-discovery
Resent-Date: Wed, 26 Jun 2013 14:56:25 +0200 (CEST)
Resent-From: [email-masked]
Resent-To: [email-masked], [email-masked]
Date: Wed, 26 Jun 2013 14:56:09 +0200
From: Stefan Winter <[email-masked]>
To: Brian Julin <[email-masked]>
CC: [email-masked]

Hi,

thanks a lot for your in-depth look at the draft spec. This is very much
appreciated!

> flavors of "error" in DNS.

2.3.3.  Definitions

   Where the algorithm states "name resolution returns with an error",
   this shall mean that either the DNS request timed out, or every
   response except a) the RR queried for and b) the positive
   acknowledgement that the record queried for does not exist (i.e.
   status: NXDOMAIN).

This would be a whitelist of stuff we want, and everything is else is
declared bad. Would that address your concern?

> Also, what to do when RRs are found
> exceeded due to an abusively byzantine NAPTR tree.

I've expanded that step in my working copy now:

   7.   Evaluate NAPTR result(s) for desired protocol tag, perform
        subsequent lookup steps until lookup yields one or more
        hostnames.

   8.   If these name resolutions return an error for any of the
        subsequent lookups (e.g. an SRV target which does not exist), or
        for other reasons does not yield a complete set of hostnames, O
        = { empty set } and terminate.

   9.   O' = (set of {hostname; port; order/preference; min{all TTLs
        that led to this result} } for all lookup results).  Keep note
        of the remaining TTL of each of the discovered records (e.g. SRV
        and AAAA).

I.e. you either get a final result for all the records, or the discovery
has failed.

Would that address your concern?

>  When, exactly, the

The underlying philosophy is: if a NAPTR with the expected tags exists,
it is the only option that is being explored. If the content of those
NAPTRs then turns out to contain nonsense, too bad. No fallback in this
case.

The algorithm makes clear that the only reason to consult the SRVs
directly is if there is no NAPTR with fitting tags.

> lookup specified in rfc2782 (the final else clause in the "Usage

Well, the draft does not refernce RFC2782, neither informatively nor
normatively. Instead, it constructs the SRV discovery inline in the
algorithm. The algorithm does not speak of "A" fallback; so it's not
performed.

> is not clear and would bear mention in the draft.

The ports are merely default ports. If someone runs a RADIUS/TLS server
on 1812 instead of 2083, then that's fine and he can announce that fact
in DNS if he so likes.

If the target he reaches then does NOT speak RADIUS/TLS, then this is an
error in DNS configuration; i.e. a failure condition "like any other".

So, I would think that your next comment is actually the generalised
version of this specific one, as it correctly reprimands me for not
having specified failure/retry mechanisms :-)

> 3.1.2.

That's right; it also misses 3.1.3 (and I've received some beating about
that on the mailing list and during the last IETF meeting :-) ).

For the 3.1.2 issue, how about this text:

2.2.1.2.  Definition of Conditions for Retry/Failure

   RADIUS is a time-critical protocol; RADIUS clients which do not
   receive an answer after a configurable, but short, amount of time,
   will consider the request failed.  Due to this, there is little
   leeway for extensive retries.

   As a general rule, only error conditions which generate an immediate
   response from the other end are eligible for a retry of a discovered
   target.  Any error condition involving time-outs, or the absence of a
   reply for more than one second during the connection setup phase is
   to be considered a failure; the next target in the set of discovered
   NAPTR targets is to be tried.

   As an exception: note that [RFC3958] already defines that a failure
   to identify the server as being authoritative for the realm is always
   considered a failure; so even if a discovered target returns a wrong
   credential instantly, it is not eligible for retry.

(I was shuffling some section numbers in my current working copy, please
ignore the 2.2.1.2 for now)

Regarding 3.1.3, I'll need to think more about it, and need another rev
of the document before it manifests.

>  Further, section 2.3.4 is open to multiple interpretations,

This was discussed during the last meeting and on the mailing list. I
have text in my current working copy which instead suggests NOT to try
this (but in Security Considerations and non-normatively, so an
implementation can make a judgement call to try it anyway). It reads:

   The algorithm has a fixed completion time-out of three seconds.
   Implementations might be tempted to continue their attempt to resolve
   DNS records even after the timeout has passed; a subsequent request
   for the same realm might benefit from retrieving the results anyway.
   Doing so exposes the server to a Denial-of-Service risk: an attacker
   might intentionally craft bogus DNS zones which take a very long time
   to reply (e.g. due to a particularly byzantine tree structure, or
   artificial delays in responses.  If an attacker generates enough
   Access-Requests for a number of such zones, it may deplete server
   sockets or other server resources.

>     these records leaves open the possibility that "relied upon" RRs
>     will expire before negative results return from DNS servers, the
>     algorithm will backtrack, and if those negative results also have
>     a short TTL or are not well cached, this process could repeat
>     itself for as long as the discovery algorithm is allowed to run.
>     The draft would do well to permit compliant implementations to
>     apply a reasonable minimum TTL value to all NAPTR and SRV records
>     received, something on the order of 60 seconds, and to advise
>     database creators to keep NAPTR and SRV record TTLs even much
>     higher than that.  One source of advice on this matter is an
>     expired draft, draft-ietf-speermint-srv-naptr-use.

Answering your point 3) first.

I believe there is a misconception on the TTLs here. Of course the
maintainers of a DNS zone should define their entries with reasonably
high TTLs to avoid thrashing. That's a normal consideration for any DNS
label and RR though, I don't think we need to engage in the business of
handholding or explain "DNS domain administration 101".

Also keep in mind that changes in the network infrastructure sometimes
REQUIRE low TTLs, e.g. when trying to change a hostname to A mapping
in-place.

The reason why I wrote misconception though is that these are not the
TTLs the algorithm gets (I tried to construct a pun involving "these are
not the TTLs you are looking for" but didn't manage :-) ). The name
resolution library on the server which does dynamic discovery is very
unlikely to get fresh entries from the respective authoritative DNS
server with the full TTL of that authoritative server. It will much more
likely get a cached reply from its resolver. And resolvers keep track of
how long *they* should cache answers received from upstream. I.e. they
will take the initial TTL from the authoritative server, and will
decrement it on their cached entry. When queried, they will reply with
their own copy's (lower) TTL.

No matter how large the TTL on the authoritative server is, it can at
any time happen that a resolver will reply in the last second of its own
cache lifetime and will rightfully state that the entry is TTLed with a
1 second duration.

So, no amount of advice towards DNS administrators will make your issue
go away.

>     says nothing of what to do during the descent.

Picking up your point from 3), it sounds like an interesting idea to
specify that any TTL < 60 seconds (or an arguable other integer) should
be considered as TTL = 60 seconds; thereby overriding the DNS caching
rules. Then your concerns would probably be mitigated.

I like that, and would put that in the draft. But I believe it should be
discussed on the mailing list first; I could imagine DNS people to be
disenchanted by an override of their data ("Why do you think you know
better than the DNS admin himself?!?!").

>   Also, it is not

If also all negative results would get at least a 60s survival time,
then this becomes a non-issue, right?

Finally, is it okay for you if I forward your mail and my response to
the IETF radext mailing list? These things are best discussed with as
many eyes on them as possible.

Greeetings,

Stefan Winter

> https://github.com/skids/freeradius-server/tree/ddds

Will take a look!

-- 
Stefan WINTER
Ingenieur de Recherche
Fondation RESTENA - Réseau Téléinformatique de l'Education Nationale et
de la Recherche
6, rue Richard Coudenhove-Kalergi
L-1359 Luxembourg

Tel: +352 424409 1
Fax: +352 422473

---

From: Stefan Winter <[email-masked]>
Date: Fri, 28 Jun 2013 17:42:46 +0200

and another follow-up; to which I have yet to write an answer.

-------- Original Message --------
Subject: RE: Mail reguarding draft-ietf-radext-dynamic-discovery
Date: Thu, 27 Jun 2013 01:47:52 +0000
From: Brian Julin <[email-masked]>
To: Stefan Winter <[email-masked]>

Stefan,

Thanks again for the followup.  Here are some thoughts.

> declared bad. Would that address your concern?

One case not covered that I know of: NXDOMAIN is not returned when an owner
has records of type other than the type you query.  What you get is a
NOERROR response without any of the requested type of RRs, and
usually an authority SOA record, and maybe some additional section RRs.
Try a dig for NAPTRs for ieee.org for example.

I'd suggest defining NXDOMAIN/NSEC/NSEC3 and the above case as a
"negative result" instead, and just say in step 4 to proceed to
step 9 on a credible negative result, step 5 if credible NAPTR RRs
were retrieved, and otherwise O = { empty set } and terminate.

(definition of "credible" depends on federation DNSSec policy)

I think this point would benefit from the attention of a jaded^Wexperienced
DNS developer, which I am not.  There could easily be other modes which I've
not seen.

>         = { empty set } and terminate.

Minor quibble -- it's presented in steps, but 8 is really happening during 7
and functioning as a break condition.

> Would that address your concern?

It still needs work.  What's a "complete set of hostnames"?  This is the
area where I'm probably going to try your patience :-).

> directly is if there is no NAPTR with fitting tags.

OK, I'll buy into that part being clear.

> algorithm. The algorithm does not speak of "A" fallback; so it's not

The words "perform SRV lookup" could mean to just launch a SRV query,
as intended, or could be taken as "follow the rules for using SRV records
in the RFC that defines them."  I don't think not mentioning 2782 gets
the draft off the hook here, because once SRV is mentioned as something that
is a known quantity, the reasonable recourse for an un-annotated external
reference is to consult relevant RFCs.

As an aside, when dealing with "s" flags, 3958 explicitly states "normal
SRV processing is applied" (2.2.3).  Were it not for the explicit provision
in 2.2.4. about declaring failure, I would take this to mean that the
Usage Rules in 2782 are supposed to be followed.

It could perhaps be rephrased "look up RRs of type SRV."

> on 1812 instead of 2083, then that's fine and he can announce that fact

I'm satisfied with that answer, and it was the one I expected.  Do
note, the language in 6613 for port 1812 in particular is a bit stronger
than merely defining a default port, but that could plausibly be a
shortcoming
in 6613's phrasing.

> of the document before it manifests.

Likewise, it will be some time until I can thoroughly investigate this.
Right now I'm working on the DNS end, and the subsystem that handles
actually
establishing connections is completely separate.  With EAP sessions wanting
connection affinity and the "State" attribute often not being implemented
by clients, that subsystem is already pretty beastly and will get more so
with the pool of eligible servers subject to change, so it is unlikely
integration with DNS processing will be very tight.

There will have to be a crankback mechanism whereby the
connection-pool/load-balancing subsystem demands more/different results
from the DNS subsystem when it decides the identified targets are unusable.
I suspect this will be the case in most designs -- errors from DNS will be
handled more directly than errors from connection setup.  Just mentioning as
something to keep in mind.

>    sockets or other server resources.

The point about even a low-depth NAPTR tree from a compromised domain
wreaking flood havoc is well taken, I had not yet considered that.

Some of these vectors will exist whether or not the implementation stops at
three seconds.  Also the 3 second value is for the benefit of RADIUS
protocol
fallback needs and should probably not be conflated with a secure value for
abating DoS attacks.  Implementations could use an arbitrary value here
that is
more than 3 seconds but still reasonably finite.  What administrators
will want
instinctively is for the resolution to continue for their usual client
retry interval,
and continue thereafter if refreshed by a retry.

In some vectors three seconds is more than enough time to cause
a lot of hurt, repeated bursts of three seconds moreso.  At the very
least the DNS caches will be subject to loading under these scenarios.
Note that RADIUS servers are likely to be exempt from many of the
safeguards applied to client machines on the DNS server side, and
they are likely to have nice fast 1G or 10G links direct to the DNS
servers with a very low RTT.  A lot of queries can happen under those
conditions in 3 seconds if the terminal records are aimed at NX records
in a locally authoritative domain.

Most mature implementations will likely opt to place arbitrary limits
on the following quantities, per realm, tunable either at compile or
runtime so that resource use can be matched to reasonable expections
within the particular federation(s) involved:

1) total number of backtracks since the last TTL expiry or
well-known-rule instigation
2) number of NAPTR records for which (summarized) TTL information is
tracked.

The width/depth of NAPTR trees will be limited by 2).  The rate of
queries for bogus domains will be limited by 1).

In addition implementations will likely limit the number of ongoing or
cached DDDS discoveries, in a FIFO fashion, failing and/or expunging the
oldest queries when new ones arrive and the cap is exceeded.  As long as
this cap is kept high enough that legitimate queries have time to complete
before they get expunged to make room for a bogus query, RADIUS service
itself is not in jeopardy.  If the rate of these queries is large enough
to spin that FIFO too fast, then the server has bigger problems on its
hands that need to be addressed at a general DoS mitigation level.

I would suggest generalizing that language to simply advise implementations
to set sensible limits to prevent over-stacking of pending DDDS discoveries.
If discussions have come up with an advised default value that is
coincidentally
also 3 seconds, distinguish this value from the limit on individual
queries and
present solid rationale as to why 10 to 30 seconds (typical for retries)
is too
long whereas 3 seconds is not.

Note that caching DDDS results for later use is required by the
draft and also a vector for resource DoS, though not for sockets.

> So, no amount of advice towards DNS administrators will make your issue

This is understood.  Do note that if a DNS server provides a lingering
record with an extremely low TTL that expires, and the authoritative TTL
is set reasonably, the re-query is only going to happen once in the
short term, as opposed to when the authoritative TTL is e.g. zero.

(Think of what happens when a sports team tour bus or two of 50 or so
eduroamers from the same institution arrives at your school and their
NAPTR TTLs are 1, then they proceed to hop repeatedly around
several autonomous access points without fast-reauth enabled.)

Advice to DNS administrators does not technically solve the issue, though,
you are correct.  It just serves (if followed) to reduce the occurence of
instances where the implementation might need to disobey the TTL -- TTLs
would more often expire in the interstices between RADIUS requests and
be refreshed with values that would endure the full DDDS discovery.

It also sets reasonable expectations on the part of DNS administrators and
federation policy makers as to how DDDS implementations are likely to
behave.
If the guidance suggests a minimum TTL, they'll be duly warned that
implementation behavior might not be reliable if they elect to do otherwise.

Your point about the scope of the specification is well taken, though.

> rules. Then your concerns would probably be mitigated.

Entirely mitigated, yes.  Another option is to say that results are sticky
during descent.  From an implementation standpoint the former is easier
since we have to keep the TTL as state anyway.

> then this becomes a non-issue, right?

Partially: it solves it for descent, but not for backtrack which can
happen much later.
This is more a technicality than an issue -- the right thing to do
(maintain a summarized
"negative" TTL for each skipped NAPTR RR which is the minimum of all
positive/negative
TTLs involved in the decision to backtrack past the RR in question, and then
check it when re-using the data) was comprehensible to me, but it isn't
stated outright and might be unfathomable to some implementers.  I probably
should not have lumped this point into 2) as it does not just apply to
initial descent.  Also it is really RFC3403s problem -- whether patching it
up in the application definition is appropriate I leave to the standards
gurus :-)

--
Brian S. Julin