Subject: [radext] Fwd: RE: Fwd: RE: Fwd: RE: Mail reguarding draft-ietf-radext-dynamic-discovery

From: Stefan Winter <[email-masked]>
Date: Wed, 10 Jul 2013 14:41:39 +0200

Hi,

Brian did a VERY thorough analysis of one particular feature of the
draft: loop prevention. See the forward below.

I believe it's worth discussing the question in Berlin (again).

My point so far is: loops can occur, so we should do all we can to
detect and prevent them. This requires all NAPTR -> SRV -> A/AAAA
lookups to be done, and look for inconsistencies in the resulting set.

Brian's counter-point is: loops can happen also outside dynamic
discovery, too; and will fix themselves by busting RADIUS packet
boundaries. In the interest of speed, let's be less thorough in finding
them and take shortcuts in DNS response evaluation wherever possible.

Greetings,

Stefan Winter

-------- Original Message --------
Subject: RE: [radext] Fwd: RE: Fwd: RE: Mail reguarding draft-ietf-radext-dynamic-discovery
Date: Tue, 9 Jul 2013 19:00:36 +0000
From: Brian Julin <[email-masked]>
To: Stefan Winter <[email-masked]>

Stefan,

Having read through the new version of the draft, I now understand what
the draft is trying to achieve WRT loop detection.

Here are my comments on that:

The motivation for performing this loop detection seems to be predominantly
aimed at dealing with mis-configured edge RADIUS proxies which are handling
local realms as though they were remote.

First let us consider the scenario where there are no failures in DNS and no
failures encountered when connecting to servers selected via the DDDS
algorithm.  In this case, only the highest matching order/preference/priority
will be used.  The possible conditions under which a loop might occur in this
scenario are as follows:

1) The DDDS algorithm selects a listening address of the same RADIUS server.
This can be detected by a safeguard to prevent tight loops performed when
inidividual connections are initiated and/or received, and does not require
examination of any of the other DDDS results.

2) The next hop is an arrangement of statically routed servers which have a static
route back to a listening address of the initiating server.  In this case there is
no clue present in any of the DDDS results that a loop has occurred.  This could
happen both on backbone federation servers, or within an institution where different
RADIUS instances perform different roles.

3) A terminal A record with multiple RRs is selected, one of which is a listening
address on the initiating server.  Since the A lookup needed to be performed
anyway, all addresses are already available for a sanity check.

4) A SRV contains multiple RRs of the same priority, one of which is a listening
address on the initiating server.  Since all A records in a SRV are not customarily
looked up, only those selected for transit, this scenario could result in loops
around a load-balancing pool, if said pool was not configured to refuse connections
from itself and other members of the pool.

5) The result of DDDS otherwise forwards the packet to another server that also
performs DDDS for the same service/protocol.  This will always result in a loop,
since DDDS for a given service/protocol must only be performed once.  (What
was said about doing the same thing multiple times and expecting different results?)

For the scenario that a backup SRV priority or a backtracked NAPTR order/preference
is selected, or a connection failure, the situations are the same as above, with the
addition that in any of the above situations the results of DDDS could differ in any
servers involved in the loop.  This could even be done intentionally or accidentally
with a split horizon DNS server.  There is also the specter of transient results during
DNS changes.  None of these are actual problems: if any of that ever matters, the
root problem is that DDDS is being performed a second time for the same
service/protocol during the routing of the same packet, when that should never be
the case.

Note that if a non-fatal DNS failure prevents a RADIUS server from seeing
its own listening address/port in the DDDS results, this can render a sanity
check ineffective.

Also note (per 2 above) that there are several scenarios where the list of addresses/ports
against which the DDDS result would need to be checked to correctly break loops is not
the same as the list of listening sockets on the server performing DDDS -- i.e. any
arrangement of multiple autonomous RADIUS servers that forwards packets internally.

When a RADIUS routing loop occurs, there is no formally defined mechanism
for breaking loops.  However, implementations have worked in some duplicate
detection for performance reasons (e.g. not sending the retransmits needed
in an unreliable UDP-based environment into a reliable TCP-based environment)
and to deal with network topologies that might generate duplicate UDP traffic.
In FreeRADIUS the duplicate detection relies on the authentication vector and
size of the packet as well as the src/dst ip/port.  This would cut short most
naturally occurring loops after a finite number of iterations -- the exception being
scenarios where hops modify transiting requests in a way that is not
eventually idempotent, without also creating a packet that exceeds any
RADIUS packet format limitations, or implementations that change their source
ports frequently.  RADIUS administrators should be advised not to disable
duplicate checking on incoming TCP connections when DDDS is in use.

Other possible safeguards for implementors to consider could include
per-realm rate limiting on transit hops, with violations leading to a purge
of all requests from that realm for a short period of time and log messages
to that effect.

General loop detection in RADIUS is an issue beyond the scope of the draft --
albeit one which does deserve some attention.

While it is a helpful safeguard to prevent a server from trying to connect
to itself, the scenarios sketched out above are likely to be rare enough that
a balance needs to be struck between the performance of the normal, functional,
case and the efficacy of safeguards, which cannot be perfect as DDDS cannot
in and of itself solve the problem.  Given the expense of performing a full
enumeration of the DDDS tree, I suggest that line should be drawn somewhat
short of that measure. with implementations given the option to be more
thorough where they deem fit.  I'd suggest that servers SHOULD lookup and
loop-guard all A/AAAA records for RRs of an active SRV priority, with a hard
failure if even one matches, and SHOULD likewise loop-guard against all
A/AAAA addresses retrieved when multiple A/AAAA RRs respond to the same
owner, and of course MUST never try to forward to the same listening address
upon which the original request was received.

I might further offer that implementations MAY shut down all attempts to connect
to a realm if they ever are asked to connect to themselves for that realm in a
manner that generates noisy complaints in logs and requires manual intervention
to clear.  In general these problems are of the type that will not self-heal and/or
they indicate a security incident which merits notification of an administrator.
Most implementations will probably elect not to permanently poison a realm for
fear that a single spoofed DNS result could use this as a DoS.  They should be
at least encouraged to log such episodes as security events.

Regards,

Brian S. Julin

---

From: Alan DeKok <[email-masked]>
Date: Thu, 11 Jul 2013 08:56:48 +0200

> them and take shortcuts in DNS response evaluation wherever possible.

  Sounds good to me.

  Some notes on Brian's text:

> naturally occurring loops after a finite number of iterations 

  It won't.  The duplicate detection is the RFC 5080 mechanism.  It
detects a NAS retransmitting the identical packet.  It doesn't affect
(or help with) loop detection.

>  -- the exception being
> eventually idempotent,

  All proxies modify the RADIUS packet.  The ID field changes (usually),
and the shared secret is different, so the authentication vector field
changes, too.

> without also creating a packet that exceeds any

  I think this advice is wrong.  RFC 6613 Section 2.6.1 forbids
retransmission of duplicate packets over TCP.  So any packet in a loop
will look different the second time around.  Or if it's the same, then
it MUST NOT be transmitted the second time, and the loop is broken.

> albeit one which does deserve some attention.

  That's very true.

> upon which the original request was received.

  That analysis presumes the *outgoing* address of the server is the
same as it's *incoming* address.  Which doesn't have to be the case.

  It may be worth updating the draft to suggest that servers using
dynamic discovery either:

  (1) use the same address for incoming and outgoing packets

  (2) publish a different DNS record indicating their outgoing address

  The SMTP world has had similar issues for a long time.  Solutions like
SPF may be applicable here.

> to clear.

  I agree.  This also brings up issues of authenticating the DNS
records.  It may be useful to have reverse IP records, which serve as a
cross-check.

  e.g.  if looking up "example.net" gets you 192.12.6.10, then doing a
reverse IP lookup on that IP should get you "example.com" (among others).

  The document doesn't discuss reverse IP records or lookups.

>  In general these problems are of the type that will not self-heal and/or

  Having a reverse IP lookup can help with this situation.  If the
forward and reverse records don't match, then that can be discovered
automatically.  And before the proxy starts a secure connection to the
home server.

  Alan DeKok.

---

From: Stefan Winter <[email-masked]>
Date: Tue, 16 Jul 2013 15:07:50 +0200

Hi,

>> them and take shortcuts in DNS response evaluation wherever possible.

I'd feel good about the more relaxed attitude if there *were* a loop
detection that works reliably, and independent of the discovery
mechanism (DDDS vs. config file).

I'd like to bring up the topic of a dedicated "TTL" attribute in Berlin
(again, as I've previously received some beating for proposing it).

I think this time the underlying situation has changed: Sam's proposal
to lift the packet size boundary on TCP/TLS links means
eternally-proxies packets now can bounce a LOT longer between hops until
their aggregated Proxy-State size busts the packet limits. Relying on
that sort of phenomenon becomes less and less useful IMHO.

>> albeit one which does deserve some attention.

Good :-) It would be very simple and straightforward to write an I-D
registering an integer attribute "Packet-TTL", with an integer value
that is to be decremented.

> SPF may be applicable here.

Yet another new DNS record would add much complexity to the draft; for
little benefit. The TTL would solve this in one go.

It would be equivalent to the SMTP's "Too many Received headers:
suspecting a loop", much more than SPF.

>   The document doesn't discuss reverse IP records or lookups.

It doesn't, because the authentication of the results learned is done by
X.509 certificate inspection during connection time. I find that a much
more reliable mechanism than reverse IP checks. (if an attacker can fake
an A record, then it may also be able to fake a PTR record).

> automatically.  And before the proxy starts a secure connection to the

Having it attempt /one/ TLS connection to the target doesn't consume
much computing power. The discovering server could also keep a list of
servers it has recently tried and found unacceptable; and save itself
the effort of trying over and over again.

This could be mentioned in Security Considerations, if you find it
useful/important to have?

Greetings,

Stefan Winter

> 
>   Alan DeKok.
> _______________________________________________
> radext mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/radext
> 

-- 
Stefan WINTER
Ingenieur de Recherche
Fondation RESTENA - Réseau Téléinformatique de l'Education Nationale et
de la Recherche
6, rue Richard Coudenhove-Kalergi
L-1359 Luxembourg

Tel: +352 424409 1
Fax: +352 422473

---

From: Alan DeKok <[email-masked]>
Date: Tue, 16 Jul 2013 15:35:42 +0200

> that is to be decremented.

  It may also be good to add a "Global-Packet-ID".  The idea would be
that it's a 64-bit or 128-bit opaque token, created by the client.  (Or
a proxy close to the client).  Other proxies would then never touch it.

  It could be cached by proxies for a period of time.  If a proxy sees
the same Global-Packet-ID in two different packets, it knows there's a
loop or retransmission via an alternate path.

> little benefit. The TTL would solve this in one go.

  The TTL wouldn't solve the forward / reverse problem.  Using a reverse
IP would catch that.

> more reliable mechanism than reverse IP checks.

  That's true.

> (if an attacker can fake

  I'm not sure I agree there.  I can fake an A record for my own domain.
 But I can't fake a reverse IP record for an IP which you control.

> the effort of trying over and over again.

  What about the discovered server?  It would be subject to attacks by
third parties.

  i.e. realm A lists proxy B as it's server via DDNS.  It then publishes
an article saying "free net access by using user@realmA".  Everyone
wanting free net access contacts proxy B.  Repeat for 10,000 domains,
and Proxy B gets a DoS.

  This should ideally be discovered *before* a using high-cost
connection like TLS.  That's why a reverse lookup is useful.

  Now... this is RADIUS, and there aren't that many RADIUS proxies.  But
if the DDNS draft makes it easier, there will be more RADIUS proxies,
and more possibility for such an attack.

> useful/important to have?

  Yes.

  Alan DeKok.

---

From: Stefan Winter <[email-masked]>
Date: Wed, 24 Jul 2013 12:06:38 +0200

Hi,

> loop or retransmission via an alternate path.

That's another approach, which would also work. There's a very small
probability that two unrelated packets happen to get the same packet-id;
that kind of indeterminism makes me a bit nervous.

Anyway: I'm glad that we're discussing loop detection /at all/. I didn't
get that far in my earlier attempts on the topic :-)

> and more possibility for such an attack.

That doesn't really help: the forward and reverse IP checks work on the
A/AAAA pair that is the "leaf" of the discovery process. Third parties
adding a NAPTR where they should not will not be detected. If the end
server is actually offering a dynamic discovery endpoint, then it will
be properly configured for its legimiate customers. Having other people
send non-legitimate customers to the same destination as well will not
be detectable with an IP address check.

>> useful/important to have?

The fact that DoS becomes possible by illegitimate clients is mentioned
in RFC6614. I take your point that a mere usage of RFC6614 with
hand-configured clients makes the problem less important, because
firewalls could prevent the session establishment.

For a next rev of dynamicdiscovery, I've added the following paragraph
for Security Considerations in my working copy:

   With Dynamic Discovery being enabled for a RADIUS Server, and
   depending on the deployment scenario, the server may need to open up
   its target IP address and port for the entire internet, because
   arbitrary clients may discover it as a target for their
   authentication requests.  If such clients are not part of the roaming
   consortium, the RADIUS/TLS connection setup phase will fail (which is
   intended) but the computational cost for the connection attempt is
   significant.  With the port for a TLS-based service open, the RADIUS
   server shares all the typical attack vectors for services based on
   TLS (such as HTTPS, SMTPS, ...).  Deployments of RADIUS/TLS with
   Dynamic Discovery should consider these attack vectors and take
   appropriate counter-measures (e.g. blacklisting known-bad IPs on a
   firewall, rate-limiting new connection attempts, etc.).

Is that okay for you?

Stefan Winter

Tel: +352 424409 1
Fax: +352 422473

---

From: Alan DeKok <[email-masked]>
Date: Wed, 24 Jul 2013 09:29:17 -0400

> that kind of indeterminism makes me a bit nervous.

  There's no good solution for global uniqueness.  It can be mitigated
via a 128-bit ID.  That means the probability of two IDs being the same
is one in 2^64, which is rare enough to not really worry about.

> be detectable with an IP address check.

  I'm not sure.  Maybe I'm missing something.  If a proxy is about to
send radsec traffic for realm A to IP address B, can't it do a DNS
lookup on IP address B to see if realm A is listed?

  Yes.

  Alan DeKok.

---

From: Stefan Winter <[email-masked]>
Date: Wed, 24 Jul 2013 16:48:16 +0200

Hi,

> lookup on IP address B to see if realm A is listed?

Well, if realmA (or anyone for that matter) wants to DoS proxy B, then
it can also see which realms B actually serves (say, "therealrealm"),
and publish an article stating "free net access for username@therealrealm".

B would have an IP reverse with the indication that it *is* serving that
realm - and the TLS connection would come up. The reverse IP check
doesn't help with that.

Besides that, I also don't like that an IP's reverse DNS lookup which is
intended to provide a *host*name would now be re-used in the different
semantics of providing a *realm*name.

>>    firewall, rate-limiting new connection attempts, etc.).

Okay, will be in -08.

Greetings,

Stefan Winter

Tel: +352 424409 1
Fax: +352 422473

---

From: Stefan Winter <[email-masked]>
Date: Wed, 24 Jul 2013 16:56:57 +0200

Hi,

> wanting free net access contacts proxy B.  Repeat for 10,000 domains,

It also just occured to me that your attack is too complex to be useful
for an attacker; it would require actual humans lured into trying to log
into realmA in masses, so that their login attempts overload proxy B. It
would require hundreds or thousands of humans trying to log in per
second so that proxy B feels the load.

If realmA really wants to DoS proxy B, they would take note of proxy B's
IP address and port, and hire a cheap botnet. The botnet would simply
not care whether there is a reverse IP to check.

In short... once your server is up in the open, it can be contacted by
anyone, like it or not. An implementation needs to cope with that.

Greetings,

Stefan Winter

Tel: +352 424409 1
Fax: +352 422473

---

From: Sam Hartman <[email-masked]>
Date: Wed, 24 Jul 2013 12:29:28 -0400

Also, how is this valuable at all without cryptographic binding of the
cert to the IP address?
We cannot be planning to have security of RADIUS over TLS devolve to an
IP ACL check.
I'm hoping I'm missing something here.

---

From: Alan DeKok <[email-masked]>
Date: Wed, 24 Jul 2013 20:57:56 -0400

> We cannot be planning to have security of RADIUS over TLS devolve to an

  It won't.  Maybe I'm missing something.

  I just want to ensure that we have a *simple* way to check if an IP
address can accept requests for a realm.  If the answer is "no", then
TLS shouldn't start.  If the answer is "yes", then TLS starts, with all
of the security mandated by TLS.

  i.e. It's a low-cost additional check before TLS.  I never proposed it
*replace* TLS.

  I'm worried about administrators mis-configuring realms.  "Hey,
company X says that server Y is a RADIUS proxy.  We'll list it for our
realm Z!"

  And then all of the traffic for these idiots goes to server Y.  No one
figures out this is wrong until after the TLS handshake.  Which can give
a DoS attack on server Y.

  A *simple* double-check can help prevent this.  The proxies get told
to go to server Y.  So they ask it.  "Do you handle realm Z"?  When the
answer is "no", they know to go bug realm Z to fix their business
relationships.  They DON'T bug server Y with useless requests.

  Alan DeKok.

---

From: Alan DeKok <[email-masked]>
Date: Wed, 24 Jul 2013 21:04:15 -0400

> second so that proxy B feels the load.

  I suppose there are no companies with 10M customers, and imperfect
administrators?

> not care whether there is a reverse IP to check.

  I'm not trying to make perfect security.  I'm trying to make it
cheaper to catch mistakes.

> anyone, like it or not. An implementation needs to cope with that.

  Yes.  There's no question there.

  I would like to discover *misconfiguration* quickly.  A reverse IP
check is cheap.  It's more expensive to have 100K customers hit your
proxy because an admin mis-configured something.

  Alan DeKok.

---

From: Stefan Winter <[email-masked]>
Date: Thu, 25 Jul 2013 09:23:32 +0200

Hi,

>   I suppose there are no companies with 10M customers, and imperfect

Even 10M customers distribute their login timestamps over the day; any
reasonable server should be able to cope with that.

And to be honest, I'd expect an ISP of this size not do such stupid
mistakes, or at least to find out about it *very* shortly after it
happened because lots of customers will be unable to authenticate when
it's happened.

> proxy because an admin mis-configured something.

Reverse IP checks are also "something completely different", and the
semantics of hostname identification is uneasy. Imagine a proxy with IP
1.2.3.4 who is AAA proxy for "cnn.com" "microsoft.com" "yahoo.com"
"telekom.de" and has corresponding PTRs to fulfill your proposed
reverse-matching. Now someone looks up the reverse of that IP address
for totally non-AAA reasons and finds that the IP address claims to be
all four domains. Huh?

Unless we invent a new PTR-like construct just for the AAA
misconfiguration catching purpose, I'm against using such a mechanism.
And frankly, inventing an own DNS RR just for that seems like enormous
overkill to me.

I take your point on hammering the server with misconfigured discovery
entries though. I see it as a deficiency in the current spec,
particularly section 2.1.1.2: Definition of Conditions for Retry/Failure

That section basically states: if client and server found out they don't
like each other, ignore the entry and try the next.

Which is fine in that instant, but doesn't speak about the persistence
of ignoring the target.

I've added a new last sentence to my working copy to ensure that there
is some backoff time:

"  If the TLS session setup to a discovered target does not succeed,
   that target (as identified by IP address and port number) SHOULD be
   ignored from the result set of any subsequent executions of the
   discovery algorithm at least until the target's Effective TTL has
   expired or until the entity which executes the algorithm changes its
   TLS context to either send a new client certificate or expect a
   different server certificate."

This provides an extra level of rate-limiting; well-behaved clients will
stop hammering the target for a while, independently of the NAI realm
that led to that server. I.e. one hotspot generates one TLS session
request, and will then be quiet for a significant amount of time.

I believe this takes the heat out of such pathological misconfigurations.

I know, this conflates the Effective TTL timer with something that's
unrelated to DNS: the retry should happen when the server-side changed
their TLS cert, because then it might work. But that condition can't be
signalled, so the timeout that DNS provides us with is the best we have.
The alternative would be to make that particular backoff time a
configuration variable.

Greetings,

Stefan Winter

Tel: +352 424409 1
Fax: +352 422473