Subject: [BEHAVE] ALTERNATE-SERVER and anycast

From: Philip Matthews <[email-masked]>
Date: Wed, 2 Apr 2008 22:22:24 -0400

At the last BEHAVE meeting, it was suggested that one use of ALTERNATE- 
SERVER was to support anycast discovery of TURN servers. I have been  
doing some inquiries, and it seems that there might be a problem with  
this idea.

The way I had been thinking this would work is that a client would  
send an Allocate request to an anycast address, and would receive an  
error response back with an error code of 300 Try Alternative and an  
ALTERNATE-SERVER attribute giving the unicast address of the server to  
contact.

The problem is that it seems there is no guarantee that two  
consecutive packets will be delivered to the same anycast server. The  
only thing that is guaranteed is a single round-trip exchange[1]. This  
means that the above idea does not work because:
a) It is not possible to contact an anycast TURN server using TCP or  
TLS over TCP, since the handshake will not necessarily complete with  
the same server.
b) It is also not possible to contact an anycast TURN server using  
UDP, since STUN says that an ALTERNATE-SERVER reply must be  
authenticated, and authentication in TURN requires a 4-packet exchange.

So this straight-forward method of doing anycast lookup in TURN does  
not seem to work.

There are probably ways to fix this by changing how ALTENATE-SERVER  
works in TURN, but I am currently inclined to forget about anycast for  
TURN right now, and let someone add it later.

Comments?

- Philip

[1] Private info received from David Oran. I have asked for a public  
reference.

_______________________________________________
Behave mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/behave

---

From: Hadriel Kaplan <[email-masked]>
Date: Wed, 2 Apr 2008 23:34:43 -0400

I am in still in favor of it, specifically for anycast support.

Can you describe the cases it won't be delivered to the same instance that you've been told of?  In my experience (which was many years in the L3 routing world, fwiw, but nowhere near as much as either of the Dave's left pinkies), anycasting fails to deliver a packet to the same server "instance" for one of two reasons I can think of off the top of my head:
        1) The network topology changes, for example the routing cost to an instance's prefix changes, or an L3 protection mechanism kicked in and temporarily selected a higher-cost path, or whatever.
        2) A router or some middle-box is doing some form of load-balancing.

For (1), it is either a transient rare event (from our use-case timescale perspective) which stabilizes, which won't matter for this use-case; or it is due to a route flap event, and route flaps take enough time to flap again, in 99% of the cases, that for this use-case timescale it won't matter.  Just as long as the mechanism is resilient to the event occurring once in its full handshake - in other words, so long as the second-trip can fail to reach the same server instance but that just triggers another round-trip restart, then (1) isn't an issue for a massive probability.  I think that is the case for ALT-SERVER use, right?  That it survives such a thing happening gracefully?

For (2), all such modern equipment I know of performs hashing or some form of keeping the same "flows" going to the same specific L3 next-hops, and do so using at least IP-dest in their decision, or IP-dest+source, or more, such that any flows to the same anycast dest address will get to the same instance virtually all of the time.  Having said that, there have been some (rather infamous) cases of ECMP or L3 load-balancing in routers which did NOT follow that model, and forwarded packets in a round-robin fashion, but I was under the impression they got "called-out" and all their hardware has since been upgraded to fix it years ago.  And when I say "anycast", I mean an IPv4 unicast address/prefix that happens to be a virtual shared address for multiple hosts, and advertised into the routing topology from multiple physical locations - I do not mean IPv6 true anycasting, and I have no idea what the modern equipment does for that or if it matters. (I've been out of routing for several years)

Regardless, I think all we need is a graceful way for the client to detect a pathological condition, should the worst happen. (ie, a way for it to give up re-tries and use an alternate server altogether)  No?

-hadriel

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Philip Matthews
> Sent: Wednesday, April 02, 2008 10:22 PM
> To: Behave WG
> Cc: Mark Townsley; David R Oran; David Ward
> Subject: [BEHAVE] ALTERNATE-SERVER and anycast
>

---

From: Cullen Jennings <[email-masked]>
Date: Thu, 3 Apr 2008 03:54:18 -0700

> TURN right now, and let someone add it later.
Sure, what you are proposing here won't work for the reasons you point  
out. I don't know when the WG decided the ALTERNATIVE-ERVER reply MUST  
be authenticated but that seems like it is not going to work for  
anycast. There are more than a few things wrong with this - for one I  
suspect the authentication you are talking about is for the server to  
authenticate the client not for the client to authenticate the server.  
As an alternative you might want to consider is something like what I  
had previously email you - I'm sure lots of other ways would work too  
and I don't really care which way we do it. No doubt the thing I  
emailed has some issues that would need to be worked out but I think  
it is closer to working.

On Apr 2, 2008, at 3:41 PM, Cullen Jennings wrote:
>
>
> 1) client is configured with TURN server of turn.example.com
>
> 2) client does an SRV lookup and finds there is no TCP service but  
> there is a UDP service at 1.1.1.1
>
> 3) client sends an UDP packet to 1.1.1.1 which happens to be an  
> anycast IP.
>
> 4) Some server gets this and responds with a redirect to 2.2.2.2  
> along with info to use TLS. The address 2.2.2.2 is not an anycast  
> address but goes to a single machine with a TURN server
>
> 5) client does TLS to 2.2.2.2. Server presents certificate with name  
> turn.example.com
>
> 6) client check certificate matches the name that it has in it's  
> config.
>
> This approach is susceptible to being DOSed by a well timed response  
> to the UDP request to 1.1.1.1 but seemed like a risk we were willing  
> to live with. Presumably some secret material would be copied from  
> request in step 3 to the response in step 4 to force the attacker to  
> be on-path.

If the client doesn't care about server authentication, then it can  
ignore TLS and do much the same with just UDP.

It's going to be very difficult to add something like this in later so  
I would argue we should deal with having at least enough placeholders  
in the base draft that we can add it later - to do this we will likely  
need to figure it out well enough that we should just add it to the  
base draft instead of trying to do it as an extension.

Cullen <with my individual contributor hat on>

---

From: Philip Matthews <[email-masked]>
Date: Thu, 3 Apr 2008 09:21:23 -0400

I am still trying to dig into the reasons the experts think that multi- 
packet exchanges won't work. My initial reaction is exactly the same  
as yours.

However, I also wondering how keen people are to have anycast support  
in TURN? I have heard that there is a draft in preparation that will  
discuss anycast discovery of servers, but it won't be out for a month  
or so, and after that there will presumably be some debate on the  
details. So this may turn out to be an area where there is not yet  
IETF consensus. If this is the case, are people willing to delay the  
TURN draft while we sort out the anycast details?

- Philip

On Wed, 2-Apr-08, at 23:34 , Hadriel Kaplan wrote:
> Can you describe the cases it won't be delivered to the same  
> instance that you've been told of?  In my experience (which was many  
> years in the L3 routing world, fwiw, but nowhere near as much as  
> either of the Dave's left pinkies), anycasting fails to deliver a  
> packet to the same server "instance" for one of two reasons I can  
> think of off the top of my head:
>        1) The network topology changes, for example the routing cost  
> to an instance's prefix changes, or an L3 protection mechanism  
> kicked in and temporarily selected a higher-cost path, or whatever.
> For (1), it is either a transient rare event (from our use-case  
> timescale perspective) which stabilizes, which won't matter for this  
> use-case; or it is due to a route flap event, and route flaps take  
> enough time to flap again, in 99% of the cases, that for this use- 
> case timescale it won't matter.  Just as long as the mechanism is  
> resilient to the event occurring once in its full handshake - in  
> other words, so long as the second-trip can fail to reach the same  
> server instance but that just triggers another round-trip restart,  
> then (1) isn't an issue for a massive probability.  I think that is  
> the case for ALT-SERVER use, right?  That it survives such a thing  
> happening gracefully?
>
> For (2), all such modern equipment I know of performs hashing or  
> some form of keeping the same "flows" going to the same specific L3  
> next-hops, and do so using at least IP-dest in their decision, or IP- 
> dest+source, or more, such that any flows to the same anycast dest  
> address will get to the same instance virtually all of the time.   
> Having said that, there have been some (rather infamous) cases of  
> ECMP or L3 load-balancing in routers which did NOT follow that  
> model, and forwarded packets in a round-robin fashion, but I was  
> under the impression they got "called-out" and all their hardware  
> has since been upgraded to fix it years ago.  And when I say  
> "anycast", I mean an IPv4 unicast address/prefix that happens to be  
> a virtual shared address for multiple hosts, and advertised into the  
> routing topology from multiple physical locations - I do not mean  
> IPv6 true anycasting, and I have no idea what the modern equipment  
> does for that or if it matters. (I've been out of routing for  
> several years)
>
> Regardless, I think all we need is a graceful way for the client to  
> detect a pathological condition, should the worst happen. (ie, a way  
> for it to give up re-tries and use an alternate server altogether)   
> No?
>
> -hadriel
>

---

From: Philip Matthews <[email-masked]>
Date: Thu, 3 Apr 2008 09:43:09 -0400

On Thu, 3-Apr-08, at 06:54 , Cullen Jennings wrote:
> Sure, what you are proposing here won't work for the reasons you  
> point out. I don't know when the WG decided the ALTERNATIVE-ERVER  
> reply MUST be authenticated but that seems like it is not going to  
> work for anycast.

Sorry. I should have provided a reference. draft-ietf- 
behave-3489bis-15, section 11 says:

11.  ALTERNATE-SERVER Mechanism

    This section describes a mechanism in STUN that allows a server to
    redirect a client to another server.  This extension is optional,  
and
    a usage must define if and when this extension is used.  To prevent
    denial-of-service attacks, this extension MUST only be used in
    situations where the client and server are using an authentication
    and message-integrity mechanism.

    A server using this extension redirects a client to another server  
by
    replying to a request message with an error response message with an
    error code of 300 (Try Alternate).  The server MUST include a
    ALTERNATE-SERVER attribute in the error response.  The error  
response
    message MUST be authenticated, which in practice means the request
    message must have passed the authentication checks.

    [Rest of section omitted]

> There are more than a few things wrong with this - for one I suspect  
> the authentication you are talking about is for the server to  
> authenticate the client not for the client to authenticate the  
> server. As an alternative you might want to consider is something  
> like what I had previously email you - I'm sure lots of other ways  
> would work too and I don't really care which way we do it. No doubt  
> the thing I emailed has some issues that would need to be worked out  
> but I think it is closer to working.
>
>> 5) client does TLS to 2.2.2.2. Server presents certificate with  
>> name turn.example.com
>>
>> This approach is susceptible to being DOSed by a well timed  
>> response to the UDP request to 1.1.1.1 but seemed like a risk we  
>> were willing to live with. Presumably some secret material would be  
>> copied from request in step 3 to the response in step 4 to force  
>> the attacker to be on-path.
>
>
> It's going to be very difficult to add something like this in later  
> so I would argue we should deal with having at least enough  
> placeholders in the base draft that we can add it later - to do this  
> we will likely need to figure it out well enough that we should just  
> add it to the base draft instead of trying to do it as an extension.
>

This works if the client can contact the server using both UDP and  
TCP. As you know, we are separately discussing the case of a client  
that can only contact TURN servers using TCP -- this is the proposed  
justification for TCP transport for a UDP allocation. If we decide to  
keep TCP transport for a UDP allocation (and right now, I am seeing  
more support for keeping it than dropping it -- but we will discuss  
this next week on the conf call), then I have a problem with something  
like this as the proposed anycast solution.

I will note that there are a couple of ways to achieve something very  
similar to this without using anycast. The first option is that  
1.1.1.1 is not an anycast address, but is a unicast address of a TURN  
server that doesn't actually accept allocations, but always redirects  
clients to other servers (using ALTERNATE-SERVER). The second is to  
use DNS to direct the client to a closer or otherwise appropriate  
server.

- Philip

---

From: Cullen Jennings <[email-masked]>
Date: Thu, 3 Apr 2008 07:22:16 -0700

>     [Rest of section omitted]

right - thanks

> use DNS to direct the client to a closer or otherwise appropriate
The point of the anycast is not to just spread stuff over multiple  
servers - a DNS A record could do that. The point is to solve in some  
way the "close" problem where the client ends up using a TURN server  
that is topologically close to the client. Anycast is a one appealing  
way to address that. What you are proposing does not seem to address  
the issues of getting to a TURN server that is topologically close to  
the client.

---

From: Cullen Jennings <[email-masked]>
Date: Thu, 3 Apr 2008 07:23:37 -0700

On Apr 3, 2008, at 6:21 AM, Philip Matthews wrote:
> in TURN?

I have not gone back and listened to the meeting recording from last  
ietf meeting but my vague recollection was the room wanted to keep  
ALTERNATVE-SERVER in to support anycast. At this point I think it will  
just be faster to go do it than argue about if we really want it or not.

---

From: Eric Rescorla <[email-masked]>
Date: Thu, 03 Apr 2008 07:30:33 -0700

At Thu, 3 Apr 2008 09:21:23 -0400,
> packet exchanges won't work. My initial reaction is exactly the same  

There's nothing complicated here:
The Internet routing system does not guarantee that two packets
sent to the same anycast address from the same source address
will be delivered to the same destination. To the extent to which
it's currently the case that that usually happens, that's an
artifact of today's routing systems. It's bad practice to design
new systems such that they depend on this behavior, because we
may wish to change it in the future.

-Ekr

---

From: Eric Rescorla <[email-masked]>
Date: Thu, 03 Apr 2008 07:33:38 -0700

At Thu, 3 Apr 2008 09:43:09 -0400,
>     message must have passed the authentication checks.

Yeah, but this doesn't contain the actual security requirement,
which is that the client be able to get data origin authentication
from the server. That does not require anti-replay for the client
request but only for the server response. I.e., it does not require
a challenge from the server.

-Ekr

---

From: Hadriel Kaplan <[email-masked]>
Date: Thu, 3 Apr 2008 11:26:55 -0400

> Of Eric Rescorla

Yeah that section confused me.  It said "to prevent denial of service attacks", and I wasn't sure if it meant (a) to prevent a malicious device from spoofing a client, in the hopes of getting the server to send responses to that legitimate client(s); or (b) if it meant to prevent a malicious device from spoofing a server and redirecting clients to bogus addresses.

For (a), I thought there was already a transaction-id to prevent that, and if it's resource exhaustion we're worried about, challenging is not going to help.
For (b), a server challenge won't stop that (right?), and DNS has this same problem already (sans DNSSEC), so I personally think it's a red herring.  It ought to be a MAY strength methinks.

-hadriel

---

From: Eric Rescorla <[email-masked]>
Date: Thu, 03 Apr 2008 08:51:34 -0700

sAt Thu, 3 Apr 2008 11:26:55 -0400,
Hadriel Kaplan wrote:
> Yeah that section confused me.  It said "to prevent denial of
> service attacks", and I wasn't sure if it meant (a) to prevent a
> malicious device from spoofing a client, in the hopes of getting the
> server to send responses to that legitimate client(s); or (b) if it
> meant to prevent a malicious device from spoofing a server and
> redirecting clients to bogus addresses.
>>
> For (a), I thought there was already a transaction-id to prevent
> that, and if it's resource exhaustion we're worried about,
> challenging is not going to help.  For (b), a server challenge won't
> stop that (right?), and DNS has this same problem already (sans
> DNSSEC), so I personally think it's a red herring.  It ought to be a
> MAY strength methinks.

I think your security analysis is right. I'll decline to comment on
ought :)

-Ekr

---

From: Hadriel Kaplan <[email-masked]>
Date: Thu, 3 Apr 2008 12:11:16 -0400

> From: Eric Rescorla [mailto:[email-masked]]
>

I think they will find all kinds of crap breaking if they try to change that behavior (people do far worse with anycasts than what we're proposing), but ok - would graceful detection and avoidance of such a case resolve this?

Like if it's using TCP/TLS, then it's possible the connection establishment will fail, or it's possible a sent request will reset the connection, but that is possible regardless of anycast and needs to be handled.

If it's using UDP and a challenge, then the challenge-responding request may well get an error response (e.g., 401 or 438?), and currently I _think_ section 10.2.3 of 3489bis-15 would just have the client keep retrying, which could be bad if anycast is broken - really it could be bad for unicast too (we've had this problem in SIP).  So maybe we should put in wording in that section to constrain that failure case, such that it moves on to another server found through SRV or configuration or whatever.

For example, if more than three 401 or 438 responses are received without a success response, consider the transaction to have failed without a response and follow section 9:
   When following these procedures, if the STUN transaction times out
   without receipt of a response, the client SHOULD retry the request to
   the next server in the ordered defined by RFC 2782.

-hadriel

---

From: Eric Rescorla <[email-masked]>
Date: Thu, 03 Apr 2008 09:23:20 -0700

At Thu, 3 Apr 2008 12:11:16 -0400,
Hadriel Kaplan wrote:
> I think they will find all kinds of crap breaking if they try to
> change that behavior (people do far worse with anycasts than what
> we're proposing),

Yes, but at their own risk. That's different from us encouraging it.

> but ok - would graceful detection and avoidance of
> such a case resolve this?
>
> Like if it's using TCP/TLS, then it's possible the connection
> establishment will fail, or it's possible a sent request will reset
> the connection, but that is possible regardless of anycast and needs
> to be handled.

Right, but it's possible that this will *never* work.

Consider a load balancing system that has two paths and uses a packet
hash to determine which path it will take. If the SYN and the SYN/ACK
end up on different paths, then anycast will totally break the system.
You'll not be able to form a TCP connection at all.

Maybe I'm missing something, but why use anycast at this layer at
all. Wouldn't it be easier to have the TURN servers be unicast and use
anycast (or just load balanced) DNS resolution instead, with different
servers giving different responses?

-Ekr

---

From: Hadriel Kaplan <[email-masked]>
Date: Thu, 3 Apr 2008 12:50:55 -0400

> Yes, but at their own risk. That's different from us encouraging it.

Yup, I concur. :)

> You'll not be able to form a TCP connection at all.

Right, and it's possible you'll never be able to create a TCP or TLS connection to a unicast server you want to, period.  Section 9 deals with that by having it try to another server I think (because the STUN transaction would fail).  I don't disagree that an anycast would need a unicast alternate, and unicasts need unicast alternates.  Maybe we could just mandate that there be one for an anycast use model, with an explanation of why?

> servers giving different responses?

It's not really just load balancing and availability, it's about distance locality (routing topology distance).  DNS gives you the former, but not the latter unless either your DNS has topological knowledge, or you deploy DNS everywhere you deploy STUN/TURN servers, and you don't do recursive queries.  I don't believe those are really tenable, but IANAE.  Regardless, all that would do is replace the ALT-SERV mechanism with DNS, and if that's kosher, then so should doing ALT-SERV without a challenge and over UDP.

-hadriel

---

From: Eric Rescorla <[email-masked]>
Date: Thu, 03 Apr 2008 10:17:45 -0700

At Thu, 3 Apr 2008 12:50:55 -0400,
Hadriel Kaplan wrote:
> Right, and it's possible you'll never be able to create a TCP or TLS
> connection to a unicast server you want to, period. 

Yes, but not because of the network being actually completely functional,
just not in a way you expect.

> Section 9 deals
> with that by having it try to another server I think (because the
> STUN transaction would fail).  I don't disagree that an anycast
> would need a unicast alternate, and unicasts need unicast
> alternates.  Maybe we could just mandate that there be one for an
> anycast use model, with an explanation of why?

Again, I think this encourages a dangerous practice.

> It's not really just load balancing and availability, it's about
> distance locality (routing topology distance).  DNS gives you the
> former, but not the latter unless either your DNS has topological
> knowledge, or you deploy DNS everywhere you deploy STUN/TURN
> servers, and you don't do recursive queries.  I don't believe those
> are really tenable, but IANAE.  Regardless, all that would do is
> replace the ALT-SERV mechanism with DNS, and if that's kosher, then
> so should doing ALT-SERV without a challenge and over UDP.

Pace DNSSEC.

-Ekr

---

From: Mark Townsley <[email-masked]>
Date: Thu, 03 Apr 2008 15:29:03 +0200

Philip Matthews wrote:
> I am still trying to dig into the reasons the experts think that 
> multi-packet exchanges won't work. My initial reaction is exactly the 
> same as yours.
>
I can say for certain that autodiscovery, is something that in many 
cases we have functionally separated out. There is often more than one 
way to skin that particular cat, there are various tradeoffs, and lots 
of opinions on what those are. Consensus is sometimes difficult to 
reach, unfortunately.

- Mark
>> Can you describe the cases it won't be delivered to the same instance 
>> that you've been told of?  In my experience (which was many years in 
>> the L3 routing world, fwiw, but nowhere near as much as either of the 
>> Dave's left pinkies), anycasting fails to deliver a packet to the 
>> same server "instance" for one of two reasons I can think of off the 
>> top of my head:
>> to an instance's prefix changes, or an L3 protection mechanism kicked 
>> in and temporarily selected a higher-cost path, or whatever.
>> enough time to flap again, in 99% of the cases, that for this 
>> use-case timescale it won't matter.  Just as long as the mechanism is 
>> For (2), all such modern equipment I know of performs hashing or some 
>> form of keeping the same "flows" going to the same specific L3 
>> next-hops, and do so using at least IP-dest in their decision, or 
>> IP-dest+source, or more, such that any flows to the same anycast dest 
>> ECMP or L3 load-balancing in routers which did NOT follow that model, 
>> and forwarded packets in a round-robin fashion, but I was under the 
>> impression they got "called-out" and all their hardware has since 
>> been upgraded to fix it years ago.  And when I say "anycast", I mean 
>> an IPv4 unicast address/prefix that happens to be a virtual shared 
>> address for multiple hosts, and advertised into the routing topology 
>> from multiple physical locations - I do not mean IPv6 true 
>> anycasting, and I have no idea what the modern equipment does for 
>> that or if it matters. (I've been out of routing for several years)
>>
>> for it to give up re-tries and use an alternate server altogether)  No?
>>
>> -hadriel
>>

---

From: Philip Matthews <[email-masked]>
Date: Mon, 7 Apr 2008 17:08:15 -0400

After some off-line discussions, it seems the following anycast  
procedure might work:

1. The TURN client uses ordinary TURN Allocate procedures to contact a  
server at address X which, unbeknownst to it, is an anycast address.  
The client can use UDP, TCP, or TLS transport. When replying, the  
server uses anycast address X as the source address of its packets  
(otherwise, this won't work through NATs).

2. The server uses ALTERNATE-SERVER to redirect the client to its own  
unicast address.

3. The client doesn’t know this is an anycast redirection, but just  
follows ALTERNATE-SERVER rules to contact new server.

Step 1 requires that the client be able to complete a multi-packet  
exchange with the same server. In other words, this requires that all  
the packets from the client get delivered to the same server (and not  
two different servers). It seems that, for the short packet exchange  
required, this is likely but not guaranteed.

However, the client can easily detect when there is a problem.  
Specifically:
* If UDP transport is used, then a correct transaction is: send  
Allocate with no credentials, gets back a 401 Unauthorized, resend  
with correct credentials, get back Allocate response. However, if the  
second Allocate is delivered to a different server, then the second  
server will reject with a 401 Unauthorized (or perhaps a 438 Stale  
Nonce).
* If TCP or TLS over TCP transport is used, then the TCP connection  
will eventually reset because the new server will not have any  
connection state and will either reply with a TCP reset or just ignore  
the packets.

In either of these cases, if the client simply retries the request  
from scratch, then it will be highly likely to succeed the second time.

So with some simple rules for ALTERNATE-SERVER processing, plus a  
little tweak to the rules for when to retry an Allocate request, it  
seems that we may be able to get anycast to work.

Comments?

- Philip

---

From: Philip Matthews <[email-masked]>
Date: Mon, 7 Apr 2008 17:49:45 -0400

A few more comments about ALTERNATE-SERVER. If anycast and redirection  
to a geographically-closer server are our intended uses for ALTERNATE- 
SERVER, then I think the more complex rules introduced in TURN-07 for  
ALTERNATE-SERVER are no longer required. Specifically, TURN-07  
proposed that ALTERNATE-SERVER could appear alongside any error code  
(not just 300 Try Alternate), which conflicts with draft-ietf-behave- 
rfc3489bis. This allowed the client to see the error code and decide  
if it wanted to retry with a modified request to the same server,  
rather than contacting the alternate server.

This was appropriate when we were thinking of ALTERNATE-SERVER as a  
way to find a server within a cluster that could handle a particular  
feature (e.g., a particularly large BANDWIDTH allocation). However,  
with the new view, this now seems less appropriate.

So I am now thinking that TURN should revert to using ALTERNATE-SERVER  
only with 300 Try Alternate, which will simplify the error processing  
rules somewhat.

The new rules would be:
* A server redirects by using error code 300 (Try Alternate) and  
including an ALTERNATE-SERVER attribute.
* The client SHOULD re-attempt the Allocate request with the alternate  
server, rather than trying the next server in the SRV list.
* That alternate server MAY redirect to a second alternate, etc,  
however the client MUST NOT try the same server twice within a 5  
minute period.
* If the chain of redirections ultimately fails, the client MAY resume  
processing the SRV list.

Comments?

- Philip

---

From: Eric Rescorla <[email-masked]>
Date: Mon, 07 Apr 2008 15:04:05 -0700

At Mon, 7 Apr 2008 17:08:15 -0400,
> from scratch, then it will be highly likely to succeed the second time.

As I said in my comments to Hadriel, this is not guaranteed to work
if the first and third packets of the TCP handshake go to different
places (e.g., because of hash-based PPLB).

> seems that we may be able to get anycast to work.

Yes. Stop trying to fix this. If it turns out we actually need this
we can do something new later.

-Ekr

---

From: Hadriel Kaplan <[email-masked]>
Date: Mon, 7 Apr 2008 18:27:33 -0400

This is an excellent idea, and a good compromise.
-hadriel

> From: Philip Matthews [mailto:[email-masked]]
> Sent: Monday, April 07, 2008 5:08 PM
> To: Behave WG
> Cc: Hadriel Kaplan; David R Oran; Eric Rescorla; Cullen Jennings
> Subject: Re: [BEHAVE] ALTERNATE-SERVER and anycast
>
> 3. The client doesn't know this is an anycast redirection, but just

---

From: "Dan Wing" <[email-masked]>
Date: Mon, 7 Apr 2008 15:45:23 -0700

> places (e.g., because of hash-based PPLB).

PPLB = per packet load balancing, I presume.

Can you point at a per-packet load balancer that would purposefully
or accidentally (through misconfiguration) send packets with the same
5-tuple to different servers?

> we can do something new later.

It has been brought up that we won't be able to introduce
Alternate-Server later if existing clients would ignore
it.  We could only, at that later date, use DNS to steer 
TURN clients to certain TURN servers.

-d

---

From: Eric Rescorla <[email-masked]>
Date: Mon, 07 Apr 2008 16:25:11 -0700

At Mon, 7 Apr 2008 15:45:23 -0700,
> 5-tuple to different servers?

Why is that relevant?

As I noted earlier, the issue isn't what the current network
does but rather that there is no guarantee that a future network
will not do this in the future. Designing a system that relies
on a property of the network that is not one of its architectural
invariants is bad, because it sticks people in the future with
the choice of breaking the system or retaining the invariant
even if it's no longer a good idea.

> TURN clients to certain TURN servers.

I didn't say anything about introducing alternate server. What I 
said was that if anycast was needed in the future, we could add
that.

-Ekr

---

From: Hadriel Kaplan <[email-masked]>
Date: Mon, 7 Apr 2008 19:27:38 -0400

> Of Dan Wing

Yes it will be much easier to know when to stop servers from doing it if the Internet routing model changes, than to know when to start doing it when it doesn't. ;)

But anyway this is whole argument is a non sequitur against ALTERNATE-SERVER.  There are other reasons for having it, regardless of anycasting.

For example:
1) Wanting to gracefully shed load temporarily from a server, and use a local alternate backup; for example to avoid service outages for a planned maintenance. (DNS caching issues would make that very difficult)
2) Having a couple load-balancers redirect clients to one of possibly hundreds of servers, without having to change DNS every time a new server is added. (e.g., think "Google")
3) Doing anycasting to servers in a physically or virtually constrained network topology; for example a 3G or 4G access provider, or a broadband provider, which can offer TURN servers to their customers while also controlling their network routing equipment.
4) Having an intelligent redirector which redirects clients based on their user/realm, or type of allocations, or whatever.  For example, if the server can do IPv4-IPv4 relaying but has a twin server which does IPv4/v6.

-hadriel

---

From: Eric Rescorla <[email-masked]>
Date: Mon, 07 Apr 2008 16:38:50 -0700

At Mon, 7 Apr 2008 19:27:38 -0400,
> 1) Wanting to gracefully shed load temporarily from a server, and use a local alternate backup; for example to avoid service outages for a planned maintenance. (DNS caching issues would make that v…

As several people have noted, this creates the possibility of loops. Moreover, there are
standard procedures here, namely load balancers.

> 2) Having a couple load-balancers redirect clients to one of possibly hundreds of servers, without having to change DNS every time a new server is added. (e.g., think "Google")

But of course this is not how Google works.

-Ekr

---

From: "Dan Wing" <[email-masked]>
Date: Mon, 7 Apr 2008 16:49:36 -0700

> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Eric Rescorla
> Sent: Monday, April 07, 2008 4:25 PM
> To: Dan Wing
> Cc: 'Behave WG'; 'David R Oran'; 'Philip Matthews'

ALTERNATE-SERVER does not require the TURN servers to use
anycast.  If, in some future where anycasting works 
differently, that anycasted TURN server can avoid using 
ALTERNATE-SERVER and can do something new and fancy.

Today, there appears to be at least one company
that has found anycast with TCP works:
http://www.nanog.org/mtg-0606/pdf/matt.levine.pdf 

> said was that if anycast was needed in the future, we could add

So you're saying we should use something other than anycast
to justify inclusion of the ALTERNATE-SERVER attribute ?

-d

---

From: Hadriel Kaplan <[email-masked]>
Date: Mon, 7 Apr 2008 23:24:58 -0400

> > 1) Wanting to gracefully shed load temporarily from a server, and use a
> local alternate backup; for example to avoid service outages for a planned
> maintenance. (DNS caching issues would make that very difficult)
>

But I thought your argument against anycast was load-balancers could screw up how they load-balance!  (just kidding!:)

Anyway, no, load-balancers don't always work for this, if you mean of the inline variety.  They can - they just restrict things topologically.  I think you're thinking central data centers, and I'm thinking distributed PoP's/CO's, where inline load-balancers aren't always reasonably worth it.

> > 2) Having a couple load-balancers redirect clients to one of possibly
> hundreds of servers, without having to change DNS every time a new server
> is added. (e.g., think "Google")
>

Oh I have no idea how Google works for TURN servers, and didn't mean they did. (I realize they don't work that way for HTTP servers)  I just meant one *could* do it that way.

-hadriel

---

From: Eric Rescorla <[email-masked]>
Date: Mon, 07 Apr 2008 21:45:03 -0700

At Mon, 7 Apr 2008 16:49:36 -0700,
> http://www.nanog.org/mtg-0606/pdf/matt.levine.pdf 

For the third time, now: the issue is not what currently works
but what the architecture of the Internet guarantees.

> to justify inclusion of the ALTERNATE-SERVER attribute ?

Uh, no.

-Ekr

---

From: David R Oran <[email-masked]>
Date: Tue, 8 Apr 2008 10:05:38 -0400

On Apr 7, 2008, at 5:08 PM, Philip Matthews wrote:
> 1. The TURN client uses ordinary TURN Allocate procedures to contact  
> a server at address X which, unbeknownst to it, is an anycast  
> address. The client can use UDP, TCP, or TLS transport. When  
> replying, the server uses anycast address X as the source address of  
> its packets (otherwise, this won't work through NATs).
>
> 2. The server uses ALTERNATE-SERVER to redirect the client to its  
> own unicast address.
>
> exchange with the same server. In other words, this requires that  
> all the packets from the client get delivered to the same server  
> (and not two different servers). It seems that, for the short packet  
> exchange required, this is likely but not guaranteed.
>
> with correct credentials, get back Allocate response. However, if  
> the second Allocate is delivered to a different server, then the  
> second server will reject with a 401 Unauthorized (or perhaps a 438  
> Stale Nonce).
> connection state and will either reply with a TCP reset or just  
> ignore the packets.
>
I'd far prefer a completely separate discovery exchange with UDP. If  
the only reason we're doing this is to not delay publication of TURN,  
I think it's a sub-optimal tradeoff. As a counter-proposal, how about  
not supporting anycast until we have a TURN protocol enhancement  
defined to do TURN server detection with anycast. Once we go down this  
path I suspect the "right thing" will never happen and we'll have yet  
another application protocol that has tight dependency on current  
routing behavior.

DaveO.

---

From: Cullen Jennings <[email-masked]>
Date: Tue, 8 Apr 2008 08:01:53 -0700

One comment. This is not a good way to do anycast for reasons  
previously given.

However, I would be very interested in understanding if anyone sees  
problems with

---

From: Jiri Kuthan <[email-masked]>
Date: Tue, 15 Apr 2008 23:19:32 +0200

> I am in still in favor of it, specifically for anycast support.

Sorry for such a late comment, still I'm hopeful better late than never.
I think use of anycast in this context may be useful and we shall not
preclude it in any way. There are surveys showing that from the
application point of view, instability (affinity) is a marginal issue.
See Ballani, Francis, Rtnsasamy: Measurement-based Deployment
Proposal for IP Anycast; Sigcomm 2006. I have seen similar observations
in other publications, alas I can't recall those right nowl.

-jiri

---

From: Jiri Kuthan <[email-masked]>
Date: Tue, 15 Apr 2008 23:25:56 +0200

> You'll not be able to form a TCP connection at all.

See my previous email -- it is a risk known empirically to be low.
I'm inclined to conclude to support it with a "use at your own risk"
disclaimer. Those who can benefit of it should be warned but not
precluded from using it.

> servers giving different responses?

That's not 100% perfect either as the DNS resolver from TURN server's
perspective may or may not be the actual client, whose proximity is
sought. Some publications suggest that practically this is mostly
fine. (Chen, Lim, Katz, Overton: on the stability of network distance
estimation, sigmetric perf. eval. rev. 2002)

-jiri

---

From: Jiri Kuthan <[email-masked]>
Date: Tue, 15 Apr 2008 23:37:04 +0200

> but what the architecture of the Internet guarantees.

I have made an interesting observation with the Internet: NATs, one of 
the topics of interest of this working group, have never been part of a 
guaranteed Internet architecture, they have been always the thing that 
currently works. I would be worried to underestimate this kind of things.

>> to justify inclusion of the ALTERNATE-SERVER attribute ?

So what's the actual suggestion?

Thanks,

-jiri

---

From: Eric Rescorla <[email-masked]>
Date: Tue, 15 Apr 2008 15:04:39 -0700

At Tue, 15 Apr 2008 23:25:56 +0200,
> disclaimer. Those who can benefit of it should be warned but not

Once again, the relevant question isn't whether this works in the
current network but whether it's guaranteed to work by the Internet
routing architecture. The answer to *that* question is "No", and
hence this is architecturally unsound.

-Ekr

---

From: Jiri Kuthan <[email-masked]>
Date: Wed, 16 Apr 2008 08:47:14 +0200

> hence this is architecturally unsound.

I respectfully disagree this argument makes a case (or even as many 
cases as many times it is repeated :-) ... I really read your point.).

The Internet is what it is and not what it is thought to be.
The NAT story, where IETF for the "guaranteed Internet architecture" 
reasons chose to ignore NATs, to find later they form the Internet more 
than an "architecture" may be a great eye-opening experience for anyone. 
Particularly with respect to the VoIP area, skype actually began to be 
the Internet application with running code whereas SIP began to be kind 
of architectural stuff.

I think empirical grounded publications, of which you have been given 
two references, show best what the Internet really is. In particular, 
they show that what is believed to be architectural problem doesn't 
occur with running code. I personally consider this research a very 
sound foundation for protocol design, and I don't think it can be 
challenged on the grounds of some routing architectures.

-jiri

---

From: Eric Rescorla <[email-masked]>
Date: Wed, 16 Apr 2008 07:25:30 -0700

At Wed, 16 Apr 2008 08:47:14 +0200,
> cases as many times it is repeated :-) ... I really read your point.).

You're entitled to your opinion, but since you're just repeating
your point again, there's no reason why I should expect it to 
be convincing.

> than an "architecture" may be a great eye-opening experience for anyone. 

Yes, strangely as a long-time participant, I'm quite aware of NATs.
I don't think that invalidates my point.

> occur with running code.

You claim you read my point, but you don't seem to understand it.  The
architectural point isn't that it happens with the current code, but
that if we assume that this is the only behavior that can happen, we
create a new invariant that will then constrain all future Internet
routing system design. Go sell the routing area that that's cool
and then we'll talk.

-Ekr

---

From: "Spencer Dawkins" <[email-masked]>
Date: Wed, 16 Apr 2008 09:25:51 -0500

I THINK I understand what's being said in this thread.

Just to up-level one notch, is there any reason that we have to mention TCP 
anycast at all in this specification?

Thanks,

Spencer

---

From: Jiri Kuthan <[email-masked]>
Date: Wed, 16 Apr 2008 17:25:49 +0200

Eric Rescorla wrote:
> routing system design. 

I don't think I misunderstand your point, I just mean that 
"architectural" arguments are weak as whole in lieu of running code 
doing things in a different way and being reserched by founded 
publications. If running code delivers and its measurement give us a 
different picture than an "architecture", there is then apparently 
something wrong in the "architecture".

Also, I don't think anyone was mandating here a requirement to do 
anycast. Instead, the suggestion here was to allow TURN to do it for 
those who wish to. I'm failing to see how allowing the anycast option in 
TURN can "constrain all future Internet routing system designs". It 
would be most helpful if you could elaborate on this.

Go sell the routing area that that's cool
> and then we'll talk.

Really rhetorics is not any helpful. It would be most helpful if you 
could continue this conversation by providing supporting arguments

-jiri

---

From: Eric Rescorla <[email-masked]>
Date: Wed, 16 Apr 2008 08:38:52 -0700

At Wed, 16 Apr 2008 17:25:49 +0200,
> something wrong in the "architecture".

I'm sorry, but this is simply not true. 

There is already existing hardware that breaks the invariants that
anycast TCP relies on. Moreover, there are known cases where its
useful to deploy such techniques. The fact that those techniques are
not currently in wide use does not mean that it's a good idea to
design systems that would preclude their use in the future.

The fact that you have measurements indicating that this is not a
widespread problem with the current designs is, frankly,
irrelevant. Using the argument you're advancing, I could as easily
argue that it was perfectly fine to build a system that relied on
grabbing all of 2/24, since after all nobody is using it right now.
An important part of good architecture is thinking about how
systems might be used in the future, not just how they are
used now.

> would be most helpful if you could elaborate on this.

I should have thought it would be fairly obvious. If, in
the future, we wish to deploy a routing system that breaks
the invariants that allows anycast TCP to function, we'll
hear from people that "oh, we need anycast TCP to work, so
you can't do that". It will have become a new invariant.

Given that there are plenty of alternatives that do not have
this side effect, designing a system which does is bad architecture.

> could continue this conversation by providing supporting arguments

It's not rhetoric. This document has to pass IESG review. Have
you asked them what they think about using anycast in this way?
      
-Ekr

---

From: Hadriel Kaplan <[email-masked]>
Date: Wed, 16 Apr 2008 14:52:33 -0400

I think we can ignore and avoid any debate about what ALTERNATE_SERVER may be used for if, as Spencer suggested and I thought as we talked about on the last call, we only discuss its anycast-specific use for UDP.  The Alt-server response may also be used for TCP, for other reasons, but we would discourage Anycasting-use of TCP in particular in the RFC.

With regards to using it for anycast, I know we debated it on the call, but I don't know why a simple UDP query-response without extraneous authentication isn't sufficient.  Yes, the server can be spoofed by a malicious attacker, such that the client goes somewhere it shouldn't.  Seems to me we have that with DNS too. Oh well.  It's up to the client if it wants to accept it or not.

We can always define a DNSSEC-style mechanism for TURN in the future; for example the server responds with some signature over the Alt-server address, so it can keep the signature statically without recomputing it every query, and the client can verify the signature by pulling a cert through TURN or HTTP from somewhere, etc., etc.  We don't need to do that for this TURN draft though, imho.

Would that be acceptable?

-hadriel

---

From: Jiri Kuthan <[email-masked]>
Date: Wed, 16 Apr 2008 21:58:58 +0200

> design systems that would preclude their use in the future.

Would you mind being more specific? It would be helpful to learn what 
type of equipment gets broken or breaks something else here. I mean some 
DNS root servers do anycast for a good reason and with good success. 
(This is really not 2/24 -- this is quite well understood *running 
code*.) Personally, if I was to choose between putting effort into 
supporting a case that is known and an unknown one and these two were 
for sure mutually exclusive, I would obviously prefer the former.

> systems might be used in the future, not just how they are

Admittedly, future-proof design is good-to-have.
However, I'm still not understanding how allowing for anycast in TURN 
would break <something> (whatever <something> ought to stand for) in the 
future. IMO, at worst, it can break itself, i.e., the TURN uses that 
chose to use anycast (even though I doubt that failure rate would be 
concerning based on the DNS/anycast reports).

It would be different if for example someone suggested that to 
accomodate anycast-based use-cases the Internet must have a new 
invariant, which is stable delivery of anycast traffic to a single 
destination. I don't think such a weak suggestion or similar has been 
made. In the field, all known anycast uses have taken this into account, 
  limit themselves to shortly-lived transactions with minimum context, 
and can cope with errors.

In fact, I believe that there are more applications than DNS that can 
benefit of anycast. anycast-powered TURN is particularly appealing to 
ASPs who are seeking to move RTP relays (or SBCs if you wish to) close 
to at least one of SIP call parties. (I guess that was the case Hadriel 
had in his mind; at least I do.)

> you can't do that". It will have become a new invariant.

Possibly we might have found the disconnect.
It is very agreeable that it makes little or actually no sense to demand 
TCP to function over anycast as good as over unicast. I don't think 
anyone is demanding this though. The residual risk with routing 
instability indeed remains there, it has proven marginal, and is to be 
traded off and entered at one's own risk for those who are willing to do so.

Concerns about some suggesting something like a "fixed anycast routing" 
are IMO of educational nature, and if considered important, may be well 
addressed by IP-tutorials and other educational material documenting 
possible risks. Which appears to be out-of-scope of this debate for me.

> this side effect, designing a system which does is bad architecture.

Actually I don't think the other alternative are equal. One of the key 
motivations for use of TURN with anycast is getting a service 
close-to-client (or at least not-getting-a-service-far-from-client). A 
survey of such may be found for example in "Freedman LAsksminarayanan, 
Mazieres: OASIS, Anycast for Any Service" (OASIS itself is based on 
anycast). Many of them typically rely on creating a "map of the 
Internet" (which has its own cans of worms), or rely on specialized 
clients (which may be in fact architecturally sounder, but quite hard to 
deploy with widely available clients).

> you asked them what they think about using anycast in this way?

It would be a surprise to me if participants of the IESG review were not 
on this mailing list.

-jiri

---

From: "Dan Wing" <[email-masked]>
Date: Wed, 16 Apr 2008 15:15:29 -0700

> [mailto:[email-masked]] On Behalf Of Hadriel Kaplan
> Sent: Wednesday, April 16, 2008 11:53 AM
> To: Behave WG
> Cc: [email-masked]; David R Oran; [email-masked]; Ward; 
> [email-masked]; Townsley
> Subject: [BEHAVE] ALTERNATE-SERVER inclusion
> 
> I think we can ignore and avoid any debate about what 
> ALTERNATE_SERVER may be used for if, as Spencer suggested and 
> I thought as we talked about on the last call, we only 
> discuss its anycast-specific use for UDP.  The Alt-server 
> response may also be used for TCP, for other reasons, but we 
> would discourage Anycasting-use of TCP in particular in the RFC.
> 
> With regards to using it for anycast, I know we debated it on 
> the call, but I don't know why a simple UDP query-response 
> without extraneous authentication isn't sufficient.  Yes, the 
> server can be spoofed by a malicious attacker, such that the 
> client goes somewhere it shouldn't. 

Such an attacker would have to know (or guess) the 96-bit
transaction id and know the client's source UDP port.

> Seems to me we have that with DNS too.  Oh well.  It's up to 
> the client if it wants to accept it or not.

Works for me.

And in the future if/when DNSSEC is deployed and TURN needs stronger
authentication (on par with DNSSEC), stronger authentication of
Alternate-Server could be added to TURN.  For example, in this DNSSEC future,
one mechanism would be for the TURN server to return a new attribute which
signs (part or all of) the STUN response with the TURN server's private key.
The TURN client could obtain the TURN server's public key via a
(DNSSEC-signed) DNS record.  That would allow the DNS server and the TURN
server to remain idempotent, and allow anycast to continue to be used.

-d

> We can always define a DNSSEC-style mechanism for TURN in the 
> future; for example the server responds with some signature 
> over the Alt-server address, so it can keep the signature 
> statically without recomputing it every query, and the client 
> can verify the signature by pulling a cert through TURN or 
> HTTP from somewhere, etc., etc.  We don't need to do that for 
> this TURN draft though, imho.
>

---

From: Jiri Kuthan <[email-masked]>
Date: Thu, 17 Apr 2008 00:17:38 +0200

> Would that be acceptable?

perfectly for me. -jiri

---

From: Jiri Kuthan <[email-masked]>
Date: Thu, 17 Apr 2008 00:18:19 +0200

> anycast at all in this specification?

I don't think so.

-jiri

---

From: Bill Woodcock <[email-masked]>
Date: Thu, 17 Apr 2008 00:22:52 -0700 (PDT)

> design systems that would preclude their use in the future.

Eric, could you catch me up on what it is that you guys are arguing about?

It may not be obvious, from a developer's perspective, how widely-used 
anycast is, and has been, since it's transparent to code and 
middle-of-the-network configuration.  Anything that breaks anycast tends 
to be ejected from the network very quickly, because any users behind it 
dislike not having, for instance, working root nameservers, or software 
downloads.  So it's not an accident that things that don't work in the 
real world don't exist much in the real world, it's natural selection.

                                -Bill