Subject: Re: [BEHAVE] Duplicate request handling

From: Philip Matthews <[email-masked]>
Date: Fri, 4 Jul 2008 13:31:56 -0400

On Wed, 25-Jun-08, at 08:56 , Philip Matthews wrote:
> I would like to come to a resolution on this issue, at least in the  
> case of a TURN Allocate request.
>
> To recap: Over UDP, requests can be retransmitted if the client does  
> not initially receive a response. To ensure that the client receives  
> a consistent answer, STUN requires that the server always return the  
> same success response. However, currently the STUN and TURN  
> specification say nothing about replying with the same failure code.
>
> Benny pointed out a case where replying with the same failure code  
> would be helpful:
>    For example,
>    imagine when TURN server first returns failure response, then
>    successful response on the second Allocate retransmission. Client  
> most
>    likely will ignore the second response retransmission, hence client
>    thinks that the Allocate has failed while the server thinks it has
>    been successful.
>
> However, Remi commented:
>    So what is the big deal? The server will just garbage collect the  
> allocation
>    eventually. This is perfectly normal operation. Clients can  
> always "vanish"
>    unexpectedly, such as if their network connection fails.
>    It's totally fine if the server thinks a request has succeeded  
> and the client
>    thinks it's failed. Only the opposite would be an issue.
>
>
> Regarding the case of a TURN Allocate request, the TURN-08 says that  
> the server must generate the same reply if the response is a success  
> response, but the spec is currently silent on what to do if the  
> response is an error response. From section 6.2 Receiving an  
> Allocate request:
>   2.  The server checks if the 5-tuple is currently in use by an
>       existing allocation, or was it in use by another allocation
>       within the last 2 minutes.  If yes, then there are two sub- 
> cases:
>
>       *  If the transport protocol in the 5-tuple is UDP, and if the
>          5-tuple is currently in use by an existing allocation, and if
>          the transaction id of the request matches the transaction id
>          stored with the allocation, then the request is a
>          retransmission of the original request.  The server replies
>          either with a stored copy of the original response, or with a
>          response rebuilt from the stored state data.  If the server
>          chooses to rebuild the response, then (a) it need not parse
>          the request further, but can immediately start building a
>          success response, (b) the value of the LIFETIME attribute can
>          be set to the current value of the time-to-expire timer, and
>          (c) the server may need to include an extra field in the
>          allocation to store the token returned in a RESERVATION-TOKEN
>          attribute.
>
>       *  Otherwise, the server rejects the request with a 437
>          (Allocation Mismatch) error.
>
> Implementing the above text does not require a lot of extra work.  
> The server already needs to keep the allocation data structure  
> containing the Transaction ID, and the server already needs to check  
> if the 5-tuple is in use by another allocation. All that is required  
> is to check if the transaction id on the incoming request matches  
> the one stored in the allocation. The response can then be easily  
> rebuilt from the stored state data.
>
> However, as I see it, replying with the same failure code requires a  
> whole new data structure that remembers recently failed Allocate  
> transactions. The code around this data structure has to handle  
> attacks like sending lots of unsuccessful Allocation request, a  
> problem that the success case does not have to worry about.  I am  
> wondering if this is really worth it, given that existing TURN  
> mechanisms will handle the uncommon case of an Allocate request that  
> fails on the first transmission but succeeds on a retransmission (as  
> Remi points out).
>
> Right now, I am leaning towards adding text pointing out this issue  
> and saying that an implementation MAY return the same failure code,  
> without requiring this behavior.
>
> Comments?
>

I received only two comments to this post, and these two comments took  
opposite positions on the issue. So I have decided to resolve this  
issue by adding the following note to the document:

       NOTE: When UDP transport is used between the client and the
       server, the client will restransmit an Allocate request if it  
does
       not receive a response within a certain timeout period
       [I-D.ietf-behave-rfc3489bis].  Because of this, the server may
       receive two (or more) Allocate requests with the same 5-tuple and
       same transaction id.  Check #2 (above) handles the case where the
       first Allocate request is accepted and generates a success
       response, but it does not handle the case where the first request
       is rejected but the second request is accepted (because  
conditions
       on the server have changed in the brief intervening time period).
       If the client receives the first (failure) response, it will
       ignore the second (success) response and believe that an
       allocation was not created.  An allocation created in this matter
       will eventually timeout, since the client will not refresh it.
       Furthermore, if the client later retries with the same 5-tuple  
but
       different transaction id, it will receive a 437 (Allocation
       Mismatch), which should cause it to retry with a different
       5-tuple.

       Server implementors MAY elect to prevent this second case by
       remembering recent failure responses and returning the saved
       failure response when receiving a retransmitted Allocate request.
       This optional behavior may be appropriate when the server
       implements some sort of charging mechanism or a per-user quota.
       Alternatively, servers may use a smaller maximum lifetime value  
to
       mimize the lifetime of this "orphaned" allocation (see below).

       Server implementors debating whether to implement this optional
       feature should be aware that there are other scenarios in TURN
       that lead to such "orphaned" allocations.

I have also added a sentence to the discussion of maximum allocation  
lifetime mentioning reasons why a server might want a maximum lifetime  
of less than 1 hour.

- Philip

_______________________________________________
Behave mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/behave

---

From: Simon Perreault <[email-masked]>
Date: Fri, 4 Jul 2008 13:57:16 -0400

> issue by adding the following note to the document:

FWIW, I think servers should remember error responses as well as successes.

The argument of the book keeping being too much of a burden is weak because 
you still have to do it for successes. In the long run, successes should 
outnumber errors and remembering errors would be no big deal. Of course there 
should be a cap on remembering responses (errors *and* successes) so that the 
server doesn't crash if it runs out of resources. I'd prefer my server to be 
non RFC-compliant for a short while until it recuperates than have it just 
stop working.

Oh, and that's how it is implemented in Numb. Both successes and errors share 
the same data structure, so there is no "new data structure that remembers 
recently failed Allocate transactions". :)

-- 
Please try Numb, a STUN/TURN server implementation.
Free access at http://numb.viagenie.ca/.
_______________________________________________
Behave mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/behave

---

From: Philip Matthews <[email-masked]>
Date: Fri, 4 Jul 2008 15:13:39 -0400

> recently failed Allocate transactions". :)

Note that the current text allows implementations to remember failures  
and replay them when appropriate, so it is likely that your  
implementation is compliant with this portion of the specification as  
it reads today.

Also note that a person's opinion on this subject seems to depend on  
the structure of their implementation. For some, keeping track of  
failures is easy to do. For others, it requires an extra data  
structure. Because other mechanisms in TURN can handle the situation  
in question, I have elected to make remembering and replaying failure  
responses optional.

- Philip

---

From: "Dan Wing" <[email-masked]>
Date: Fri, 4 Jul 2008 13:12:19 -0700

> FWIW, I think servers should remember error responses as well 

If we require this, we are requiring the TURN server to retain state for
errors -- which creates a vector for an attacker to cause a denial of service.

-d

> The argument of the book keeping being too much of a burden 
> is weak because you still have to do it for successes. 
> In the long run, successes should 
> non RFC-compliant for a short while until it recuperates than 
> have it just stop working.
>

---

From: Simon Perreault <[email-masked]>
Date: Fri, 4 Jul 2008 16:23:14 -0400

> errors -- which creates a vector for an attacker to cause a denial of

This is STUN-specific so:

How is it any different from requiring the STUN server to retain state for 
non-indempotent successes?

There are those for which the server can regenerate the same response from 
other state that it needs anyway (i.e. a TURN Allocate response), but what 
about the potential others?

Just put a cap on the amount of resources consumed for remembering responses, 
be they errors or successes.

---

From: "Dan Wing" <[email-masked]>
Date: Fri, 4 Jul 2008 13:49:25 -0700

> How is it any different from requiring the STUN server to 
> retain state for non-indempotent successes?

Anything an attacker can use that will cause state in a 
shared server is a problem.  This is what causes TCP SYN
attacks to take down TCP servers.

-d

---

From: Cullen Jennings <[email-masked]>
Date: Fri, 4 Jul 2008 20:03:21 -0700

I think many people will implement STUN by not remember state even for  
success and just making sure that the recomputed answer is the same  
for retransmissions as the original request. The reason I think  
implementations will end up this way is for exactly the reason Dan  
mentions below.

A very common problems for servers is where a broken client (but one  
that is not even a malicious attacker) sends in a request that fails,  
and the client instantly forms a new transaction and more or repeats  
the same mistake as a new transaction. Servers often see these failed  
series of transactions coming in at a rate determined by whatever the  
RTT to the client is.

Cullen <in my individual contributor roll>

---

From: Rémi Denis-Courmont <[email-masked]>
Date: Mon, 7 Jul 2008 10:24:39 +0300

On Friday 04 July 2008 20:57:16 ext Simon Perreault, you wrote:
> you still have to do it for successes.

I don't see the point. For a successful response, you need to allocate 
(limited) resources anyway - for instance an operating system socket handle 
so you better do book keeping.

Please point me to a TCP stack that keeps track of the TCP SYN/RST packets it 
sent...

> In the long run, successes should outnumber errors and remembering errors
                             ^^^^^^
You won't build reliable and scalable solutions through wishful thinking.

-- 
Rémi Denis-Courmont
Maemo Software, Nokia Devices R&D
_______________________________________________
Behave mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/behave

---

From: Jonathan Rosenberg <[email-masked]>
Date: Mon, 07 Jul 2008 14:53:48 -0400

Well, SIP actually requires the server to statefully remember responses, 
  except for authentication challenges. Clients can still create lots of 
state in a server via failed requests but at least they are authenticated.

That said, I am not suggesting we change the stun spec to require the 
server to send the same response for failure responses too. However, I 
do think rfc3489bis has to say something about error responses. I 
suggest it says that the server SHOULD send the same response, and that 
this is only a SHOULD to allow servers to avoid storing state for failed 
transactions. For the Binding method it is trivial to re-generate error 
responses without retaining state, this may be more complicated for 
other methods.

And then TURN can describe the issues for Allocate.

-Jonathan R.

-- 
Jonathan D. Rosenberg, Ph.D.                   499 Thornall St.
Cisco Fellow                                   Edison, NJ 08837
Cisco, Voice Technology Group
[email-masked]
http://www.jdrosen.net                         PHONE: (408) 902-3084
http://www.cisco.com
_______________________________________________
Behave mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/behave

---

From: "Dan Wing" <[email-masked]>
Date: Mon, 7 Jul 2008 11:59:51 -0700

> -----Original Message-----
> From: Jonathan Rosenberg [mailto:[email-masked]] 
> Sent: Monday, July 07, 2008 11:54 AM
> To: Dan Wing
> Cc: 'Simon Perreault'; 'Behave WG'; 'Philip Matthews'
> Subject: Re: [BEHAVE] Duplicate request handling
> 

Yes, and remembering state for authenticated users is a different
situation.

> And then TURN can describe the issues for Allocate.

Sounds great.  Thanks,
-d

---

From: Jonathan Rosenberg <[email-masked]>
Date: Tue, 08 Jul 2008 16:58:06 -0400

We need to allow people to implement however they want. Here is the text 
I put into -17. Does this look OK?:

	<t>If the request uses UDP transport and is a retransmission
	of a request for which the server has already generated a
	success response within the last 40 seconds, the server MUST
	retransmit the same success response, and for an error
	response with the last 40 seconds, the server SHOULD
	retransmit the same error response. One way for a server to do
	this is to remember all transaction IDs received over UDP and
	their corresponding responses in the last 40 seconds. Another
	way is to reprocess the request and recompute the
	response. The latter technique MUST only be applied to
	requests which are idempotent (a request is considered
	idempotent when the same request can be safely repeated
	without impacting the overall state of the system) and result
	in the same success response for the same request. The SHOULD
	strength requirement for error responses allows servers to
	avoid storing transaction state for failed non-idempotent
	requests. The Binding method is considered to be idempotent
	(even though certain rare network events could cause
	the reflexive transport address value to change). Extensions
	to STUN SHOULD state whether their request types are idempotent.</t>

-Jonathan R.

---

From: Simon Perreault <[email-masked]>
Date: Tue, 08 Jul 2008 15:18:02 -0700

>     to STUN SHOULD state whether their request types are idempotent.</t>

A big +1.

Cheers,
Simon

---

From: "Dan Wing" <[email-masked]>
Date: Tue, 8 Jul 2008 15:57:12 -0700

> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Simon Perreault
> Sent: Tuesday, July 08, 2008 3:18 PM
> To: Jonathan Rosenberg
> Cc: Behave WG

Works for me, too.

(as individual contributor.)

-d

---

From: Rémi Denis-Courmont <[email-masked]>
Date: Wed, 9 Jul 2008 11:02:20 +0300

> 	to STUN SHOULD state whether their request types are idempotent.</t>

Not very good to me, but I can live with that.

---

From: Jonathan Rosenberg <[email-masked]>
Date: Wed, 09 Jul 2008 09:23:10 -0400

I thought this text was in line with what you were saying in this 
thread. Can you comment on what part you don't like?

Thanks,
Jonathan R.

---

From: Philip Matthews <[email-masked]>
Date: Wed, 9 Jul 2008 18:41:13 -0400

On Tue, 8-Jul-08, at 16:58 , Jonathan Rosenberg wrote:
> We need to allow people to implement however they want. Here is the  
> text I put into -17. Does this look OK?:
>

I now think that the implications of request retransmission is  
something that needs to be considered on a method-by-method basis. The  
STUN specification should give general guidance, but specific guidance  
needs to be given for each method.

For the Binding method, the STUN text needs makes it clear that it is  
OK to return different mapped addresses in response to a retransmitted  
Binding request. The text above does this, though I suggest we remove  
the term "idempotent" and just say this directly.

For the Allocate method, the TURN spec now makes it clear that if an  
Allocate request is successful, then retransmissions will return a  
success response. However, if the first Allocate fails, the second MAY  
succeed and this is OK.

For the Refresh method, I have clarified that it is OK to delete a non- 
existent allocation. Thus subsequent Refreshes with a zero lifetime  
simply delete a non-existent allocation, while subsequent Refreshes  
with a non-zero lifetime are the same a rapid refreshes.

For the ChannelBind method, retransmitted ChannelBind requests are not  
a problem since it is OK to rebind a channel to a peer.

The one question that we need to decide is exactly how to detect a  
retransmitted request. In the current TURN spec, this is done by  
comparing the incoming transaction id with the transaction id of the  
request that created the allocation. If the two values match, then  
this is considered a retransmission, and the spec says that the server  
can immediately stop parsing the incoming request.

At present, there is no concept of time associated with this test, so  
a request with the same transaction id arriving 8 minutes later is  
also considered a retransmission. This is probably not good, since not  
everyone uses a good random number generator.

I see some possible solutions to this:

a) Require that the request arrive within 40 seconds of the original  
request. In the case of Allocate, this would be adding an extra timer  
to some implementations. And I don't really like this idea since a  
random number generator is allowed to generate the same value in rapid  
succession.

b) Have the client compute the transaction id using a 64-bit random  
number and a 32-bit sequence number. This would guarantee that the  
transaction id would not repeat for quite some time.

c) Combine (a) and (b).

I currently lead towards option (c).

- Philip

---

From: Simon Perreault <[email-masked]>
Date: Thu, 10 Jul 2008 09:22:38 -0700

Philip Matthews wrote:
> request that created the allocation. If the two values match, then this 
> is considered a retransmission, and the spec says that the server can 
> immediately stop parsing the incoming request.

It should also be mentioned that for two requests with the same ID to be 
considered duplicates they must have the same 5-tuple.

Cheers,
Simon

---

From: Jonathan Rosenberg <[email-masked]>
Date: Sat, 12 Jul 2008 09:57:38 -0400

inline:

Philip Matthews wrote:
> I now think that the implications of request retransmission is something 
> that needs to be considered on a method-by-method basis. The STUN 
> specification should give general guidance, but specific guidance needs 
> to be given for each method.

It more or less said that, but I have clarified to say:

Extensions to STUN SHOULD discuss the
	implications of request retransmissions on servers which do
	not store transaction state.

> 
> For the Binding method, the STUN text needs makes it clear that it is OK 
> to return different mapped addresses in response to a retransmitted 

Well, idempotency is important here. But I added a sentence to mention 
this fact specifically:

Note that, there are certain rare network events
	that could cause the reflexive transport address value to
	change, resulting in a different mapped address in different
	success responses.

> For the Refresh method, I have clarified that it is OK to delete a 
> non-existent allocation. Thus subsequent Refreshes with a zero lifetime 
> simply delete a non-existent allocation, while subsequent Refreshes with 
> a non-zero lifetime are the same a rapid refreshes.
> 
> For the ChannelBind method, retransmitted ChannelBind requests are not a 
> problem since it is OK to rebind a channel to a peer.
> 
> 
> 

This is also how it works in STUN. This is what the transaction ID is for.

> 
> At present, there is no concept of time associated with this test, so a 
> request with the same transaction id arriving 8 minutes later is also 
> considered a retransmission. This is probably not good, since not 

THis is a non-issue. The transaction ID is 96 bits. It will not repeat 
anytime soon. The spec requires randomness for this field. Thus, any 
implementation that reuses the same value in 8 minutes is non-compliant. 
We do not need to do anything further to deal with this.

> request. In the case of Allocate, this would be adding an extra timer to 
> some implementations. And I don't really like this idea since a random 
> number generator is allowed to generate the same value in rapid succession.
> 

No - there is no problem here and neither TURN nor STUN need to say 
anything more.

-Jonathan R.