Subject: Re: [BEHAVE] TURN - some proposed changes for improvement and simplification

From: "Mayank Dev Sondhi" <[email-masked]>
Date: Thu, 8 Nov 2007 15:09:02 +0530

> Now, that said, TCP has some additional complexities that UDP does not.
> First and foremost is that you need to be very careful with how you
> handle buffering. We want to avoid the case where data arrives quickly
> on the connection from the client to the TURN server, but the TCP
> connection from the TURN server to the remote peer is slow, and so data
> is lost at the TURN server when buffers overflow. To handle this, we
> will specify a buffering requirement and a possiblemechanism. There
> would be a maximum buffer size  per connection set in the kernel (for
> example). The TURN server application code reads  off of this buffer and
> writes to the other TCP connection. If it cannot write because the
> window is full, it doesn't read data from the incoming
> connection. This will eventually cause the buffers in the kernel to
> fill, and then packets are dropped (and not ACKed), causing backpressure
> on the TCP connections. This effectively bridges the congestion control
> between the connections, and ensures that, TURN server failures aside,
> no packets get dropped in the TURN server. We believe that this kind of
> mechanism is used in other similar boxes like HTTP proxies.

The point on HTTP proxies presents an interesting scenario. The
recommended buffer size is a difficult number to come up with just
because some ISA proxies for example, set the maximum window
size(buffer) for their proxied connection whereas the client actually
connecting via the proxy may have advertised a "lesser" buffer size.
So Client A connecting via Proxy A may advertise a 20KB Window Size
and the proxied connection would however advertise a 64KB Window Size
to the TURN server. If the TCP connection to the remote peer is slow,
the backpressure from the TURN server will take more time to affect
the buffer on the sending client and this could potentially create a
problem.(degraded user experience and increased latency in signal for
example) Is there a recommended way to advertise the least window size
over the connection, or potentially have a "sweet spot" number that
would be the IETF recommendation? I confess that this may not even be
an issue, but just wanted to call it out.

Thanks
Mayank

On Sep 12, 2007 10:29 PM, Jonathan Rosenberg <[email-masked]> wrote:
>
> As Dan mentioned at the last IETF meeting, we've formed a small team to
> actively work on TURN over the next month, in order to accelerate its
> completion and produce a version ready for WGLC by the end of September.
>
> As part of this, we took a step back and looked a bit at some of the
> more troubling aspects of TURN. Three in particular seemed important:
>
> * There are several different methods of framing and encapsulation of
> messages with a number of exceptions based on transport protocol,
> contents of the data, and the active destination.  The overall effect is
> confusing and rather complex.
>
> * The active destination concept is inelegant, confusing, complex, and
> during forking results in undesirable overhead and at least one
> timing-related race condition.
>
> * the use of multiple TCP connections from the TURN server towards
> peers, when there is one TCP connection from the TURN server to the
> client, has been a concern on the list recently.
>
> We have a proposal on some non-trivial changes to TURN on these topics.
>
> Firstly, we can eliminate the entire need for active destinations (and
> the Send and Data primitives as well) by introducing the concept of a
> 'channel'. Each channel is a stream of application data sent to and from
> a remote IP and port. In the case of UDP, we'd add a SINGLE BYTE to the
> front of every packet to and from the TURN server, which is used to
> signal the channel number. Channel 0 is reserved for TURN signaling to
> and from the client and TURN server. The remaining 255 channels are for
> remote destinations and carry pure data after the 1-byte channel number.
> So when a client does a Send or a Connect, the server allocates a
> remaining channel number and passes this channel number back to the
> client. From that point on, when the TURN server receives data from the
> peer, it sends it to the client and adds only the byte up front
> containing the channel number. Similarly, when the client wants to send
> data, it just adds this byte up front indicating the channel number, and
> the TURN server sends the data to the corresponding remote peer. There
> is no encapsulation of data as a TURN message ever. Thus, the Data
> indication is removed. TURN currently requires the server to check for
> the magic cookie in packets received from remote peers, and when it sees
> it, encapsulate the resulting data in a Data indication. That
> requirement would also be eliminated.
>
> Consider a forking use case where A calls a URI and this forks to B and
> C. Both send a 200 OK. A will end up running ICE simultaneously with
> both B and C. All of the STUN checks and any resulting RTP would just be
> sent over different channels, all with identical overhead (1 byte).
> There would be no need to re-invite, as there would typically be in the
> current TURN mechanism.
>
> This also fixes the complexity that was raised on the list for
> packetcable; the current mechanism complicates their QOS mechanisms
> because the bandwidth for the stream is not deterministic. Here, it is
> deterministic. Every packet always has an extra 1 byte per packet overhead.
>
> The main reason we've avoided any kind of framing for UDP up until now
> has been overhead. So what kind of overhead does this introduce? Well if
> we assume something really nasty like g729 with 20ms framing, thats 20
> bytes of payload plus 40 for RTP/UDP/IP gives us 60, which now goes to
> 61. Thats a 1.6 percent penalty which is pretty good, considering
> especially the huge overhead you are already paying for RTP/UDP. Anyway
> a more realistic number is g711 with 30ms framing, and there is .35%
> overhead. This cost seems reasonable given the simplifications it brings.
>
> Another possible concern with the 1-byte overhead is path MTU. If a UDP
> packet was sent from a remote peer and it was exactly equal to the path
> MTU from the remote peer to the TURN server, this will now fragment on
> the TURN server to client link. Its not clear that this is a real
> problem. But if you are aware of UDP applications that send data exactly
> at the MTU for some reason, please speak up.
>
> For TCP, the channel mechanism would work the same. The current framing
> already has a 'type' field which we'd just turn into a channel number.
> Thus both UDP and TCP would have framing, the difference is only that
> TCP includes a length whereas UDP does not. Protocol operation is
> otherwise more or less the same, and this would forge closer alignment
> between UDP and TCP. As with UDP, this allows multiple TCP connections
> from the TURN server to remote peers, each of which would have its own
> channel number, allocated to the client when permissions are created for
> a remote IP address, or when a connect request is sent. Connection
> status indications would all come over the control channel, and each
> such indication would contain, as an attribute, the channel number that
> the status indication is associated with.
>
> Now, that said, TCP has some additional complexities that UDP does not.
> First and foremost is that you need to be very careful with how you
> handle buffering. We want to avoid the case where data arrives quickly
> on the connection from the client to the TURN server, but the TCP
> connection from the TURN server to the remote peer is slow, and so data
> is lost at the TURN server when buffers overflow. To handle this, we
> will specify a buffering requirement and a possiblemechanism. There
> would be a maximum buffer size  per connection set in the kernel (for
> example). The TURN server application code reads  off of this buffer and
> writes to the other TCP connection. If it cannot write because the
> window is full, it doesn't read data from the incoming
> connection. This will eventually cause the buffers in the kernel to
> fill, and then packets are dropped (and not ACKed), causing backpressure
> on the TCP connections. This effectively bridges the congestion control
> between the connections, and ensures that, TURN server failures aside,
> no packets get dropped in the TURN server. We believe that this kind of
> mechanism is used in other similar boxes like HTTP proxies.
>
> Now, lets consider the case of multiple TCP connections to a
> multiplicity of remote peers. There have been suggestions on the list to
> just disallow this. We cannot. It happens as an artifact of forking.
> Consider client A that sends an offer. Amongst its candidates is a
> passive TCP candidate learned from the TURN server. THis INVITE forks to
> B and C, each of which return a single active candidate. Now, A will
> need to open permissions on the TURN server for B and C's IP addresses
> listed in the SDP answer, and both B and C will open a TCP connection
> towards the TURN server in order for ICE checks to proceed. Note well,
> that since ICE processing happens ideally prior to call answering, this
> case happens NOT JUST when both B and C answer at the same time, but
> whenever there is forking at all with ICE-tcp. So this is actually a
> common case and not a corner case.
>
> In this case of multiple TCP connections to remote peers, the congestion
> control bridging described above gets complicated, and head-of-line
> blocking gets introduced. Congestion on the connection from the TURN
> server to one peer can slow down the connection to the client, and thus
> limit the throughput to the other peer. This is only a problem when we
> are transferring real data, not so much for things like STUN messages or
> TLS handshakes. So the recommendation is going to be that, if a client
> ends up in a case with multiple TCP connections that survive past the
> connectivity checks, in that case the spec will recommend the client
> allocate a new TCP candidate from the TURN server and restart ICE with
> one of the peers, in order to eliminate the HOL blocking.
>
> We're looking for feedback on this proposal (and questions and
> clarifications of course) ASAP - within the next week. If this seems
> like a good direction, we can work these changes in and address the
> other comments and still deliver a revision on schedule.
>
> Thanks,
> Jonathan R.
> --
> Jonathan D. Rosenberg, Ph.D.                   600 Lanidex Plaza
> Cisco Fellow                                   Parsippany, NJ 07054-2711
> Cisco Systems
> [email-masked]                              FAX:   (973) 952-5050
> http://www.jdrosen.net                         PHONE: (973) 952-5000
> http://www.cisco.com
>
>
> _______________________________________________
> Behave mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/behave
>

_______________________________________________
Behave mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/behave

---

From: Philip Matthews <[email-masked]>
Date: Thu, 8 Nov 2007 08:04:59 -0500

The TURN design team has decided to split TURN into two drafts. The  
base TURN draft will handle just "UDP allocations" while a new TURN- 
TCP draft will talk about "TCP allocations". We have already talked  
with the ADs and they are happy with this decision.

The base draft will handle the case where the connection to the  
remote destination is over UDP, however it will allow the connection  
between the TURN client and the TURN server to be TCP or TLS over TCP.

This decision allows us to rapidly advance the UDP case, which people  
seem to be most interested in right now, while giving us a chance to  
get the TCP stuff right.

We ARE hoping to get both drafts out for this IETF, and are madly  
working away. The TURN-TCP draft (due on Monday!) might be, well, a  
bit drafty, but should give some meat for us to discuss in Vancouver.  
The base TURN draft we are hoping will be solid and suitable for  
rapid advancement.

- Philip

---

From: "Alfred E. Heggestad" <[email-masked]>
Date: Thu, 08 Nov 2007 14:16:33 +0100

Philip Matthews wrote:
> The TURN design team has decided to split TURN into two drafts. The base 
> TURN draft will handle just "UDP allocations" while a new TURN-TCP draft 
> will talk about "TCP allocations". We have already talked with the ADs 
> and they are happy with this decision.
> 
> The base draft will handle the case where the connection to the remote 
> destination is over UDP, however it will allow the connection between 
> the TURN client and the TURN server to be TCP or TLS over TCP.
> 

this makes sense.

can I ask that we move the "turn-ipv6" spec into the base turn spec?

/alfred

> seem to be most interested in right now, while giving us a chance to get 
> the TCP stuff right.
> 
> working away. The TURN-TCP draft (due on Monday!) might be, well, a bit 
> drafty, but should give some meat for us to discuss in Vancouver. The 
> base TURN draft we are hoping will be solid and suitable for rapid 
> advancement.
> 
> - Philip
> 
> 
> 
>