Subject: [BEHAVE] TURN Issue #6: Framing header for actual TURN messages

From: Jonathan Rosenberg <[email-masked]>
Date: Fri, 07 Dec 2007 18:04:59 -0500

This issue was not in my slides. It was brought up at the mike in the 
WG, and was discussed there. It was not discussed further at the ad-hoc.

As of turn-05, we introduced a framing mechanism, which is a 4 byte 
header consisting of 2 bytes channel and 2-bytes length. This framing 
header precedes ALL packets sent between a client and its TURN server, 
including both application data and TURN messages themselves, which are 
sent on channel zero.

Ekr raised a question of why this header was needed for actual TURN 
messages. After all, he argued, we went through all this trouble to make 
STUN (and thus TURN) differentiatable from each other. Indeed, others 
noted that the framing header itself looks very similar to the top 4 
bytes of the STUN message format anyway:

         0                   1                   2                   3
         0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
        |0 0|     STUN Message Type     |         Message Length        |
        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
        |                         Magic Cookie                          |
        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
        |                                                               |
        |                     Transaction ID (96 bits)                  |
        |                                                               |
        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

and the proposed framing header:

         0                   1                   2                   3
         0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
        |        Channel                |         Message Length        |
        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

So an alternative design is to make the framing header look like:

         0                   1                   2                   3
         0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
        |1|      Channel                |         Message Length        |
        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

So that there are 15 bits of channel instead of 16. Placing a 1 in that 
position makes the framing header always a non-match for STUN messages. 
Thus, it would not be even needed to do the more complex matching 
operations defined in STUN; a 1-bit check would do the job to separate 
out application data from STUN/TURN.

The benefits of this change would be:

   1. We would, once again, be able to run STUN and TURN on the same 
port on the server. As of turn-05 they MUST run on different ports 
because the framing header was unique to TURN. This could be a feature 
or a bug depending on your perspective:

          FEATURE: TURN and STUN are actually very different; using 
separate ports would allow a firewall to block one and not the other.

          BUG: practically speaking separate ports doesnt help since 
with DNS the ports can be dynamic anyway. In addition, its always good 
to use fewer ports on a server. Also its weird that an extension to a 
protocol would run on a different port; that kind of tells you its not 
an extension.

   2. Reduces the size of TURN messages by 4 bytes. This is a minor 
benefit and the real one is having a single port.

   3. it'd be a little clearer to explain to people that TURN is an 
extension to STUN, rather than saying, TURN is a STUN extension that is 
run over a new framing header

The drawbacks of this change would be:

   1. cuts the number of channels in half

An alternative design would be to use the full 16 bits for channel 
numbers, and then rely on the full suite of STUN differentiation 
techniques to demux. However, my preference in general is to not do that 
when its not actually needed. It complicates message processing and I 
don't think its worth it. 32767 channels is also a LOT for our 
envisioned use cases.

This was somewhat inconclusive at the microphone. I'm personally on the 
fence and don't feel that strongly. I probably midly prefer making this 
change.

Please comment on whether you prefer this change or not.

Thanks,
Jonathan R.

-- 
Jonathan D. Rosenberg, Ph.D.                   499 Thornall St.
Cisco Fellow                                   Edison, NJ 08837
Cisco, Voice Technology Group
[email-masked]
http://www.jdrosen.net                         PHONE: (408) 902-3084
http://www.cisco.com

_______________________________________________
Behave mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/behave

---

From: Rémi Denis-Courmont <[email-masked]>
Date: Mon, 10 Dec 2007 12:32:13 +0200

Le Saturday 08 December 2007 01:04:59 ext Jonathan Rosenberg, vous avez 
> Please comment on whether you prefer this change or not.

I am weakly supporting the change.

I don't think reducing from 65535 to 32767 channels is a problem. I hope IESG 
will agree that you have to recreate a new TURN allocation when you have 
exhausted the channel ranges, regardless of the total number of channels (?).

-- 
Rémi Denis-Courmont

---

From: Jonathan Rosenberg <[email-masked]>
Date: Mon, 10 Dec 2007 09:57:32 -0500

> exhausted the channel ranges, regardless of the total number of channels (?).

I don't see why IESG per se would have an opinion on it, but you never 
know...

-Jonathan R.

---

From: "Derek MacDonald" <[email-masked]>
Date: Mon, 10 Dec 2007 18:46:00 -0800

I like this change because it will simplify the implementation of a client
that uses both ICE/TURN as two modes of TURN/ICE packet writing will no have
to be supported.

I do not believe the reduction in the total # of channels is relevant; 32767
channels is half of the available port space, which is much more than an
endpoint would expect from a shared(enterprise) NAT.

The TURN server will probably run out of tuples well before the client does.

-Derek

---

From: Eric Rescorla <[email-masked]>
Date: Mon, 10 Dec 2007 19:43:09 -0800

At Fri, 07 Dec 2007 18:04:59 -0500,

I'm EKR and I approve of this change.

-Ekr

---

From: Jonathan Rosenberg <[email-masked]>
Date: Wed, 12 Dec 2007 23:14:24 -0500

32767 vs. 65535 is irrelevant for any SIP use cases we have to date.

Where this is a potential issue, is the following. Lets say you are a 
client and go to your TURN server and get an ip address and port. You 
basically want to hold on to that IP address and port for a really long 
time. Think of it like a nailed up VPN. Sufficiently long that the 
client might want to publish its IP/port in DNS or something (obviously 
requiring SRV records).

Now, the client can't run a server, but presuming some rendezvous 
mechanism, for each corresponding peer, it opens a permission, 
communicates with them, and then its done. Next day, a different peer, 
new permission created, etc.

In that use case, the client will eventually burn through all 32767 
channels, and obviously will do so in half the time it would if all 
65535 were there.

Now, I personally don't see this use case as being real, but I just 
wanted to point it out.

-Jonathan R.

Derek MacDonald wrote:
> I like this change because it will simplify the implementation of a 
> client that uses both ICE/TURN as two modes of TURN/ICE packet writing 
> will no have to be supported.
> 
> I do not believe the reduction in the total # of channels is relevant; 
> 32767 channels is half of the available port space, which is much more 
> than an endpoint would expect from a shared(enterprise) NAT.
> 
> On Dec 7, 2007 3:04 PM, Jonathan Rosenberg <[email-masked] 
> <mailto:[email-masked]>> wrote:
> 
>            |                         Magic Cookie                      
>        |
>     [email-masked] <mailto:[email-masked]>
>     [email-masked] <mailto:[email-masked]>

---

From: Philip Matthews <[email-masked]>
Date: Thu, 13 Dec 2007 09:15:03 -0500

I think this use-case might be real (well, not the put it in DNS  
part...). I think that TURN clients might well want to do a single  
allocation and use it for a long period of time.

So I am somewhat opposed to making this change at all. I don't think  
that any of the three advantages are strong, and I am worried about  
cutting the number of channels in half.

- Philip

On 12-Dec-07, at 23:14 , Jonathan Rosenberg wrote:

> Where this is a potential issue, is the following. Lets say you are  
> a client and go to your TURN server and get an ip address and port.  
> You basically want to hold on to that IP address and port for a  
> really long time. Think of it like a nailed up VPN. Sufficiently  
> long that the client might want to publish its IP/port in DNS or  
> something (obviously requiring SRV records).
>
> communicates with them, and then its done. Next day, a different  
> peer, new permission created, etc.
>
>> I like this change because it will simplify the implementation of  
>> a client that uses both ICE/TURN as two modes of TURN/ICE packet  
>> writing will no have to be supported.
>> I do not believe the reduction in the total # of channels is  
>> relevant; 32767 channels is half of the available port space,  
>> which is much more than an endpoint would expect from a shared 
>> (enterprise) NAT.
>>             0                   1                    
>> 2                   3
>>            |                         Magic  
>> Cookie                             |
>>             0                   1                    
>> 2                   3
>>             0                   1                    
>> 2                   3

---

From: "Derek MacDonald" <[email-masked]>
Date: Thu, 13 Dec 2007 11:42:17 -0800

Is the concern the # of channels that can be used over the lifetime of an
allocation or the # of concurrent channels used in an allocation?

I think that channel re-use is easier w/ ekr's proposal....let the client
re-use the channel after no keep-alive traffic has been sent for twice the
keepalive interval...don't know if we need this, but it could be done.  If
its # of concurrent channels...there are lots of state issues w/ 64 active
channels in an allocation.

>From an implementation perspective, the new framing approach is far
superior.  It doesn't require different encoding mechanisms of STUN
packets..which would be *very* annoying.

If we really have a use case for long-lived allocations, we need to solve
it, not just make the pain twice as slow.

-Derek

---

From: Philip Matthews <[email-masked]>
Date: Thu, 13 Dec 2007 16:06:54 -0500

On 13-Dec-07, at 14:42 , Derek MacDonald wrote:

> Is the concern the # of channels that can be used over the lifetime  
> of an allocation or the # of concurrent channels used in an  
> allocation?

The former (= # over total lifetime). I think it would be very hard  
for a client to keep even 32000 channels alive simultaneously.

The design team working on NAT traversal for HIP, for example, is  
thinking about doing an allocation and essentially keep it alive  
forever as the node connected to various peers. A SIP node could do  
the same thing.

>
>
> I think that channel re-use is easier w/ ekr's proposal....let the  
> client re-use the channel after no keep-alive traffic has been sent  
> for twice the keepalive interval...don't know if we need this, but  
> it could be done.  If its # of concurrent channels...there are lots  
> of state issues w/ 64 active channels in an allocation.
>

I guess I don't see this. Everything comes inside a channel, so you  
strip off the channel header and then direct the packet as appropriate.

>
>
> If we really have a use case for long-lived allocations, we need to  
> solve it, not just make the pain twice as slow.

Is solving it correctly worth the extra complexity?
Right now, I am thinking that just making it a rare occurrence is  
good enough.

---

From: "Derek MacDonald" <[email-masked]>
Date: Thu, 13 Dec 2007 13:40:03 -0800

Inline.

On Dec 13, 2007 1:06 PM, Philip Matthews <[email-masked]> wrote:

> The former (= # over total lifetime). I think it would be very hard for a
> client to keep even 32000 channels alive simultaneously.
>
> The design team working on NAT traversal for HIP, for example, is thinking
> about doing an allocation and essentially keep it alive forever as the node
> connected to various peers. A SIP node could do the same thing.
>
>
>
> I guess I don't see this. Everything comes inside a channel, so you strip
> off the channel header and then direct the packet as appropriate.
>

The pain is in the encoding side, not as mch for decoding.

> Right now, I am thinking that just making it a rare occurrence is good
> enough.
>

If it's going to be common enough to worry about, 1/2*common is not rare....

-Derek

---

From: Philip Matthews <[email-masked]>
Date: Thu, 13 Dec 2007 17:29:09 -0500

I still don't really see the argument for this approach, but I will  
note that if we change the approach to use "11" as the initial two  
bits, then we are only throwing away 16535 channels, rather than 32767.

Given this, perhaps it is worthwhile finding a way to reuse old  
channel numbers.

- Philip

On 13-Dec-07, at 16:40 , Derek MacDonald wrote:

>> Is the concern the # of channels that can be used over the  
>> lifetime of an allocation or the # of concurrent channels used in  
>> an allocation?
>
>> client re-use the channel after no keep-alive traffic has been  
>> sent for twice the keepalive interval...don't know if we need  
>> this, but it could be done.  If its # of concurrent  
>> channels...there are lots of state issues w/ 64 active channels in  
>> an allocation.
>>
>> From an implementation perspective, the new framing approach is  
>> far superior.  It doesn't require different encoding mechanisms of  
>> STUN packets..which would be *very* annoying.
>
>> If we really have a use case for long-lived allocations, we need  
>> to solve it, not just make the pain twice as slow.
>
>> >>  On Dec 7, 2007 3:04 PM, Jonathan Rosenberg < [email-masked]
>> >>            +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- 
>> +-+-
>> >> +-+-+-+-+
>> >>            |                         Magic
>> >> Cookie                             |
>> >>            +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- 
>> +-+-
>> >> +-+-+-+-+
>> >>
>> >> |                                                               |
>> >>            |                     Transaction ID (96
>> >> bits)                     |
>> >>
>> >> |                                                               |
>> >>            +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- 
>> +-+-
>> >> +-+-+-+-+
>> >>            +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- 
>> +-+-
>> >> +-+-+-+-+
>> >>            |        Channel                |         Message
>> >> Length           |
>> >>            +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- 
>> +-+-
>> >> +-+-+-+-+
>> >>            +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- 
>> +-+-
>> >> +-+-+-+-+
>> >>            |1|      Channel                |         Message
>> >> Length           |
>> >>            +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- 
>> +-+-
>> >> +-+-+-+-+

---

From: Jonathan Rosenberg <[email-masked]>
Date: Thu, 13 Dec 2007 23:55:49 -0500

Philip Matthews wrote:
> I still don't really see the argument for this approach, but I will note 
> that if we change the approach to use "11" as the initial two bits, then 
> we are only throwing away 16535 channels, rather than 32767.

How is that? I must be missing something. This would throw even more 
away since it leaves only 14 bits for channel ID.

> 
> Given this, perhaps it is worthwhile finding a way to reuse old channel 
> numbers.

I remain unconvinced in this use case really. THe main reason is that 
you STILL need some kind of rendezvous protocol before you can use the 
allocation with a new peer, and this rendezvous can provide the 
allocated IP and port anyway, so what is the big deal about using a new 
one for new peers when you run out of channels? Even in the HIP case, 
you are exchanging the allocated IP/port so I don't see the problem.

I have thought about reuse of channel and it is possible. Fortunately 
the framing header allows us to get rid of the timing and race 
conditions that plagued set active destination.

We'd steal a bit from the channel space and make this the 'flip' bit. 
For any new channel binding this value is zero in ChannelBind and zero 
in the framing header.

At some point later the client can change the binding. To do that, it 
sends a new CHannelBind request for the same channel, but a different 
peer IP/port. It also flips the value of the flip bit (from zero to 
one). Once the client gets a response, it can begin sending on the new 
channel, but does so with the value of the flip bit changed (now a 1) in 
the framing header. When the server gets data on the channel, it knows 
to send to the old peer while the flip bit is zero, and the new peer 
when its one. Similarly, when the server sends to the client, it sends 
with the flip bit of zero prior to receipt of the new CHannelBind, and 
then afterwards, with a value of 1. Data from the old peer after the 
flip gets sent encapsulated. The client knows that data packets with 
flip bit zero mean the old peer, and a one mean new peer.

The client would be limited in that it could not flip again for some 
duration (say one minute), to make sure any old packets in the network 
clear out. After a minute both client and server forget about the old 
remote peer.

So, even though this is possible there is some complexity and I don't 
see the value.

I do see the value in having a single port however and I think this 
makes the protocol simpler.

-Jonathan R.

---

From: Cullen Jennings <[email-masked]>
Date: Mon, 17 Dec 2007 22:25:22 -0800

I'm not following the use case for a large number of channels. If the  
issues is number of simultaneous channels, you just just form another  
connection form client to TURN server and you have a fresh set of  
channels (assuming the TURN server has enough ports). If the issues  
if you want a certain port to be long term stable, then TURN does not  
look like a good protocol to use because if you loose the port, there  
is no way to get a specific port back. And this limitation seems to  
be a positive design features of protocol.

I'm not making an argument here one way or the other but could you  
clarify what might be a use case for the more than say a few hundred  
channels?

Thanks, Cullen <with my individual contributor hat on>

On Dec 13, 2007, at 8:55 PM, Jonathan Rosenberg wrote:

>
>
> Philip Matthews wrote:
>> I still don't really see the argument for this approach, but I  
>> will note that if we change the approach to use "11" as the  
>> initial two bits, then we are only throwing away 16535 channels,  
>> rather than 32767.
>
> How is that? I must be missing something. This would throw even  
> more away since it leaves only 14 bits for channel ID.
>
> I remain unconvinced in this use case really. THe main reason is  
> that you STILL need some kind of rendezvous protocol before you can  
> use the allocation with a new peer, and this rendezvous can provide  
> the allocated IP and port anyway, so what is the big deal about  
> using a new one for new peers when you run out of channels? Even in  
> the HIP case, you are exchanging the allocated IP/port so I don't  
> see the problem.
>
> I have thought about reuse of channel and it is possible.  
> Fortunately the framing header allows us to get rid of the timing  
> and race conditions that plagued set active destination.
>
> We'd steal a bit from the channel space and make this the 'flip'  
> bit. For any new channel binding this value is zero in ChannelBind  
> and zero in the framing header.
>
> At some point later the client can change the binding. To do that,  
> it sends a new CHannelBind request for the same channel, but a  
> different peer IP/port. It also flips the value of the flip bit  
> (from zero to one). Once the client gets a response, it can begin  
> sending on the new channel, but does so with the value of the flip  
> bit changed (now a 1) in the framing header. When the server gets  
> data on the channel, it knows to send to the old peer while the  
> flip bit is zero, and the new peer when its one. Similarly, when  
> the server sends to the client, it sends with the flip bit of zero  
> prior to receipt of the new CHannelBind, and then afterwards, with  
> a value of 1. Data from the old peer after the flip gets sent  
> encapsulated. The client knows that data packets with flip bit zero  
> mean the old peer, and a one mean new peer.
>
> The client would be limited in that it could not flip again for  
> some duration (say one minute), to make sure any old packets in the  
> network clear out. After a minute both client and server forget  
> about the old remote peer.
>
> So, even though this is possible there is some complexity and I  
> don't see the value.
>

---

From: Philip Matthews <[email-masked]>
Date: Tue, 18 Dec 2007 08:40:57 -0500

Consider the following scenario. Node X is behind a "bad" NAT. Node X  
might be a peer in a p2psip overlay wanting to form peer protocol  
connections with other peers, or it might be a node running HIP that  
wants to communicate with other nodes. Node X allocates a relay  
address then signals that address to other nodes.

_IF_ there was a way to de-allocate channels, and _IF_ TURN supported  
a sufficiently large number of channels, then node X could get away  
with just the one allocation. When it wanted to talk to a new node,  
it would signal with the one relay address it got from the  
allocation, rather than getting a new allocation.

The point is that the code could be written to just deal with a  
single allocation.

On 18-Dec-07, at 01:25 , Cullen Jennings wrote:

>
>
> I'm not following the use case for a large number of channels. If  
> the issues is number of simultaneous channels, you just just form  
> another connection form client to TURN server and you have a fresh  
> set of channels (assuming the TURN server has enough ports).

For most nodes, 65535 is plenty of channels. Most nodes will never  
need more than this number of simultaneous channels.

> If the issues if you want a certain port to be long term stable,  
> then TURN does not look like a good protocol to use because if you  
> loose the port, there is no way to get a specific port back. And  
> this limitation seems to be a positive design features of protocol.

The applications mentioned above do not care which port number they get.

> clarify what might be a use case for the more than say a few  
> hundred channels?

If we are writing general-purpose code that has a hard limit of no  
more than N channels simultaneously, then it would be nice to pick N  
so that we are confident we will never need to exceed that number.

Perhaps I have missed something, but at least in the HIP case, 65535  
is large enough for most uses. As I see it, it is insufficient in  
precisely those cases where 65535 ports per IP address is  
insufficient -- for example, for a node acting as a web server with  
lots and lots of web clients. Arguing that fewer channels is  
sufficient is just like arguing that a node with a public IP address  
only needs fewer port numbers -- it works in most cases, but there  
are cases where that is not enough ports.

So what I am saying is that, if we had a way to deallocate channels  
and we had 65535 channels available, then most nodes could get away  
with just one allocation, in the same way that most nodes not using  
TURN can get away with just 65535 ports and one IP address.

Does this make any sense?

- Philip

---

From: Cullen Jennings <[email-masked]>
Date: Tue, 18 Dec 2007 09:01:18 -0800

I'm assuming we will probably add  a way to deallocate or close a  
channel for the error generation reasons I mentioned before but  
ignoring that ...

I'm still not understanding the use case. Let's say we had 200  
channels. After every 200 binding requests, you would have to do an  
single alloc request. Is this a problem?

---

From: Philip Matthews <[email-masked]>
Date: Tue, 18 Dec 2007 13:43:53 -0500

> single alloc request. Is this a problem?

It is a problem only in that I now have to keep a list of  
allocations, rather than just one. It makes the code a bit more   
complex. If there is any possibility of forking or equivalent, then I  
should get a new allocation once my last allocation down to say 10  
channels left, in case the number of peers I get back is more than  
10. It probably makes debugging from traces more complex, since a  
node now has multiple relay addresses and thus multiple sockets. It  
raises the question of which server-reflexive address to use (e.g.,  
the one from the latest allocation, or the one from the first  
allocation?).

None of these are big problems. Jonathan's argument is that the  
complexity of adding channel deallocation will trump them, and I  
suspect he might be right, especially if long-lived allocations are a  
rare use.

- Philip

---

From: Jonathan Rosenberg <[email-masked]>
Date: Tue, 18 Dec 2007 18:42:44 -0500

inline.

Philip Matthews wrote:
> It is a problem only in that I now have to keep a list of allocations, 
> rather than just one.

You are already having to maintain a list of channels, so I don't see 
why including the allocation in the list makes this harder.

You are already needing the code to obtain an allocation, and you need 
the code to include it in the rendezvous protocol.

Thus, I do not see where the savings is.

> It makes the code a bit more  complex. If there is 
> any possibility of forking or equivalent, then I should get a new 
> allocation once my last allocation down to say 10 channels left, in case 
> the number of peers I get back is more than 10. It probably makes 
> debugging from traces more complex, since a node now has multiple relay 
> addresses and thus multiple sockets. 

Why does that make debugging more complex?

Also, lets take this p2psip case. With 32767 channels, given the number 
of connections to other peers that an p2psip client needs, how long will 
it take before this 32767 is exhausted? I bet its WAY longer than the 
amount of time the client would even be up.

> It raises the question of which 
> server-reflexive address to use (e.g., the one from the latest 
> allocation, or the one from the first allocation?).

The client should be redoing the server reflexive prior to each ice 
exchange. I think someone else commented, that there are cases where 
this can change over time. The ICE philosophy is not to assume anything. 
So don't assume that your old server reflexive candidate remains valid 
forever.

-Jonathan R.