Subject: Re: [Speermint] TLs performance

From: Cullen Jennings <[email-masked]>
Date: Tue, 1 Aug 2006 00:03:49 -0700

Klaus, first of all thank you for producing some real results we can  
discuss.  Just to validate some of your results - this is not the  
first time I have seen performance results where TLS (or more  
specifically TCP) has better performance that UDP. I have seen this  
on other products too and when I think about it, it is not that  
surprising.

Cullen

On Jul 24, 2006, at 10:59 AM, Klaus Darilion wrote:

> Hi!
>
> I made some tests with sipp+openser:
>
> scenario:
>
> UAC <-TLS-> openser <-TLS-> UAS (all of them got its own 2xP3  
> 1,26GHz machine, linked with 100MBit). Simple forwarding  
> configuration (like a loadbalancer), no DNS lookups or DB queries.
>
> UDP: ~620 cps
> TLS: ~680 cps (single TLS connection)
>
> cps=calls per second = INVITE-180-200-OK-ACK-BYE-200
>
> I think TLS performs better as because of UDP retransmissions,  
> which "kills" the proxy if it becomes slow.
>
> More data when I have it.
>
> regards
> klaus
>
> _______________________________________________
> Speermint mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/speermint

_______________________________________________
Speermint mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/speermint

---

From: Benny Prijono <[email-masked]>
Date: Tue, 01 Aug 2006 10:38:42 +0100

Hi all,

just to share my data, fwiw. In my benchmark, TCP (with single 
connection) performs almost consistently better than UDP in 
transactions and calls processing, but only when UAC and UAS are in 
local machine. When UAC and UAS are separated into different boxes, 
TCP performance is only about half of UDP performance.

For me, my data makes perfect sense, because in local machine, TCP 
transactions has lower cost to setup (e.g. less timers), so it should 
be faster. While on the network, the TCP round-trip probably prevents 
transaction to complete as fast as the UDP version, so TCP should be 
slower. So I'm a bit surprised that in your benchmarks TCP 
outperformed UDP, but perhaps it's just some tuning in the socket 
settings (any info on these will be appreciated).

Btw my benchmark report can be found here:
http://www.pjsip.org/testing/0.5.6.5/bench.htm

-benny

Cullen Jennings wrote:
> discuss.  Just to validate some of your results - this is not the first 
> time I have seen performance results where TLS (or more specifically 
> TCP) has better performance that UDP. I have seen this on other products 
> too and when I think about it, it is not that surprising.
> 
> Cullen
> 
> 
>> UAC <-TLS-> openser <-TLS-> UAS (all of them got its own 2xP3 1,26GHz 
>> machine, linked with 100MBit). Simple forwarding configuration (like a 
>> loadbalancer), no DNS lookups or DB queries.
>>
>> UDP: ~620 cps
>> I think TLS performs better as because of UDP retransmissions, which 
>> "kills" the proxy if it becomes slow.
>>

---

From: [email-masked]
Date: Tue, 01 Aug 2006 11:57:07 +0200

"I think TLS performs better as because of UDP retransmissions, which 
"kills" the proxy if it becomes slow. "

That's why I think it is not possible to make performance testing  in 
UDP without activating some kind of flow regulation mechanism... 
overwise the results are very difficult to interpret: what does it mean 
to test a proxy overloaded by retransmissions?.... Should always test 
performances on a stable rate, overwise this is a resistance/torture test...

Cullen Jennings wrote:
> specifically TCP) has better performance that UDP. I have seen this on 
> other products too and when I think about it, it is not that surprising.
>
> Cullen
>
>
>> machine, linked with 100MBit). Simple forwarding configuration (like 
>> a loadbalancer), no DNS lookups or DB queries.
>>
>> UDP: ~620 cps

---

From: Klaus Darilion <[email-masked]>
Date: Wed, 02 Aug 2006 10:44:38 +0200

Benny Prijono wrote:
> connection) performs almost consistently better than UDP in transactions 
> and calls processing, but only when UAC and UAS are in local machine. 
> When UAC and UAS are separated into different boxes, TCP performance is 
> only about half of UDP performance.
> 
> transactions has lower cost to setup (e.g. less timers), so it should be 
> faster. While on the network, the TCP round-trip probably prevents 
> slower. So I'm a bit surprised that in your benchmarks TCP outperformed 
> UDP, but perhaps it's just some tuning in the socket settings (any info 
> on these will be appreciated).

Hi Benny!

Are you using a new TCP connection for each call or do you use 
connection reuse? In my case, all calls where sent over the same TLS 
connection, thus only one TCP/TLS handshake per test case, not per call.

regards
klaus

---

From: Klaus Darilion <[email-masked]>
Date: Wed, 02 Aug 2006 10:47:19 +0200

> performances on a stable rate, overwise this is a resistance/torture 

Yes. Thus in my tests, I increased the call/second slowly (watching 
retransmissions) until I reach the limit (and the call rate is getting 
down due to proxy overload.). And this limit is the maximum number of calls.

btw: does someone know another tool beside sipp (whose TCP 
implementation is buggy)?

regards
klaus

---

From: Benny Prijono <[email-masked]>
Date: Wed, 02 Aug 2006 10:18:45 +0100

Hi Klaus!

> connection, thus only one TCP/TLS handshake per test case, not per call.

Same, I also use one TCP socket for all calls.

 > implementation is buggy)?

I have a tool to send bulk requests/transactions/calls to a server 
(with UDP or TCP) and measure the completion time of the 
responses/transactions/calls. It's called pjsip-perf, and it's 
included in the source tarball from http://www.pjsip.org/download.htm

-benny

---

From: Cullen Jennings <[email-masked]>
Date: Fri, 4 Aug 2006 15:15:36 -0700

Well if you are benchmarking how fast a single transaction can be  
process, what you are saying makes sense. However, I think we were  
benchmarking the maximum number of transactions per second (or call  
per second) a proxy could support in a loaded network which  often  
comes up with different results.

On Aug 1, 2006, at 2:38 AM, Benny Prijono wrote:

> transactions has lower cost to setup (e.g. less timers), so it  
> should be faster. While on the network, the TCP round-trip probably  
> prevents transaction to complete as fast as the UDP version, so TCP  
> should be slower. So I'm a bit surprised that in your benchmarks  
> TCP outperformed UDP, but perhaps it's just some tuning in the  
> socket settings (any info on these will be appreciated).
>
>> Klaus, first of all thank you for producing some real results we  
>> can discuss.  Just to validate some of your results - this is not  
>> the first time I have seen performance results where TLS (or more  
>> specifically TCP) has better performance that UDP. I have seen  
>> this on other products too and when I think about it, it is not  
>> that surprising.
>> Cullen

---

From: Benny Prijono <[email-masked]>
Date: Sat, 05 Aug 2006 00:11:33 +0100

Not sure if I understand you correctly (why would anyone benchmark how 
fast a single transaction can be processed), but my observation was 
based on the time to complete 10,000 transactions or calls, then 
calculate the tps/cps. If you think this doesn't make sense, then I 
think we have something interesting to discuss.

-benny

Cullen Jennings wrote:
> benchmarking the maximum number of transactions per second (or call per 
> second) a proxy could support in a loaded network which  often comes up 
> with different results.
> 
>