Subject: [hybi] WebSockets Test Suite / Browser Status

From: Tobias Oberstein <[email-masked]>
Date: Wed, 17 Aug 2011 09:04:57 -0700

Regarding WS test suites for servers and browser status ..

Cheers,
Tobias

1.
Autobahn WebSockets now (0.3.2) includes an automated test suite for both clients and servers.

http://www.tavendo.de/autobahn/testsuite.html

2.
The fixes for Chrome have landed on v15 .. all green now.

http://www.tavendo.de/autobahn/testsuite/report/clients/index.html

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Thu, 18 Aug 2011 16:44:40 +0200

> http://www.tavendo.de/autobahn/testsuite.html

Really nice.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Brian <[email-masked]>
Date: Thu, 18 Aug 2011 16:52:50 -0700

Very nice!  The test suite helped me identify and fix a few bugs in my
implementation.  All the tests now pass for WebSocket-Node except about 5 of
the performance related ones.  Turns out my implementation doesn't do so
well with large messages sent with small frame sizes, but seems to do much
better with larger frame sizes.  I guess I have some optimization to do with
handling smaller fragments.

Results compared with Autobahn:

http://worlize.github.com/WebSocket-Node/test-report/servers/

Brian

On Wed, Aug 17, 2011 at 9:04 AM, Tobias Oberstein <
[email-masked]> wrote:

> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi
>

---

From: Tobias Oberstein <[email-masked]>
Date: Fri, 19 Aug 2011 00:06:41 -0700

> implementation.  All the tests now pass for WebSocket-Node except about 5 of

cool! nice to hear that is has some value.
 
> the performance related ones.  Turns out my implementation doesn't do so well
> with large messages sent with small frame sizes, but seems to do much better
> with larger frame sizes.  I guess I have some optimization to do with handling
> smaller fragments.
> 

interesting: for the performance tests (apart from heavily
fragmented/chopped up ones), your implementation is twice as fast.

that was under equal conditions? and that is node.js/v8?

could you point me to the place in your code that does the demasking?

i'd be interested, since that's the bottleneck in my code .. did already
experiment with a couple of options. something which is trivial to do
performant in C isn't the same with Python.

---

From: Brian <[email-masked]>
Date: Fri, 19 Aug 2011 00:47:19 -0700

> that was under equal conditions? and that is node.js/v8?

My implementation is indeed for node.js/v8.
Yes, equal conditions.  I ran the Autobahn echo server on port 9000 and my
echo server on port 8080, then configured fuzzing_client_spec.json to test
both consecutively.  The tests were run on my MacBook Pro (Summer 2009):

Core 2 Duo 2.66 GHz
8GB Memory
Mac OS X 10.7.1
Python v2.7.1
Node.js v0.4.10

My echo server is here:
https://github.com/Worlize/WebSocket-Node/blob/master/test/echo-server.js

> could you point me to the place in your code that does the demasking?

Sure.  Here it is:
https://github.com/Worlize/WebSocket-Node/blob/996c70e442a35dfc8c61e55e5afa7ca0b319afd4/lib/WebSocketFrame.js#L146

> performant in C isn't the same with Python.

Interesting.  I think the bottleneck in my code is memory allocations and
buffer copying.  :-/  Probably going to refactor how I'm receiving bytes off
the wire, add a streaming interface, and refactor my frame aggregation code
to use the streaming interface.

Cheers,
Brian

---

From: Sylvain Hellegouarch <[email-masked]>
Date: Sat, 20 Aug 2011 00:09:07 +0200

> http://www.tavendo.de/autobahn/testsuite/report/clients/index.html

Thanks a lot Tobias. Your test suite is very useful. I've found a few issues
with my WebSocket support in ws4py [1]. Now all the tests pass against the
CherryPy server aside from some performance tests which I need to address.

-- 
- Sylvain
http://www.defuze.org
http://twitter.com/lawouach

[1] https://github.com/Lawouach/WebSocket-for-Python

---

From: Peter H Thorson <[email-masked]>
Date: Fri, 19 Aug 2011 16:56:27 -0500

On 2011-Aug-18, at 18:52, Brian wrote:

> Very nice!  The test suite helped me identify and fix a few bugs in my implementation.  All the tests now pass for WebSocket-Node except about 5 of the performance related ones.  Turns out my implementation doesn't do so well with large messages sent with small frame sizes, but seems to do much better with larger frame sizes.  I guess I have some optimization to do with handling smaller fragments.
> 
> On Wed, Aug 17, 2011 at 9:04 AM, Tobias Oberstein <[email-masked]> wrote:

I want to second this, this server test suite found a number of oversights for me to fix and has been very helpful overall for debugging a websocket server.

For anyone interested, here is a report done of the autobahn server vs my websocketpp server written in C++ using boost asio's asynchronous networking library. So far my effort has been focused on correctness and clarity and less on performance. Unsurprisingly, unmasking in C++ seems to be less a bottleneck than the rest of the frame header and network io overhead.

http://zaphoyd.com/websocketpp/autobahn-report/

Lastly, can you clarify the difference between fragments and "chops", is the difference just that the chops are very small?

Thanks again,

Peter

---

From: Brian McKelvey <[email-masked]>
Date: Fri, 19 Aug 2011 16:20:54 -0700

As far as I can tell, chops are referring to the underlying tcp packet size.

Sent from my iPhone

---

From: Tobias Oberstein <[email-masked]>
Date: Sat, 20 Aug 2011 09:00:34 -0700

I'm trying to answer/comment all recent mails regarding the topic in one mail, since I'm a
bit afraid that this thread could be seen as OT. I apologize.

So if there are further questions specifically related to Autobahn Testsuite, I'm inviting you
to join http://groups.google.com/group/autobahnws

If you have doubts regarding test case behavior, that should still get to this list, when it's
doubts routed in the spec or interpretation thereof i.e.

==

That said, I'm glad that the little tool finds practical use.

I think protocol conformance, and by that interoperability is import for WebSockets to reach
broad adoption. I really want to see WebSockets suceed.

I also think that for adoption, there is the user side - developers _using_ WS implementations.

We should make things easy and transparent for them.

So, since a couple of you already posted links to test reports, and I think that users of your
libraries might be interested in that, but also other information (like features in general),
I've started a

http://en.wikipedia.org/wiki/User:Oberstet/Comparison_of_WebSocket_implementations

where we can include pointers to test results.

This page is not meant as a shootout of any kind … there are so many factors involved
that there is enough room for many implementations. I.e. Sylvain's impl. is Python
as is (currently) Autobahn, but his has no external deps., mine has Twisted - which
is both an adv. and disadv. depending on how you look at it.

IMHO having a Wikipedia page can serve well as a neutral ground to host such stuff.

Goal is to make information there correct, fair and comprehensive - serving implementors
and users.

Please edit that / add your stuff … it's not published yet, _anyone_ can edit it.
Feel free to add more comparison aspects, criticise or discuss using the Wikipedia
discussion feature.

Cheers,
Tobias

==

@Andy:
I've added libwebsockets, but left everything as ?.
I added, since your's is both on http://en.wikipedia.org/wiki/Websockets#Libraries and you are on this list.
If it's unwelcome, drop me a line. If not, but you're too lazy, give me the facts rgd. libwebsockets, I'll
hack it into the awkward Wiki syntax. I think libwebsockets should really be there ..

@Sylvain:
As said above, it's very useful to have a no-frills, low-entry barrier, no-dependencies WS Python client library.
When I get that right, there is the _option_ of running within Tornado/CherryPy? If so, the Wikipage is
wrong .. pls help ..

@Brian:
Node.js is obviously important .. pls come on board and help with the Wikipage.

@Peter:
1.
C++/ASIO vs poor little Python/Twisted .. unfair fight;) No, indeed interesting to see the relative performance!

2.
Fragments/Chops
fragments = payload of WS frames of a fragmented WS message
chops = "bytes on wire"

Chopped tests serve 2 purposes:

a) The idea is to stress the implementation along three axis: message size, fragment size and chop size.

b) The intention with chops additionally is: test whether the implementation is agnostic to in what
chops it receives bytes. It must be "TCP clean".

Caution: strictly controlling the chops which get on the wire is hard to next impossible. I probably
post something on a yet-to-be-written Autobahn FAQ.

Practically, the deal is: if you see some case red for the chopped tests, something is buggy with
the implementation. If it's green, that does not necessarily mean it is TCP clean.

3. the Wikipedia comparison
I have not included any pointers to your report/impl. since I guess it's commercial, and I don't
know your plans. If you want it there, feel free of course to include it though.

Von: [email-masked] [mailto:[email-masked]] Im Auftrag von Peter H Thorson
Gesendet: Freitag, 19. August 2011 23:56
An: [email-masked]
Betreff: Re: [hybi] WebSockets Test Suite / Browser Status

Brian

On Wed, Aug 17, 2011 at 9:04 AM, Tobias Oberstein <[email-masked]<mailto:[email-masked]>> wrote:

Cheers,
Tobias

1.

2.

Thanks again,

Peter

---

From: Sylvain Hellegouarch <[email-masked]>
Date: Sun, 21 Aug 2011 16:16:43 +0200

> ** **
>
> That said, I'm glad that the little tool finds practical use.****
>
> ** **
>
> I think protocol conformance, and by that interoperability is import for
> WebSockets to reach****
>
> broad adoption. I really want to see WebSockets suceed.****
>
> ** **
>
> I also think that for adoption, there is the user side - developers _using_
> WS implementations.****
>
> ** **
>
> We should make things easy and transparent for them.****
>
> ** **
>
> So, since a couple of you already posted links to test reports, and I think
> that users of your****
>
> libraries might be interested in that, but also other information (like
> features in general),****
>
> I've started a****
>
> ** **
>
>
> ****
> ** **
> where we can include pointers to test results.****
>
> ** **
>
> This page is not meant as a shootout of any kind … there are so many
> factors involved****
>
> that there is enough room for many implementations. I.e. Sylvain's impl. is
> Python****
>
> as is (currently) Autobahn, but his has no external deps., mine has Twisted
> - which****
>
> is both an adv. and disadv. depending on how you look at it.****
>
> ** **
>
> IMHO having a Wikipedia page can serve well as a neutral ground to host
> such stuff.****
>
> ** **
>
> Goal is to make information there correct, fair and comprehensive - serving
> implementors****
>
> and users.****
>
> ** **
>
> Please edit that / add your stuff … it's not published yet, _anyone_ can
> edit it.****
>
> Feel free to add more comparison aspects, criticise or discuss using the
> Wikipedia****
>
> discussion feature.
>

That's a great resource and I like the idea. I'll add more details about
ws4py.

@Sylvain:****
>
> As said above, it's very useful to have a no-frills, low-entry barrier,
> no-dependencies WS Python client library.****
>
> When I get that right, there is the _option_ of running within
> Tornado/CherryPy? If so, the Wikipage is****
>
> wrong .. pls help ..****
>
> **
>

Will do.
By the way, here are the latest results of ws4py (CherryPy server) against
Autobahn.

http://www.defuze.org/oss/ws4py/testreports/servers/cherrypy/