Subject: [hybi] Update: WebSocket test suite reports

From: Tobias Oberstein <[email-masked]>
Date: Thu, 18 Oct 2012 02:08:26 -0700

Hello,

we have updated reports for WebSocket protocol tests for browsers (Chrome, Firefox, IE10):

http://autobahn.ws/testsuite/reports/clients/index.html

The results are quite encouraging: all green;)

Mobile and server reports will follow up.

Cheers,
Tobias

Couple of notes:

1. The "fail" of IE10 for Case 2.10 is actually testsuite fault (the spec does not require respnding to each and every ping).

2. As can be seen from the 6.4.* Cases, IE10 does fail fast on invalid UTF-8 _per frame_. Firefox and Chrome only fail per-message (no fail fast). Autobahn does fail-fast even within frames, on the very first offending octet.

3. As can be seen from Cases 9.1.*/9.2.*, Chrome Canary has improved it's large payload processing efficiency a lot compared to current released Chrome.

4. The "non-strict" case outcomes for Firefox are due to it's asynchronous processing of WebSocket. That's perfectly fine. The spec does not require to process strictly everything before a break of the protocol. All other clients however seem to do that.

5. The handling of close codes is somewhat inconsistent/different between clients.

6. As can be seen from comparing the columns AutobahnClient and AutobahnClient/PyPy, running PyPy vastly improves Autobahn's performance for large payloads. The main bottlenecks improved are masking, UTF-8 validation and general bit banging/shuffling.

Caveat: the AutobahnClient/PyPy was running locally next to Autobahn Testsuite, different from all the other clients.

7. Safari is not included, since Apple seems to have stopped shipping Safari for Windows (and stopped delivering updates for my old Macbook).

Test setup:

Machine 1:
 - Core i7, 3.4 Ghz, 12GB Ram
 - Win8 Rel. Prev. 64 Bit
 - Browsers and Autobahn on CPython

Machine 2:
 - Core i7, 3.4 Ghz, 12GB Ram
 - VirtualBox VM running Ubuntu 10.04 LTS on Win7 64-Bit host
 - Autobahn Testsuite running on PyPy (current trunk compiled from source)
 - Autobahn Client PyPy

Network: 1GbE fully switched

---

From: Peter Thorson <[email-masked]>
Date: Thu, 18 Oct 2012 13:07:43 -0500

For those curious about Safari 6, it behaves identically to Chrome 22 on Autobahn tests. (Hurray!)

Raw results: (test suite run on cpython so 9.x performance tests may not be super accurate)
http://zaphoyd.com/websocketpp/autobahn-report-client/

On Oct 18, 2012, at 04:08 , Tobias Oberstein <[email-masked]> wrote:

> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi