Subject: [hybi] Testsuite update

From: Tobias Oberstein <[email-masked]>
Date: Mon, 31 Oct 2011 08:05:49 -0700

New release 0.4.3 of Autobahn WebSockets including updates to test suite:

+ supports Hybi-17
+ detailed tracking of close behavior
+ initial set of 30 cases for close handshake
+ various test suite and protocol options

Special thanks to Peter Thorson for significant code contributions
and fruitful discussions!

Peter works on a  C++/ASIO-based WebSocket client/server framework
you may checkout here https://github.com/zaphoyd/websocketpp

Updated reports are available under

http://www.tavendo.de/autobahn/testsuite.html

including mobile clients and servers.

Updated Python package is here:

http://pypi.python.org/pypi/autobahn

Since "close behavior" is new coverage for the test suite, there will
likely be questions/issues .. feel free to give feedback on

http://groups.google.com/group/autobahnws

Cheers,
Tobias

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 31 Oct 2011 16:42:39 +0100

> http://pypi.python.org/pypi/autobahn

Hi Tobias, I've installed autobhan 0.4.3 with easy_install (python
2.7). Now, where is supposed to be the Autobahn/ directory??

The egg is installed under
/usr/local/lib/python2.7/dist-packages/autobahn-0.4.3-py2.7.egg
(Ubuntu) but there is not the Autobahn/ directory with the testsuite.

I got it some time ago but don't remember now.

Thanks a lot.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Micheil Smith <[email-masked]>
Date: Mon, 31 Oct 2011 16:02:34 +0000

You're probably better off cloning the git repository and running the tests 
(I couldn't work it out with easy_install so I've done that instead).

Regards,
Micheil Smith
-- 
Pusher.com

On 31 Oct 2011, at 15:42, Iñaki Baz Castillo wrote:

> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 31 Oct 2011 17:31:28 +0100

> + various test suite and protocol options

It's great to see that my server implementation passes all the new
tests without requiring modifications :)

  http://public.aliax.net/oversip/websocket-autobahn-results/

NOTE: Upon receipt of a close frame I send a close frame with not
status code (rather than using 1000). IMHO both are valid.

Regards.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Tobias Oberstein <[email-masked]>
Date: Mon, 31 Oct 2011 09:47:23 -0700

> NOTE: Upon receipt of a close frame I send a close frame with not status
> code (rather than using 1000). IMHO both are valid.

just a note: so i.e. with 7.5.1, where the close reason sent is invalid utf-8,
you reply by close without code/reason, but for invalid utf-8 in msgs,
you close with 1007. is that by design?

another note: this release only has the "simple cases" for closing.
more interesting stuff (like i.e. timeouts) will be in next release;)

---

From: Peter Thorson <[email-masked]>
Date: Mon, 31 Oct 2011 11:52:38 -0500

> status code (rather than using 1000). IMHO both are valid.

The websocket spec indicates that both 1000 and "no code" are valid. The test suite will allow either as passing behavior and will note which of the passing behaviors the endpoint being tested used. If you send a close code that is illegal or does not make sense for the situation or do not send a close frame when one was required the close behavior portion of the test will be marked failed.

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 31 Oct 2011 18:18:12 +0100

> you close with 1007. is that by design?

Yes, do you think it is wrong?

> more interesting stuff (like i.e. timeouts) will be in next release;)

Great.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 31 Oct 2011 18:19:24 +0100

> Yes, do you think it is wrong?

Also, I suggest that the column in which you write "1XXX" or "none"
you also write "null" in case no close frame has been received (but
just TCP disconnection).

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 31 Oct 2011 18:38:17 +0100

> Yes, do you think it is wrong?

Sorry, I didn't understand test 7.5.1. Indeed it sends a close frame
with invalid UTF-8 in the description.

Yes, my server does not inspect that.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Tobias Oberstein <[email-masked]>
Date: Mon, 31 Oct 2011 11:07:56 -0700

> >> just a note: so i.e. with 7.5.1, where the close reason sent is
> >> invalid utf-8, you reply by close without code/reason, but for
> >> invalid utf-8 in msgs, you close with 1007. is that by design?
> >

My reading/interpretation of the spec currently is:

Upon receiving a close frame with i.e. invalid utf8 in reason, a peer may

1. reply with close, no code/reason
2. reply with close, code = 1002/1007
3. immediately drop TCP

A reply with code != 1002/1007, i.e. 1000 is wrong.

Of above 3 options, my preference would be

Do 2., when the implementation also responds gracefully to protocol violations
in different areas (i.e. receive invalid utf8 in text message).

Do 3., when the implementation also responds brutally to protocol violations
in different areas.

The reasoning is: then the implementation behaves consistently .. either
be brutal, or be informative.

Anyway, behavior 1. is also valid wrt spec.

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 31 Oct 2011 19:11:09 +0100

> A reply with code != 1002/1007, i.e. 1000 is wrong.

I agree. I've already fixed it in my server.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Micheil Smith <[email-masked]>
Date: Mon, 31 Oct 2011 18:37:19 +0000

Hi Tobias,

Nice work with the new version, I've just been running it against em-websocket today, 
and picked up on a fair few areas that we could improve on.

However, one thing that I'm curious about is how I can test multiple versions of the 
websocket protocol at once (em-websocket has some support for most major versions).

When passing an options object into a server, the test runner crashes: 
https://gist.github.com/1328400

Regards,
Micheil Smith
--
Pusher.com

---

From: Tobias Oberstein <[email-masked]>
Date: Mon, 31 Oct 2011 11:52:49 -0700

> However, one thing that I'm curious about is how I can test multiple versions
> of the websocket protocol at once (em-websocket has some support for
> most major versions).
> 

I'm not sure if I get it, since the spec you posted

{
   "servers": [
               {"agent": "EMWebSocket", "url": "ws://localhost:8080", "options": {"version": 17}},
               {"agent": "AutobahnServer", "url": "ws://localhost:9000", "options": {"version": 17}}
               ],

   "cases": ["*"],
   "exclude-cases": ["2.*"],
   "exclude-agent-cases": {}
}

should work just fine.

You can "options" globally, and per server.

For the options available, see
http://www.tavendo.de/autobahn/doc/python/websocketclient.html#client-factory
setProtocolOptions()

===

It wont test multiple versions though. You can only do that (currently) by starting your server
multiple times on different ports and then have a spec like this:

{
   "servers": [
               {"agent": "EMWebSocket1", "url": "ws://localhost:8080", "options": {"version": 10}},
               {"agent": "EMWebSocket2", "url": "ws://localhost:8081", "options": {"version": 13}},
               {"agent": "EMWebSocket3", "url": "ws://localhost:8082", "options": {"version": 17}},
..

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 31 Oct 2011 20:05:19 +0100

> + various test suite and protocol options

Hi, I cannot connect to a wss URI, the client script fails with:

2011-10-31 20:04:05+0100 [-]   File
"/usr/local/lib/python2.7/dist-packages/autobahn-0.4.3-py2.7.egg/autobahn/websocket.py",
line 142, in connectWS
2011-10-31 20:04:05+0100 [-]     raise Exception("Secure WebSocket
connection requested, but no SSL context factory given")
2011-10-31 20:04:05+0100 [-] Exception: Secure WebSocket connection
requested, but no SSL context factory given

There is no connection attempt.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Micheil Smith <[email-masked]>
Date: Mon, 31 Oct 2011 19:18:07 +0000

I can't get it to work at all, this is with python 2.6.1 and autobahn 0.4.3.

I'm going to try upgrading to 2.7.2, and I'll respond off list with any 
further problems. (or on an Autobahn mailing list, if one exists).

– Micheil

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 31 Oct 2011 20:20:48 +0100

2011/10/31 Micheil Smith <[email-masked]>:
> (or on an Autobahn mailing list, if one exists).

Yes there is:

-- 
Iñaki Baz Castillo
<[email-masked]>