Subject: [hybi] websocket test suite

From: Greg Wilkins <[email-masked]>
Date: Fri, 29 Jul 2011 10:02:25 +1000

All,

I'd like to restart the conversation around developing a standard set
of tests for websocket implementations.  The draft I had previously
published has expired, and reading it again now, I'm not sure it is a
good starting point as the approach adopted was a bit laissez faire
and was just a few test that was felt to check the bulk of the
specified behaviour.   I now think that what is needed is a more
rigorous approach with specific tests designed to verify each section
of the specification.  ie there should be traceability from a clause
in the spec to the test(s) that verify it.

So firstly, does anybody have suggestions of good tests suites
specified for other IETF efforts that we could use as a model to base
our tests on?

Anybody else interested in collaborating on this?

cheers

---

From: Justin Lee <[email-masked]>
Date: Thu, 28 Jul 2011 19:26:36 -0600

I'm actually building up a set of unit tests in grizzly that i'm going to
use for testing 1.9 clients against 2.0 servers, and vice versa.  I was
hoping to test grizzly client code against jetty servers as well
libwebsocket servers, too.  However, having a standard set of tests to
validate against would increase my confidence that I'm implementing
everything correctly on my end.

On Thu, Jul 28, 2011 at 6:02 PM, Greg Wilkins <[email-masked]> wrote:

> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi
>

-- 
You can find me on the net at:
http://antwerkz.com
http://antwerkz.com/+
http://antwerkz.com/linkedin
http://antwerkz.com/twitter

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 29 Jul 2011 13:15:18 +1000

Here is a first cut at a list of tests for section 4 of the spec.  Are
these the kinds of tests people were thinking about?
If so, then I'm happy to go the next step of detail and start
providing more detail for each test.

Obviously some implementation are not going to be able to send some of
the invalid frames or force fragmentation, so some suites will have to
record some tests as skipped, or only testing in one direction etc.

4.2 Base Framing Tests
----------------------

* 7 bit length: send/receive frames of length 0-125
* 16 bit length: send/receive frames of length 126-128, 65536
* 63 bit length: send/receive frames of length 65537

* Non zero RSV: send frames with non zero reserved bits and no
extension. Verify connection is failed.

* Known Opcodes: send/receive frames with text,binary,ping,pong
opcodes (continuation & close tested separately).
* Unknown Opcodes: send frames with unknown opcodes 3-7, B-F. Verify
connection is failed.

4.3 Client to Server Masking
----------------------------

???

4.4 Fragmentation
-----------------

* 1 fragment message: send/receive message in 1 fragment
* 2 fragment message: send/receive message in 2 fragments
* 3 fragment message: send/receive message in 3 fragments
* Injected control: send/receive 2 fragment message with unsolicited
pong message between frames
* Fragmented control: send fragmented ping. Verify that connection is failed.
* Interleaved fragments: send a fragment with FIN set and Continuation
opcode. Verify that connection is failed.
* Handled control: send/receive 2 fragment message with ping message
between frames and last frame delayed. Verify pong is received before
last frame is sent.
* 7 bit fragment: send/receive message with initial/middle/final
fragments of sizes 0-125
* 16 bit fragment: send/receive message with initial/middle/final
fragments of sizes 126-128,65536
* 64 bit fragment: send/receive message with
initial/middle/finalfragments of size 65537

4.5.1 Close Frames
------------------
* Orderly Close: On an idle connection, send a close and receive a close.
* Busy Close: send a close to an endpoint that has sent the first
fragment of a message.
* Simultaneous Close: send a close on both ends of a connection.
* Message after close: attempt to send a message after a close. Verify
message is not received.
* Idle close: ??
* Error Close: ??

4.5.2 Ping
----------
* Empty ping: Send an empty ping, verify pong is received.
* Non empty ping: Send a non empty ping, verify pong is received.

4.5.3 Pong
----------
* Empty pong: Send an empty ping, verify pong is received and is empty.
* Non empty pong: Send a non empty ping, verify pong is received and
has matching content.
* Outstanding ping: Send two pings with different content, reply with
content for only the second. Verify connection stays open.
* Unsolicited pong: Send/recv unsolicited pong.

4.5.4 Data Frames
-----------------
* UTF-8 Text 0000-007f: Send/receive text message with characters in
code point range  U+0000 to U+007F
* UTF-8 Text 0080-07ff: Send/receive text message with characters in
code point range  U+0080 to U+07FF
* UTF-8 Text 0800-ffff: Send/receive text message with characters in
code point range  U+0800 to U+FFFF
* UTF-8 Text 010000-10ffff: Send/receive text message with characters
in code point range  U+010000 to U+10FFFF
* Illegal bytes UTF-8: Send message with invalid UTF-8 byte sequence.
Verify message is not received.
* Illegal codes UTF-8: Send message with invalid UTF-8 code points.
Verify message is not received.
* Binary frames: Send/receive message containing illegal UTF-8 bytes
sequences and code points as binary message.