Subject: Re: [hybi] websocket test suite

From: Salvatore Loreto <[email-masked]>
Date: Fri, 5 Aug 2011 14:07:30 +0300

On 7/29/11 3:02 AM, Greg Wilkins wrote:
> So firstly, does anybody have suggestions of good tests suites
> specified for other IETF efforts that we could use as a model to base
> our tests on?
Hi Greg and all,

I asked Larry Masinter about some good link,

and he told me that

http://www.ietf.org/iesg/implementation-report.html

has all of the IETF implementation reports, including the one for HTTP:
http://www.ietf.org/iesg/implementation/report-http-v11-spec-rev.txt

cheers
/Sal

-- 
Salvatore Loreto
www.sloreto.com

---

From: Ahmed Soliman <[email-masked]>
Date: Tue, 9 Aug 2011 09:45:09 +0000

Greg,

I have the following comments:
1. Scenario to add:
	* Receive/send control frames (e.g. close frame) within a fragmented message. Verify that client/server responds with a close frame.

2. Focusing on interoperability testing:  Shouldn't we narrow the tests below to focus on interoperability testing?  I think some of the negative tests below  fit better in functional testing rather than interoperability testing? What do you think? 
3. Standard process/test implementation: I think this will make it easier to share test implementations and verify interoperability between different protocol implementations. Any thoughts around this?
(e.g. cre ating server endpoints per scenario for server implementation,  using sub-protocol to select between fragmented message/non-fragmented message, .... etc)

Thanks,
Ahmed

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Greg Wilkins
Sent: Thursday, July 28, 2011 8:15 PM
To: Hybi
Subject: Re: [hybi] websocket test suite

Here is a first cut at a list of tests for section 4 of the spec.  Are these the kinds of tests people were thinking about?
If so, then I'm happy to go the next step of detail and start providing more detail for each test.

Obviously some implementation are not going to be able to send some of the invalid frames or force fragmentation, so some suites will have to record some tests as skipped, or only testing in one direction etc.

4.2 Base Framing Tests
----------------------

* 7 bit length: send/receive frames of length 0-125
* 16 bit length: send/receive frames of length 126-128, 65536
* 63 bit length: send/receive frames of length 65537

* Non zero RSV: send frames with non zero reserved bits and no extension. Verify connection is failed.

* Known Opcodes: send/receive frames with text,binary,ping,pong opcodes (continuation & close tested separately).
* Unknown Opcodes: send frames with unknown opcodes 3-7, B-F. Verify connection is failed.

4.3 Client to Server Masking
----------------------------

???

4.4 Fragmentation
-----------------

* 1 fragment message: send/receive message in 1 fragment
* 2 fragment message: send/receive message in 2 fragments
* 3 fragment message: send/receive message in 3 fragments
* Injected control: send/receive 2 fragment message with unsolicited pong message between frames
* Fragmented control: send fragmented ping. Verify that connection is failed.
* Interleaved fragments: send a fragment with FIN set and Continuation opcode. Verify that connection is failed.
* Handled control: send/receive 2 fragment message with ping message between frames and last frame delayed. Verify pong is received before last frame is sent.
* 7 bit fragment: send/receive message with initial/middle/final fragments of sizes 0-125
* 16 bit fragment: send/receive message with initial/middle/final fragments of sizes 126-128,65536
* 64 bit fragment: send/receive message with initial/middle/finalfragments of size 65537

4.5.1 Close Frames
------------------
* Orderly Close: On an idle connection, send a close and receive a close.
* Busy Close: send a close to an endpoint that has sent the first fragment of a message.
* Simultaneous Close: send a close on both ends of a connection.
* Message after close: attempt to send a message after a close. Verify message is not received.
* Idle close: ??
* Error Close: ??

4.5.2 Ping
----------
* Empty ping: Send an empty ping, verify pong is received.
* Non empty ping: Send a non empty ping, verify pong is received.

4.5.3 Pong
----------
* Empty pong: Send an empty ping, verify pong is received and is empty.
* Non empty pong: Send a non empty ping, verify pong is received and has matching content.
* Outstanding ping: Send two pings with different content, reply with content for only the second. Verify connection stays open.
* Unsolicited pong: Send/recv unsolicited pong.

4.5.4 Data Frames
-----------------
* UTF-8 Text 0000-007f: Send/receive text message with characters in code point range  U+0000 to U+007F
* UTF-8 Text 0080-07ff: Send/receive text message with characters in code point range  U+0080 to U+07FF
* UTF-8 Text 0800-ffff: Send/receive text message with characters in code point range  U+0800 to U+FFFF
* UTF-8 Text 010000-10ffff: Send/receive text message with characters in code point range  U+010000 to U+10FFFF
* Illegal bytes UTF-8: Send message with invalid UTF-8 byte sequence.
Verify message is not received.
* Illegal codes UTF-8: Send message with invalid UTF-8 code points.
Verify message is not received.
* Binary frames: Send/receive message containing illegal UTF-8 bytes sequences and code points as binary message.
_______________________________________________
hybi mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/hybi

---

From: Ahmed Soliman <[email-masked]>
Date: Wed, 31 Aug 2011 21:33:53 +0000

Greg,

Are you still planning to add any of these scenarios to the jetty server?

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Ahmed Soliman
Sent: Tuesday, August 09, 2011 2:45 AM
To: Greg Wilkins; Hybi
Subject: Re: [hybi] websocket test suite

Greg,

Thanks,
Ahmed

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Greg Wilkins
Sent: Thursday, July 28, 2011 8:15 PM
To: Hybi
Subject: Re: [hybi] websocket test suite

4.2 Base Framing Tests
----------------------

???

4.4 Fragmentation
-----------------

4.5.1 Close Frames
------------------

4.5.2 Ping
----------

4.5.3 Pong
----------

4.5.4 Data Frames
-----------------

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 1 Sep 2011 10:47:31 +1000

Ahmed,

The autobahn test suite looks to be way ahead of where I got to.
My plan was to integrate jetty to autobahn tests and then review those
tests to see if they can be improved or not.

cheers