Subject: [hybi] Autobahn feedback

From: Piotr Kulaga <[email-masked]>
Date: Sat, 10 Dec 2011 00:03:23 +0000

Hi Tobias,

I went through the Autobahn client tests and I have some feedback regarding some of them:

*         2.10: The test requires to reply to every ping which is against the spec. It should check that the client replied just to the last one.

*         3.2, 3.3, 3.4, 4.1.3, 4.1.4, 4.1.5, 4.2.3, 4.2.4, 4.2.5, 5.15: The tests may return non-strict result because of a race. All of them are constructed in a similar way: send a valid message followed by an invalid frame. The expectation is to receive a valid message, but it races with an RST sent due to terminating the connection because of the invalid frame.

*         7.9.10, 7.9.11: Error codes 5000+ are not explicitly forbidden, so I think that they should be allowed (at least on the protocol layer, the browser behavior is different).

Thanks,
Piotr

---

From: Tobias Oberstein <[email-masked]>
Date: Tue, 13 Dec 2011 01:15:25 -0800

> * 2.10: The test requires to reply to every ping which is against the spec. It should check that the client replied just to the last one.

Yes, the case should report non-strict when not every ping is responded to. The fact that the case currently does not behave
so is only documented in the case description currently (and the test report I did for IE10 some time ago).

My problem with constructing a tight test is .. the spec says:

   If an endpoint receives a Ping frame and has not yet sent Pong
   frame(s) in response to previous Ping frame(s), the endpoint MAY
   elect to send a Pong frame for only the most recently processed Ping
   frame.

So if an implementation just pretends to never have had time to process a received ping and just waits for the next one
(for that being the last ping) would be formally valid ...

Iow: what is the formal condition upon which a pong is _required_, and not just optional?

> * 3.2, 3.3, 3.4, 4.1.3, 4.1.4, 4.1.5, 4.2.3, 4.2.4, 4.2.5, 5.15: The tests may return non-strict result because of a race. All of them are constructed in a similar way: send a valid message followe…

I'm not sure if I can follow you here. You can see the difference strict vs non-strict for those cases (in part) when comparing Firefox
and Chrome. Firefox will not (always) process data received before an invalid frame but still unprocessed. Chrome and Autobahn
will strictly process all data up to the invalid frame. So the test suite does recognize the difference.

Are you saying that the testsuite _itself_ has a race bug which makes it fail to recognize a strict behavior for those cases and
incorrectly marks cases as non-strict for some implementation? If so, are we talking about IE 10? If so, does the latest
update to Windows 8 Developer Preview allow me to check that for myself?

> * 7.9.10, 7.9.11: Error codes 5000+ are not explicitly forbidden, so I think that they should be allowed (at least on the protocol layer, the browser behavior is different).

Yes, right. The spec only talks about the status code being an integer and certain ranges being reserved for certain purposes.
It does not talk about the rest of the integer range (>= 5000). My implicit assumption was that this is forbidden. Since thats
not supported by the spec, I'd make the case outcome non-strict.

---

From: Peter Thorson <[email-masked]>
Date: Tue, 13 Dec 2011 06:41:08 -0600

> not supported by the spec, I'd make the case outcome non-strict.

The IANA registration section of RFC6455 includes the following: 

[Page 63]
Status Code
      The Status Code denotes a reason for a WebSocket connection
      closure as per Section 7.4 of this document.  The status code is
      an integer number between 1000 and 4999 (inclusive).

Section 7.4 itself though, as pointed out, does not explicitly forbid codes in the range >=5000.