Subject: [hybi] Draft websocket interoperability tests

From: Greg Wilkins <[email-masked]>
Date: Mon, 11 Apr 2011 11:36:57 +1000

As promised, here is a draft of the interoperability tests that I've proposed

    http://www.ietf.org/staging/draft-wilkins-hybi-websocket-tests-00.txt

I'd really appreciate co-authors to step forward to both improve the
quality of this document and to suggest more tests

cheers

---

From: Micheil Smith <[email-masked]>
Date: Mon, 11 Apr 2011 12:04:38 +1000

Greg: do you have a sample implementation of a few of these tests, some of the language used 
is a little bit confusing here.

Although, I'm sure it's just my lack of skill in reading IETF drafts.

– Micheil

On 11/04/2011, at 11:36 AM, Greg Wilkins wrote:

> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 11 Apr 2011 11:07:00 +0900

>      http://www.ietf.org/staging/draft-wilkins-hybi-websocket-tests-00.txt

That's a strange URI. 'staging' means it may not be correctly published. 
Can you please make sure it's published correctly and you get an URI such as

http://www.ietf.org/id/draft-wilkins-hybi-websocket-tests-00.txt or
http://tools.ietf.org/html/draft-wilkins-hybi-websocket-tests-00

or so before you tell us to read it?

Thanks,    Martin.

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 11 Apr 2011 16:05:34 +1000

> That's a strange URI. 'staging' means it may not be correctly published.

I must have transgressed some written rule of ietf draft submission as
it wants manual process and says that it will take 2 days.
You can view status here:

   https://datatracker.ietf.org/idst/status.cgi?passed_filename=draft-wilkins-hybi-websocket-tests

cheers

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 11 Apr 2011 16:14:10 +1000

> Although, I'm sure it's just my lack of skill in reading IETF drafts.

More likely my lack of skill in writing them.

Jetty has an implementation of these tests and you can download and run it with:

  wget -O jetty-all.jar
https://oss.sonatype.org/content/groups/jetty-with-staging/org/eclipse/jetty/aggregate/jetty-all/7.4.0.RC0/jetty-all-7.4.0.RC0.jar
  wget --user-agent=something
http://repo2.maven.org/maven2/javax/servlet/servlet-api/2.5/servlet-api-2.5.jar
  java -cp jetty-all.jar:servlet-api-2.5.jar
org.eclipse.jetty.websocket.TestServer --help
  java -cp jetty-all.jar:servlet-api-2.5.jar
org.eclipse.jetty.websocket.TestServer --port 8080

then in another window

 java -cp jetty-all.jar:servlet-api-2.5.jar
org.eclipse.jetty.websocket.TestClient --help
 java -cp jetty-all.jar:servlet-api-2.5.jar
org.eclipse.jetty.websocket.TestClient --port 8080 --verbose
--protocol echo-fragment  --size 12 --fragment 3

cheers

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 12 Apr 2011 08:00:02 +1000

On 12 April 2011 05:43, SM <[email-masked]> wrote:
> Hi Greg,
> At 18:36 10-04-2011, Greg Wilkins wrote:
>>
> I suggest taking a look at RFC 5657 as it provides some guidance about
> interoperability and implementation reports.

Well I wasn't intending to presume to be making an interoperability
report for the protocol (but it looks like such a thing would be
helpful to move the protocol through the process).

RFC5657 does briefly talk about tests and appears to imply that for
report purposes it  prefers either realworld feedback or tests derived
from real world data, rather than unit tests designed to test
particular features.    This suggests that the testing approach I'm
advocating may have limited applicability when the time comes for
writing the real interoperability report.

However, I do believe it is very useful to have a series of known
tests so that we can quickly check at least the basics between
implementations.  If such testing does not fit in the IETF process,
I'm more than happy to drop the ietf from these test names and make
them an informal agreement between implementers.

cheers

---

From: SM <[email-masked]>
Date: Mon, 11 Apr 2011 23:24:04 -0700

Hi Greg,
>helpful to move the protocol through the process).

Yes.

>writing the real interoperability report.

Ok.

>them an informal agreement between implementers.

It can be published as an individual submission if you want 
that.  You could leave the test names as they are and see how it goes. :-)

Regards,
-sm

---

From: Andy Green <[email-masked]>
Date: Mon, 25 Apr 2011 22:35:24 +0800

> quality of this document and to suggest more tests

Hi -

I started looking at this since I have some time stuck in a hotel room.

I think the echo one might be misconceived.  It has this idea that 
frames must be kept as they came in, and that there is a frame size 
limit in the server.  Both seem wrong to me, but I'm willing to hear 
reasons why that's because I am a chump.

By the time a protocol handler is getting payload coming, the framing 
boundaries are burned off.  By nature too the framing flags if not gone 
will have to have been converted to something less 
protocol-version-specific.

In the same way, is it natural that the protocol handler, this is like 
"superchat", can twiddle reserved bits in the framing?  That's what echo 
is asking for right now, seems wrong.  An extension might have that 
power but not the protocol layer.

How about redefining it to only echo payload, with no promise to keep 
framing structure that came in?

-Andy

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 27 Apr 2011 11:36:32 +1000

Andy,

I guess the issue here is that protocols really cannot (or should not)
deal with framing.  It so happens that the jetty implementation does
have the option of handling framing, hence I've been able to implement
them.

But generally speaking, I don't think that will be the case.  So
perhaps to test the framing/fragmentation we need to specify
extensions rather than protocols,

I'll reconsider the tests in that light in the next few days.

cheers

On 26 April 2011 00:35, Andy Green <[email-masked]> wrote:
> I think the echo one might be misconceived.  It has this idea that frames
> must be kept as they came in, and that there is a frame size limit in the
> server.  Both seem wrong to me, but I'm willing to hear reasons why that's
> because I am a chump.
>
> boundaries are burned off.  By nature too the framing flags if not gone will
> have to have been converted to something less protocol-version-specific.
>
> "superchat", can twiddle reserved bits in the framing?  That's what echo is
> asking for right now, seems wrong.  An extension might have that power but
> not the protocol layer.
>

---

From: Andy Green <[email-masked]>
Date: Wed, 27 Apr 2011 15:27:58 +0800

> I'll reconsider the tests in that light in the next few days.

Any extension will have to "take over" the link and use the OOB protocol 
naming to agree what it's doing on the link.

It'll have to take it over because it will generate valid traffic 
itself, and it'll have to specify in what it's doing in protocol so its 
peer knows what result to expect so it can confirm it.

I would worry though that you're testing extension oranges when what the 
victi...uh..user experiences is protocol apples.  It's quite possible 
actual traffic to protocol layer is broken and these extension-based 
tests tell you everything is cool because they're using totally 
different code.

You can solve it well just by prioritizing this code coverage action 
over the test definition, ie, just define "echo" to give you the same 
payload back (in the same text or binary mode) but say nothing about how 
many frames it is coming in, etc - you can't control that anyway if 
someone inbetween muxed it and it's all reframed.  Then you just have a 
real simple protocol handler and the test is testing something you 
really want to know about the critical code paths.

-Andy

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 27 Apr 2011 18:32:36 +1000

Andy,

my current thinking is that the protocols should be things like echo
and echo-broadcast, which operate on entire messages.
Then I think we can have some extensions that do influence framing, at
least on the sending side.

cheers

On 27 April 2011 17:27, Andy Green <[email-masked]> wrote:
> It'll have to take it over because it will generate valid traffic itself,
> and it'll have to specify in what it's doing in protocol so its peer knows
> what result to expect so it can confirm it.
>
> victi...uh..user experiences is protocol apples.  It's quite possible actual
> traffic to protocol layer is broken and these extension-based tests tell you
> everything is cool because they're using totally different code.
>
> You can solve it well just by prioritizing this code coverage action over
> the test definition, ie, just define "echo" to give you the same payload
> back (in the same text or binary mode) but say nothing about how many frames
> it is coming in, etc - you can't control that anyway if someone inbetween
> muxed it and it's all reframed.  Then you just have a real simple protocol
> handler and the test is testing something you really want to know about the
> critical code paths.
>
> -Andy
>

---

From: Andy Green <[email-masked]>
Date: Wed, 27 Apr 2011 16:48:50 +0800

> least on the sending side.

Right but normally the protocol layer can itself influence frame 
fragmentation anyway by being selective about how it presents its 
payload to the lower layers.  Ie, gets told about 256 bytes that just 
came in, can decide to send a 256 byte lump, or 16 x 16 byte lumps or 
random size ones and each lump sent down the stack will translate to a 
frame.

Because incoming monster frames are still allowed, basically the echo 
server will have to spill at some small arbitrary limit, ie, start 
sending the reply at the time the first packet of the original came.

So we can talk about "operating on entire messages" and that's kind of 
accurate, but what's really going on is more in the way of shoving stuff 
back out as soon as anything has come in, until it stops coming in.

I don't know what we are asking this proposed extension to actually do 
for us then, more than fragmentation we can control enough from 
protocol-land anyway.  It seems to me a small and selfcontained 
protocol-layer-only implementation can be done if we realize we are not 
guaranteeing anything other than the whole payload must come back and 
server did chop up the payload randomly in the fraggle-type case.  (And 
mux might have chopped it further, or some intermediary re-coalesce it 
for us, but we need only test the payload is still golden at message level).

-Andy