Subject: Providing two-way communication back to a server from a Web page

From: Ian Hickson <[email-masked]>
Date: Fri, 9 Jan 2009 22:35:00 +0000 (UTC)

Hi,

In collecting requirements for developing the HTML5 specification, on 
particular problem kept coming up: how to get two-way communication back 
with a server in a cleaner way than repeated client-pull. It can be 
summarised as these requirements:

 - The ability for Web pages to send arbitrary text to the server from a 
   script in a Web page, in the form of ordered discrete blocks.

 - The ability for Web servers to send arbitrary text to a Web page that 
   has established a connection, in the form of ordered discrete blocks.

 - Text messages should support all of Unicode.

 - The protocol should support running over ports 80 and 443, ideally 
   with the ability to share the port on the server with an HTTP server.

 - It must be possible to implement a fully-conforming server-side 
   component for this in a few dozen lines of scripting code, in the case 
   where sharing the connection with an HTTP server isn't required.

 - The must be extensible so that it can be extended to support binary 
   data and structured data in both directions in the future, without 
   breaking deployments from the first generation.

 - It must not be possible for the mechanism to be used to successfully 
   connect to existing servers and send them arbitrary commands. In 
   particular, SMTP, HTTP, HTTPS, IMAP, POP, and similar protocols must be 
   safe from this.

 - It must be possible to connect to arbitrary remote hosts, but 
   connections to these hosts must only be allowed if the remote host 
   explicitly opts in to allowing that Web page to connect (ideally using 
   the 'origin' convention).

I would be interested in any suggestions people may have along these 
lines. Are there existing protocols that satisfy these requirements?

There is a straw-man proposal for such a protocol in HTML5, which I 
recently converted to Internet-Draft form:

   http://www.ietf.org/internet-drafts/draft-hixie-thewebsocketprotocol-01.txt

Any comments on this, or suggestions for a better way to do this, would be 
welcome.

For what it's worth, the API that is being proposed (somewhat independent 
of the protocol) is currently at:

   http://dev.w3.org/html5/websockets/

You can see both together here:

   http://www.whatwg.org/specs/web-apps/current-work/multipage/#network

Cheers,
-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

---

From: "Robert Brewer" <[email-masked]>
Date: Fri, 9 Jan 2009 15:15:48 -0800

Ian Hickson wrote:
>    particular, SMTP, HTTP, HTTPS, IMAP, POP, and similar protocols
must
>    be safe from this.
>    explicitly opts in to allowing that Web page to connect (ideally
>    using the 'origin' convention).
> 
> Any comments on this, or suggestions for a better way to do this,
would
> be welcome.
> 
> For what it's worth, the API that is being proposed (somewhat
> independent of the protocol) is currently at:
> 

Hi Ian,

I've already been getting some CherryPy users asking if the next version
of CP will support WebSockets. My initial thoughts:

First, just want to agree that finding a better bidirectional mechanism
would be nice.

Second, Python web frameworks at least have pretty much standardized now
on the Web Server Gateway Interface [1], which is strictly
request/response. Rewriting WSGI, and/or extending the servers, to use
WebSockets would be nontrivial. I imagine most other existing web
frameworks in other languages are similarly bound to that model.

Third, I've seen too many custom TCP protocols in my day, and am quite
concerned this would be a step backwards into some pretty ugly RPC. Not
just hard to read, but lacking any visibility or most of the other
network-architectural benefits we've come to rely on in HTTP.

Fourth, if I were implementing bi-directional communication in an HTTP
environment that was machine-to-machine (as opposed to
user/agent-to-machine), I would naturally consider making both ends
expose an HTTP server interface: the "origin server" would expose a
POST'able subscription resource, and the "client" would POST a URL from
its own space. The "server" could then POST updates to that URL; that
is:

    client.user.com
    ---------------
    POST server.biz.com/resource/poll_me HTTP/1.1
    ...
    myurl=client.user.com/listeners/A

...then, at intervals when there is new data to be pushed:

    server.biz.com
    ---------------
    POST client.user.com/listeners/A
    ...
    eventtype=update&data=blahblahblah

Synthesizing those, I'd feel better about WebSockets if the messages
passed in both directions were HTTP, not opaque binary formats. I also
wonder if this couldn't be better achieved by creating a spec (HTTP 2?)
that allowed pipelined HTTP request and response messages in both
directions arbitrarily.

Robert Brewer
[email-masked]

[1] http://www.python.org/dev/peps/pep-0333/

---

From: Jamie Lokier <[email-masked]>
Date: Sat, 10 Jan 2009 01:23:16 +0000

> frameworks in other languages are similarly bound to that model.

Can you overlay something like Bayeux
(http://svn.cometd.com/trunk/bayeux/bayeux.html) over WSGI?

Ugly, but as a transition mechanism may be usable.

> passed in both directions were HTTP, not opaque binary formats.

People don't have to opaque, binary RPC over it.  If they want, they
can send HTTP style messages over it.  The question is, do you want to
suggest a common way to do that, or attempt to force it?

> I also wonder if this couldn't be better achieved by creating a spec
> (HTTP 2?)  that allowed pipelined HTTP request and response messages
> in both directions arbitrarily.

If you do that, you probably need an ability to send out-of-order
responses tagged by their corresponding request, possibly multiplex
responses, and if multiplexing also you'll need per-response flow
control to avoid starvation/unbounded-buffer problems.

Personally I think it's a good idea :-)  But that's because I have an
application which roughly does that, inspired by HTTP and those sort
of extensions, and it works (but drifted from HTTP a bit in the process).

One of the downsides of your suggestion is you don't always want a
response to every message.  That can be wasteful.  So you also want
responsless messages.  Publish-subscribe methods which don't require a
HTTP round-trip just to setup and teardown have uses too.  In complex
mobile applications, you want to keep down the round trip latency
cascade.

With a compatibility mechanism, two-way messaging extensions can be
overlaid on top of HTTP/1.1 if you're feeling brave, using any of the
existing two-way communication hacks.  Stretch and hand-waving, I can
just about imagine a HTTP/2 doing it neatly, with implementations able
to fall back to a HTTP/1.1 overlay hack when a HTTP/2 path to the
server isn't available.

-- Jamie

---

From: Mark Baker <[email-masked]>
Date: Mon, 12 Jan 2009 16:21:37 +0000

Ian,

>   script in a Web page, in the form of ordered discrete blocks.

This one comes for free with XHR/HTTP, no?  Unless you meant something
more than what's written there.

>   has established a connection, in the form of ordered discrete blocks.

This seems to be the big value-add, and I agree it's desirable.

> lines. Are there existing protocols that satisfy these requirements?

Depends what you mean by "existing".  Waka[1] sounds like it would
address all those requirements.  It has the advantage of being a full
HTTP replacement protocol too, meaning that it's still the Web and so
still suitable for use over ports 80 and 443 (whereas Web Sockets is a
tunnel for potentially arbitrary applications).  It also means that
developers will be starting with something quite feature rich and so
will be less prone to reinventing those features poorly (possibly
inefficiently, or worse, with security problems).

On the downside, Roy hasn't published the protocol yet.

 [1] http://en.wikipedia.org/wiki/Waka_(protocol)

Mark.