Subject: [ogpx] content negotiation for LLSD over HTTP connections

From: Infinity Linden <[email-masked]>
Date: Fri, 26 Jun 2009 16:48:00 -0700

so... we have three serialization formats for LLSD messages over HTTP:
XML, JSON and Binary. each has it's own uses and constituency.

but it _does_ leave implementers and deployers with questions like:

* must i support each and every LLSD defined serialization?
* under what circumstances should i choose one over the other?
* what if i am a server and i only want to use one serialization?
* what if i am a client and i only want to use one serialization?

i have some opinions (of course). but i wanted to ask implementer's
opinions on these questions... so here's a starting point for
discussion...

THE BACKSTORY

way back in the day, some people noticed SL's proto-LLSD "annotation
format" looked an aweful lot like JSON. users with a copy of ethereal
/ wireshark could see lotsa XML flying around too. so when public
discussions of OGP came along, people naturally thought that LLSD
SHOULD be serialized using JSON ("cause... you know.. like the whole
web is movign to JSON, dude.") well... except those people who thought
it should be serialized using XML, since XML was the king of all
structured information interchange formats. and then there were the
people who said, "oh noes! you have to use a binary format to get
efficiency!"

all these different stakeholders have valid points. and that's part of
why we have LLSD as an abstract type system with three defined
serializations today.

THE MORE RECENT BACKSTORY

then last year when we started the interop work, software engineers
had to code message parsers and the like. for the most part, we
ignored JSON and binary serializations and focused on XML. but.. we
know we can't do that moving forward.

THE PROBLEM

what we only started handling last year was addressing the questions
above: should my service support ALL serializations? what do i do if i
get a serialization i don't like? or can't understand?

THE SOLUTION (we wanted to implement last year but didn't have time for)

inspired by the words "an implementation should be conservative in its
sending behavior, and liberal in its receiving behavior," [RFC760] we
figured a generally available service SHOULD accept messages
serialized with all three defined serializations (XML, JSON & Binary,)
but SHOULD probably try to send things using one serialization.

we're using the word "SHOULD" here in case you're building a service
where you control both the producer and consumer of a service and only
want to support one serialization (maybe you have a super-fancy
hardware XML parser and you choke when you get JSON or something.) i
think there's general agreement that we don't want to force deployers
/ implementers in these situations to use a serialization that will be
difficult for them.

and maybe you're a developer building a new client, and you were able
to get the JSON library to build on your system, but expat remains
unavailable. we don't want to force you to support XML in this case.

but... in all these cases there are probably very good ideas to
support the other serializations, but we're going to let you analyze
the trade-offs.

so... back to the questions above... should a well defined service
accept requests using all three LLSD serializations? i vote "yes." it
is in keeping with internet tradition and a generally good idea. when
a client uses HTTP to transport an OGP message from client to server
via PUT or POST, it may include a Content-Type: header to tell the
server about the serialization it is using. if the client wants a
response using a particular serialization, it can use the Accept:
header.

so.. question... what should happen if the server doesn't want to use
the serialization(s) specified in the Accept: header? or the client
identifies a serialization the server just doesn't understand? Maybe a
406 or 415 response?

anyway. this is just an initial post to get the discussion started.

if you have any opinions one way or another, please voice them here.

-cheers
-meadhbh

---

From: "Hurliman, John" <[email-masked]>
Date: Fri, 26 Jun 2009 19:30:15 -0600

This is how we're using LLSD. It will accept any of the three serializations: XML, JSON, and binary. Notation support was (thankfully) removed when the LLSD/LLIDL draft was posted. It uses a basic algorithm to try and figure out what content type the data is (since no one sets their content type headers correctly or can even agree what the content type string should be). However, only JSON data is sent. The standard "application/json" mime type is used since we're using a subset of LLIDL that is always compliant with the JSON RFC.

John

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]] On Behalf Of
>Infinity Linden
>Sent: Friday, June 26, 2009 4:48 PM
>To: [email-masked]
>Subject: [ogpx] content negotiation for LLSD over HTTP connections
>
>_______________________________________________
>ogpx mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/ogpx
>

---

From: Infinity Linden <[email-masked]>
Date: Fri, 26 Jun 2009 23:00:20 -0700

there are actually some OGP constructions that are single
(non-collection) types (and thus, not RFC4627 compliant JSON.)

for what it's worth... none of the browsers we tested (two versions of
firefox, one version of safari, two versions of IE) seemed to reject
non-collections when receiving and evaluating application/json media
types. in other words, there seems to be a fine history in the browser
world of ignoring RFCs.

still... i just noticed that RFC4627 is informational, which _may_
cause a problem in the future as we listed it as a normative reference
in the LLSD draft. maybe i'll write another, normative draft for a
version of JSON that "does the right thing" and lists the encoding in
the "standard" way by adding it to the media type.

at the end of the day, if we can develop an algorithm for identifying
the serialization without peeking at the media type, we'll probably
all be fine. thought i'll ask that developers at Linden "do the right
thing" and properly identify the media type in headers and would ask
you to do the same with developers you work with.

---

From: Carlo Wood <[email-masked]>
Date: Sat, 27 Jun 2009 12:47:31 +0200

> * what if i am a client and i only want to use one serialization?

In the past I designed a client-server protocol negotiation sheme
that allows for arbitrary protocol enhancements over time, without
an every growing protocol negotiation payload or the need to
keep supporting legay protocol; namely, it is possible to shift
the 'base' protocol and drop support for (very) old protocol.

This sheme was originally written for IRC, and wasn't adopted
because the people who had to decide if they liked it weren't
able to understand it... and lacking an overview, it gave them
the uneasy feeling that it was "too complicated".

However, it is not too complicated at all. It's mathematically
perfect and rather easy way to allow for protocol changes in
the future.

One part of it involves negotiation of optional (non-mandatory)
features / protocol aspects, so you could use it for example to
negotiate the serialization in the case that both, server and
client have their preferences and/or limitations.

Basically, it works as follows:

The client connects and "requests" a protocol (version) and
specific features. The server then has two options: either
it tells the client that it is incompatible (too old), or
it sends back the set of feautures that will be used.

The details can be found here (though be warned, this is my
original design and applies to IRC):

http://www.xs4all.nl/~carlo17/irc/prot.html

-- 
Carlo Wood <[email-masked]>

PS Don't tell me it's "too complex" ;).
   This is a very flexible, very powerful way to make sure
   that grids being developed by different groups will not
   be get stuck in some legacy protocol.

---

From: Carlo Wood <[email-masked]>
Date: Sat, 27 Jun 2009 13:01:12 +0200

> http://www.xs4all.nl/~carlo17/irc/prot.html

Let me add, because this was a source of confusion in the
past too, the mention of webpages in that document has
nothing to do with the actual protocol negotation (there
is no need for either side to connect to and parse
those websits).  I just added what webpages should exist
for and how they'd link to *documentation*. The idea there
is that if some client coder connects to a new network
they will be able to find the documentation of the
protocol, including any negotiable feature, with a few
clicks. It's just documentation thus.

If this sheme was to be implemented for ogpx, then
the negotation messages would be XML (LLSD/LLIDL)
of course :).

---

From: Carlo Wood <[email-masked]>
Date: Sat, 27 Jun 2009 13:51:11 +0200

Ok, I made a 'translation' of that IRC page for the purpose of ogpx:

For documentation purposes there would exist a (central) webpage,
for example http://protocols.ogpx.net, that lists all (accepted)
ogpx server implementations (basically meaning more or less
independent groups working on their own ogpx server versions).

For example, this page could list the labels:
SecondLife, OpenSim, BlueMoon, which link to websites
controlled by the aforementioned group. Lets call these
"implementation labels" for now.

Upon connection to a server, the server should send a message
to the client as early as possible (even before authentication,
or order to allow also the authentication method to be altered
in the future) indicating which protocol base it is using.
This would exist of a (list) of triplets exists of a label (string)
a version (integer) and a 'compatibility version' (integer).

For human readable and documentation reasons, the label should
always start with an "implementation label". It's highly suggested
to just add an integer behind that to complete the label and
make it unique. Lets call these labels "version labels".
For example, the version labels could read "SecondLife23",
"OpenSim1", etc.

For documentation purposes, each of the aforementioned websites
owned by the different implementer groups should list each of
these version labels with a link to more information about them.

The version string together with the version defines the (supported)
protocol of the server: every feature, mandatory of optional.

Neither side has to worry about the mandatory parts, since the
versions will take care of that: a 'compatibility version' is
oldest version that supports all mandatory stuff.

The client then reacts with the version label and protocol version
it 'requests', as well as all the optional features it wants.
The server confirms which optional features will be used, after
which the protocol is switched to the negotiated protocol.
The client acknowledges this and also switches to the final
protocol.

As an example, lets assume that one implementation (opensim)
adds a feature that allows clients to better synchronize animations
with sound. They assign the label "anisndsync" to it, which only
really has to be unique within the namespace of OpenSim.
It is made a negotiable feature of 'OpenSim1 version 30'.

A viewer that has no notion of this feature will logically not
request it, and therefore it will not be used.

If some grid makes the feature mandatory, the negotation fails
with such a client and the user is told to upgrade to a viewer
that supports 'anisndsync'. I do not consider it very likely
that anyone would make it mandatory however, until -say- 90%
of all viewers already support it, and the 4 year-old viewers
that still don't become really annoying.

If the client supports it, it will request it (if it's an optional
feature, still) and the server will grant it. Obviously the
server will know about it (it sent 'OpenSim1 30'), or else
the viewer would not have requested it.

However, then this feauture turns out to not to work - and
the opensim developers decide to remove it again from version
31. Removing a negotiable feature doesn't break compatiblity,
so there is nothing to indicate to a viewer (that was written
during version 30) that version 31 doesn't support it anymore.
What would happen is this: The server says: Opensim1 31 10.
(where 10 is the compatibility version in this example).
The viewer, only knowing about version 30 and before (that is,
version 10 to 30) doesn't have a clue what changed in
version 31, but it knows that 10 < 30, the version it does
support - and hence it is allowed to react as if the server
is version 30. So, it requests "anisndsync" as before.
This time however, the server doesn't acknowledge it, and
the feature is not used.

What worried people in the past with this sheme is:
How do the clients keep track of all the features that are
mandatory and/or negotiable for every version?

Well, this is really very very simple. For each implementated
feature the coder had to sit down and read the documenation.
Along with that documentation come the label and version
number. So, all that is needed is a map from "label + version"
to which *implemented* features are mandatory or negotiable.
The maintenance load of that is neglectable. It will be like
adding two lines of code extra per feature:

  protocol.register("OpenSim1", 30, "anisndsync", &handle_anisndsync);

and common code that takes care of the whole negotiation
will do the rest.

Mapping "OpenSim1" to the mandatory and optional features
might involve more code lines, but this code can simply be
copy&pasted from the documentation website of "OpenSim1".
Also this will be something that is easy to maintain (adding
and removing lines per added/delete feature).

---

From: Joshua Bell <[email-masked]>
Date: Sat, 27 Jun 2009 08:31:00 -0700

> types. in other words, there seems to be a fine history in the browser

FWIW, the ECMAScript 5 folks are formalizing that relaxation in the  
ES5 JSON parser, at least based on the latest discussions.

https://mail.mozilla.org/pipermail/es-discuss/2009-June/009481.html  
(Allen Wirfs-Brock, ES5 editor)

and:

https://mail.mozilla.org/pipermail/es-discuss/2009-June/009443.html  
(Douglas Crockford, author of the JSON RFC)

and the summary that indicates we're not crazy:

https://mail.mozilla.org/pipermail/es-discuss/2009-June/009445.html

Also in the thread, Crockford indicates no intention to issue a new  
JSON RFC. (That's my interpretation of the thread, mea culpa if I mis- 
implied.)

There is also some tooth gnashing about considering U+2028 and U+2029  
as whitespace.