Subject: Re: [vwrap] [ogpx] type-system : version tag and handling unknown tags

From: "Robert G. Jakabosky" <[email-masked]>
Date: Thu, 1 Apr 2010 00:08:33 -0800

The only use cases I can see for a version field are:
1. A new more compact binary encoding (i.e. remove closing tags, remove 'k' 
tags, variable-length encoded sizes, support for compression codecs).
2. A new basic/container type is added to LLSD.

Case 1 would make the new version completely incompatible with old readers.  
In this case the sender could downgrade to the old format if the reader 
rejected a 2.0 format.

Case 2 would require some type of extension tag in version 1.0 to allow old 
readers to even skip/ignore the new type.  This case would also require 
changes to the JSON & XML encoding formats.  It might not be possible to 
downgrade the LLSD content in this case.  XML readers can ignore unknown 
tags, but what about JSON readers?  LLSD uses all the JSON types, there isn't 
anyway to add an extension that will be ignored by 1.0 readers.

Since LLSD already has three encoding formats defined, there will need to be a 
way (MIME-type/Content-Type) to identify which format was used to encode a 
LLSD message and a way for the client & server to negotiate which formats 
they accept.  So I see no reason for adding a version tag to the binary 
format, when a new MIME-type will provide the same functionality.

-- 
Robert G. Jakabosky

---

From: Carlo Wood <[email-masked]>
Date: Tue, 6 Apr 2010 14:03:11 +0200

Yet another place where protocol negotiation would come in handy.
If I had more time, I'd write a draft for the protocol negotiation
that I have in mind :/

On Wed, Mar 31, 2010 at 04:27:48PM -0700, Mark Lentczner wrote:
> In trying to think thought the scenarios where a version field might be workable, I'm coming up short:
> 
> A v.n only reader encountering v.n+1 could only make two choices: Ignore everything it doesn't understand and assume that the semantics of the content aren't affected, or reject the transmission entirely. We'd have to choose now how it should react.
> 
> If it is ignoring things, then it seems that any v.n+1 extension would have to be defined like "If you understand v.n+1, then ignore some (or all) of the v.n stuff and substitute this richer data." But at that point, we are mostly using this version stuff as a way of stuffing two or more versions of the data in the same container. Extending this beyond one or two revisions is likely to get out of hand.
> 
> If it is rejecting things, then this has little use here, since there must be some signaling and protocol for negotiation to handle this case in some outer scope (HTTP, say). But then, why not just embed the version information at the higher scope (MIME-type, say)?
> 
> The "ignore by preserve" semantic works well for processing streams, and in cases where the vast bulk of the data is understood, and the ignored, foreign markup, isn't semantically important. VWRAP's use of LLSD doesn't seem to fit that model.
> 
> 
> Mark Lentczner
> Sr. Systems Architect
> Technology Integration
> Linden Lab
> 
> [email-masked]
> 
> Zero Linden
> [email-masked]
> 
> 
> 
> 
> 
> _______________________________________________
> vwrap mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/vwrap

-- 
Carlo Wood <[email-masked]>

---

From: Meadhbh Hamrick <[email-masked]>
Date: Tue, 6 Apr 2010 07:14:13 -0700

what do systems that only understand xml 1.0 do if they encounter xml 1.1?

the purpose of the version tag and the rule that you ignore but
preserve tags you don't understand is to provide the hope for
extension for situations we do not currently foresee.

-cheers
-meadhbh
--
meadhbh hamrick * it's pronounced "maeve"
@OhMeadhbh * http://meadhbh.org/ * [email-masked]

---

From: Mark Lentczner <[email-masked]>
Date: Tue, 6 Apr 2010 10:05:46 -0700

> what do systems that only understand xml 1.0 do if they encounter xml 1.1?

In the original XML sepc, systems were to signal an error and reject the entire document. In the 5th revision, they are allowed to accept a document with version "1.1" if and only if the document doesn't use anything that isn't in 1.0. (see "Extensible Markup Language (XML)" §2.8)

> extension for situations we do not currently foresee.

There is no such provision in XML. DTDs and schemas built with XMLSchema or Relax generally don't have the facility to handle unexpected elements and attributes (with the notable exceptions of XML applications that expect namespace delimited elements in very particular places). XSLT processing provides an facilities that match any element or attribute, and reproduce it in the output, though that is thoroughly at the discretion of the XSLT writer: They can just as easily drop such unknown fragments.

----

Within the context of LLSD, the problem is that there is semantics to the structures of array and map. For example, what does the application reading the following binary sequence see:

	"[" %x00.00.00.03 "1" "z" %x00.00.00.01 "!" "0" "]"

("z" here represents an unknown tag, and follows the recommendation that all new unknown tags have a four byte length so that they can be parsed.)

Is that an array of three items, or two? What does "skipping but preserving" mean? It is tempting to treat unknown tags as values. The middle value is seen by the application as "undef". (Or, perhaps it is seen as yet some other type we'd need to introduce into LLSD like "foreign" with a value similar to binary: "1 byte of %x21")

However, this leads to questions in maps:

	"{" %x00.00.00.01 "k" $x00.00.00.03 "abc" "z" $x00.00.00.01 "!" "}"

This would be clear under the "value" interpretation, but what about:

	"{" %x00.00.00.01 "z" $x00.00.00.01 "!" "1" "}"

Is that invalid (as the unknown tag appears in a non-value location) or is it a "new key type", and hence is seen as the key "undef", only that isn't a valid key. Is the whole key/value pair dropped? Does that make this equivalent to the empty map?

If we are to implement new structures, presumably they'd be wrapped up inside one of these things with the count extending over past all the values. For example, an unordered set might be introduced by:

	"w" %x00.00.00.0E %x00.00.00.02 "i" 0x00.00.01.3A "i" 0x00.00.00.2A

The first count includes all the bytes of the construct, the second count is the number of elements of the set, the two elements are the numbers 314 and 42. To an older application, this would be seen as the "undef" value.

----

A wholly different choice of handling unknown tags is to simply ignore it. This allows the older app to see a "substitute" value, and requires that newer tags that want to be a value to have the semantic of "skip the next value you read".  Hence, the above examples might become:

	"[" %x00.00.00.03 "1" "z" %x00.00.00.01 "!" "1" "0" "]"
	-- this is an array of three booleans, True, True, False,
	-- where there is a newer value type that substitutes for the middle True

	"{" %x00.00.00.01 "k" $x00.00.00.03 "abc" "z" $x00.00.00.01 "!" "1" "}"
	-- this is a map of "abc" to True
	-- where there is a newer value type that substitutes for the True

	"{" %x00.00.00.01 "z" $x00.00.00.01 "!" "k" $x00.00.00.03 "abc" "1" "}"
	-- this is also a map of "abc" to True
	-- where there some newer tag that may have some unknown effect on the map or the key

	"w" %x00.00.00.00 "[" %x00.00.00.02 "i" 0x00.00.01.3A "i" 0x00.00.00.2A "]"
	-- an array of two integers, 314 and 42
	-- where there is a tag in front that indicates to treat the array as a set

In these cases, the newer tags are simply ignored, and the remaining tags form a valid binary serialization. It brings into question what it would mean to "preserve" these. Perhaps, it a very delimited set of future semantics ("all new tags must 'apply' to the next value or key/value pair") LLSD implementations could attached these tags as auxiliary data to existing LLSD types and constructs.

----

But for all the effort such definitions would take, I think it is better to not allow them all, since I don't see a scenario where this is favorable to content negotiation or protocol negotiation.

In any such scenario, the "extension" would have to be minor enough to not warrant protocol negotiation - the "extension" couldn't provide any semantically important information: If newer code wants to pass an unordered set rather than an array, then presumably the existing semantics of the data block are that they array is to be treated as an unordered set anyway. (If not, then we've got a new protocol.) If there are new data types, whose values will be taken as "undef" in older code, either that data is optional (in which case they should be under new map keys) or it is crucial and hence needs to be under a new protocol.

In short, it seems to me that ad hoc extension of LLSD at the layer of elements within a serialization isn't that useful. Such extension should occur at the content or protocol levels, both of which have existing negotiation systems.

	- Mark

Mark Lentczner
Sr. Systems Architect
Technology Integration
Linden Lab

[email-masked]

Zero Linden

---

From: Meadhbh Hamrick <[email-masked]>
Date: Tue, 6 Apr 2010 10:18:56 -0700

isn't this what "unknown" in wheat was used for?
--
meadhbh hamrick * it's pronounced "maeve"
@OhMeadhbh * http://meadhbh.org/ * [email-masked]

---

From: Meadhbh Hamrick <[email-masked]>
Date: Tue, 6 Apr 2010 11:39:39 -0700

mark,

XML defines "well formed" as being distinct from being "valid". up til
now we have been talking on the list about what to do with documents
that are not "well formed."

well-formed documents do not refer to a schema, and as i am sure you
are aware, do not check or enforce a sequence of tags. that is
something that's done to check the document's validity.

are we suggesting that the two should be conflated (or well-formedness
be ignored) for the purposes of LLSD serialization?

should we NEVER allow for the extension of the LLSD XML schema?

it just seems a bit short sighted to say the equivalent of "we know of
all uses for this serialization scheme at the current moment and do
not need to add the capability for expansion."

-cheers
-meadhbh