Subject: Fwd: Re: DataRaw?

From: "Joel M. Halpern" <[email-masked]>
Date: Fri, 8 Jul 2005 20:11:55 -0400

Jamal and I have been batting back and forth ideas for how the contents of 
the dataraw TLV should be constructed, so as to cope with complex 
circumstances lake arrays with holes and structures with optional elements.
The attachment is our current best suggestion.

Please take a look and let us know.
Thanks,
Joel

---

From: Robert Haas <[email-masked]>
Date: Tue, 12 Jul 2005 10:56:43 +0200

Joel, Jamal,
The syntax looks ok to me. But I am not sure though how optional 
elements are the way of life in ForCES as some aspects are still unclear 
to me, for instance:

Do we assume that default values are returned for optional elements 
unless they are explicitely set, or is there a reserved <not_set> value ?
Does the default value appear in the LFB definition for each optional 
element ?

What do we mean by "optional LFB element" exactly ?
Does it mean that from the LFB definition perspective, all optional 
elements MUST be supported by the FE and they MAY be used by the CE ?
Or can we have optional elements of an LFB that a particular FE 
implementation does not support ?

Regards,
-Robert

Joel M. Halpern wrote:

> Jamal and I have been batting back and forth ideas for how the 
> contents of the dataraw TLV should be constructed, so as to cope with 
> complex circumstances lake arrays with holes and structures with 
> optional elements.
>------------------------------------------------------------------------
>
>
>A Data TLV has a 16 bit Type = DATARAW or PACKED-DATARAW, and a 16bit Length
>followed by the data value/contents.
>In the case of the DATARAW (note: not packed) each DATA element in the 
>Value part will be further encapsulated in an ILV. An ILV is like a 
>TLV except it has a 32 bit Index (or element ID) and a 32 bit Length field.
>The rationale behind ILVs is to make it easier to encapsulate optionaly
>appearing Data elements (which are expected to be the way of life in Forces).
>
>Rules:
>
>1) Both ILVs and TLVs MUST 32 bit aligned. If need be they SHOULD be padded
>to achieve the alignement.
>
>2) Packed-Dataraw TLV may be used at a particular path only if 
>every element at that path level is present.  This requirement 
>holds whether the fields are fixed or variable length, mandatory or optional.
>
>3) An assumed optimization is that the encoder(sender) and the decoder(receiver)
>both understand the definition of the data elements being shipped.
>
>a) If a Packed-Dataraw TLV is used, the encoder MUST layout data for each 
>element ordered by the order in which the data was defined. This ensures
>the decoder is guaranteed to retrieve the data.
>
>b) In the case of a non-packed Dataraw we dont need to order since the
>I in the ILV uniquely identifies the element.
>
>4) Inside a Packed-Dataraw TLV
>    a) the values for atomic, fixed-length fields are given without any TLV or
>       ILV encapsulation.
>    b) the values for atomic, variable-length fields are given inside 
>       Dataraw TLVs.
>
>5) Inside a Dataraw TLV (not packed)
>    -> the values for atomic fields may be given with ILVs (32-bit index, 
>       32-bit length)
>
>6)  Any of the Dataraws can contain a ILV but an ILV cannot contain a DATARAW.
>This is because it is hard to disambiguate ILV since an I is 32
>bit and a T is 16 bit.
>
>7) A DATARAW can also contain a DATARAW for variable sized elements. 
>The decoding disambiguation is assumed from rule #3 above.
>
>
>We give a few examples below but dont show the padding.
>
>==========
>Example 1:
>==========
>
>Structure with three fixed-length, mandatory fields.
>
>
>struct S {
>  uint16 a
>  uint16 b
>  uint16 c
>}
>
>
>(a) Describing all fields using DATARAW
>with ILVs (index/length values)
>
>Path to an instance of S ...
>  Dataraw TLV
>    ILV: ElementID(a), length(a), 
>         value(a)
>    ILV: ElementID(b), length(b),
>         value(b)
>    ILV: ElementID(c), length(c),
>         value(c)
>
>(b) Describing a subset of fields 
>
>Path-Data TLV to an instance of S ...
>  Dataraw TLV
>    ILV ElementID(a), length(a), 
>        value(a)
>    ILV: ElementID(c), length(c),
>         value(c)
>
>Note: Even though we have non-optional elements in structure S, since
>we can uniquely identify elements, we can selectively send element of
>structure S (eg in the case of an update from CE to FE).
>
>(c) Describing all fields using a Packed-Dataraw TLV
>
>Path to an instance of S ...
>  Packed-Dataraw TLV
>    value(a)
>    value(b)
>    value(c)
>
>==========
>Example 2:
>==========
>
>Structure with three fixed-length fields, one mandatory, two optional.
>
>
>struct T {
>  uint16 a
>  uint16 b (optional)
>  uint16 c (optional)
>}
>
>This example is identical to Example 1, as illustrated below.
>
>
>(a) Describing all fields using ILVs (index/length values)
>
>Path to an instance of T ...
>  Dataraw TLV
>    ILV: ElementID(a), length(a)
>         value(a)
>    ILV: ElementID(b), length(b)
>         value(b)
>    ILV: ElementID(c), length(c)
>         value(c)
>
>(b) Describing a subset of fields using ILVs (index/length values)
>
>Path to an instance of T ...
>  Dataraw TLV
>    ILV (ElementID a, length, value of a)
>    ILV (ElementID c, length, value of c)
>
>(c) Describing all fields using a Packed-Dataraw TLV
>
>Path to an instance of T ...
>  Packed-Dataraw TLV
>    (value of a)
>    (value of b)
>    (value of c)
>
>Note: Packed-Dataraw TLV _cannot_ be used unless all 
>fields are being described.
>
>==========
>Example 3:
>==========
>
>Structure with a mix of fixed-length and variable-length fields, 
>some mandatory, some optional (would be a good idea to show unions maybe).
>
>
>struct U {
>  uint16 a
>  string b (optional)
>  uint16 c (optional)
>}
>
>
>(a) Describing all fields using ILVs
>
>Path to an instance of U ...
>  Dataraw TLV
>    ILV (ElementID a, length, value of a)
>    ILV (ElementID b, length, value of b)
>    ILV (ElementID c, length, value of c)
>
>
>(b) Describing a subset of fields using ILVs
>
>Path to an instance of U ...
>  Dataraw TLV
>    ILV (ElementID a, length, value of a)
>    ILV (ElementID c, length, value of c)
>
>
>(c) Describing all fields using a Packed-Dataraw TLV
>
>Path to an instance of U ...
>  Packed-Dataraw TLV 
>    (value of a)
>    Packed-Dataraw TLV (value of b)
>    (value of c)
>
>Note: The variable-length field requires the addition of a TLV withing the
>packed data.
>
>
>==========
>Example 4:
>==========
>
>Structure containing an array of another structure type.
>
>
>struct V {
>  uint32 x
>  uint32 y
>  struct U z[]
>}
>
>
>(a): Describing all fields of V using ILVs, with two instances of z[], also
>described with ILVs, assuming only the 10th and 15th subscript of z[].
>
>path to instance of V ...
>  Dataraw TLV
>    ILV (ElementID x, length, value of x)
>    ILV (ElementID y, length, value of y)
>    ILV (ElementID z)
>      ILV (index = 10)
>        ILV (ElementID a, length, value of a)
>        ILV (ElementID b, length, value of b)
>        ILV (ElementID c, length, value of c)
>      ILV (index = 15 )
>        ILV (ElementID a, length, value of a)
>        ILV (ElementID c, length, value of c)
>
>Note the holes in the elements of z (10 followed by 15). Also note the
>gap in index 15 with only elements a and c appearing but not b.
>
>  
>

-- 
Dr. Robert R. Haas
IBM Zurich Research Laboratory
Säumerstrasse 4
CH-8803 Rüschlikon/Switzerland
phone +41-44-724-8698  fax +41-44-724-8578  http://www.zurich.ibm.com/~rha

---

From: Robert Haas <[email-masked]>
Date: Tue, 12 Jul 2005 11:13:40 +0200

Searching through the new model draft I have found answers to some of my 
previous questions:

I wrote:

> elements are the way of life in ForCES as some aspects are still 
> unclear to me, for instance:
>

There is no <not_set> value defined, apparently.

> Does the default value appear in the LFB definition for each optional 

Not clear if a default value MUST appear for each optional element though.

> implementation does not support ?

According to the model draft, it is the latter (a particular LFB 
instance in a FE implementation does not have to support all optional 
elements). This makes sense, although using too many of these optional 
elements will lead to complex CE software if the software wants to 
benefit from whatever optional elements are provided.

>> In the case of the DATARAW (note: not packed) each DATA element in 
>> the Value part will be further encapsulated in an ILV. An ILV is like 
>> a TLV except it has a 32 bit Index (or element ID) and a 32 bit 
>> Length field.
>> 2) Packed-Dataraw TLV may be used at a particular path only if every 
>> element at that path level is present.  This requirement holds 
>> whether the fields are fixed or variable length, mandatory or optional.
>>
>> a) If a Packed-Dataraw TLV is used, the encoder MUST layout data for 
>> each element ordered by the order in which the data was defined. This 
>> ensures
>>    -> the values for atomic fields may be given with ILVs (32-bit 
>> index,       32-bit length)
>>
>>    ILV: ElementID(a), length(a),         value(a)
>>    ILV ElementID(a), length(a),        value(a)
>> Note: Packed-Dataraw TLV _cannot_ be used unless all fields are being 
>> described.
>>
>> ==========
>> Example 3:
>> ==========
>>
>> Structure with a mix of fixed-length and variable-length fields, some 
>> mandatory, some optional (would be a good idea to show unions maybe).
>>
>>
>> struct U {
>>  uint16 a
>>  string b (optional)
>>  uint16 c (optional)
>> }
>>
>>
>>  Packed-Dataraw TLV    (value of a)

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 12 Jul 2005 07:41:31 -0400

An optional element is a component (LFB attribute, part of a structure, 
...) that an FE is not required to support.
A request to GET the component will return an error (<target-not-supported> 
or equivalent.).  GETting the whole structure will return the supported 
fields, and not the unsupported fields.  A SET request which includes 
writing that field will return an error.
A not supported optional element is not the same as a read-only element 
with a defaul value.

Yours,
Joel

At 04:56 AM 7/12/2005, Robert Haas wrote:
>Joel, Jamal,
>The syntax looks ok to me. But I am not sure though how optional elements 
>are the way of life in ForCES as some aspects are still unclear to me, for 
>instance:
>
>Do we assume that default values are returned for optional elements unless 
>they are explicitely set, or is there a reserved <not_set> value ?
>>Jamal and I have been batting back and forth ideas for how the contents 
>>of the dataraw TLV should be constructed, so as to cope with complex 
>>Value part will be further encapsulated in an ILV. An ILV is like a TLV 
>>except it has a 32 bit Index (or element ID) and a 32 bit Length field.
>>element at that path level is present.  This requirement holds whether 
>>the fields are fixed or variable length, mandatory or optional.
>>
>>    b) the values for atomic, variable-length fields are given 
>> inside       Dataraw TLVs.
>>
>>7) A DATARAW can also contain a DATARAW for variable sized elements. The 
>>decoding disambiguation is assumed from rule #3 above.
>>
>>

---

From: Weiming Wang <[email-masked]>
Date: Tue, 12 Jul 2005 22:55:09 +0800

Joel, and Jamal,

I'm glad to see that an ILV is used which directly uses ID as the index for element, which matches my very early thought that we use attribute ID for index directly rather than a path. 

I like this example:

  Dataraw TLV

By using such nestd ILV format, again, my question is, do we need the Path format anymore? 

Thank you.
Weiming

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 12 Jul 2005 17:21:45 -0400

We still need paths for selecting which element is wanted.  If we always 
wanted to get or set the whole LFB, we would not need paths.

One way to think of this is that the path represent a sequence of I fields 
from ILVs where the lengths would be redundant and the V is just the single 
next ILV.

Yours,
Joel

At 10:55 AM 7/12/2005, Weiming Wang wrote:
>Joel, and Jamal,
>
>I'm glad to see that an ILV is used which directly uses ID as the index 
>for element, which matches my very early thought that we use attribute ID 
>for index directly rather than a path.
>
>I like this example:
>
>   Dataraw TLV

---

From: Weiming Wang <[email-masked]>
Date: Wed, 13 Jul 2005 07:39:47 +0800

Joel,

----- Original Message ----- 
From: "Joel M. Halpern" <[email-masked]>
To: "Weiming Wang" <[email-masked]>; <[email-masked]>

> from ILVs where the lengths would be redundant and the V is just the single 

The ILVs with length =0 at the terminal can  repsent the path function well in a get operation. 
As for efficiency, representation of paths less than 3 layers (e.g. 1.2.3) is not as efficient as nested ILV format.

Another very important feature for ILV format is, we may use the following format to , e.g., query a number of fields for values at once, like:

> >   Dataraw TLV
> >     ILV (ElementID x, length=0)
> >     ILV (ElementID y, length=0)
> >     ILV (ElementID z)
> >       ILV (index = 10)
> >         ILV (ElementID a, length=0)
> >         ILV (ElementID b, length=0)
> >       ILV (index = 15 )
> >         ILV (ElementID a, length=0)
> >         ILV (ElementID c, length=0)

The get-response is then simplified to fill out all the value part in the ILVs.

Thanks,
Weiming

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 12 Jul 2005 20:22:18 -0400

a) In my opinion, it is helpful to have a clear sesparation between "this 
is the thing I am interested in" (path) and "this is the value of the thing 
I am interested in" (dataraw or packed-dataraw).
b) We could use just the ILV in Set Request and Get Response 
messages.  However, if there is any depth there will be a lot of redundant 
length fields and more complex parsing to realize that there are not 
additional elements at each level.
c) We could use the ILV with length 0 in GET requests to indicate "this is 
the field I want to get", but that seems a very misleading encoding of a 
very natural concept.

Basically, the path structure gives us a clear and compact representation 
of an important concept in the protocol activity.

Yours,
Joel

At 07:39 PM 7/12/2005, Weiming Wang wrote:
>As for efficiency, representation of paths less than 3 layers (e.g. 1.2.3) 
>is not as efficient as nested ILV format.
>
>Another very important feature for ILV format is, we may use the following 
>format to , e.g., query a number of fields for values at once, like:
>
> > >   Dataraw TLV
> > >By using such nestd ILV format, again, my question is, do we need the 
> Path
> > >format anymore?
> > >
> > >Thank you.
> > >Weiming
> > >
> contents of