Subject: updated path data text

From: Jamal Hadi Salim <[email-masked]>
Date: Mon, 17 Jan 2005 16:09:47 -0500

Attached updated version.
The outstanding issues are listed in the text.
Please read the text.

Weiming! I am still waiting to hear from you on this. I have heard from
Joel and he is pretty much set to move forward. Please make your
comments, suggested modifications etc.

cheers,
jamal

---

From: "Wang,Weiming" <[email-masked]>
Date: Fri, 21 Jan 2005 14:26:32 +0800

Hi Jamal,

Thank you for the progressive efforts. Pls see comments inline for the format:

OPER TLV : = <PATH-DATA>
[Weiming] I suppose you mean <PATH-DATA>+. I see more is, we may even need a
DATA more as <PATH-DATA>+ [DATA]. This last DATA may be needed by some Query
Responses.
PATH-DATA := <PATH> [DATAINFO]
PATH := flags IDcount <IDs>
[Weiming]Here, my thinking is the IDs should only be FieldIDs rather than any
Indices included. Indices are represented only in the followed INDINFO TLV. I
see to include index in this first Path may makes the path not unique for
parsing. Moreover, if we use universal 32bit for IDs, we may not need the count.
DATAINFO TLV := [SELECTOR]  [DATARAW]
[Weiming] The DATARAW here I see may better still as DATA, because we still have
a case the data here are not purely raws for table. It's still possible it is a
whole entry for a table.
SELECTOR :=  [BLKINFO] [KEYINFO]+ | [INDINFO]>
[weiming]I see there may be only one KEYINFO, instead of KEYINFO+ here, but not
very sure. Maybe Joel can comment more.  I'v seen below that you'v included the
sub fieldIDs in the INDINFO. I think the index and the sub fieldIDs are still
possibly exsisting at the same time, therefore to separate it as [INDINFO] and
[SUBPATH] is still that I suggest.
DATARAW := <THEDATA> [DATAINFO]
THEDATA := [somedata]* [VARENCDATA TLV]*
[Weimig]I'm not very sure of the meaning here. Maybe it has someting to do with
data encoding. Hope others can comment more.
INDINFO TLV := index | fieldID
[Weiming]Let it just for Index, and add another optional TLV as
    SUBPATH TLV= <fieldIDs>
KEYINFO TLV := KEYID KEY_DATA
[Weiming]That's a small confusion to me, I supposed the DATA even for the key
should be in the DATAINFO. we may need more explain from Joel also.
BLKINFO TLV := [BLK_RANGE_INDEX] | [BLK_LIST_INDEX] | [BLK_CNT_INDEX]
[Weiming]We may need to dig more into it. I'm not sure if the expression is
unique for parsin or not. maybe we need some flags.

Thank you.
Weiming

---

From: Jamal Hadi Salim <[email-masked]>
Date: Fri, 21 Jan 2005 10:47:15 -0500

On Fri, 2005-01-21 at 01:26, Wang,Weiming wrote:
> [Weiming] I suppose you mean <PATH-DATA>+.

As the note in the text i posted says: This is an issue for discussion.
(In my opinion if we do this we have met about 99% of what you want;
however, the discussion is about reaching a consensus).
I am indifferent about this. Joel may have issues with it.

>  I see more is, we may even need a

Thats covered i believe. Look at some of the examples.

> Indices included. Indices are represented only in the followed INDINFO TLV.

You have said this a few times now. I apologize its not
well taken care of in the current BNF and examples - This is because
mostly i never understood the desire for this. 
 
> I see to include index in this first Path may makes the path not unique for
> parsing. 

Note this is only in the first path. I dont see any ambiguity. Maybe you
can provide me an example?

> Moreover, if we use universal 32bit for IDs, we may not need the count.

We need to know the boundaries. How do you know where the IDs end?
We could put things in a TLV but that would have to include flags as
well and waste an extra 32 bits.

> [Weiming] The DATARAW here I see may better still as DATA, 

If i understood you, you dont like the name DATARAW and you just want
the name changed to DATA?

> because we still have

This is the controvesial piece which is now constrained to be only
within KEYINFO. 

> [weiming]I see there may be only one KEYINFO, instead of KEYINFO+ here, 

Did you look at the examples where KEYINFO is used? I thought you had a
very different opinion so far. This is also tagged as an issue in the
text. Are you agreeing with Joel now?

> but not very sure. Maybe Joel can comment more.  

Joel?

> I'v seen below that you'v included the

Good catch, how does this look:
IDINFO := [FIELDINFO]* [INDINFO]* 

> data encoding. Hope others can comment more.

Its also explained in the text with an example. Please read the text in
full - it will help speed up these discussions.
Essentially, this is to encode non-fixed sized data. 

> [Weiming]Let it just for Index, and add another optional TLV as

Refer to what i said above on  IDINFO := [FIELDINFO]* [INDINFO]* 

> should be in the DATAINFO. we may need more explain from Joel also.

I will wait for Joels view - the idea behind keyinfo is to restrict the
usage of keys.

> unique for parsin or not. maybe we need some flags.

Agreed; some of these BLKINFO maybe optimized later with flags. 

Weiming and Joel when you make chnages to the BNF, please walk onto the
use cases i provided and say how they change if they are affected.
And read the text closely as well so we are all in sync. 

cheers,
jamal

---

From: "Wang,Weiming" <[email-masked]>
Date: Sat, 22 Jan 2005 15:08:56 +0800

Jamal,

----- Original Message -----
From: "Jamal Hadi Salim" <[email-masked]>
Subject: Re: updated path data text

> I am indifferent about this. Joel may have issues with it.
I'm sorry I'v made a confusion by myself. Actually I'm indifferent of
<PATH-DATA> or <PATH-DATA>+, IF ONLY we are going to have Subpaths with
SubfieldIDs included in the followed DATA field.
> Thats covered i believe. Look at some of the examples.
This also follows above thought. If we have, as you mentioned, the Path Selector
Data Selector Data mode, this is not a problem.
> > PATH-DATA := <PATH> [DATAINFO]
According to your followed explanation, I suppose it should be:
PATH-DATA := <PATH> [DATAINFO]*
Is it?

> can provide me an example?

Let me just try to use your example to show the possible ambiguity.

table2: type array, ID = 4
	elements are:
	j1, type u32, ID = 1
	j2, type u32, ID = 2

Then, according to your scheme, I understand, to access the #1 entry, we may use
the Path=4.1, but I can understand this also as trying to access the j1 row of
the table, which Path is also 4.1.

> well and waste an extra 32 bits.
We have a different view here. I think the Path here should be represented as a
TLV. That is,
PATH TLV = flags <FieldIDs>
I don't see any waste for bits here.
> This is the controvesial piece which is now constrained to be only
My understanding of KEYINFO and the sub FIELDINFO I proposed are only optional,
therefore Data are not purely raws.
> text. Are you agreeing with Joel now?
After read your examples provided, I confess I'v understood more. But I have to
say the examples you set on KEYINFO is actually what I mean the FIELDINFO. I
don't think Joel's KEY is the same as you provided in the e.g. Even so, the
example is still valuble to me, for I understand what you mean the '+' here. I
suppose it should also apply to FIELDINFO below, i.e., [FIELDINFO] should be
[FIELDINFO]+ .
> IDINFO := [FIELDINFO]* [INDINFO]*
Yes, that's it,though I think the SUBPATH is more relavent to the FIELDINFO
meaning here.
> > should be in the DATAINFO. we may need more explain from Joel also.
after read your examples, I can accept to move some DataINFO here. Then, I try
to suggest the SUBPATH TLV (or you call FIELDINFO TLV) as:
    SUBPATH TLV := counts <fieldIDs> The_Data
Note that there are possibly moer than one level of fieldIDs here, just thinking
in your example of table2, j1 is structure instead of an atom.

Thank you.
Weiming

---

From: "Joel M. Halpern" <[email-masked]>
Date: Sat, 22 Jan 2005 10:09:46 -0500

I'll comment on some of the other issues later, but I wanted to get these 
two highlighted.

Firstly, I would strongly prefer to use <PATH-DATA>+, i.e. multiple 
occurrences of the PATH-DATA TLV rather than the Subpaths with SubPath IDs.

Secondly, with regard to the indexing and content addressing, as described 
there is some ambiguity which must be removed.  There are several ways to 
remove it.
1) We can assign an identifier, declared with the array type, for each form 
of indexing.  Thus, the path 4 would be the array, 4.1 would be the array, 
indexed by the subscript, and 4.2 would be the array with the element 
sleected by a specific declared key.  Then the j1 element is 4.1.1 or 4.2.1.
2) We can declare that 4 means the whole array and 4.1 means the array with 
some form of indexing or content selection, where the key / index is 
identified in the keying / indexing TLV.  Then 4.1.1 is the j1 element no 
matter what key is being used for the selection.

In both those cases,  the index value or content lookup value is in its own 
TLV.
A third alternative becomes possible if we place restrictions on when 
content keying is used.  Some of the advocates for content keying wanted it 
only used for leaf selection.  In that case, we would use a flag to 
indicate that content keying is to be applied to the path after it is fully 
parsed.  In that case, to reference the j1 element of the second index off 
array 4 one would use 4.2.1.  However, to reference the structure in array 
4 ata certain content key one would use the <content-key-flag>4 as the 
path.  This would not permit referencing just the j1 element when doing 
content keying.
This has the advantage of allowing us to use the indexing in the path 
sequence, making the common case simpler.  However, it has the disadvantage 
that it restricts the application of content keying.
I mention this because at least two different discussants who have not been 
active recently had asked for this particular combination in earlier rounds 
of this debate.

Yours,
Joel M. Halpern

>SubfieldIDs included in the followed DATA field.

...

---

From: Weiming Wang <[email-masked]>
Date: Mon, 24 Jan 2005 21:57:15 +0800

Joel,

----- Original Message -----
From: "Joel M. Halpern" <[email-masked]>
Subject: Re: updated path data text

> occurrences of the PATH-DATA TLV rather than the Subpaths with SubPath IDs.
I certainly can not get the reasons why Subpaths are so unacceptable to you. Let
me try to use an example quite identical to Jamal's and with your index included
to explain more on the advantage with the subpaths. Also I hope you can also
show some of its disadvantages, and  some advantages with the multiple separate
path-data approach.

A table attribute for an LFB as defined:

 j3, type u32, ID = 3
 j4, type u32, ID = 4

Now, the configure task is: to set the j1 to 100, and j2 to 200, j3 to 300, and
remain j4 as it is,  for  Index of the table (1, 3, 5, 7, 9).

To use the separate Path-Data way, we have a sequence of Path-Data as:
OpTLV= 'Set' ...
Path(4.1) IndexTLV (1,3,5,7,9) Data(100)
Path(4.2) IndexTLV (1,3,5,7,9) Data(200)
Path(4.3) IndexTLV (1,3,5,7,9) Data(300)

To use a subpath way, we have a path-data as:
Path(4) IndexTLV(1,3,5,7,9)
Subpath(1) Data(100)
Subpath(2) Data(200)
Subpath(3) Data(300)

Obviously, I can see the advantage for the latter approaches.

I have to say a path with subpaths expression, that is a tree structure in
nature,  is more relavant to LFB attribute nature, because they are also
naturally tree structured with the AttributeID as the node and the FieldIDs
their leaves.

Yours,
Weiming

---

From: Jamal Hadi Salim <[email-masked]>
Date: Mon, 24 Jan 2005 12:44:06 -0500

Weiming,

> the table, which Path is also 4.1.

Aha!
I think this is the CORE source of all miscommunication we have had.
So i am gonna delete everything we have said so far and try to settle
this issue first.

4.1 is a path to "table2 row 1" _NOT_ to "any cell j1 in table2" as you
have it. 4.1 can be used only when row 1 exists and not before.
In the case of "table2 row 1" it may also be addressed as path=4 with
index=1 - this is not an ambiguity, rather an alternative.

I think we need to settle this first or we will talking about different
things.

cheers,
jamal

---

From: Robert Haas <[email-masked]>
Date: Mon, 24 Jan 2005 19:15:17 +0100

>index=1 - this is not an ambiguity, rather an alternative.
Indeed, but my assumption was that 4.1 is the element with index 1 in 
table2, i.e., using the rule:
- If the object x is an array, then x.y refers to element with index y 
in the array.
- else (object x is not an array), then x.y refers to element with ID y 
in the object.

So to address element j1 in the 1st entry of table2, one would write: 
4.1.1 (or more explicitely: ID4.index1.ID1).

In order to address the entire row of elements j1, one should make an 
explicit difference between index and ID:
ID4.index1 (1st entry in the array) is different from ID4.ID1 (first row 
of the array).

The less flexible alternative is to change the definition of table2  to 
include instead two arrays j1 and j2, but then one cannot address the 
first entry j1,j2, of course.

Regards,
-Robert

-- 
Dr. Robert R. Haas
IBM Zurich Research Laboratory
Säumerstrasse 4
CH-8803 Rüschlikon/Switzerland
phone +41-1-724-8698  fax +41-1-724-8578  http://www.zurich.ibm.com/~rha

---

From: Jamal Hadi Salim <[email-masked]>
Date: Mon, 24 Jan 2005 14:17:50 -0500

> 4.1.1 (or more explicitely: ID4.index1.ID1).

nod.

> ID4.index1 (1st entry in the array) is different from ID4.ID1 (first row 

I didnt follow: Did you mean "first column" in the case of ID4.ID1?

Note, so far we have been defering talking about columns or cells within
a row (subset of a column). Maybe a good time to start discusing this; i
believe that a block operation complimented by a FIELDINFO TLV would
suffice to describe the column coordinates of interest. In othewise this
would be a cartesian x,y coordinate scheme with the block describing the
indices(X) and the fieldinfo will describe the column(Y).

> first entry j1,j2, of course.

I think this will complicate things further.

cheers,
jamal

---

From: Jamal Hadi Salim <[email-masked]>
Date: Mon, 24 Jan 2005 14:31:23 -0500

I was gonna have Joel take this but couldnt resist, sorry ;->

> Obviously, I can see the advantage for the latter approaches.

If by subpaths you mean fieldinfos (or is it selctor in general), then i
think after last exachange weve had they will show up in the BNF (recall
the IDINFO construct). 

Let me for example show how the above will be incorporated in the BNF i
posted (with addition of FIELDINFO):

operation = SET
	Path: Flags = F_REPLACE, IDCount = 1, IDs=4
	SELECTOR := 
		// select the rows
        	BLK_LIST_INDEX, L = XXX
		   	V = {1,3,5,7,9}
		// select the columns
        	FIELDINFO = {1,2,3}
		// data to use
	DATAINFO TLV, Length = XXXX
		V = {100, 200, 300}

cheers,
jamal

---

From: Robert Haas <[email-masked]>
Date: Tue, 25 Jan 2005 19:03:18 +0100

>I didnt follow: Did you mean "first column" in the case of ID4.ID1?
no, I meant the entire first row, i.e., the array composed of the j1 
elements only.
First column (=1st entry of the array) is ID4.index1.

Regards,
-Robert

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 25 Jan 2005 14:04:21 -0500

Once can imagine arbitrary joins, arbitrary selections, etc.

But, given what ForCES is trying to do, does the CE really need this kind 
of flexibility?

Yours,
Joel

---

From: "Wang,Weiming" <[email-masked]>
Date: Thu, 27 Jan 2005 11:25:29 +0800

Jamal,

----- Original Message ----- 
 
> DATAINFO TLV, Length = XXXX

To put the sub fieldIDs together and the data together will bring ambiguity. Thinking of an example as below:

table2: type array, ID = 4
 elements are:
 j1, type struct Add, ID = 1
 j2, type u32, ID = 2
 j3, type u32, ID = 3
 j4, type u32, ID = 4

Where struct Add may be globally defined as:
struct Add {
d1, type u32, ID = 1
d2, type u32, ID = 2
} 

where the IDs are numbered independantly of table2.

Then, when we use the FIELDINFO as {1,2,3},it will lead to different parsing result:

{{1, 2}, 3}  --- meaning to address the {j1.d2, j3}
{1, 2, 3} --- meaning to address the {j1,j2,j3}

Obviously, this will also lead to parsing error of Data TLV.

By using the subpath-data separate way, above two cases are differently as:

Subpath(1.2) Data(2000) ; suppose to set j1.d2=2000
Subpath(3) Data(300)

and 
 subpath(1) Data(100)
Subpath(2) Data(200)
Subpath(3) Data(300)

Note that this approach are also actually not bits consuming very much. By defining a subpath-data with a TLV (just identical to what you suppose Key-data does) as:

as I supposed in last post.

Thanks,
Weiming

---

From: Jamal Hadi Salim <[email-masked]>
Date: Thu, 27 Jan 2005 15:50:53 -0500

Weiming,

> Obviously, this will also lead to parsing error of Data TLV.

You keep bringing some very good points. I am hoping the end result is
we will have some good encoding and reach a good consensus.

We can allow FIELDINFO TLV to also contain FIELDINFO inside it;
i.e from BNF i posted earlier for Joel as well as in my last email on
the DATARAW, lets say we have:

OPER TLV : = <PATH-DATA>+
PATH-DATA := <PATH> [DATAINFO]
PATH TLV := flags IDcount <IDs> 
DATAINFO TLV := [SELECTOR]  [DATARAW]
SELECTOR :=  [BLKINFO] [KEYINFO]* [IDINFO] 
DATARAW := somedata* [DATARAW TLV]*
IDINFO := [INDINFO] [FIELDINFO]
FIELDINFO TLV := <zero or more fieldIDs [FIELDINFO]*>
INDINFO TLV :=  one or more indexes
KEYINFO TLV := KEYID KEY_DATA
BLKINFO TLV := [BLK_RANGE_INDEX] | [BLK_CNT_INDEX]

Then:
{{1, 2}, 3} becomes:

FIELDINFO TLV, L = XXXX
	V = {{FIELDINFO TLV, L = XXX, V = {1, 2}},3}
	      
And 
{1, 2, 3} is:

FIELDINFO TLV, L = XXXX
	V = {1, 2, 3}

Note again the FIELDINFO in this case is just a further filter to
the target path

> Subpath(1.2) Data(2000) ; suppose to set j1.d2=2000

PATH TLV
	point to table
BLKINFO or INDINFO to select row(s) in table
FIELDINFO TLV, L = XXXX
	V = {{FIELDINFO TLV, L = XXX, V = {1, 2}},3}
DATARAW TLV, L = XXXX
	V = {2000, 300 }

It is expected that by reading the selector you know
_exactly_ how to decode the data.

Note also now that we allow for <PATH-DATA>+
You can repeat those multiple times 	

> and 
>  subpath(1) Data(100)
> Subpath(2) Data(200)
> Subpath(3) Data(300)
> 
> Note that this approach are also actually not bits consuming very much. By defining a subpath-data with a 
> TLV (just identical to what you suppose Key-data does) as:
> 

I fixed the BNF to show TLV as you said last time. Sorry took me a while
to understand. Essentially this:

---
PATH-DATA := <PATH> [DATAINFO]
PATH TLV := flags IDcount <IDs> 
---

is like your SUBPATH TLV and you can do encode it as you describe also
with <PATH-DATA>+ now being allowed multiple times.

cheers,
jamal

---

From: "Wang,Weiming" <[email-masked]>
Date: Sat, 29 Jan 2005 16:42:27 +0800

Jamal,

Let me firstly explain the BNF for Subpaths that I proposed a litlle more. Pls

> BLKINFO TLV := [BLK_RANGE_INDEX] | [BLK_CNT_INDEX]

To simplify things, I just modify it with my thoughts inside as:

   OPER TLV : = <PATH-DATA>+
   PATH-DATA := <PATH> [DATAINFO]
   PATH TLV := flags IDcount <IDs>
   DATAINFO TLV := [SELECTOR]  [DATARAW]
   SELECTOR :=  [INDINFO] [SUBPATHINFO]* [KEYINFO]*
!I think the BLKINFO can be included in INDINFO anyway, which all belongs to
index ops.
   DATARAW := somedata* [DATARAW TLV]*
   INDINFO TLV:= [ENUM_INDEX] | [BLK_RANGE_INDEX] | [BLK_CNT_INDEX]
!ENUM_INDEX TLV is for enumerated indexs.
   SUBPATHINFO TLV:= IDcounts <IDs> SUBPATH_DATA
   KEYINFO TLV := KEYID KEY_DATA
!A comment here is, I do think a KEYID is a kind of FIELDID, therefore
SUBPATHINFO format can include what KEYINFO does, note that the SUBPATHINFO TLV
is identical to KEYINFO TLV only with a IDcounts added to allow more than one
IDs. As a result, I even think we do not need the KEYINFO format in BNF here.
The restriction imposed on KEYs can be defined by Model document (Just my
thinking, I know it will be strongly objected.)

Following above BNF, addressing to things like { {1,2},3} actually goes with two
separate SUBPATHINFOs as:

SUBPATHINFO TLV
V = {IDcounts(2), IDs(1,2), Data(2000)}

SUBPATHINFO TLV
V = {IDcounts(1), IDs(3), Data(300)}

Pls also see more a small comments inline below.

Cheers,
Weiming

> V = {{FIELDINFO TLV, L = XXX, V = {1, 2}},3}

I'm afraid this still does not solve the uniqueness issue, for the nested
FIELDINFO TLV can still not be uniquely parsed, the V part still can be parsed
as with or without the nested TLV.

---

From: Jamal Hadi Salim <[email-masked]>
Date: Sun, 30 Jan 2005 14:55:37 -0500

Weiming,

> !I think the BLKINFO can be included in INDINFO anyway, which all belongs to

sure.

> !ENUM_INDEX TLV is for enumerated indexs.

Can you provide an example of ENUM_INDEX?

>    SUBPATHINFO TLV:= IDcounts <IDs> SUBPATH_DATA

So SUBPATH_DATA is actually data to be used for setting as opposed 
for searching/matching? Did i understand correctly?

> IDs. As a result, I even think we do not need the KEYINFO format in BNF here.

I am not sure i understood what SUBPATH_DATA is to be used for.
KEY_DATA is the key value to compare against. In other words, you do not
set anything with that data but you use it to refine entities of
interest.

> thinking, I know it will be strongly objected.)

Agreed.

> V = {IDcounts(1), IDs(3), Data(300)}

So now i understand that SUBPATH_DATA is actually data that is to be
set. in which case:
How do you describe content that is to be searched? If you can meet that
requirement I can live with it.
Rewrite the grammar to include this as well.

cheers,
jamal