Subject: path data textt update

From: Jamal Hadi Salim <[email-masked]>
Date: Wed, 26 Jan 2005 13:32:18 -0500

Changelog:
---------
$Log: data_path2.txt,v $
Revision 1.7  2005/01/26 06:27:10  hadi
fixed INDINFO definition as caught by Weiming
Added FIELDINFO
Added example on selective setting based on posting from Weiming.

Diff and text both attached.

Please read it. 

cheers,
jamal
--- data_path2.txt	2005/01/17 21:05:58	1.6
+++ data_path2.txt	2005/01/26 18:25:41
@@ -28,25 +28,33 @@
 PATH-DATA := <PATH> [DATAINFO]
 PATH := flags IDcount <IDs> 
 DATAINFO TLV := [SELECTOR]  [DATARAW]
-SELECTOR :=  [BLKINFO] [KEYINFO]+ | [INDINFO]> 
+SELECTOR :=  [BLKINFO] [KEYINFO]+ [IDINFO] 
 DATARAW := <THEDATA> [DATAINFO]
-THEDATA := [somedata]* [VARENCDATA TLV]*
-INDINFO TLV := index | fieldID
+THEDATA := somedata* [VARENCDATA TLV]*
+IDINFO := [INDINFO] [FIELDINFO]
+FIELDINFO TLV := one or more fieldIDs
+INDINFO TLV :=  one or more indexes
 KEYINFO TLV := KEYID KEY_DATA
 BLKINFO TLV := [BLK_RANGE_INDEX] | [BLK_LIST_INDEX] | [BLK_CNT_INDEX]
 
 -> IDs := a series of 32bit IDs (whose count is given by IDcount)
 defining the main path. These IDs could be field IDs (staticaly defined
-at LFB definition) or dynamically created (table indices).
+at LFB definition) or dynamically created table indices.
+The IDs essentially define the _start_ point in a hierachy. They
+may also point to the final target path in the case of a terminal.
+Refer further to examples below.
+
 -> flags are used to further refine the operation to be applied
 on the ID_TLV.
--> SELECTOR := optional different BLK, KEY or Index/fieldID 
-to extend/refine/filter the path.
+-> SELECTOR := optional construct that further defines the path.
+Different BLK, KEY or Index/fieldID  extend/refine/filter the path.
 The SELECTOR has been reffered to as subpath in some email discussions.
 The path information in the SELECTOR indicates what information is being 
-used for selection.
+used for selection. Refer to examples for further info.
+The KEYINFO is the only selector that contains content search information.
 -> DATAINFO: = Optional variable sized data dependent on PATH
-and applied operations. As an example a GET or DEL may not have data.
+and requested/applied operations. As an example a GET or DEL may not 
+have data.
 -> DATARAW contains "the data" whose path is selected. This may
 recursively contain DATAINFO such that we can have a series
 of "path data selector data selector data ..."
@@ -147,12 +155,17 @@
 [a,b], [a,N] or a list [a,c,e,f]
 These modes could be optimized or more added; the idea is to capture
 here for the moment.
+At the moment as well the Block operations apply to Indices and
+not field IDs. We could add BLK_XXX_FIELD later.
 
 1) BLK_RANGE_INDEX TLV
 T = BLK_RANGE_INDEX, length of 4+8
     start // "a"
     end // "b" 
 
+The above Block operation will be removed since it is redundant
+with INDIFO TLV
+
 2) BLK_CNT_INDEX TLV
 T=BLK_CNT_INDEX, length = 4+8
     start // "a"
@@ -244,6 +257,15 @@
 	someid, type u32, ID = 1
 	name, type string variable sized, ID = 2
 
+table4: type array, ID = 6
+elements are:
+j1, type u32, ID = 1
+j2, type u32, ID = 2
+j3, type u32, ID = 3
+j4, type u32, ID = 4
+
+
+
 1) To get foo1
 Path: IDCount = 1, IDs = 1
 
@@ -277,7 +299,7 @@
 To create entry 0-2 of table2 (I am going to ignore error conditions for
 now)
 
-Path: flags = F_CREATE | F_REPLACE, IDCount = 1, IDs=4
+Path: flags = F_CREATE | F_EXCL, IDCount = 1, IDs=4
 BLK_RANGE_INDEX TLV
     start = 0
     end = 2
@@ -493,6 +515,24 @@
 	j1value,j2value
 	j1value,j2value
 
+15) Selective setting (Example posted by Weiming)
+on table 4 -- for indices 1, 3, 5, 7, and 9.
+Replace j1 to 100, j2 to 200, j3 to 300. Leave j4 as is.
+
+operation = SET
+Path: Flags = F_REPLACE, IDCount = 1, IDs=4
+SELECTOR := 
+	// select the rows
+	BLK_LIST_INDEX, L = XXX
+		V = {1,3,5,7,9}
+	// select the columns
+	FIELDINFO, L = XXXX
+		V = {1,2,3}
+// data to use
+DATAINFO TLV, Length = XXXX
+	V = {100, 200, 300}
+
+
 TODO:
 
 - column and cell access in a table use cases

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 26 Jan 2005 17:10:32 -0500

A scattering of comments:

1) I am not sure the nesting of DATAINFO inside DATARAW is either necessary 
nor particularly useful.  I am afraid it may lead to very confusing constructs.

2) Please, can we use one size length field for all TLVs?  16 bits?  If we 
actually have a block of information larger than 64K to send, I think that 
we are solving some other problem than normal ForCES operation, and should 
probably use a different notion.  (I can imaging a message sequence 
intended to convey a large block of information in an agreed format.)  If a 
GET actually would produce more than 64K bytes, the FE should return a "too 
large" error, and let the CE get it in pieces.  Among other things, the 
head of line blocking by large data may be a (admittedly minor) issue.

3) On the Delete operation, I do not normally expect it to include "data to 
be Deleted".  It will include the index or the content selector, but that 
is not the same as the data to b deleted.

4) I can live with <PATH_DATA>+.  I thought I said so on the list.  Sorry 
if I was not sufficiently clear.  Compared with most of what we are 
discussing, this is small potatoes.

5) On the flags, I can not tell the difference between F_EXCL and F_CREATE?
5') I strongly object to F_APPEND as described.  Keys or indices shall be 
unique.  I would have no problem with F_APPEND if the meaning were:
     use only with indexed operation
     ignore the index specified in the operation
     use the smallest free index in the table
     return the index used in the result
and all set operations would have the requirement that fields declared as 
key shall be unique in the table even if they are not the fields used for 
the entry selection.

6) When getting a whole table, I would prefer that the structure returned 
to represent the table included the indices, rather than using the BLK TLC 
alongside the data.  The reason for this is that if we have tables inside 
tables (clearly necessary in some cases), we need to be able to return all 
the entries of the inner tables.  When the get used the BLK, obviously the 
BLK is included in the results to indicate what is returned at the 
outermost layer.

7) In example 10, it says it is getting table1, and then says t1value, 
t2value, j1value, j2value.  But there are no ji fields in table1?  from the 
later text, this was envisioned as a transparent indirection.  That fact 
that the t2 values can be used as indices in table2 should not affect the 
results of this get.  (Note that if there was a t3value that was an alias 
for the table2 entry, that would be a different matter.  But without the 
alias, table2 things should not be returned when one accesses table1.)

note: examples 11 and 12 relate to the uniqueness debate.

8) rather than using special encoding, I believe we will need to agree on a 
general encoding for arrays, struct, bytestrings, strings, etc.  All data 
types and constructs defined in the model must have an agreed encoding for 
use in the data portion (SET requests, GET responses, and content indexing 
all should use the same encoding rules.)

9) I find the implicit iteration binding in example 15 confusing.  While I 
think we can write rules to permit this, I think it is going to be a major 
source of implementation errors.

Yours,
Joel

PS:  Jamal, thanks very much for all the work keeping this on track and 
organized.

---

From: Jamal Hadi Salim <[email-masked]>
Date: Thu, 27 Jan 2005 14:46:58 -0500

Joel,

Thanks for the input; i will split response email into 2 so its more
readable.

> nor particularly useful.  I am afraid it may lead to very confusing constructs.

Based on #4 below we can kill this now.
So how does this look:

OPER TLV : = <PATH-DATA>+
PATH-DATA := <PATH> [DATAINFO]
PATH TLV := flags IDcount <IDs> 
DATAINFO TLV := [SELECTOR]  [DATARAW]
SELECTOR :=  [BLKINFO] [KEYINFO]* [IDINFO] 
DATARAW := somedata* [VARENCDATA TLV]*
IDINFO := [INDINFO] [FIELDINFO]
FIELDINFO TLV := one or more fieldIDs
INDINFO TLV :=  one or more indexes
KEYINFO TLV := KEYID KEY_DATA
BLKINFO TLV := [BLK_RANGE_INDEX] | [BLK_CNT_INDEX]

I have killed BLK_LIST_INDEX as well.
Should also be noted that the SELECTOR is both a content(using KEYINFO)
as well as path selector.

> head of line blocking by large data may be a (admittedly minor) issue.

The TLV is still 16 bit byte-lenght (16 bit T and 32 bit alignement for
data); the suggestion to use the T to imply that the Length was other
than byte-length was as you note in your email - to stash large amounts
of data. It is not unusual to find table with 100K entries. Expound a
little on the "too large" error usage with 100K entries and we can throw
the non-byte-Length hack out.

> is not the same as the data to b deleted.

The data may be included in the SELECTOR to further refine what it is
that is being deleted. Example 12 provides such usage. To save time:

---
12) delete  first 10 entries with j1=100, and j2=200 in table 2,
starting at index 10. 

Path: IDCount = 1, IDs=4
SELECTOR := 
        BLK_CNT_INDEX TLV = {10,10} 
        KEYINFO = {KEYID =1 KEY_DATA=100}
        KEYINFO = {KEYID =2 KEY_DATA=200}

--------

Is this fine with you? Did you mean something else?

> 4) I can live with <PATH_DATA>+.  I thought I said so on the
list.Sorry 
> discussing, this is small potatoes.

Done. 

cheers,
jamal

---

From: Jamal Hadi Salim <[email-masked]>
Date: Thu, 27 Jan 2005 15:07:53 -0500

> 5) On the flags, I can not tell the difference between F_EXCL and F_CREATE?

The wording is unclear even to me now when i read it;-> 
Let me explain then maybe we can have better wording.

Its a unixism derived from filesystem operations (look at man open).
Adopted by BSD Route sockets in the next point in history. Extensively
used in netlink (borrowed from Route sockets) and i believe in some
databases.... 
As an example, if this was an operation to add to a table row: CREATE is
essentially instantiating of the row object (and in this case populating
the entries in that object) but not putting it in the table.
A lookup in the targeted table (using a given key) may show that the
table already contains an entry with a matching key. 
EXCL says if you do find the match in that case, do not stash the
instantiated object in the table. In unix you return a -EEXIST i.e entry
already exists. So if you want to add a brand new object you would have
the CREATE|EXCL together. If all you wanted was to test for existence
you could do a SET with just EXCL flag. 
A CREATE|REPLACE on the other hand will expect such an entry to exist
already and perhaps replace the row or parts of it.

> key shall be unique in the table even if they are not the fields used for 

Theres actually ASICs out there that will accept entries that have the
same key but may differ in one or more fields in a row.
Theres also a few s/ware firewall types (such as Linux iptables) that
do this.
I cant remember the exact details but i know people play tricks with
these kinds of setups that depend on such layouts.

I would suspect such a setup would probably just go ahead and add
entries anyways with or without this flag. If theres consensus to remove
this then will be fine with me.

> BLK is included in the results to indicate what is returned at the 

I dont think i understood you %100 on the benefit of tables inside
tables.
The motivation was to have the exact same format in the SET as it is
the DELete or GET results. As an example, lets take example #7

7a) to GET entry 0-2 of table2 using BLK operations

Path = IDCount = 1, IDs=4
followed by BLK_RANGE_INDEX TLV
T = BLK_INDEX_RANGE, length of 4+8
    start = 0
    end = 2

Result:
Path = IDCount = 1,  IDs=4
T = BLK_INDEX_RANGE, length of 4+8
    start = 0
    end = 2
DATAINFO TLV, Length = XXXX
Values:
        j1value,j2value
        j1value,j2value
        j1value,j2value

7b) To DELete the same entries you could send _exactly_ the 
same msg as 7a) just different operation

7c) To SET. you would send _exactly_ the same thing as the
Result of GET in 7a).

So that was the motivation - cleaner, less code involved.

Clearly putting the indices in the DATAINFO TLV would save some
space. What i didnt quiet get is how it helps in
the case of table of table ...

cheers,
jamal

---

From: Jamal Hadi Salim <[email-masked]>
Date: Thu, 27 Jan 2005 15:28:53 -0500

Sorry I said two but this is 3 now ..

> BLK is included in the results to indicate what is returned at the 

More thought since i typed last email:

We could have DATARAW TLVs inside of DATARAW TLVs. This way we could
kill the VARENC TLV and should be able to cover table of table setup
i.e BNF extension on what i showed earlier ..

Would be:
DATAINFO TLV := [SELECTOR]  [DATARAW]
SELECTOR :=  [BLKINFO] [KEYINFO]* [IDINFO] 
DATARAW := somedata* [DATARAW TLV]*

> t2value, j1value, j2value.  But there are no ji fields in table1?

t2value being table index of some row in table 2; just as would an L3LPM
table probably have an index to NH table. Did i answer what you asked?

>   from the 

I would have no issues with that approach as it would simplify things
greatly. I would go as far as saying returning table1 should only return
just table1 and not interpret anything. Desire to have both tables
should be fulfiled by two GETs. Thoughts?

> note: examples 11 and 12 relate to the uniqueness debate.

Nod. 
Those examples show that KEYINFO is used to restrict search of content
in the SELECTOR. Any further restriction(in the LFB definition) should
result in rejection of the used KEYINFO. Does this sound reasonable?

> all should use the same encoding rules.)

The last time a proposal such as this(from Zsolt/Steve), it become
problematic for two reasons
a) Lead to innefficiencies/overhead in encoding. Workaround to try and
dump things like tables resulted in tricks like those used in marshaling
serialized java bytecode - which we dont suffer from at the moment.
b) Not pretty at all dealing with variable sized data (such as the
middle cell in a row being a variable sized entity).

Resolve those and we could be back on track. Actually I am worried of
spending even more time on that ;-> If theres some way we can do it
without stalling progress, I wouldnt object.

> source of implementation errors.

If you look at the SELECTOR as a "further" path/content selector/filter
then should be fine, no? 

cheers,
jamal

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 27 Jan 2005 15:33:23 -0500

variable chunk length fields for TLVs is such a large invitiation to 
trouble that I would strongly prefer not to go there.  I understand that 
some tables may be large.  Routing / Forwarding tables are more than 
64K.  There rae two reasonable answers that I can see:
Always use 32 bit lengths for TLVs.
Or require the CE to fetch the large tables in piecs.
I actually think the latter is more sensible.  Note that if one is fetching 
a megabyte table over TCP, that is going to produce a noticeable blockage 
for any other transactions with that FE.

With regard to deletion, I agree that deletion by key is supposed to be 
supported.  I had read the pair "DEL data to be deleted" as referring to 
something else.  Except for my desire that there shall only be one entry 
with j1=100 and j2=200 (if the j1, j2 pair is a table key), I agree with 
the formulation of being able to delete by that key as per example 12.

Yours,
Joel

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 27 Jan 2005 17:01:44 -0500

Comments in line.

>already and perhaps replace the row or parts of it.

This changes my confusion, but still leaves questions.  Lets try listing 
cases for SET
If CREATE is not set, then by definition we are doing a replace, and an 
error will be returned if the selected entry does not exist.
If CREATE|EXCL then the create will work if there is no entry, and will 
fail if there is an existing entry.
If CREATE without EXCL then it will create if thee is no entry, and update 
if there is an entry.  Or else that is CREATE|REPLACE.
In either case, whichever setting is used for "create or replace", the 
other flag combination seems to have no meaning.

I suspect that the Unix flags have grown over time and we can define a 
cleaner set for ourselves without trying to emulate the sockets behavior.

>these kinds of setups that depend on such layouts.

In my opinion, values which can occur on multiple top level things should 
not be Keys.  From a model perspective you can use nested tables, or not 
use those values as keys.  Remember that the implementation will often be 
different from the model.  Coping with that is the FE control logic's 
primary job.

>this then will be fine with me.

I believe we should remove this (F_APPEND).

>the DELete or GET results. As an example, lets take example #7

What I am looking for is that "GET whole table" uses a simpler structure 
than "GET range".
Get RANGE does exactly what you describe.
However, Get Whole table should, I think, look like

Path = IDCount = 1, IDs = 4

result:
Path = IDCount = 1, IDs = 4
Values:  (structured defined in protocol document for array and structure 
values)
     # of entries
     subscript (0), j1value, j2value  (where j1value, j2value is the 
representation
                                       for a structure of the table entry 
contents.)
     subscript(1), j1value, j2value

[subscript(0) refers to the subscript of the first extant entry of the table.]
The table construct is a packed sequence starting with a number of entries 
in the table, followed by pairs, where each pair is a subscript value 
followed by the packed representation of that entry in the table.

In this case, it does not include the BLK construct.
Note that this can be used in a SET operation.  The FE will understand that 
the whole table is to be replaced.  (This is important since a SET of Blk 
range 0..2 does not mean one should delete entries after 2.)
Hence, this is a more reliable way to do a GET that can be used as a SET.
(There is no need to worry about delete.  Delete for the whole table, if 
defined, just uses the path to the table, with no value elements needed.)

The other advantage of this is when there is a table2 inside 
table1.    Suppose that table1's entries are a j1value and a table with x1 
and x2 values.  Then,

GET
Path = IDCount = 1, IDs = 4

result:
Path = IDcount = 1, IDs - 4
Values:
    # of entries in table1
    subscript(0), j1value
         # of entries of this table2 instance
         subscript(0), x1value, x2value
         subscript(1), x1value, x2value
         ...
    subscript(1), j1value
         # of entries of this table2 instance
         subscript(0), x1value, x2value
         ...
    ...

This gives a consistent representation, that can be used with SETs, and 
which copes with nesting without trying to create weird matching between TLVs.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 27 Jan 2005 17:04:09 -0500

At 03:28 PM 1/27/2005, Jamal Hadi Salim wrote:
>should be fulfiled by two GETs.

Agreed.

Extracted and repeated so anyone else who disagrees gets a clear chance to 
speak.

Yours,
Joel

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 27 Jan 2005 17:15:43 -0500

Actually, the more I look at this, and read your further response to 
Weiming, the less I like it.
The identification of what is being set goes in the data path.  Yes, there 
is a little bit of redundancy if one is trying to set fields x, y, and z of 
elements 1, 3, and 5 of the same structure.  However, as soon as we go down 
this view, we now have to assemble identification from several places.

This seems to be a case of making the protocol more complex in order to 
save a few bytes in a moderately rare corner case. <DATAPATH>+ seems 
sufficient.

I recommend removing the FIELDINFO.

Note that there is actually an inconsistency between example 15 and earlier 
example.  In the earlier examples, when there was a BLK_LIST_INDEX, there 
was a corresponding value for each value in the list.  In example 15 we 
added a fieldinfo, and suddenly there is a value for each field, but the 
system is to implicitly duplicate them across the BLK_LIST_INDEX.  That 
changes the semantics.

Note that if we remove FIELDINFO, then we have the option of adding a 
semantic game.  Given a BLK_LIST (Or count, or range), if there is only one 
occurrence of the value (whether it is a simple value or a structure does 
not matter) then the FE will understand to duplicate the value for all 
entries in the BLK.  This is actually a useful way to optimize for the case 
that you and Weiming have asked for.  As long as we do not also try to do 
FIELDLIST at the same time.  Please.

Yours,
Joel

---

From: Jamal Hadi Salim <[email-masked]>
Date: Thu, 27 Jan 2005 18:41:40 -0500

> for any other transactions with that FE.

Since we allow to have multi path-data within the same operation should
be fine for an FE actually just breaking it into 64K chunks.
Do you see any issues on that?

cheers,
jamal

---

From: Jamal Hadi Salim <[email-masked]>
Date: Thu, 27 Jan 2005 19:15:29 -0500

> cleaner set for ourselves without trying to emulate the sockets behavior.

Let me doodle on it and get back to you; I think you may be right but i
just wanna make sure we dont forget some scenario.

> different from the model.  Coping with that is the FE control logic's 

I just found that the Linux routing tables as well do support an APPEND
(which appends to the end of the table) and PREPEND which inserts to the
very top of the table. Note that this is a key based table.
This worries me about totaly ignoring the existence of such a construct.
Give me some short time to investigate its usage and how widespread that
is and whether we can provide the same effect some other way. BSD as
well seems to do this. Of course both constructs are dangerous.
If you know the history behind this and could shed some light - that
would save me some time.

> I believe we should remove this (F_APPEND).

The next person who says reasons and gives a reason will result in
killing it. I'll just study the kernel code a little and see how useful
this is.

> Get RANGE does exactly what you describe.

Ok - gotcha. 
What you described is fine too but rather implicit. Could we have an
explicit flag to say that this is happening or a new TLV type to
indicate it - something like T=BULKDATA?

> the whole table is to be replaced.

Agreed.

>   (This is important since a SET of Blk 

nod.

> Path = IDcount = 1, IDs - 4

We would still need to encap whats below in a TLV, DATARAW or whatever
other name that would recognize what this is.

>          subscript(1), x1value, x2value

Using "# of entries in tableX" - why not just use a TLV there?

Something like
Path = IDcount = 1, IDs - 4
BULKDATA TLV, L = XXXX
	Values:
	subscript(0), j1value, BULKDATA TLV, L = XXXX
		Values
			subscript(0), x1value, x2value
			subscript(1), x1value, x2value
			.....
			....
	subscript(1), j1value, BULKDATA TLV, L = XXXX
		Values
			subscript(0), x1value, x2value
			subscript(1), x1value, x2value
			.....
			....
	...
	....
	...

cheers,
jamal

---

From: Jamal Hadi Salim <[email-masked]>
Date: Thu, 27 Jan 2005 19:44:27 -0500

> I recommend removing the FIELDINFO.

I agree on removing the nesting;
I think that it would be hard to totaly remove it: <PATH> is really
flags followed by FIELDINFO except we make it _mandatory_. 
Recall Weimings example:

table2: type array, ID = 4
elements are:
  j1, type struct Add, ID = 1
  j2, type u32, ID = 2
  j3, type u32, ID = 3
  j4, type u32, ID = 4
 
 Where struct Add may be globally defined as:
 struct Add {
 d1, type u32, ID = 1
 d2, type u32, ID = 2
 } 
 
And:
 {{1, 2}, 3}  --- meaning to address the {j1.d2, j3}

The addressing of the above is the tricky part without nesting.

If we remove the restriction of IDcount <IDs> being mandatory and
substitute that with a single optional FIELDINFO, then this would
satisfy both what you are saying as well as what Weiming wants IMO. i.e
grammar goes from:

---- 
FIELDINFO TLV := <zero or more fieldIDs [FIELDINFO]*>

to:

---- 
OPER TLV : = <PATH-DATA>+
PATH-DATA := <PATH> [DATAINFO]
PATH TLV := flags [FIELDINFO] 
DATAINFO TLV := [SELECTOR]  [DATARAW]
SELECTOR :=  [BLKINFO] [KEYINFO] [IDINFO] 
DATARAW := somedata* [DATARAW TLV]*
IDINFO := [INDINFO] [FIELDINFO]
FIELDINFO TLV := zero or more fieldIDs
INDINFO TLV :=  one or more indexes
KEYINFO TLV := KEYID KEY_DATA
BLKINFO TLV := [BLK_RANGE_INDEX] | [BLK_CNT_INDEX]
---

What the above grammar is saying now is
1) select and/or filter the path to the target); 
essentially everything from PATH on but not including DATARAW
then optinally
2) tell me what data to apply to that target.

It could be rearranged to be more readable.

Example 
Then path to that {{1, 2} becomes:
PATH TLV: L = XXXX,
	V = flags, FIELDINFO TLV, L=XXX V = {1.2}
// if specific row are being targeted then
// INDINFO or one of the BLKs will follow here.
DATAINFO TLV: L = XXXX,
	V= {2000}

Thoughts?

cheers,
jamal

---

From: Jamal Hadi Salim <[email-masked]>
Date: Thu, 27 Jan 2005 19:55:28 -0500

> system is to implicitly duplicate them across the BLK_LIST_INDEX.  That 

The intent was to show a target filter; i agree that it may be
confusing. The first part:  fieldinfo, BLK**, KEY** just says
"let me describe the data target for you"; they each have slightly
different semantics. KEYINFO for example will appear before the data to
say what the key match is. 
The end goal is to describe the data ...
We could change how it is used.

> FIELDLIST at the same time.  Please.

;->

Look at the reconstruction; since we allow the repeatable path-data i
think we should be able to simplify. 
If we can agree that a path-data is a target-slector followed by
optional data then we are on our way. 
I see that the target filter will describe:
a) the how to get to the start of the data (via the PATH - whose keyinfo
we now make optional), 
b) what to look for when you get there (search key via KEYINFO) 
c) constraints (INDINFO, BLK** or further FIELDINFO); 

Not forgeting the flags of course

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 27 Jan 2005 20:01:32 -0500

I agree that is one of many acceptable solutions available to the CE.

Yours,
Joel

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 27 Jan 2005 20:48:07 -0500

Given that PATH-DATA can occur multiple times, I don't see the value in 
allowing the FIELDINGO in the IDINFO.  Just repeat the path and the 
subscript.  The saving in reduced repetition is not, to me, worth the 
complexity.
Admittedly, this is a personal judgement call.  What has been proposed can 
be made to work.

Yours,
Joel

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 27 Jan 2005 20:55:46 -0500

I think that there are two separate things mixed here.  If I disentangle 
them, we may get clearer, and I suspect end up near agreement.

We have to define an encoding for data.  This has to be able to handle 
simple values (uint32, uint8), variable length character strings, byte 
strings, structures, and tables.  After all, I can GET a structure which in 
turn contains a number and a table.  So I have to be able to represent 
within the data, a table.
Whether this is a packed binary structure, a nested TLV structure, or some 
other mechanism we can imagine makes relatively little difference to 
me.  The generator and receiver will both know the relevant data types.  So 
they can use that to help with parsing.  Or we can encode redundant type 
and length information to make things more consistent and more robust.  Or ....

There does need to be a block of data sent on a SET request and returned in 
a successful GET response.  This will need to be in a TLV inside the 
PATH-DATA TLV.  If the data encoding uses a packed structure, then this TLV 
will be the DATA TLV>  If the data encoding uses TLVs for array, structure, 
string, etc then this will presumably be the correct top level TLV needed 
for the type of the target of the operation.
In either case, there presumably will be as many of these TLVs as there are 
entries in a BLK request.  (Or, if we keep it, entries in a nested 
FIELDINFO request.)

In either case, when fetching or setting an entire table we will not need 
the BLK construct as the data respresentation will need to be complete 
enough to handle it.

Yours,
Joel M. Halpern

---

From: Jamal Hadi Salim <[email-masked]>
Date: Thu, 27 Jan 2005 21:43:19 -0500

> Admittedly, this is a personal judgement call.  What has been proposed can 

It does sound reasonable to get rid of the internal FIELDINFO;
I will have to run the use cases again to double check. This will lead
the grammar to be:

SELECTOR :=  [BLKINFO] [KEYINFO] [INDINFO]

Weiming this should meet you desire to do:

path(1.2) Data(2000) ; suppose to set j1.d2=2000
path(3) Data(300)

and 
path(1) Data(100)
path(2) Data(200)
path(3) Data(300)

Ive substituted the word subpath with path from above grammar.

cheers,
jamal

---

From: Jamal Hadi Salim <[email-masked]>
Date: Thu, 27 Jan 2005 22:11:05 -0500

Agreed on no need for BLK for entire tables and that those are
optimizations really for ranges;
The knowledge by the receiver/transmitter about what the data being
carried is in my opinion the main optimization. In otherwise, if you can
be told where on the LFB definition the data comes from, you can 100%,
unambigously, intepret that data type. So defining DATARAW in addition
to the path coordinates is sufficient for _all_ details. I though a
little about the table dump you pointed to earlier and I dont think we
need a new BULKDATA TLV; i.e by looking at the path Field ID, we know
its table4 and we can therefore decode it. So a DATARAW is sufficient. 

The other way to do it (I think maybe thats what you are alluding to
here) is to define containers for different DataTypes;
The advantage of this case is that it is more structured data that can
infact be decoded by any entity with not much knowledge of the LFB data
definitions.
The disadvantage i think is we will end up being relatively less
efficient and we are more than likely going to run into the same issues
as the Zsolt/Steve proposal. Like i said earlier: not religious if we
dont get bogged down. 

cheers,
jamal

---

From: Jamal Hadi Salim <[email-masked]>
Date: Fri, 28 Jan 2005 13:46:46 -0500

Some small gotcha in the BNF found while walking through the use cases..
To use Joels example below:

On Thu, 2005-01-27 at 17:01, Joel M. Halpern wrote:

> table1.    Suppose that table1's entries are a j1value and a table2 with x1 
> and x2 values.  

To give some values:
Table1: ID 1
	j1: ID 1, table2: ID2
		x1: ID 1, x2: ID2

Lets say i wanted to set cell x1 of table2's index 4 which is contained
in table1 index 10.

The old way i would have specified a path with a mix of both field and
index ids together. i.e

1.10.2.4.1 i.e a field.index.field.index.field

And the data would contain whatever i want to set to x1.

The only way to get this working now is to allow one of:

a) PATHINFO := <[FIELDINFO][INDINFO]>*

i.e you could have zero or more combos of FIELDINFO and INDINFO
TLVs; A Field info preceeds a INDINFO
--> 20 bytes overhead for example i gave

b) Both FIELDINFO and INDINFO to be able to recursively contain each
other i.e:

FIELDINFO := fieldvalue(s) [INDINFO]
INDINFO := index(es) [FIELDINFO]

No difference in overhead; still 20 bytes

c) restore or allow as a substitute the original format we had i.e
the 1.10.2.4.1 example i gave
MIXEDINFO:= a mix of fieldvalue(s) and indexes

4 bytes of overhead and rather intuitive to me. My prefered mode really.

Disdvantage is now in the simple case we have a simple case such as
1.10 being describable by either FIELDINFO followed by INDINFO
or by a single MIXEDINFO TLV.

I am worried this is getting out of control; but should be noted that it
is absolutely not an obscure case - we actually do use such a table
layout.

I believe if we can solve this, we are done on path data.

cheers,
jamal

PS:
As a reminder the standing BNF so far is:

---

From: "Wang,Weiming" <[email-masked]>
Date: Sat, 29 Jan 2005 17:16:32 +0800

Joel,

----- Original Message -----
From: "Joel M. Halpern" <[email-masked]>
Subject: Re: [3]Re: path data textt update

> subscript.  The saving in reduced repetition is not, to me, worth the
I'v seen several times you mentioned the complexity the subpaths may add. I'v
been thinking of the complexity you might thought, but the more I think, the
more I find things are on the contrary, i.e., the repeated PATH-DATA will add
much unnecessary complexity for system, not only the bits consuming. I'v made a
post explaining the advantage for subpaths, and also hope you can reply with the
contrary views, but a pity is I still didn' t get. I know you are quite
experienced, but it may be difficult to fully understand what you mean before
you  show with more details.

To more clearly express my views, pls allow me mention the example again as:

To use the separate Path-Data way, we have a sequence of Path-Data as:
OpTLV= 'Set' ...
Path(4.1) IndexTLV (1,3,5,7,9) Data(100)
Path(4.2) IndexTLV (1,3,5,7,9) Data(200)
Path(4.3) IndexTLV (1,3,5,7,9) Data(300)

To use a subpath way, we have a path-data as:
Path(4) IndexTLV(1,3,5,7,9)
Subpath(1) Data(100)
Subpath(2) Data(200)
Subpath(3) Data(300)

To compare with:
1. the difference for bits consuming is very obvious.
2. The PATH-DATA + repeating mode also results in the complexity for resources
consuming. Just consider we may need unnecessarily repeat the searching work for
Index(1,3,5,7,9) for three times. Just as I mentioned an example before, why we
need to drive three times to deliver three pieces of  pisa to a family to the
father, mother, and son?

Cheers,
Weiming

---

From: "Wang,Weiming" <[email-masked]>
Date: Sat, 29 Jan 2005 17:22:03 +0800

Jamal,

One more thing I have to mention is, the KEY you used in your example is
actually also a fieldID anyway. To use KEYINFO this way is actually the same as
Subpaths.  Quoted from your example:

j1, type u32, ID = 1
j2, type u32, ID = 2

2) delete  first 10 entries with j1=100, and j2=200 in table 2,
starting at index 10.

Path: IDCount = 1, IDs=4
SELECTOR :=

This is exactly what I mean the thought for subpaths, only with that subpaths
allow more than one level fieldID addressing.

I'm afraid what Joel means for KEY is a little different from what you mean :)

Cheers,
Weiming

---

From: "Joel M. Halpern" <[email-masked]>
Date: Sun, 30 Jan 2005 11:50:22 -0500

You are correct that if content searching is common, and particularly if 
arbitrary content searching is allowed, then the repetition of the content 
search clause would represent a significant increase in work.

In contrast, if all of the element identifiers are in the path description, 
then the logic for parsing / interpreting paths and determining the class 
of object being manipulated is in one place, controlled by one field.  This 
produces a significant increase in consistency and accuracy as compared 
with assembling paths from multiple components.

And I normally expect that even the references to multiple indexes, and the 
setting of multiple discrete items in a single LFB are both moderately 
rare.  If we have done our job right, items which are manipulated together 
will be defined in common structures.

After looking over the discussion, I can probably live with the 
subpath.  It would have to be structured along the lines of your example, 
rather than the earlier BNF proposal.  That is, the two multiply occurring 
elements (indexing and subpathing) have to be neszted and arranged with the 
data in such a way that it is very clear what is being repeated, and how 
the data relates to the repetition.  (Part of my concern with the previous 
example, was the odd grouping that had crept into the organization of 
information as we tried to understand each other.)

I would ask at the same time whether you can live with a more restricted 
form of content addressing.

Yours,
Joel M. Halpern

---

From: Jamal Hadi Salim <[email-masked]>
Date: Sun, 30 Jan 2005 15:05:04 -0500

> allow more than one level fieldID addressing.

KEYINFO does not set any value; it is merely describing how to search
and find what to delete.
Looking at your previous posting, subpath is used to also carry data to
be used for setting - unless i misunderstood.
How do you say "first 10 entries with j1=100, and j2=200 in table
2,starting at index 10¨ with subpath?

> I'm afraid what Joel means for KEY is a little different from what you mean :)

The KEY merely describes search content and NEVER data to be set. I
think this may be the misunderstanding. You seem to be saying subpath
can have construct which can be used to be both filter as well as 
data to be set. Maybe you can describe this angle a little more.

What Joel wants is in addition to restrict what fieldID can be a key
(which could be done in the LFB document).

cheers,
jamal