Subject: [icnrg] link to updated FLIC draft (was Re: On FLIC and ISO manifest differences

From: [email-masked]
Date: Tue, 1 Dec 2015 09:03:56 -0800 (PST)

BTW: Chris and I updated the FLIC draft, see

https://github.com/tschudin/icn-flic-rfc/blob/master/draft-tschudin-icnrg-flic-00.txt

It contains explanations for 4 special use cases:

3.  Advanced uses of FLIC manifests . . . . . . . . . . . . . . .   9
   3.1.  Block-level de-duplification  . . . . . . . . . . . . . .   9
   3.2.  Growing ICN collections . . . . . . . . . . . . . . . . .   9
   3.3.  Re-publishing a FLIC under a new name . . . . . . . . . .  10
   3.4.  Data Chunks of variable size  . . . . . . . . . . . . . .  10

best, christian

---

From: <[email-masked]>
Date: Tue, 1 Dec 2015 18:47:55 +0000

On 11/30/15, 11:20 PM, "[email-masked]" <[email-masked]> wrote:

>Hi Nacho,
>
>On Tue, 1 Dec 2015, [email-masked] wrote:
>
>> Forcing all objects to be nameless does sound nice and it simplifies 
>> stuff.  However, it also means that you have to host all the data 
>> yourself… all the time.  I don’t want to impose that restriction.
>
>Picking this one first: would you like to have partial storage of an ISO 
>involving multiple hosting sites, instead of storing the full set at a 
>single site?
>
>This is supported in FLIC by having a "locator entry" in a HashGroup's 
>metadata. In the example below all chunks would be fetched from where 
>the root manifest was fetched (/parc/west) except for the manifest with 
>the metadata locator entry: this manifest and all its descendent nodes 
>will be fetched at /parc/east.

Ah, perfect, this is (mostly) what I want.   There was an implication from Chris that this was not possible and only nameless objects were allowed.

>But I would advise against such a practice because you nail down 
>(hardcode) part of an ISO to a locator, which means that you cannot 
>switch the hoster of the *whole* ISO anymore by just putting the root 
>manifest elsewhere. Switching then comes at the price of recomputing 
>several manifest nodes.

I agree this is a tradeoff, but it’s something that I don’t want to force by the design. Let the use dictate if people want to do this.  In the FLIC case you’re doing this via HashGroup locators, in ISO this is done via basenames.

There are a few cases where I think having names is useful and I don’t want to lose that by forcing everything to be nameless.

>The second point - byte count - is minor, I should have left it out at 
>this stage of the discussion. But the observation is simply this: If in 
>ISO each pointer can have a metadata entry, a dataptr needs at least two 
>levels of TLVs.
>
>(dataptr
>   (hashvalue OCTET[32])
>   (metadata
>      (entry1 ...)
>   )
>)
>
>which holds even if no metadata section is present. While in FLIC, it's 
>always
>
>(dataptr OCTET[32])
>
>
>You ask specifically for manifest pointers with size information: here 
>we have equality. Using CCNx encoding when showing byte counts:
>
>ISO:
>(mfstptr                  2+2
>   (hashvalue OCTET[32])   2+2+32
>   (metadata               2+2
>      (size ...)           2+2+ intLen
>   )
>)
>
>FLIC:
>(HashGroup                2+2
>   (metadata               2+2
>     (size ...)            2+2+ intLen
>   )
>   (mfstptr OCTET[32])     2+2+32
>)
>
>ISO could save 2+2 bytes if all metadata entries would be at the same 
>level as the hashvalue entry. (But so could FLIC).

Well, first of all, I don’t want to have “metadata” per pointer, I just want to have size.  So, in your example we would save the 2+2 if we just used size and if we decided to have more metadata it would go in the same level (as you state).  So, for this small example you would see 4 bytes of saving in ISO.

If FLIC applied the same it would be the same as well.  A little more confusing because of the construction of a hashgroup but still doable.

One quick note is that I didn’t make a case for a 2+2 TLV (Neither for ISO or for FLIC). And hence my comment about defining encodings before comparing.

The main disagreement that I have with Chris is that I don’t think grouping buys us much. Why organize the data that way?  What is the big benefit of the grouping vs having the layout be more flat?

The only place I see the system potentially saving bytes is a place where there is a mixture of various uniform groups that interlace at a granular level. Something like:

10 pointers to leaf nodes of the same size
1 pointer to a manifest node
10 pointers to leaf nodes of the same size
1 pointer to a manifest node
10 pointers to leaf nodes of the same size

Because, if the blocks are large, then just let the “group” just be the whole manifest.
If the blocks are not uniform, you’re going to need many groups, effectively forming one group per pointer (as with the pointers to manifest nodes, which apparently, from what I understood, need a separate group to be able to define a size).

So, given this, I don’t see a value in the complexity of the groups.

I would rather have each manifest be on it’s own.  Each entry with its own information (like size).  If there is some metadata that applies to the group, then let it apply to the whole manifest node.  For example, if we determined that it would be worthwhile to save the space required by having a “block size” that applies to all leaf nodes in a manifest.

And to be clear, I’m trying to separate the encoding part with what the semantics are.

Nacho

>
>> On 11/30/15, 3:48 PM, "icnrg on behalf of [email-masked]" <[email-masked] on behalf of [email-masked]> wrote:
>>
>>
>>
>>
>>>
>>> (ManifestMsg
>>>    (HashGroup
>>>        (MetaData (Blocksize 2048) (Totalsize 4001))
>>>        (DataPtr 0xabcd....)
>>>        (DataPtr 0x2334....)
>>>    )
>>>    (HashGroup
>>>        (MetaData (Totalsize 64987))
>>>        (MfstPtr 0x8772....)
>>>    )
>>>    (HashGroup
>>>        (MetaData (Blocksize 2048) (Totalsize 2999))
>>>        (DataPtr 0xbcde....)
>>>        (DataPtr 0x3345....)
>>>    )
>>> )
>>>
>>> Note that this trick of adding a separate HashGroup consumes the same
>>> number of encoded bytes as in ISO. But we save many more bytes in the
>>> normal HashGroup case because DataPtr directly leads to the 32 digest
>>> bytes instead of having internal structure.
>>
>> I’m not sure why you claim that it consumes the same number of bytes 
>> since we haven’t defined an encoding yet.
>>
>> Not to mention that it’s probably not true that it needs the same 
>> number of bytes. It would probably consume more bytes if you are 
>> encoding an array of MfstPrts (couldn’t you spell this out?) each one 
>> inside a HashGroup.
>>
>> Also, I’m unsure that you save more bytes in a normal HashGroup. 
>> (Purely for the fact of being in a HashGroup, as opposed to lack of 
>> features, which could lead to byte savings).
>>
>>
>>> To summarize, we believe the ISO approach of mixing named and nameless
>>> objects *in the manifest tree* should not be done for single object
>>> manifests. This is not to say this approach won’t be useful for the
>>> “advanced manifest” to come later.
>>
>> Forcing all objects to be nameless does sound nice and it simplifies 
>> stuff.  However, it also means that you have to host all the data 
>> yourself… all the time.  I don’t want to impose that restriction.
>>
>> Nacho
>>
>>
>>
>> --
>> Nacho (Ignacio) Solis
>> Protocol Architect
>> Principal Scientist
>> Palo Alto Research Center (PARC)
>> +1(650)812-4458
>> [email-masked]
>> _______________________________________________
>> icnrg mailing list
>> [email-masked]
>> https://www.irtf.org/mailman/listinfo/icnrg

---

From: <[email-masked]>
Date: Tue, 1 Dec 2015 19:00:11 +0000

On December 1, 2015 at 10:47:56 AM, Solis, Ignacio ([email-masked]) wrote:

<snip>

> Ah, perfect, this is (mostly) what I want. There was an implication from Chris that this  
> was not possible and only nameless objects were allowed.

We only suggested not to mix things. The design supports it if really needed. 

> I agree this is a tradeoff, but it’s something that I don’t want to force by the design.  
> Let the use dictate if people want to do this. In the FLIC case you’re doing this via HashGroup  
> locators, in ISO this is done via base names.

Great! Both schemes do what we want. :-)

> forcing everything to be nameless.

Right--Neither design makes this assertion.

<snip> 

>  
> Well, first of all, I don’t want to have “metadata” per pointer, I just want to have size.  
> So, in your example we would save the 2+2 if we just used size and if we decided to have more  
> metadata it would go in the same level (as you state). So, for this small example you would  
> see 4 bytes of saving in ISO.
>  
> If FLIC applied the same it would be the same as well. A little more confusing because of  
> the construction of a hashgroup but still doable.
>  
> One quick note is that I didn’t make a case for a 2+2 TLV (Neither for ISO or for FLIC). And  
> hence my comment about defining encodings before comparing.
>  
>  
> The main disagreement that I have with Chris is that I don’t think grouping buys us much.  
> Why organize the data that way? What is the big benefit of the grouping vs having the layout  
> be more flat?
>  
> The only place I see the system potentially saving bytes is a place where there is a mixture  
> of various uniform groups that interlace at a granular level. Something like:
>  
> If the blocks are not uniform, you’re going to need many groups, effectively forming  
> one group per pointer (as with the pointers to manifest nodes, which apparently, from  
> what I understood, need a separate group to be able to define a size).
>  

It doesn’t seem complex to me. It can be made simple--one group for the whole manifest, or one group per pointer--or anything in between. The design shouldn’t restrict how to user wants to group things. 

Best,
Chris