Subject: [icnrg] [CCNx-TWG] Meeting on 12/9/15 at 11am PST

From: <[email-masked]>
Date: Mon, 7 Dec 2015 18:47:51 +0000

A CCNx-TWG teleconference will take place on Wednesday 12/9 at 11am PST.

If you are interested in participating, please join the conference call using
the instructions below. Meeting notes will be made available on Thursday 12/10.

Please be prepared for the agenda items by familiarizing yourself with the
related material. If you have any questions, concerns, or comments about this
meeting, please reply to this email.

Best,
Chris

--
Date & Time: 12/9/15 at 11am PST

Purpose: Continue the discussion of the FLIC and ISO manifests. Propose planned
changes to the CCNx Semantics and Messages documents. Time pending, discuss
the obstacles to interoperability and identify the next step for the "advanced
manifests."

Agenda:

- Continue discussion and evaluation the FLIC and ISO Manifest designs, with a
  focus on de-duping and sizes per pointer.
- Propose changes to the CCNx Semantics and Messages documents.
- Briefly revisit interoperability barriers (link protocol, forwarder
  discovery, etc.) and identify the gaps.
- Briefly discuss advanced manifest use cases.
- Identify volunteers for the “advanced” Manifest. (This design work
  will be done in parallel.)

Related material:

- FLIC vs ISO email (naming-notes.txt)
- FLIC specification (https://github.com/tschudin/icn-flic-rfc/blob/master/draft-tschudin-icnrg-flic-00.txt)
- ISO overview (iso-overview.txt)
- CCNx Semantics (https://www.ietf.org/id/draft-irtf-icnrg-ccnxsemantics-00.txt)
- CCNx Messages in TLV Format (https://www.ietf.org/id/draft-irtf-icnrg-ccnxmessages-00.txt)

Attendance instructions:

Dial-in number: 1-800-268-4017
Participant code: 3768636189

---

From: [email-masked]
Date: Wed, 9 Dec 2015 00:00:07 -0800 (PST)

On Mon, 7 Dec 2015, [email-masked] wrote:

> ...

For tomorrow's discussion, I attach a figure from the "venti" archival 
file system of Plan9 that uses the manifest concept we are talking about 
for de-duplification at disk block level.

Some notes from implementation land (and a notice that the FLIC draft 
has been updated this evening, now also talking about binary trees):

- I was able to stream HD video over the atlantic (200-300 msec RTT) using
   FLIC manifests.

   This answers one concern, namely the "data digests per second" question:
   It is high enough, both in the skewed tree and the binary tree shape,
   such that the retrieving client always has enough data pointers to request
   content in time.

   To this end I added a simple sliding window receiver to FLIC with TCP-like
   window scaling. Running a streaming demo is as simple as
   ./ccn-lite-flic -s ndn2013 -u 127.0.0.1/7777 /movie/elephants_dream.mp4 | cvlc -vvv -

- As a side effect I had to implement a repo which (mainly) serves
   digest-based queries and also (sporadicly) named-based queries. The
   latter is useful for getting the root manifest.

   If you need a high-volume repo, checkout the dev-master branch of
   ccn-lite. I tested it with up to 1GB content (in a single FLIC file
   with some 1.3 million ICN packets).

   A feature of repo256 is that it can also serve queries based on the
   *content's hash* (which is something different from querying via the
   packet's hash).

Best, christian

Links:
   Venti paper: http://plan9.bell-labs.com/sys/doc/venti/venti.pdf
   FLIC draft: https://github.com/tschudin/icn-flic-rfc/blob/master/draft-tschudin-icnrg-flic-00.txt
   ccn-lite: dev-master branch at https://github.com/cn-uofbasel/ccn-lite/tree/dev-master

---

From: [email-masked]
Date: Wed, 9 Dec 2015 09:41:31 -0800 (PST)

Hi again,

Regarding dedup, I wanted to share with you this nice observation
about manifest benefits:

One of the movies I played with had the following sequence of (data
packet) digests:

[I] 0.0044: db5cf7ac1529524e632a559b9e8c77e29a33af8179857cc788bc85665f0ea1b7
[I] 0.0084: a42498bcbdb0b5fdf23dc15270c56ccae2be20d636c3f8df6b5ffb1963cca7e0
[I] 0.0085: a42498bcbdb0b5fdf23dc15270c56ccae2be20d636c3f8df6b5ffb1963cca7e0
[I] 0.0086: a42498bcbdb0b5fdf23dc15270c56ccae2be20d636c3f8df6b5ffb1963cca7e0
[I] 0.0087: a42498bcbdb0b5fdf23dc15270c56ccae2be20d636c3f8df6b5ffb1963cca7e0
[I] 0.0088: a42498bcbdb0b5fdf23dc15270c56ccae2be20d636c3f8df6b5ffb1963cca7e0
[I] 0.0089: a42498bcbdb0b5fdf23dc15270c56ccae2be20d636c3f8df6b5ffb1963cca7e0
[I] 0.0090: a42498bcbdb0b5fdf23dc15270c56ccae2be20d636c3f8df6b5ffb1963cca7e0
[I] 0.0091: a42498bcbdb0b5fdf23dc15270c56ccae2be20d636c3f8df6b5ffb1963cca7e0
[I] 0.0092: a42498bcbdb0b5fdf23dc15270c56ccae2be20d636c3f8df6b5ffb1963cca7e0
[I] 0.0093: a42498bcbdb0b5fdf23dc15270c56ccae2be20d636c3f8df6b5ffb1963cca7e0
[I] 0.0094: a42498bcbdb0b5fdf23dc15270c56ccae2be20d636c3f8df6b5ffb1963cca7e0
[I] 0.0095: a42498bcbdb0b5fdf23dc15270c56ccae2be20d636c3f8df6b5ffb1963cca7e0
[I] 0.0096: a42498bcbdb0b5fdf23dc15270c56ccae2be20d636c3f8df6b5ffb1963cca7e0
[I] 0.0097: a42498bcbdb0b5fdf23dc15270c56ccae2be20d636c3f8df6b5ffb1963cca7e0
[I] 0.0098: a42498bcbdb0b5fdf23dc15270c56ccae2be20d636c3f8df6b5ffb1963cca7e0
[I] 0.0171: aaad495d807d19a0ff2e21ef2fa7f2a5486bee8448df6afbb96f573103f87119
[I] 0.0203: ece08662be53ad518fcd31eb1a1c30f560cd9b2c5ef012c5dedbf9c0368bc388

That is, the 2nd to the 15th packet are identical. This permits for
double deduplification:

a) suppress identical interests: The first dedup is to *not* send
multiple queries. Note that this is a "stream of digests" view: such
data pointer repetition might occur across manifest boundaries.

b) internal expansion: The second dedup is to service with one
incoming packet *all* the slots where the data for that
0xa4249... hash is required. Note that this also handles the situation
where a data hash D was gathered from an incoming manifest *after* the
first interest for D was sent out.

The current FLIC receiver does this double-dedup without maintaining a
packet cache:

Case a) Outgoing queries are checked in the small out queue and
identical interest packets are suppressed. (BTW: the ccn-lite
forwarder does this by default). This catches the presented case
above.

Case b) An incoming packet will satisfy all slots in the send window,
regardless whether an interest has already been sent out or not. That
is, even future interests can be avoided. This goes beyond treating
the receiver as being PIT-driven (because PIT entries vanish after
they have been satisfied and do not exist before a first interest is
sent out).

In other words: (within some window size) the content dedup that
exists at the server side is preserved over the network. Nice! If one
wishes to enlarge this window, one simply would have to add a "receive
cache".

Note that all this is only possible if objects are nameless. As soon
as there is a unique name for each data packet (that is independent of
the packet's content, e.g. a sequence number), all the dedup
opportunities described above are lost.

best, c

BTW: I will be challenged to join the phone call today because of
travel, but will try.

On Wed, 9 Dec 2015, [email-masked] wrote:

> For tomorrow's discussion, I attach a figure from the "venti" archival file 
> system of Plan9 that uses the manifest concept we are talking about for 
> de-duplification at disk block level.
>
> Some notes from implementation land (and a notice that the FLIC draft has 
> been updated this evening, now also talking about binary trees):
>
>  FLIC draft: 
> https://github.com/tschudin/icn-flic-rfc/blob/master/draft-tschudin-icnrg-flic-00.txt
>  ccn-lite: dev-master branch at 
> https://github.com/cn-uofbasel/ccn-lite/tree/dev-master
>
>
>> Attendance instructions:
>>

---

From: <[email-masked]>
Date: Wed, 9 Dec 2015 18:44:07 +0000

Christian,

This is a great example of de-dupe with hash based names!

Nacho and I have talked at times about putting an in-line client cache in the system (at
the application’s request) which would allow the consumer to just issue interests, even if they
are duplicates, and the client-side cache would handle them.

If one is running Metis on the client with a content store, that would have the same effect, but not be under the application’s direct control.

It’s a trade off of where you want to maintain the historical knowledge of what has already been asked for.  I think for correctness, the manifest handler needs to have that knowledge for your (b) item of one incoming content object satisfying all pending manifest entries of the same hash.  But, I think the manifest handler should not be keeping a historical window beyond the pending interests.  That’s the job of a cache somewhere (either in the forwarder or in the stack or as a separate component below the manifest handler).

Marc

> On Dec 9, 2015, at 9:41 AM, [email-masked] wrote:
> 
> Hi again,
> 
>> For tomorrow's discussion, I attach a figure from the "venti" archival file system of Plan9 that uses the manifest concept we are talking about for de-duplification at disk block level.
>> 
>> Some notes from implementation land (and a notice that the FLIC draft has been updated this evening, now also talking about binary trees):
>> 
> _______________________________________________
> icnrg mailing list
> [email-masked]
> https://www.irtf.org/mailman/listinfo/icnrg