Subject: [icnrg] Update of draft-irtf-icnrg-evaluation-methodology to address IRSG review comments

From: Börje Ohlman <[email-masked]>
Date: Fri, 11 Mar 2016 13:10:49 +0000

Aaron,
Thanks again for your constructive comments. To address them we have restructured the document and made several updates and clarifications. Please let us know if you think we managed to address your comments properly.

ICNRG,
Please have a look at the document and let us know what you think.

Dirk (the document shepherd),
Unless we get mayor comments during the next few days I would propose that you start an RG last call during next week.

Börje

On 30 dec. 2015, at 00:49, Aaron Falk <[email-masked]<mailto:[email-masked]>> wrote:

Hi Dirk-

Here's my review.  It's an interesting doc.  I'm happy to correspond about any of my comments.

--aaron

High level comments:

  *   I'm not quite sure this document provides a methodology.  A better title might be “Evaluation Considerations”

We agree and have changed the title. We actually changed it to “Evaluation and Security Considerations” to better reflect the full contents of the document.

We also rewrote the abstract.

  *   Section 2 isn't required by Section 3 and is a little off the focus of the document.  I recommend moving Section 2 to after Section 4, possibly to an appendix.

Moved to after section 4. Updated section references in the intro.

Other comments:

Section 1:

   This document incorporates input from ICNRG participants and their
   corresponding text contributions; it has been reviewed by several
   ICNRG active participants (see section 7<https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#section-7>), and represents the
   consensus of the research group.

[AF] This is  first mention of the ICNRG and comes without expansion or definition.  Since you say earlier that the audience is the "ICN ... researchers and practitioners" I wouldn't expect them to know about the RG (indeed, RFCs live longer than RGs and the ICNRG may be a distant memory).  Thus, I suggest adding a short paragraph describing the RG’s goals and members.

Done.

For sections 2.2 and 2.3 (now 4.2 & 4.3) I’ve introduced sub-headings for the respective tools/facilities to make the document easier to read and to make them easier to find in the index.

Section 2.2:
[AF] The section starts with a nice summary of requirements but the descriptions of the various tools should include some discussion of how well the requirements are met.

   The Icarus simulator [ICARUS<https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#ref-ICARUS>] implements ProbCache [PROBCACH<https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#ref-PROBCACH>],

   centrality-based in-network caching [CL4M<https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#ref-CL4M>]

[AF] "centrality-based" must be a ICN phrase.  It isn't meaningful to me.  Is there a less jargon-y phrase that can be used?
Added text to address this.

Section 2.3:
[AF] Given the discussion of requirements for tools, why no requirements for facilities?  E.g., low cost of use, reproducible configuration, easy-to-use tools, sharable, available background traffic…
Added text to address this.

   Often, central to the assessment of the features provided by
   a novel mechanism, lies the consideration of how it improves over
   already existing technologies, and by "how much."

[AF] Strike “Often,"
Done.

Section 3:

   This section considers techniques and options available for several
   key aspects of any evaluation method.

[AF] Given the draft title is "Information-centric Networking: Evaluation Methodology", I assume this section is the meat of the document and yet there is only a trivial introduction.  Throw the reader a bone, eh?  What am I going to hear about?  Am I to select one of the approaches described?  Or are these considerations for any evaluation technique?  Is there a relationship between the tools and facilities of the prior sections and methodologies?
Added text to address this.

Section 3.1.  Topology Selection
[AF] There's been a lot of prior work on topologies for simulation & performance evaluation.  Does ICN change any of that?  This section is a little of a hodgepodge of information.  It's hard to take away much insight on how topology selection applies to ICN evaluation.  It would benefit from a re-write that is clearer on what questions the section is supposed to answer (and then answers them).
Rewrote section.

Section 3.2.  Traffic Load

Table III. Content catalog

   Traffic | Catalog |  Mean Object Size  |  Popularity Distribution
   Load    | Size    |  [L4<https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#ref-L4>][L5][L7<https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#ref-L7>][L8]  |  [L3<https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#ref-L3>][L5][L6<https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#ref-L6>][L11][L12<https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#ref-L12>]
           | [L1<https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#ref-L1>][L2]|  [L9<https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#ref-L9>][L10]         |
           | [L3<https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#ref-L3>][L5]|                    |
   ==================================================================
   VoD     |  10^4   | Object: ~100 MB    | Zipf, 0.65 <= alpha <= 1
   ==================================================================

[AF] Video traffic has largely migrated to HTTP Adaptive Streaming where chunks are separate web objects and may be cached separately.  For example, the first 10 minutes of a movie may be more popular than the rest and cached in more places.  Treating VoD as a single 100MB object doesn’t reflect this.
Added text to address this.

Section 3.3.  Choosing Relevant Metrics

   ICN is a networking concept that arose from the desire to align the
   operation model of a network with the model of its typical use.  For
   TCP/IP networks, this implies changing the mechanisms of data access
   and transport from a host-to-host model to a user-to-information
   model.  The premise is that the effort invested in changing models
   will be offset, or even surpassed, by the potential of a "better"
   network.  However, such a claim can be validated only if it is
   quantified.

[AF] This is a great introductory paragraph.  Suggest moving it to Section 1.
Done.

   ICN approaches have used system metrics, but unfortunately
   the situation is not as coherent as with the traffic metrics.

[AF] This statement isn't supported.  How is the use of system metrics not coherent?  What does this statement mean?
Removed the sentence.

Some additional things we did (new section numbering):

Sec. 3.1:
Updated NetInf security text and added a reference.

Sec 3.3:
Added some text clarifying that it is only the first hop router that can see who request what.

Sec. 4:

I added the following text and created the corresponding Wiki page.

An updated version of the information in this section will be maintained at the ICNRG Wiki page: https://trac.tools.ietf.org/group/irtf/trac/wiki/IcnEvaluationAndTestbeds

Börje

On Wed, Nov 25, 2015 at 8:07 AM, Dirk Kutscher <[email-masked]<mailto:[email-masked]>> wrote:

Hello IRSG,

ICNRG has reached consensus on moving forward with another document that now needs IRSG review:

https://datatracker.ietf.org/doc/draft-irtf-icnrg-evaluation-methodology/

This document surveys the evaluation tools currently available to

researchers in the information-centric networking (ICN) area and

provides suggestions regarding methodology and metrics.  Further,

this document sheds some light on the impact of ICN on network

security.

Looking at our review assignment table, next in line would be Aaron.

Aaron, do you have a chance to review this?

Thanks,

Dirk