Subject: Re: [icnrg] [irsg] IRSG review for draft-irtf-icnrg-evaluation-methodology

From: Dirk Kutscher <[email-masked]>
Date: Fri, 4 Dec 2015 09:34:00 +0000

Thanks, Aaron!

Dirk

From: Aaron Falk [mailto:[email-masked]]
Sent: Freitag, 4. Dezember 2015 00:08
To: Eggert, Lars
Cc: Dirk Kutscher; [email-masked]; Internet Research Steering Group ([email-masked])
Subject: Re: [irsg] IRSG review for draft-irtf-icnrg-evaluation-methodology

Whoops.  Lost this in my inbox.  Sure, I will review it.

--aaron

On Mon, Nov 30, 2015 at 8:24 AM, Eggert, Lars <[email-masked]<mailto:[email-masked]>> wrote:
Aaron?

On 2015-11-25, at 14:07, Dirk Kutscher <[email-masked]<mailto:[email-masked]>> wrote:
>
> Hello IRSG,
>
> ICNRG has reached consensus on moving forward with another document that now needs IRSG review:
>
> https://datatracker.ietf.org/doc/draft-irtf-icnrg-evaluation-methodology/
>
> This document surveys the evaluation tools currently available to
> researchers in the information-centric networking (ICN) area and
> provides suggestions regarding methodology and metrics.  Further,
> this document sheds some light on the impact of ICN on network
> security.
>
> Looking at our review assignment table, next in line would be Aaron.
>
> Aaron, do you have a chance to review this?
>
> Thanks,
> Dirk

---

From: Aaron Falk <[email-masked]>
Date: Thu, 3 Dec 2015 18:08:05 -0500

--aaron

On Mon, Nov 30, 2015 at 8:24 AM, Eggert, Lars <[email-masked]> wrote:

> Aaron?
>
> On 2015-11-25, at 14:07, Dirk Kutscher <[email-masked]> wrote:
> >
> > Hello IRSG,
> >

---

From: Aaron Falk <[email-masked]>
Date: Tue, 29 Dec 2015 18:49:38 -0500

Hi Dirk-

Here's my review.  It's an interesting doc.  I'm happy to correspond about
any of my comments.

--aaron

High level comments:

   - I'm not quite sure this document provides a methodology.  A better
   title might be "Evaluation Considerations"
   - Section 2 isn't required by Section 3 and is a little off the focus of
   the document.  I recommend moving Section 2 to after Section 4, possibly to
   an appendix.

Other comments:

*Section 1: *

>    This document incorporates input from ICNRG participants and their
>    corresponding text contributions; it has been reviewed by several
>    ICNRG active participants (see section 7 <https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#section-7>), and represents the
>    consensus of the research group.
>
>
[AF] This is  first mention of the ICNRG and comes without expansion or
definition.  Since you say earlier that the audience is the "ICN ...
researchers and practitioners" I wouldn't expect them to know about the RG
(indeed, RFCs live longer than RGs and the ICNRG may be a distant memory).
Thus, I suggest adding a short paragraph describing the RG's goals and
members.

*Section 2.2:*
[AF] The section starts with a nice summary of requirements but the
descriptions of the various tools should include some discussion of how
well the requirements are met.

   The Icarus simulator [ICARUS
<https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#ref-ICARUS>]
implements ProbCache [PROBCACH
<https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#ref-PROBCACH>],
>
>    centrality-based in-network caching [CL4M <https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#ref-CL4M>]
>
>
[AF] "centrality-based" must be a ICN phrase.  It isn't meaningful to me.
Is there a less jargon-y phrase that can be used?

*Section 2.3:*
[AF] Given the discussion of requirements for tools, why no requirements
for facilities?  E.g., low cost of use, reproducible configuration,
easy-to-use tools, sharable, available background traffic...

   Often, central to the assessment of the features provided by
>    a novel mechanism, lies the consideration of how it improves over
>    already existing technologies, and by "how much."
>
>
[AF] Strike "Often,"

*Section 3:*

>    This section considers techniques and options available for several
>    key aspects of any evaluation method.
>
>
[AF] Given the draft title is "Information-centric Networking: Evaluation
Methodology", I assume this section is the meat of the document and yet
there is only a trivial introduction.  Throw the reader a bone, eh?  What
am I going to hear about?  Am I to select one of the approaches described?
Or are these considerations for any evaluation technique?  Is there a
relationship between the tools and facilities of the prior sections and
methodologies?

*Section 3.1.  Topology Selection*
[AF] There's been a lot of prior work on topologies for simulation &
performance evaluation.  Does ICN change any of that?  This section is a
little of a hodgepodge of information.  It's hard to take away much insight
on how topology selection applies to ICN evaluation.  It would benefit from
a re-write that is clearer on what questions the section is supposed to
answer (and then answers them).

*Section 3.2.  Traffic Load*

> Table III. Content catalog
>
>    Traffic | Catalog |  Mean Object Size  |  Popularity Distribution
>    Load    | Size    |  [L4 <https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#ref-L4>][L5][L7 <https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#ref-L7>][L8]  |  [L3 <https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#ref-L3>][L5][L6 <https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#ref-L6>][L11][L12 <https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#ref-L12>]
>            | [L1 <https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#ref-L1>][L2]|  [L9 <https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#ref-L9>][L10]         |
>            | [L3 <https://tools.ietf.org/html/draft-irtf-icnrg-evaluation-methodology-03#ref-L3>][L5]|                    |
>    ==================================================================
>    VoD     |  10^4   | Object: ~100 MB    | Zipf, 0.65 <= alpha <= 1
>    ==================================================================
>
>
[AF] Video traffic has largely migrated to HTTP Adaptive Streaming where
chunks are separate web objects and may be cached separately.  For example,
the first 10 minutes of a movie may be more popular than the rest and
cached in more places.  Treating VoD as a single 100MB object doesn't
reflect this.

*Section 3.3.  Choosing Relevant Metrics*

>    ICN is a networking concept that arose from the desire to align the
>    operation model of a network with the model of its typical use.  For
>    TCP/IP networks, this implies changing the mechanisms of data access
>    and transport from a host-to-host model to a user-to-information
>    model.  The premise is that the effort invested in changing models
>    will be offset, or even surpassed, by the potential of a "better"
>    network.  However, such a claim can be validated only if it is
>    quantified.
>
>
[AF] This is a great introductory paragraph.  Suggest moving it to Section
1.

   ICN approaches have used system metrics, but unfortunately
>    the situation is not as coherent as with the traffic metrics.
>
>
[AF] This statement isn't supported.  How is the use of system metrics not
coherent?  What does this statement mean?