Subject: [bmwg] draft-ietf-bmwg-ipg-dataplane-conv-term-02.txt

From: "Robert Holley" <[email-masked]>
Date: Sat, 29 May 2004 07:05:35 -0400

I've used the proposed last call comment template below.
I think the template may need some modifications in order
to clarify what are the template sections and what are the
comments provided by the reviewer. In the text below, I've added
the words "template" and "comments" to try and held distinguish between
the template text and my comments.

Last Call Review Template

I-D Title(s):
Terminology for Benchmarking IGP Data Plane Route Convergence

Filename(s):
draft-ietf-bmwg-igp-dataplane-conv-term-02.txt

Reviewer Name: Bob Holley

Date: May 26, 2004

Please organize your comments in the following categories below.

Review Summary:

Overall:

     * Does/Do the draft(s) provide clear identification of the
       scope of work? E.g., is the class of device, system, or
       service being characterized clearly articulated.

Comment;

The text provides a good description of the intent,
however, the figure provided is unclear and causes
more confusion. I just skipped the figure and
didn't even try to figure it out. Not sure, it
this was just a formatting issue.

Template:

     * If a terminology memo, are the measurement areas clearly
       defined or otherwise cited?  Is the working set of
       supporting terminology sufficient and correct?  To your
       knowledge, are the areas of the memo that may conflict
       with other bodies of work? Are there any measurements or
       terminology that are superfluous?  Are any missing?

Comment;

Section 3.1 Convergence Event
Implies that convergence events are network
initiated, should add clarification that it may
be due to configuration change.

"better next hop" should be clarified,
I believe you are talking about a protocol update
message that results in the re-calculation of the
route tables.

Section 3.2 Network Convergence
This seems like a fairly large approximation.
The section states that network convergence is the sum
Route Convergence <defined later> of all routers
in the network.

In several sections the convergence state is
defined by the "forwarding rate" equal to the
"offered load". May be better to clarify that
the offered load must be less than or equal to the
capacity of the device.

Section 3.3 Route Convergence
This term seems misleading. The definition
implies the convergence of the routing function
on a single device. But "route convergence" implies
a source-destination path (e.g. multiple devices).

Section 3.4 Full Convergence
The term "converge an entire route table" to
me implies completion of an algorithm.

Section 3.5 Convergence Packet Loss
The units in this and other sections are
measured in absolute packet counts. Would it
be feasible to consider percent of offered load?

Section 3.7 Convergence Recovery Instant
This section contains a hard constant.
This should be changed to a "provisional" or
suggested value.

Section 3.9 Convergence Event Transition
Is it possible that the forwarding rate may
never reach zero? In that case would not a
percentage of load or slope be more appropriate?

Also, the discussion of this term is not clear

It seems that this term might better be called
"Degradation Transition" to imply the downward
slope of the forwarding rate.

Section 3.10 Convergence Recovery Transition
For both this section and 3.9, it may be
more flexible to define this as a slope and a
percentage of forwarding rate. In that case,
the "threshold" crossed to determine state
could be manipulated. For example, instead of
defining this time as an absolute 100% recovery
of the forwarding state, it might be useful in some cases
to define recovery as 95% recovery of the forwarding state.

Section 3.11 Loss-Derived Convergence Time
the phrase "converging many routes simultaneously".
I assume you mean processing multiple update messages?

Section 3.13 Restoration Convergence Time
Why is this a special case. It seems that this
would be the same as the other convergence measurements
but in this case, we would be transitioning to a previously
obtained (assumed desirable steady state).

Template:

     * If a methodology memo, does the methodology AND its
       corresponding terminology adequately define a benchmarking
       solution for its application area? Do the methodologies present
       sufficient detail for the experimental control of the benchmarks

Comment;

       NA this is a terminology memo

Template:

     * If neither a terminology or methodology, does the offered
       memo offer complementary information important to the use
       or application of the related benchmarking solution?

Comment;

       NA this is a terminology memo

Template:

     * Do you feel there are undocumented limitations or caveats to
       the benchmarking solution being proposed?  If so, please
       describe.

Comment;

       NA this is a terminology memo

Template:

     * Does the memo attempt to define acceptance criteria for
       any of the benchmark areas?

Comment;

      There are some hard values in the paper. those were
      commented on above.

Template:

 Technical Content:  (Accuracy, Completeness of coverage)

     Are definitions accurate? Is the terminology offered relevant?

Comment;

     See above.

Template:

     To your knowledge, are there technical areas that are erroneous?
     Are there questionable technical areas that need to be re-examined
     or otherwise scrutinized.

Comment;

     There are some terms that can be clarified. see above.

Template:

     Does the solution adequately address IPv6?

Comment;

     IPv6 was not mentioned specifically. Since this is a
     terminology memo, it doesn't seem to apply

Template:

     Do you feel the memo(s) being offered are technically mature enough
     for advancement to informational RFC?

Comment;

     The paper is well along in it's development. Additional consideration
     in the measurement units should be investigated. Additional flexibility
     such as using rates, slopes, and thresholds would allow for
modification
     to the requirements for a device to "pass" the benchmark. Also, there
     should be "averages" defined so that a test can be conducted
     several times and the average or mean results can be recorded.

Template:
 Clarity and Utility:

    If you had a need, would you utilize the benchmarking solutions
    advocated by this and its related memos?  If not, why?

Comment;

    NA this is a terminology memo

Template:

    Conformance to BMWG principles: (see BMWG charter)
    http://www.ietf.cnri.reston.va.us/html.charters/bmwg-charter.html

Comment;

    Appears compliant.

Template:

    Do you have confidence that the benchmarks, as explicitly
    defined, will yield consistent results if repeated on the
    same device (DUT/SUT), multiple times for a given test condition.
    If not, cite benchmark(s) and issue(s).

Comment;

    Not a high degree of confidence. The device configuration,
    including Operating System scheduling and timer modifications,
    features activated etc. would change the test results.

Template:

    Do you have confidence that the benchmarks, if executed for a
    given test condition, utilizing the documented methodology
    on multiple test infrastructure (e.g., test equipment), would
    yield correct and consistent results on the same DUT/SUT?
    (Said differently, are the benchmark's methodology written
    with enough exacting detail, that benchmark implementation
    differences do not yield a difference in the measured quantities?)
    If not, cite benchmark(s) and issue(s).

Comment;

    The benchmarks talk about offered load but do not clarify
    if the offered load is max load for the device or a
    smaller load. Seems like the relationship between offered
    load and max load would impact the results.

Template:

    Do you feel that the benchmarks form a basis of comparison between
    implementations of quantity being characterized? (I.e., are the
    benchmarks suitable for comparing solutions from different vendors.)

Comment;

    One thing the memo does not clarify is the type of traffic being offered
    It may be the case that the type of traffic may impact the
    convergence time due to the fact that additional processing
    resources are required on the device.

    Essentially, you are measuring software algorithms, which run
    in processes, which compete with scheduling functions.

Template:

    If not, cite benchmarks and issues.

Comment;

Template:

    For those benchmarks cited above, do you feel that the benchmarks,
    as specified, have universal applicability for the given
    behavior being characterized?  (i.e., benchmarks might not form
    a basis for cross-vendor comparison, can be used universally
    in a different role.)

Comment;

    The benchmarks do not seem to completely isolate the
    function. In other words, traffic forwarding may be impacted
    by some other characteristic in addition to route convergence

    The benchmarks would not distinguish those cases clearly.

    However, at this point in time, I cannot think of any
    alternatives which would correct that. It may be sufficient
    to note that in the paper.

 Editorial Comments:
 (includes any deficiencies noted w.r.t. I-D Nits, spelling, & grammar)

Comment;
  There is inconsistent capitalization of certain terms.
  For example, "route convergence" and "Route Convergence"
  on page 1.

============================================
Robert Holley
Network Consulting Engineer
Global Solutions Engineering

Cisco Systems
7025 Kit Creek Road.
Research Triangle Park, NC 27709
Phone:  +1 919-392-2359
Pager:   +1 800-796-7363 pin 100-9187
============================================

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: Al Morton <[email-masked]>
Date: Sat, 29 May 2004 09:42:30 -0400

>the template text and my comments.

Thanks, Bob.
I see your point, especially if the reviewer
indents their responses to the prompts.

Perhaps we could suggest that reviewers "reply"
to the template in their mail tool, and ensure the
original message/template is quoted or marked with ">".

Another point for Editors -- if you respond to a
specific reviewer's comment, please excerpt
the relevant text and don't reply the entire template
at once (or we will all spend lots of time searching
for replies).  Thanks!

Status of the Experiment: So far, so good.
I have passed Tanju Cataltepe's reviews along to the
editors, and we have heard from 2 other volunteers.

Al