Subject: Re: [bmwg] Last Call: <draft-ietf-bmwg-2544-as-04.txt> (RFC 2544 Applicability Statement: Use on Production Networks Considered Harmful) to Informational RFC

From: Al Morton <[email-masked]>
Date: Wed, 01 Aug 2012 17:32:50 -0400

BMWG,

We received comments from Stewart and Adrian during IETF Last Call.

After some thought and discussion, we proposed the text below
which expands on some of the difficulties with applying
RFC 2544 methods to production networks, and identifies
additional forms of harm that are highly likely should our
advisory be ignored.

We are awaiting a response.

Al, for the Co-authors

-=-=-=-=-=-=-=-

5.  Advisory on RFC 2544 Methods in Production Networks

    The tests in [RFC2544] were designed to measure the performance of
    network devices, not of networks, and certainly not production
    networks carrying user traffic on shared resources.  There will be
    unanticipated difficulties when applying these methods outside the
    lab environment.

<insert>
One such difficulty stems from reliance on frame loss as an indicator of
resource exhaust in RFC 2544 methods. In practice, link-layer and physical-
layer errors prevent production networks from operating loss-free.
The RFC 2544 methods will not correctly assess Throughput when loss
from uncontrolled sources is present. Frame loss occurring at the SLA levels
of some networks could affect every iteration of Throughput testing
(when each step includes sufficient packets to experience facility loss).
Flawed results waste the time and resources of the testing service user,
and of the service provider when called to dispute the measurement.
These are additional examples of harm that compliance with
this advisory should help avoid.

<end_insert>

    Operating test equipment on production networks according to the
    methods described in [RFC2544], where overload is a possible outcome,
    would no doubt be harmful to user traffic performance.  These tests
    MUST NOT be used on production networks and as discussed above, the
    tests will never produce a reliable or accurate benchmarking result
    on a production network.

    [RFC2544] methods have never been validated on a network path, even
    when that path is not part of a production network and carrying no
    other traffic.  It is unknown whether the tests can be used to
    measure valid and reliable performance of a multi-device, multi-
    network path.  It is possible that some of the tests may prove valid
    in some path scenarios, but that work has not been done or has not
    been shared with the IETF community.  Thus, such testing is contra-
    indicated by the BMWG.

-=-=-=-=-=-=-=-