Subject: Re: WG Last Call: New Work Proposal on Benchmarking DiffServ Mechanisms

From: Kevin Dubray <[email-masked]>
Date: Mon, 04 Dec 2000 11:10:22 -0500

When I look at the minutes of the last BMWG meeting, 
    http://www.ietf.org/proceedings/00jul/index.html

there was interest in an individually submitted (non-WG) I-D:
    http://www.ietf.org/internet-drafts/draft-khurana-bmwg-diffservm-00.txt

Have the proposers of the new work item regarded this work?  Are they
different? The same? Are there areas of collaboration? 

Kevin Dubray wrote:
> 
> BMWG'ers:
> 
> A WG Last Call period for the new work proposal presented to the BMWG:
> 
>      Benchmarking Differentiated Services Mechanisms
> 
> will be open from 29 Nov until 6 Dec.
> 
> If you feel that there is reason that this new work should not be
> undertaken by the BMWG, please respond to this list or
> [email-masked].
> 
> =============================================================================
> The Benchmarking Differentiated Services Mechanisms Proposal intends to
> address new problems created by differentiated service mechanisms.  These
> problems are not evident when testing with RFC2544 or RFC2889.  The main
> problems we propose to tackle are:
> 
> 1. Extending current metrics, which assume the absence of congestion,
> to address the issues of DUT behavior during congestion. Metrics like
> Throughput (RFC1242), measure the point just before DS (Differentiated
> Services) has to make a decision on who to forward and who to drop.
> Throughput is very useful to find out how much a DUT can handle until
> a DS kicks in.  The new metrics we propose have the following qualities:
> 
>      - Defined as a per hop behavior (PHB).
>      - Defined explicitly for congested traffic where the DUT MUST drop
>        frames.
>      - Several PHBs can be combined to approximate a path segment.  There is
>        no intent to represent end-to-end measurements.  Rather, a goal of
>        this effort is to evaluate what effect each DUT's metrics have on,
>        say, a backbone network as a whole.
> 
> 2. Defining specific methods to activate the DS in order to get an
> "apples-to-apples" comparison.  Different forms of DUT congestion will
> produce different results.  Main factors are traffic distribution, amount
> of over-subscription, and number of code points used.  On the issue of code
> points, a DUT might have 2, 4 or 8 queues to handle all 64 code points.  If
> a DUT only has 4 queues, then how does it handle 64 different services?
> 
> 3. Creating one set of metrics for both IP precedence and DSCPs.  IP
> precedence is currently deployed.  DSCP is just starting to be deployed.
> By defining one methodology for both technologies, we help the latest
> networks while supporting existing ones.
> 
> 4. Defining test scenarios for multiple DSCPs.  Most tests today just test
> 2 priorities: protected and unprotected.  But DSCP can theoretically handle
> priorities.  A new model is needed to replace the current binary model.
> 
> 5. Creating a foundation for new technologies that do a DS-type forwarding
> decision not based in the IP header.  These technologies can build upon the
> DSCP test methodology.  One such technology is MPLS, where the DSCP functionally
> is embedded in a label just before the IP header.
> 
> The first terminology Internet-Draft on this work is proposed for January 2001.
> The first methodology Internet-Draft on this work is proposed for July 2001.

---

From: "David Newman" <[email-masked]>
Date: Mon, 4 Dec 2000 09:49:33 -0800

> different? The same? Are there areas of collaboration?

Kevin and fellow bmwg'ers,

We did indeed review the Khurana I-D. It raised for us some of the same
concerns that were raised by others about our previous effort -- too broad
in scope and too liberal in its use of the term "quality of service" (a term
we've deliberately avoided this time). Further, we felt the Khurana draft
(and, admittedly, our previous effort) took the too-simple approach of
applying existing metrics to a new field of measurement. We also had many
other concerns with the Khurana draft. We can enumerate this if the list is
interested. However, we felt that it would be more constructive to submit a
revised, and more focused, proposal of our own.

Jerry Perser
David Newman

---

From: Sumit Khurana <[email-masked]>
Date: Mon, 4 Dec 2000 18:34:14 -0500 (EST)

Jerry, David,

This is the first time that we have heard any comments on the ID
since it was discussed at the last BMWG meeting, where there was
initial support for the draft. May I suggest that it
would be useful for the working group for you to enumerate the
specific concerns that you have so that similar mistakes are avoided
if and when the work is taken up by the working group. After all, the
draft is the first tangible work to address the problem as opposed to the
proposals.

As far as the broad concern that you have raised regarding application of
existing metrics for the measurements, that is not necessarily a bad thing
if it captures the properties that one is trying to measure. 
Indeed we have "adapted" existing metrics for the measurements. It is
also true, that there could  be other properties for which  one may
require completely new  metrics and they could be incorporated into
the methodology. Suggestions for such properties and metrics would 
be welcomed.    

We would like to suggest that the draft be taken up as the basis for the
proposed work item. We look forward to comments on the contents of the
draft from the mailing list.

Thanks,
Sumit

---

From: "David Newman" <[email-masked]>
Date: Mon, 4 Dec 2000 16:21:48 -0800

> specific concerns that you have

Please see below, where we have enumerated a rather large number of concerns
with your draft.

so that similar mistakes are avoided
> draft is the first tangible work to address the problem as opposed to the

Well, no, it's not the first:

1. The first tangible work in this area was posted on the reflector by Scott
Poretsky in 1999.

2. We submitted a workplan early this past spring and had a draft ready to
go before Adelaide. However, bmwg members raised strong objections to the
term "quality of service" and to the notion of attempting to define QOS
metrics within the context of the bmwg. We were surprised, therefore, to see
another attempt using the same terminology that met with such strong
objection earlier.

3. We submitted a revised workplan a few weeks ago; it is now in last call.

> Indeed we have "adapted" existing metrics for the measurements.

As noted below, there are areas where existing metrics are a
less-than-perfect fit.

It is
> proposed work item.

Then please submit a workplan, as per bmwg custom.

We look forward to comments on the contents of the
> draft from the mailing list.

Here are our concerns with your current draft. Because of the volume and
severity of these concerns -- and being mindful of the numerous suggestions
made when we began our work in this area some time ago -- we felt it more
constructive to develop a new, and better focused proposal.

1. The Khurana I-D doesn't follow longtime bmwg practice of submitting a
workplan first. What are the authors' goals for this document? Is this one
document in a series? What's the timetable?

2. The Khurana I-D focuses exclusively on measuring effects of diff-serv
codepoints; it does not describe measurement of the other major
network-layer prioritization mechanism, IP precedence.

DSCPs undoubtedly will be an important classification mechanism in the
future, but today we are at a relatively early stage of diff-serv
development and implementation. IP precedence is currently more widely used
than diffserv, and as yet there is no standard means of measuring the
performance of devices that implement it

3. The Khurana I-D mixes terminology (1 definition) and methodology, while
historically bmwg documents have used separate I-Ds for each. As described
later, a grand total of one definition is not adequate to cover the breadth
of tests described here.

4. The Khurana I-D uses but does not define the term "quality of service."
Two problems here: First, absent a definition it is not clear what this term
describes. Second, numerous bmwg participants strenuously objected to the
presence of the Q-term in a previously submitted workplan, citing concerns
of a mismatch between end-to-end QOS metrics and the per-hop nature of most
metrics defined in bmwg documents.

5. The Khurana I-D does not clearly distinguish between methodologies for
measuring per-hop behaviors and those that result from multiple PHBs. A test
that describes the aggregate result of multiple PHBs is of little use unless
the behavior of each component PHB is well understood.

6. The Khurana I-D does not specify whether the DUT or DUTs are at the edge
of a network (network boundaries, per RFC 2474), or at the core, or whether
this distinction is relevant.

7. Section 4 states as goal "to characterize how well the devices implement
the forwarding engine." This is an improper assertion. BMWG documents do not
attempt to describe or judge implementation quality; rather, they focus on
externally observable performance metrics.

8. Section 4 also notes several specific goals (a through d) but does not
include the ability to correctly classify and mark packets in the first
place.

9. Section 5.1 is vague on distinctions between "features" and "mechanisms."
Better to state that the goal is to enable equivalent PHBs on each device
and leave it at that.

10. The terminology in section 5.1's examples requires clarification. What
is "absolute relative"?

11. Section 5.3, paragraph 1, describes a test that concurrently measures
bandwidth and latency. A single PHB is likely to describe one metric or the
other but not both at the same time. It is important to give examples that
isolate one variable (in this case, one PHB) at a time for measurement.

12. Section 5.4.1 defines the term "delay jitter." However, the definition
and the discussion both make clear that in fact the term describes
*instantaneous* jitter between just two packets. Since jitter (and latency,
for that matter) may vary significantly over the test duration, it is also
necessary to define terminology and methodology for measurements of mean or
median latency and jitter.

Presumably this is omitted because of the authors' wish to follow the
framework set out in RFC 2544. At the time that document was written the
means did not exist to measure jitter of all packets throughout a trial.
Such technology does now exist and is very useful in measuring both average
latency and jitter.

13. Section 6.1 describes a methodology for testing throughput without
naming the number of transmit/receive interfaces involved, or the traffic
orientation (one-to-one, one-to-many, partial mesh, full mesh, etc.).

14. In addition to the throughput test described in Section 6.1, the I-D
lacks a test of bandwidth allocation. In some situations, a strict upper
bound on bandwidth is more desirable than simply using all available
bandwidth. The I-D should include a test to measure a DUT's ability to
enforce specific bandwidth allocation targets.

This is also relevant to Section 7.3 in evaluating bandwidth consumption by
multiple traffic classes.

15. Section 6.1, procedure, third paragraph, states that the test should be
repeated with different frame sizes but doesn't specify whether these are
different trials with a single frame size (a la RFC 2544) or one trial with
a mix of frame sizes. If the latter, the frame size distribution MUST be
specified.

16. Section 6.2 on a latency test methodology has the same problem as
section 5.4.1 - it describes only instantaneous latency rather than mean or
median latency over the life of the test duration.

Presumably this is because of the authors' wish to follow the framework set
out in RFC 2544. At the time that document was written the means did not
exist to measure latency of all packets throughout a trial. Such technology
does now exist and is very useful in measuring both average latency and
jitter.

17. Section 6.3 describes a frame loss test that calculates loss using a
ratio of different classes of offered loads. It would be much more
meaningful to calculate these ratios based on received rather than offered
loads.

18. Section 6.4 on back-to-back testing describes a burst test with bursts
lasting at least 2 seconds, again following RFC 2544. That suggestion was
made mainly in the context of testing 10-Mbit/s Ethernet. At gigabit
Ethernet rates, nearly 3 million 64-byte frames can be transmitted in 2
seconds' time. By any reasonable definition, 3 million frames is not a
burst. The requirement of a minimum duration should be reduced, made
proportional to the media rate, or eliminated.

19. As noted in the objection to section 5.4.1, the methodology for jitter
only describes instantaneous jitter. A separate test should be defined for
average jitter.

20. Section 7.1 describes tests for one code point and section 7.2 describes
concurrent tests for two code points. A more meaningful test would evaluate
from 1 to N code points, not just 1 to 2.

21. The objectives described for Section 7.3 are really for two separate
tests: one to determine whether capacity allocated to one traffic class is
available for other classes if it is not in use, and the other to determine
whether lower-priority classes are starved. As currently stated the
benchmark attempts to determine both in a single test. In fact the two
metrics are separate and not necessarily related, and thus separate tests
should be run for each.

Respectfully,

Jerry Perser
David Newman

> -----Original Message-----
> From: Sumit Khurana [mailto:[email-masked]]
> Sent: Monday, December 04, 2000 3:34 PM
> To: David Newman
> Cc: [email-masked]; BMWG (E-mail); Scott Poretsky;
> Perser, Jerry; Tony Bogovic; Shobha Erramilli
> Subject: RE: WG Last Call: New Work Proposal on Benchmarking DiffServ
> Mechanisms
>
>
> Jerry, David,
>

---

From: Shobha Erramilli <[email-masked]>
Date: Wed, 06 Dec 2000 10:20:42 -0500

David,

Responses to your specific concerns with our draft are given below.
In general though, we submitted the draft to the BMWG because we believe
that it addresses a new and relevant area, and still believe that it is
a good baseline document for benchmarking boxes that support
Differentiated Services based on our own experience in testing
Diffserv boxes.

> constructive to develop a new, and better focused proposal.

And we do support better focus. We differ on the severity of the concerns
that you have raised, however. In either case it is good to bring up the
issues to the working group as admittedly you too had problems with your
first proposal, which we were unaware of at the time of submitting
the draft. It also gives us an opportunity to provide clarifications.

> document in a series? What's the timetable?

This custom is not documented anywhere. It is not in the charter of the
bmwg and based on our experience that is not how other working groups
operate. It is usually acceptable for an individual submission to be taken
up as a working group item.
We see this document as a generic methodology for benchmarking
devices supporting DS, and propose its use as a baseline for a WG document.
We don't see a series of such documents, though it could grow into more
depending
on WG feedback, e.g. separate terminology and methodology documents.

> network-layer prioritization mechanism, IP precedence.

IP Precedence, at least the current practice of using it,
has been subsumed into RFC2474 - Class Selector Code Points. So our
draft does not ignore Precedence, and in fact views the Class Selector PHB
as the first application of this methodology.

> later, a grand total of one definition is not adequate to cover the breadth

A separate terminology document can be derived from this doc.

> metrics defined in bmwg documents.

I don't believe our draft is trying to measure end-to-end QoS. We use the term
to
describe mechanisms that need to be enabled on the DUT. None of the tests
has a dependency on an end-to-end definition of a service. I believe all
references
to the word "service" can be replaced with PHB with no impact on the
methodology.

> the behavior of each component PHB is well understood.

We do have single code-point tests, followed by combination tests.

> this distinction is relevant.

We are only benchmarking the PHBs and not the traffic conditioners,
so this distinction is not relevant. Clarification can be added to state
this.

> externally observable performance metrics.

This is a rewording fix. The methodology itself has nothing in it that needs to

know implementation details. It only uses externally observable behavior.

> include the ability to correctly classify and mark packets in the first

This is covered in 5.1.

> Better to state that the goal is to enable equivalent PHBs on each device

This is a word fix, IMO. The word "mechanisms" can be eliminated.

> 10. The terminology in section 5.1's examples requires clarification. What

Agreed that we could probably use a better term. All it is trying to convey is
an expression
for X < 1.5Y.

> isolate one variable (in this case, one PHB) at a time for measurement.

A PHB can have more than one metric associated with it, but a given test
should isolate one variable at a time. Sec 5.3 suggests how the overall results

should be interpreted, not a particular test.

> Such technology does now exist and is very useful in measuring both average

True. The test can be modified to measure jitter for the duration of the test
and
report the average or some other summary measure of jitter.

> orientation (one-to-one, one-to-many, partial mesh, full mesh, etc.).

True, our document is not described at that level of detail. We are prepared
to work with the BMWG in evolving this document and adding in more detail
or more sections to the document.

> enforce specific bandwidth allocation targets.

That is measured and reported in the described test.

> a mix of frame sizes. If the latter, the frame size distribution MUST be

More clarification needs to be added, again. We are not proposing trials with
mixed
frame sizes.

> does now exist and is very useful in measuring both average latency and

Same as the jitter test.

> meaningful to calculate these ratios based on received rather than offered

Could you clarify this? Frame loss makes more sense as a ratio of offered
load, IMO.

> proportional to the media rate, or eliminated.

This needs clarification. Here we could have input rate exceeding output at
configured rate.

> only describes instantaneous jitter. A separate test should be defined for

Agreed.

> from 1 to N code points, not just 1 to 2.

Section 7 provides a minimal set of recommended tests and states why they
are the minimal sets. How would one test for N codepoints if the DUT
doesn't support N queues? Ofcourse one may carry out the tests with N
codepoints  using the methodology stated in section 6, beyond what is
in the minimal set of section 7.

> metrics are separate and not necessarily related, and thus separate tests

 Yes, this group of tests does cover both effects - 1) what bandwidth share
class x
gets  in the presence of traffic from other classes (issue of starvation) and
2) what happens to unused capacity of the other classes. It is possible to add
more detail into the test description to draw out that distinction.

> > > different? The same? Are there areas of collaboration?

We think they cover the same area of interest, and there are areas of
collaboration.

Shobha Erramilli