Subject: [bmwg] WG Last Call: draft-ietf-bmwg-dsmterm-05

From: Kevin Dubray <[email-masked]>
Date: Tue, 04 Mar 2003 09:42:16 -0500

Metric Mavens:

A WG Last Call period for the Internet-Draft regarding
"Terminology for Benchmarking Network-layer Traffic Control
Mechanisms":
              <draft-ietf-bmwg-dsmterm-05.txt>

will be open from 4 March 2003 until 25 March 2003.

Please weigh in on whether or not you feel the Internet-Draft
should be given to the Area Directors for consideration in
progressing the draft to an Informational RFC.  Send your comments
to this list or [email-masked].

A URL for the Internet-Drafts is:
  http://www.ietf.org/internet-drafts/draft-ietf-bmwg-dsmterm-05.txt

-Kevin

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: "Perser, Jerry" <[email-masked]>
Date: Fri, 14 Mar 2003 10:48:07 -0800

Al,

Rebuttal inline:

> 3.2.5 Jitter
> It would help to add that if packet n is lost,
> then jitter is undefined for both packet pairs
> that use packet n.

In the discussion, we state "The measurement does not include lost packets."
We can include your comment to be clearer.  But you raised an interesting
question, what does the definition mean by "two consecutive packets"?

Suppose we transmit packets with a sequence [n-1], [n], [n+1].  The delay
for each of the packets will be D[n-1], D[n], and D[n+1].  Packet n is lost.
Jitter |D[n] - D[n-1]| and |D[n+1] - D[n]| are undefined.  But could you
report the jitter from "two consecutive packets" received?  Is |D[n+1] -
D[n-1]| a valid jitter measurement?

My initial response is yes.  Maybe you could find some time at the IETF
meeting to discuss this with Scott and I.
 
> 3.3.1 In-sequence packet and
> 3.3.2 Out-of-Order Packet
> The definition of the expected packet should be explicit,
> something like:
> "The expected sequence number is the last in-sequence number + 1."

The reason expected packet is NOT explicit is based upon
draft-ietf-ippm-reordering-02.txt.  In section 3.2, "the Next Expected
Sequence number at the Dst, in units of messages, time, or bytes."  By
stating "The expected sequence number is the last in-sequence number + 1.",
we exclude time and bytes units.

Someone implementing this may chose to do it in hardware at speeds of
OC-768.  An incrementing sequence number would require checksum to be
calculated on every packet of the stream.  This particular implementation
would use an sequence algorithm that is intellectual property to avoid
changing the checksum.  In order to comply with RFC2026 section 10, we have
excluded explicit implementation that may be considered intellectual
property.
 
> There should be agreement between these definitions and the
> reordering metrics being drafted in ippm (so far, so good).
> 
> Al
>

---

From: "Perser, Jerry" <[email-masked]>
Date: Fri, 14 Mar 2003 10:48:07 -0800

Al,

---

From: Al Morton <[email-masked]>
Date: Fri, 14 Mar 2003 15:44:36 -0500

>we exclude time and bytes units.

Thanks for your response, Jerry.
So then, I think what needs to be more explicit is
the reference to the ippm draft (which hopefully progresses
and doesn't hold up dsmterm in a reference-wait state).

Al

---

From: Al Morton <[email-masked]>
Date: Fri, 14 Mar 2003 15:44:36 -0500

Al

---

From: Debby Stopp <[email-masked]>
Date: Wed, 19 Mar 2003 09:41:08 -0800

I have some comments regarding this definition:

     3.2.4 Delay

        Definition:
          The time interval starting when the last bit of the input IP
          packet reaches the input port of the DUT/SUT and ending when
          the last bit of the output IP packet is seen on the output
          port of the DUT/SUT.

*	 According to this def, we need to be able to determine the time at
which the last bit of the packet reaches the DUT/SUT - isn't this white box?
How are we supposed to find out this time w/out asking the DUT/SUT?  Same
discussion for the other goalpost; how are we supposed to find out the time
that  the last bit left the DUT/SUT? 

*	I question the appropriateness of defining 'delay' specifically in
this document.  The word 'delay', is a well-known & highly utilized term in
the internet community.  I suggest that you append an modifier on this def,
such as "IP Delay" in order to differentiate it from the well-known
terminology usage. 

*	In the same section:

Delay is delay, regardless of the technology being measured. 

 This is an ambiguous statement; either remove it or crisp up exactly what
you were trying to say.

& lastly -  If delay happens and no one measures it, is it still delay?
:) 

Debby Stopp

---

From: Tony DeLaRosa <[email-masked]>
Date: Tue, 25 Mar 2003 11:51:00 -0800

Kevin and Al,
 
I don't believe that the "draft-ietf-bmwg-dsmterm-05.txt" is ready to be
submitted to the area directors for consideration.  There are still a number
of issues that need to be clarified or resolved.  I heard at the IETF
meeting that a number of changes were being made to the draft.  I do
anticipate that revised document will be resubmitted to the reflector for
review before it goes to last call.  This draft has stirred quite a debate
and it is important that the input from reflector be considered.
Unfortunately, some of the concerns voiced on the reflector were either
discounted or ignored.  This insensitivity does not encourage dialog and
collaboration but rather creates hostility and antipathy.  Below are some of
my comments and suggestions:
 
<<<Section 3.1.3 Forwarding Congestion>>>
 
<<ISSUE>>
This definition argues that overloading is too limited for this application.
Unfortunately, the cited references used to make this assertion have been
taken out of context.  
 
<<RECOMMENDATION>>
Delete the section and use the existing definition for overloading.
Otherwise, rewrite the section and cited references appropriately.
 
<<RATIONAL>>
[MA98] is cited, stating its overload definition assumes congestion is
introduced strictly by the tester on ingress of a DUT/SUT.  However, I did
not see any restriction in [MA98] definition.  The reference to introducing
congestion on ingress interfaces in the third paragraph of the discussion
section is just an example.  Clearly, the first paragraph clearly
articulates how one or more egress interfaces can be overloaded:
"Overloading can serve to exercise buffers and buffer allocation algorithms
as well as congestion control mechanisms.  The number of input interfaces
required to overload one or more output interfaces of a DUT/SUT will vary
according to the media rates of the interfaces involved."
 
[BR91] is stated to deal with only ingress interfaces.  No where does it
state this in the definition of overloading behavior.  In contrast, the
measurement units call out the following:  "Measurement units:  Description
of behavior of device in any overloaded states for both input and output
overload conditions."
 
[Ra99] reference is cited to reinforce the assertion that is impractical to
build a black-box test to externally observe Incipient Congestion in a
router.   Yet, I did not find any statement that would support this
assertion.  What [Ra99] states is "Treating the network as a "black-box" and
treating loss as an indication of congestion in the network is appropriate
for pure best-effort data carried by TCP which has little or no sensitivity
to delay or loss of individual packets."  I don't believe the author meant
the inverse was true.
 
<<ISSUE>>
Draft asserts that any packet loss above throughput is due to forwarding
congestion (AKA overloading).  As Barbara Denny, John Dawson and Mark
Robinson pointed out, it is presumptuous to assume that the only reason
packets are being dropped above throughput is because of forwarding
congestion.  One can not assume that a DUT forwarding switching fabric is
100 percent bug free.  Packet loss can be caused by any errors from the
physical layer to state machine implementation.
 
<<Recommendation>>
Insert a caution or disclaimer stating that packet loss will be used as an
indication of forwarding congestion although it may not be the sole reason.
Additional clarification may be required on why packet loss alone is enough
for indicating forwarding congestion.  The reference to [RA99] pertains to
only packet loss on best-effort data carried by TCP.
 
<<Question>>
Why not use both the proposed term Delay and Packet Loss as an indicator of
forwarding congestion (AKA overloading)?  [RA99] does not state that
incipient congestion is impractical.  Plus dsmterm states that the proposed
term "Delay" will be a black-box measurement so I don't any reason why both
metrics can be used as an indication of overloading. 
 
Section 3.2.1 Channel Capacity
<<ISSUE>>
The definition of "Channel Capacity" is the same as for throughput.  
 
<<RECOMMENDATION>>
I can only presume that this is for measuring throughput for a set of flows
or class of service.  I would recommend changing the name to "Flow Capacity"
or "CoS Capacity"
 
Section 3.2.4 Delay
<<ISSUE>>
Delay is a common term used pervasively throughout many RFCs to indicate a
time measurement between two specified endpoints.  I agree with Debby Stopp
that the draft should a add modifier in front of Delay to make it specific
for the particular measurement.  I can imagine the confusion in methodology
document when trying to distinguish the difference between delay and Delay.
So does the BMWG wish to restrict Delay to this definition?    
 
<<RECOMMENDATION>>
Please add a modifier in front of delay.  
 
<<ISSUE>>
Bullet number 1, unclear on the following statement "Delay is delay,
regardless of the technology being measured"
 
<<RECOMMENDATION>>
Replace with "Delay measures the elapsed time between two arbitrary
goalposts, regardless of the technology being measured."
 
<<ISSUE>>
Bullet number 3, I am unclear on how Delay measures the IP datagram only.  
 
<<RECOMMENDATION>>
Please clarify on this assertion
 
<<ISSUE>>
Bullet number 4.  Latency is measured only at the throughput level.  Yes, I
would agree that [BR99] measures latency at throughput but neither [BR91]
nor [BR99] precludes measuring latency at any other offered loads.  
 
<<RECOMMENDATION>>
Delete bullet number 4
 
<<ISSUE>>
Bullet number 5.  Should not be considered a bullet since this paragraph is
more of a note.  
 
<<RECOMMENDATION>>
Change bullet number 5 to a paragraph
 
Section 3.2.5 Jitter
<<ISSUE>>
See discussion on using common terms as a definition in Section 3.2.4 Delay.
 
<<RECOMMENDATION>>
Please add a modifier in front of Jitter.  
 
I look forward to working with the authors on addressing these issues and
concerns.  
 
Regards,
 
Tony

-----Original Message-----
From: Kevin Dubray [mailto:[email-masked]] 
Sent: Tuesday, March 04, 2003 6:42 AM
To: [email-masked]
Subject: [bmwg] WG Last Call: draft-ietf-bmwg-dsmterm-05

Metric Mavens:

A WG Last Call period for the Internet-Draft regarding "Terminology for
Benchmarking Network-layer Traffic Control
Mechanisms":
              <draft-ietf-bmwg-dsmterm-05.txt>

Please weigh in on whether or not you feel the Internet-Draft should be
given to the Area Directors for consideration in progressing the draft to an
Informational RFC.  Send your comments to this list or [email-masked].

-Kevin

---

From: Tony DeLaRosa <[email-masked]>
Date: Tue, 25 Mar 2003 11:51:00 -0800

Kevin and Al,

 
<<ISSUE>>
 
<<RECOMMENDATION>>
 
<<RATIONAL>>

 
<<ISSUE>>
 
<<Recommendation>>
 
<<Question>>

<<RECOMMENDATION>>
 
Section 3.2.4 Delay
<<ISSUE>>
 
<<RECOMMENDATION>>
 
<<ISSUE>>
 
<<RECOMMENDATION>>
 
<<ISSUE>>
 
<<RECOMMENDATION>>
 
<<ISSUE>>
 
<<RECOMMENDATION>>
Delete bullet number 4
 
<<ISSUE>>
 
<<RECOMMENDATION>>
 
Section 3.2.5 Jitter
<<ISSUE>>
 
<<RECOMMENDATION>>

Regards,
 
Tony

-----Original Message-----
From: Kevin Dubray [mailto:[email-masked]] 
Sent: Tuesday, March 04, 2003 6:42 AM
To: [email-masked]
Subject: [bmwg] WG Last Call: draft-ietf-bmwg-dsmterm-05

Metric Mavens:

-Kevin

---

From: Dana Heins <[email-masked]>
Date: Wed, 19 Mar 2003 10:14:57 -0800

While we're cleaning up the Delay defintion, perhaps there is a more clear
phrasing of the phrase: 'bit of the IP packet'.  Does 'bit' refer to binary
digit or is it slang meaning  a 'piece of the packet'?  Perhaps it would be
more clear with phrases like:
 
Start of the IP Header (what if it's tunneled?)
Start of packet (but only if qualified IP packet)

-----Original Message-----
From: Debby Stopp [mailto:[email-masked]]
Sent: Wednesday, March 19, 2003 9:41 AM
To: Bmwg (E-mail)
Subject: RE: [bmwg] WG Last Call: draft-ietf-bmwg-dsmterm-05

     3.2.4 Delay

        Definition:

*	In the same section:

Debby Stopp

---

From: "Perser, Jerry" <[email-masked]>
Date: Wed, 26 Mar 2003 11:55:31 -0800

Tony,

You were at the IETF meeting last week.  I got up and presented the changes
and action items to dsmterm-05.  I asked if there were and other comments or
issues.  Do you remember the response?  None.  The general consensus was all
concerns were being addressed.

Could you list all the "concerns voiced on the reflector" that "were either
discounted or ignored"?  To my knowledge, each and every one has been
addressed.  Please state the year, month, and author of the concern that you
feel were not addressed and I will review them with my co-authors.

For you new comments, I address them inline.  I will cut a lot of the text

> Unfortunately, the cited references used to make this 

Please read RFC2285 section 3.5.4 Overloading.  It limits the scope to
"maximum rate of transmission allowed by the medium".  What happens when the
traffic control mechanisms polices at a level below the medium?  I can have
Forwarding Congestion with OR without Overloading.
  
> physical layer to state machine implementation.

Packet loss due to physical layer has been outside the scope of the BMWG.
RFC1944, RFC2544, RFC2889, and mcastm-11 assume no physical layer errors.

Packet loss due to switching fabric bugs can not be external observed.  To
ask the DUT if it meant to forward a packet is considered white box testing.
dsmterm proposes no metrics based on white box testing.
  
> only packet loss on best-effort data carried by TCP.

I will propose the disclaimer in mcastm-11 to the other co-authors.  I am
having a hard time finding it in section 4.3.  Can you point it out to me?

> The definition of "Channel Capacity" is the same as for throughput.  

This is a John Dawson concept back in 2000.  Throughput measure the rate on
the ingress.  The DUT/SUT internal buffering will artificially raise the
throughput level.  John proposed measuring the rate at the egress
(Forwarding rate).  He also proposed the term "Channel Capacity".  Ingress
is not the same as egress.
  
> document when trying to distinguish the difference between 

Can you state the RFCs?

> So does the BMWG wish to restrict Delay to this definition?    

Yes.  Look at the charter.  It states "... present the requirements for the
common, unambiguous reporting of benchmarking results."

> regardless of the technology being measured"

I understood it when a co-author wrote it.  I see Debby's point that it may
not be clear.  I will ping the co-author to see if he can come up with
better wording.
  
> goalposts, regardless of the technology being measured."

Disagree.  Are you proposing that each implementation pick "two arbitrary
goalposts"?  What if two implementations pick different goalposts?

> Bullet number 3, I am unclear on how Delay measures the IP 

Does this sound better:

3. Delay is referenced to the end of IP packet only, unlike [Br91], which
also includes link layer overhead.

> nor [BR99] precludes measuring latency at any other offered loads.  

When you say "either [BR91] nor [BR99] precludes measuring latency at any
other offered loads", you meant that it is outside their scope.

Read [BR99] section 26.2, the procedure.  It is very specific about "at the
determined throughput rate".  Other rate can be measured outside of [BR99].
They are not precluded, they just to comply with RFC2544.

> Bullet number 5.  Should not be considered a bullet since 

I will ask the other co-author for a consensus on this.  It is not a
technical issue.  If the majority wants to change it to a paragraph, you
will see it on dsmterm-06.

> See discussion on using common terms as a definition in 

Only after Delay discussion is resolved.  No consensus yet.

Jerry.

___________________________________________________________ 
Jerry Perser                         Spirent Communications
Director of Test Methodology         SmartBits Division 
Phone: 818.676.2320                  26750 Agoura Road 
Lab:   818.676.2337                  Calabasas, CA, 91302 
Cell:  818.292.6457                  Corp:  818.676.2300

---

From: "Perser, Jerry" <[email-masked]>
Date: Wed, 26 Mar 2003 11:55:31 -0800

Tony,

  

  

  

Can you state the RFCs?

  

Does this sound better:

Jerry.

---

From: Tony DeLaRosa <[email-masked]>
Date: Thu, 27 Mar 2003 16:02:37 -0800

Jerry,

The last time I checked, the deadline for dsmterm's last call was March 25,
and not the date of the IETF meeting.  Also, "no response" at the IETF
meeting does not mean consensus was achieved.

Concerns that were either discounted or ignored
Date	   Author			Subject
Notes
03/03/06 Barbara Denny 		Forwarding Congestion
No response yet to her last set of questions and no resolution to her
issue(s) raised
03/03/06 Mark E. Robinson	Forwarding Congestion
No response yet
03/03/14 Mark E. Robinson 	Forwarding Congestion
First question answered but dismissed the remaining questions.  
03/03/21 Al	Morton 		Latency
Responded 3/27/2003
03/03/25 Sunil Kalidindi	Latency
No response yet
03/03/25 Sunil Kalidindi	Delay
No response yet

Section 3.1.3 Forwarding Congestion
<<Overloading versus Forwarding Congestion>>
My objection to this section was that the justification used to define
forwarding congestion was weak.  Your original premise for forwarding
congestion was that existing definitions were only applicable to ingress
interfaces which are not correct.  Your clarifying explanation in your
recent email is an improvement, however unfortunately, not documented in
this section.  As David Newman stated on a 3/14/2003 email, no methodology
should assume that "the test person ...understand[s] what he/she is doing."
Hence, the section should be rewritten to clear up any ambiguities.   

<<Packet Loss Indicates Congestion>>
Yes, I agree with you that packet loss due to physical layer errors is
outside the scope of BMWG and that packet loss due to switching fabric
cannot be externally observed.  However, the RFCs you mentioned measured
packet loss for a specific configuration or condition.  In contrast, the
discussion in your section is very broad.  You are stating that ANY packet
loss indicates congestion.  This is where you open yourself up to comments
about all the reasons packet loss may occur.  This is why I suggest you
place a disclaimer if you wish to make that assertion.  Otherwise, I would
suggest providing scenarios and examples in the discussion on when exactly
packet loss is an appropriate symptom to indicate congestion.  Also, I would
avoid generic statements such as:

"Packet Loss, not increased delay, is the metric to indicate condition of
forwarding congestion", or 
"...any device that cannot forward packets on one or more egress interfaces
is under forwarding congestion"  

My proposal for rewriting this section (in this order)
1. Beef up the paragraph on why the overload definition is inappropriate
when enabling traffic control mechanisms 
2. Add more description on the types of measurements used to indicate
Forwarding Congestion (delay and packet loss) 
3. Add a more detailed explanation on why packet loss is preferred over
delay (BTW, RA99 is the wrong reference to use) 
4. Expand on when packet loss is appropriate to indicate Forwarding
Congestion.  (i.e. If I send 5 packets and one or more gets dropped, does it
mean the egress interface is congested?) 

Section 3.2.1 Channel Capacity
Can you point me to the appropriate thread archive/documentation?  I am not
entirely convinced that John Dawson's comments have been interpreted
correctly.

Section 3.2.4 Delay
<<Delay Usage>>
rfc956
rfc955
rfc813
rfc795
rfc793
rfc3449
rfc3439
rfc3096
rfc3095
rfc3019
rfc3002
rfc2682
rfc2626
rfc2508
rfc2461
rfc2330
rfc2213
rfc2205
rfc1850
rfc1801
rfc1633
rfc1479
rfc1470
rfc1323
rfc1257
rfc1165
rfc1123
rfc1077
rfc1045
rfc1017
rfc1005
rfc3414
rfc3272
rfc3246
rfc3148
rfc3134
rfc3125
rfc2757
rfc2214
rfc2194
rfc2021
rfc1821

<<does the BMWG wish to restrict Delay to this definition?>>
Sorry Jerry, this question was not meant to you but to the reflector.  I
know your position.

<<Delay is delay>>
I did not say that the implementer defines the arbitrary goalposts.  I was
just offering the replacement text for "Delay is delay".  You may use my
recommendation and then follow it with a statement saying "For this
definition, the goalposts defined will be the last bit... and the first
bit..."

<<How Delay measures the IP datagram>>
Generally the term "IP Packet" refers only to the IP header +
datagram/payload portion exclusive of the da/sa/crc portion of the frame.
Is this what you mean?  Can you give me an example how this would be
measured?

<<Latency is measured only at the throughput level>> 
[BR99] is a procedure that describes measuring latency at throughput.  It
does not say that latency is invalid at any other offered rates.  [BR91]
does not define latency at a particular offered rate.  Let's take it to the
reflector and solicit a vote on how many persons associate latency with
throughput as a MUST criteria.

<<Bullet number 5>>
Thanks

<<Jitter>>
I agree that we postpone this discussion until the Delay discussion is
resolved.  

Regards,

Tony
-----Original Message-----
From: Perser, Jerry [mailto:[email-masked]] 
Sent: Wednesday, March 26, 2003 11:56 AM
To: 'Tony DeLaRosa'; 'Kevin Dubray'; 'Al Morton'
Cc: [email-masked]'
Subject: RE: [bmwg] WG Last Call: draft-ietf-bmwg-dsmterm-05

Tony,

For you new comments, I address them inline.  I will cut a lot of the text
out so we can get to the heart of the issues:

> This definition argues that overloading is too limited for this 
> application. Unfortunately, the cited references used to make this 
> assertion have been taken out of context.

> <<ISSUE>>
> Draft asserts that any packet loss above throughput is due to 
> forwarding congestion (AKA overloading).  As Barbara Denny, John 
> Dawson and Mark Robinson pointed out, it is presumptuous to assume 
> that the only reason packets are being dropped above throughput is 
> because of forwarding congestion.  One can not assume that a DUT 
> forwarding switching fabric is 100 percent bug free.  Packet loss can 
> be caused by any errors from the

  
> <<Recommendation>>
> Insert a caution or disclaimer stating that packet loss will be used 
> as an indication of forwarding congestion although it may not be the 
> sole reason. Additional clarification may be required on why packet 
> loss alone is enough for indicating forwarding congestion.  The 
> reference to [RA99] pertains to

  
> Section 3.2.4 Delay
> <<ISSUE>>
> Delay is a common term used pervasively throughout many RFCs to 
> indicate a time measurement between two specified endpoints.  I agree 
> with Debby Stopp that the draft should a add modifier in front of 
> Delay to make it specific for the particular measurement.  I can 
> imagine the confusion in methodology

Can you state the RFCs?

  

Does this sound better:

> <<ISSUE>>
> Bullet number 4.  Latency is measured only at the throughput level. 
> Yes, I would agree that [BR99] measures latency at throughput but 
> neither [BR91] nor [BR99] precludes measuring latency at any other 
> offered loads.

> <<ISSUE>>
> Bullet number 5.  Should not be considered a bullet since this 
> paragraph is more of a note.

Jerry.

---

From: Tony DeLaRosa <[email-masked]>
Date: Thu, 27 Mar 2003 16:02:37 -0800

Jerry,

Section 3.2.4 Delay
<<Delay Usage>>
rfc956
rfc955
rfc813
rfc795
rfc793
rfc3449
rfc3439
rfc3096
rfc3095
rfc3019
rfc3002
rfc2682
rfc2626
rfc2508
rfc2461
rfc2330
rfc2213
rfc2205
rfc1850
rfc1801
rfc1633
rfc1479
rfc1470
rfc1323
rfc1257
rfc1165
rfc1123
rfc1077
rfc1045
rfc1017
rfc1005
rfc3414
rfc3272
rfc3246
rfc3148
rfc3134
rfc3125
rfc2757
rfc2214
rfc2194
rfc2021
rfc1821

<<Delay is delay>>

<<Bullet number 5>>
Thanks

<<Jitter>>

Regards,

Tony
-----Original Message-----
From: Perser, Jerry [mailto:[email-masked]] 
Sent: Wednesday, March 26, 2003 11:56 AM
To: 'Tony DeLaRosa'; 'Kevin Dubray'; 'Al Morton'
Cc: [email-masked]'
Subject: RE: [bmwg] WG Last Call: draft-ietf-bmwg-dsmterm-05

Tony,

  

  

  

Can you state the RFCs?

  

Does this sound better:

Jerry.