Subject: RE: [bmwg] dsmterm-05 - Forwarding Congestion

From: Kevin Dubray <[email-masked]>
Date: Mon, 03 Mar 2003 15:06:16 -0500

Do you all agree w/ Jerry's assessment?  The short answer is
in the last paragraph :-)

-Kevin

-------- Original Message --------
Subject: RE: [bmwg] dsmterm-05 - Forwarding Congestion
Date: Mon, 3 Mar 2003 11:10:08 -0800
From: "Perser, Jerry" <[email-masked]>
To: "'Kevin Dubray'" <[email-masked]>
CC: "David Newman (E-mail)" <[email-masked]>,   "Scott Poretsky 
(E-mail)" <[email-masked]>,   Shobha Erramilli	 
<[email-masked]>,   Sumit Khurana <[email-masked]>

 > But why the reticence to call the term "Lossful Forwarding
 > Congestion."  And why not call out the lossful attribute
 > out explicitly in the definition?
 >
 > See: http://www.ietf.org/proceedings/02nov/152.htm
 >
 >
 > -Kevin
 >

The focus on Congestion is centered around Traffic Control Mechanisms.  By
externally observing the DUT/SUT ability to forward packets, it can be
determined whether Traffic Control Mechanisms are active.

Active and Enabled are two different things.  At low OLOAD, the Traffic
Control Mechanisms may be enabled, but not effecting the forwarding of
packets.  Stating it a different way, enabling and disabling the Traffic
Control Mechanisms will have no externally observable affects.

Going back to Traffic Control Mechanisms, what are their purpose?  On a per
hop bases, Traffic Control Mechanisms use some type of intelligence to
modify packet streams by either delaying the forwarding process or by not
forwarding select packets.

The purpose of the I-D is to benchmark the Traffic Control Mechanisms'
intelligence on a per hop behavior.

Another point is that RFC1242 and RFC2432 do not call out their lossful
attribute out explicitly in the definition.  Why not call it Mixed Class
Zero-loss Throughput?  I have seen reports that have Throughput numbers with
a loss < 0.01%.  Looking the industry reports, there are Throughput numbers
and there are Zero-loss Throughput numbers.  I agree the precedence set by
RFC1242 and RFC2432.

Last point I would like to make is the notion of "lossless forwarding
congestion".  If there is lossful, is there also lossless?  The answer is
no.  By dropping the lossful/lossless notion from the definition does not
change the meaning.

Jerry.

___________________________________________________________
Jerry Perser                         Spirent Communications
Director of Test Methodology         SmartBits Division
Phone: 818.676.2320                  26750 Agoura Road
Lab:   818.676.2337                  Calabasas, CA, 91302
Cell:  818.292.6457                  Corp:  818.676.2300

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: Kevin Dubray <[email-masked]>
Date: Mon, 03 Mar 2003 15:06:16 -0500

-Kevin

-------- Original Message --------
Subject: RE: [bmwg] dsmterm-05 - Forwarding Congestion
Date: Mon, 3 Mar 2003 11:10:08 -0800
From: "Perser, Jerry" <[email-masked]>
To: "'Kevin Dubray'" <[email-masked]>
CC: "David Newman (E-mail)" <[email-masked]>,   "Scott Poretsky 

Jerry.

---

From: Debby Stopp <[email-masked]>
Date: Tue, 4 Mar 2003 08:52:39 -0800

Mon, 11 Nov 2002 , Fred Baker stated that:
"A congested interface is one for which the queuing delay is measurably 
non-zero, and as a result causes issues for applications running over the
connected internet fabric. It is hard to meaningfully discuss without
bringing the effects on the applications into the discussion."

W/out the 'Lossful' modifier as suggested by Kevin, the term definition for
'Forwarding Congestion":
          "Packet Loss, not increased Delay, is the metric to indicate
          the condition of Forwarding Congestion.  Packet Loss is a
          deterministic indicator of Forwarding Congestion."

is, at best, inaccurate.

- Debby Stopp

> -----Original Message-----
> From: Kevin Dubray [mailto:[email-masked]]
> Sent: Monday, March 03, 2003 12:06 PM
> To: [email-masked]

---

From: "Perser, Jerry" <[email-masked]>
Date: Tue, 4 Mar 2003 10:49:56 -0800

Debby,

I was reading draft-ietf-bmwg-mcastm-10.txt and found several metrics (I
counted 6) that have the same 'Lossful' attribute as Forwarding Congestion.
For example, Aggregated Multicast Throughput requires loss to equal zero.

Do you believe that Mixed Class Throughput, Aggregated Multicast Throughput,
Encapsulation Throughput, etc. are also "at best, inaccurate."?

If not, why?

Jerry.

> From: Debby Stopp [mailto:[email-masked]]
> Sent: Tuesday, March 04, 2003 8:53 AM
> To: [email-masked]

---

From: "Perser, Jerry" <[email-masked]>
Date: Tue, 4 Mar 2003 10:49:56 -0800

Debby,

If not, why?

Jerry.

---

From: Michele Bustos <[email-masked]>
Date: Tue, 4 Mar 2003 13:40:14 -0800

In section 3.1.3 Forwarding Congestion:

"At offered rates above throughput, the DUT/SUT is considered congested."

Can you explain this?

In my understanding of the classic definition of throughput and congestion,
the DUT/SUT could be congested long before any packets are dropped. 

thx,
/m

---

From: Scott Poretsky <[email-masked]>
Date: Tue, 04 Mar 2003 18:46:13 -0500

>the DUT/SUT could be congested long before any packets are dropped.

This is called "Incipient Congestion" as defined in RFC 2481.  This is the 
case in which queues are building, but not full.  Incipient Congestion is 
externally observable as increased delay.  A full queue results in packet 
loss.  We have defined the term Forwarding Congestion to describe the 
condition in which queues are full and packet loss is externally 
observable.  RFC 2481 states that black-box measurement of Incipient 
Congestion is not practical.  Increased delay is non-deterministic 
indicator of congestion, but packet loss is a deterministic indicator of 
congestion.  For the practical purposes of DiffServ measurement, we have 
specified Forwarding Congestion as the indicator for congestion.  For 
reference, here is the discussion from the draft....
Packet Loss, not increased Delay, is the metric to indicate the condition 
of Forwarding Congestion.  Packet Loss is a deterministic indicator of 
Forwarding Congestion.  While increased delay may be an indicator of 
Forwarding Congestion, it is a non-deterministic indicator of Forwarding 
Congestion.  External observation of increased delay to indicate congestion 
is in effect external observation of Incipient Congestion. [Ra99] states 
that it is impractical to build a black-box test to externally observe 
Incipient Congestion in a router.  For the purpose of "black-box" testing a 
DUT/SUT, Packet Loss as the indicator of Forwarding Congestion is used.

---

From: Scott Poretsky <[email-masked]>
Date: Tue, 04 Mar 2003 18:46:13 -0500

---

From: Michele Bustos <[email-masked]>
Date: Tue, 4 Mar 2003 16:54:27 -0800

Dear Scott,

Thank you for your clear explanation. :)

Unfortunately, I did not gain that same level of understanding from reading
the doc.

"Throughput is the maximum offered rate with no Forwarding Congestion."

It appears from your explanation that the point of the throughput reference
is to provide an example, however the doc reads as though it's a
redefinition of throughput in relation to the more general definition of
'congestion', as suggested in an earlier thread by Kevin, defined in
rfc1254.

http://www1.ietf.org/mail-archive/working-groups/bmwg/current/msg00283.html

kind regards,

/m

-----Original Message-----
From: Scott Poretsky [mailto:[email-masked]]
Sent: Tuesday, March 04, 2003 3:46 PM
To: Michele Bustos; Jerry Perser (E-mail)
Cc: [email-masked]
Subject: RE: [bmwg] dsmterm-05 - Forwarding Congestion

Can you explain this?

This is called "Incipient Congestion" as defined in RFC 2481.  This is the
case in which queues are building, but not full.  Incipient Congestion is
externally observable as increased delay.  A full queue results in packet
loss.  We have defined the term Forwarding Congestion to describe the
condition in which queues are full and packet loss is externally observable.
RFC 2481 states that black-box measurement of Incipient Congestion is not
practical.  Increased delay is non-deterministic indicator of congestion,
but packet loss is a deterministic indicator of congestion.  For the
practical purposes of DiffServ measurement, we have specified Forwarding
Congestion as the indicator for congestion.  For reference, here is the
discussion from the draft....

Packet Loss, not increased Delay, is the metric to indicate the condition of
Forwarding Congestion.  Packet Loss is a deterministic indicator of
Forwarding Congestion.  While increased delay may be an indicator of
Forwarding Congestion, it is a non-deterministic indicator of Forwarding
Congestion.  External observation of increased delay to indicate congestion
is in effect external observation of Incipient Congestion. [Ra99] states
that it is impractical to build a black-box test to externally observe
Incipient Congestion in a router.  For the purpose of "black-box" testing a
DUT/SUT, Packet Loss as the indicator of Forwarding Congestion is used.

thx,
/m
_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg
<https://www1.ietf.org/mailman/listinfo/bmwg>

---

From: Debby Stopp <[email-masked]>
Date: Wed, 5 Mar 2003 11:42:10 -0800

My point was merely that the discussion following the definition for the
term "Forwarding Congestion" has some ambiguous references to 'congestion',
which MAY lead to confusion.

d.

> From: Perser, Jerry [mailto:[email-masked]]
> Sent: Tuesday, March 04, 2003 10:50 AM
> To: [email-masked]

---

From: Debby Stopp <[email-masked]>
Date: Wed, 5 Mar 2003 11:42:10 -0800

d.

---

From: "David Newman" <[email-masked]>
Date: Wed, 5 Mar 2003 12:58:07 -0800

> which MAY lead to confusion.

As Kevin has indicated, the ambuigity is whether the term is Forwarding
Congestion or Lossful Forwarding Congestion.

In a sense, prepending the redundant "lossful" to "forwarding congestion"
actually might increase confusion. Comments like "oh, so if there's lossful
there must be lossless too, right?" may follow.

This is hardly a hypothetical. I have spent many years explaining (with
varying degrees of success) to various marketdroids and the test labs in
their employ that there is no such thing as lossful throughput. I am not
eager to repeat that exercise.

dn

---

From: "David Newman" <[email-masked]>
Date: Wed, 5 Mar 2003 12:58:07 -0800

dn

---

From: "Perser, Jerry" <[email-masked]>
Date: Wed, 5 Mar 2003 15:29:58 -0800

Just to be clear, your point was not related to the 'Lossful' modifier.  The
term 'congestion' is ambiguous.  The term 'congestion' could be confused
with Congestion Collapse, Forwarding Congestion, Incipient Congestion, or
other type of congestion.

I found three uses of the term 'congestion', when in fact we meant
'Forwarding Congestion'.  The other uses of the term 'congestion' are not
implied 'Forwarding Congestion', but the general concept of congestion.  For
example, paragraph 1, the overload definition in RFC2285 does differentiate
between Congestion Collapse, Forwarding Congestion, and Incipient
Congestion.

Statements like "is, at best, inaccurate." are judgmental and don't belong
in a technical discussion.  An alternate method would be to supply suggested
text.  I learned this from Mr. Dubray.  Here is and example:

3.1.3 Forwarding Congestion

- Second paragraph, the sentence "Depending on traffic patterns, the edge
devices may induce congestion on multiple egress interfaces on the core
device." may be clearer is worded "Depending on traffic patterns, the edge
devices may induce Forwarding Congestion on multiple egress interfaces on
the core device."

- Fourth paragraph, the sentence "At offered rates above throughput, the
DUT/SUT is considered congested."  Suggested change: "At offered rates above
throughput, the DUT/SUT is considered to be in a state of Forwarding
Congestion".

- Fifth paragraph, last sentence, consider: "Even though ingress interfaces
accept all packets without loss, Forwarding Congestion is present in this
hypothetical device."

Jerry.

P.S. I am proposing these three change to the other authors and BMWGers.

> Sent: Wednesday, March 05, 2003 11:42 AM
> To: [email-masked]

---

From: "Perser, Jerry" <[email-masked]>
Date: Wed, 5 Mar 2003 15:29:58 -0800

Jerry.

---

From: "Perser, Jerry" <[email-masked]>
Date: Thu, 6 Mar 2003 09:44:30 -0800

Barbara,

This I-D as well as its predecessors (RFC1242, RFC2285, RFC2432, RFC2647)
have ignored physical layer errors.  The focus has been to benchmark the
device.  Physical errors can affect the metrics, but in practice don't.  In
RFC2554 or RFC2889 test where physical layer errors were present, I have
corrected the error and rerun the tests.

Another point is that we (the test operator) externally observe the metrics
at the ingress and egress of the DUT/SUT.  If the packet is corrupted going
into the ingress of the DUT/SUT, it is observed and not counted as a valid
offered packet.

If "a packet gets dropped due to buffer or queue limits being exhausted",
weather it the IP layer or lower layers, then is falls under Forwarding
Congestion.  If 'Good in' > 'Good Out', the DUT/SUT filtered the packet
based on congestion, not on errors.

Jerry.

> From: David Newman [mailto:[email-masked]]
> Sent: Thursday, March 06, 2003 9:01 AM
> To: Barbara Denny
> Cc: [email-masked]
> On Thu, 6 Mar 2003, Barbara Denny wrote:
> 
> > I sent David some more info but maybe I should
> > at least mention to the group that this kind of
> > loss will look the same to IP.  You lost a packet.
> > At IP, how will you know if it was due to congestion
> > or a problem at the link layer due to interference,
> > etc? Are you requiring that lower layers notify
> > IP a packet gets dropped due to buffer or queue
> > limits being exhausted?
> 
> Not at all. One of OSI's few redeeming features was layer 
> separation --
> one variable at a time is a very good thing indeed.
> 
> Let's begin by agreeing that loss induced by events at other 
> layers occurs
> all the time, and some of these events could affect the sole variable
> under observation here -- the DS field in the IP header. 
> Examples include
> LOF in Sonet, backoff algorithm timeouts in half-duplex 
> Ethernet, and RF
> issues in wireless.
> 
> If I understand your question correctly, it is essentially 
> "aren't these 
> cases where loss exists without congestion?" and of course 
> the answer is 
> yes. 
> 
> Still, this issue is orthogonal to the definition at hand. 
> Let's look at 
> the definition of forwarding congestion again:
> 
> "A condition in which one or more egress interfaces are offered more
> packets than are forwarded."
> 
> In the cases above, the egress interfaces are offered too FEW packets,
> not too many. Given this underfeeding, we would expect the egress
> interfaces to be able to forward all packets without loss.
> 
> My statement was that loss on the egress interfaces ALWAYS indicates
> forwarding congestion. I don't believe anything we've discussed here
> contradicts that statement. I could be wrong. If I've missed something
> please let me know -- especially if there is a case we've 
> missed where 
> loss on egress != forwarding congestion. Thanks.
> 
> dn
> 
> ps. By the way, the issue of extraneous variables isn't 
> addressed in other
> bmwg documents either. I could test a switch or router and 
> get throughput
> of 0 bit/s by unplugging a cable, yet still conform to 2889 
> or 2544.  On
> the general question of whether we should make all efforts to 
> factor out
> extraneous variables like this, I think we're in complete agreement.
> 
> 
>

---

From: "John Dawson" <[email-masked]>
Date: Thu, 6 Mar 2003 10:42:08 -0800

> If the packet is corrupted going into the ingress of the DUT/SUT, 
> it is observed and not counted as a valid offered packet.

Are you saying that offered load is now dependent upon whether the DUT
detects an error with the packet?  That seems like a substantial change in
methodology -- I use the word "seems" as I don't think the RFCs have
agreement on how offered load is actually to be measured.  Previously the
entire measurement process was independent upon the DUT's view of the world
-- we could measure everything from an external perspective.  The above
statement would be a substantial departure from that philosophy.

I would say that the group has avoided problems with bit error rate for far
too long.  I argue that some measure of protection is already warranted.
For instance, including an automatic retry for a failing test case if the
measured loss is less than the expected BER for the transmission media.

Although current tests (with an occasional retry) do work for 30 to 60
seconds without significant CRC disruptions, is that going to hold up as we
move to systems with many copper gigabit ports?  Is throughput going to
become a meaningless measure if the number of offered bits exceeds practical
BER limits?

John

-----Original Message-----
From: Perser, Jerry [mailto:[email-masked]] 
Sent: Thursday, March 06, 2003 9:45 AM
To: Newman, David; Barbara Denny
Cc: [email-masked]
Subject: RE: [bmwg] dsmterm-05 - Forwarding Congestion

Barbara,

Jerry.

---

From: "John Dawson" <[email-masked]>
Date: Thu, 6 Mar 2003 10:42:08 -0800

John

-----Original Message-----
From: Perser, Jerry [mailto:[email-masked]] 
Sent: Thursday, March 06, 2003 9:45 AM
To: Newman, David; Barbara Denny
Cc: [email-masked]
Subject: RE: [bmwg] dsmterm-05 - Forwarding Congestion

Barbara,

Jerry.

---

From: "Perser, Jerry" <[email-masked]>
Date: Thu, 6 Mar 2003 11:20:48 -0800

John,

I'll start with a quote from RFC2285, section 3.5.2 Offered load:

      Frames which are not successfully transmitted by an external
      source of traffic to a DUT/SUT MUST NOT be counted as transmitted
      frames when measuring forwarding rates.

How do you define "successfully transmitted"?

My opinion is frames that has a CRC errors, Dribble Bit Errors, Alignment
Errors, or are either Oversize or Undersize lengths (RFC2889 section 5.9.3),
"are not successfully transmitted ... to a DUT/SUT".  The Data link layer
SHOULD filter these frames such that they never appear at the egress of the
DUT/SUT.

Jerry.

-----Original Message-----
From: John Dawson [mailto:[email-masked]]
Sent: Thursday, March 06, 2003 10:42 AM
To: [email-masked]
Subject: RE: [bmwg] dsmterm-05 - Forwarding Congestion

John 
-----Original Message----- 
From: Perser, Jerry [mailto:[email-masked]] 
Sent: Thursday, March 06, 2003 9:45 AM 
To: Newman, David; Barbara Denny 
Cc: [email-masked] 
Subject: RE: [bmwg] dsmterm-05 - Forwarding Congestion 

Barbara,

---

From: "John Dawson" <[email-masked]>
Date: Thu, 6 Mar 2003 12:28:16 -0800

My assumption is that the traffic generator is going to be running properly
and not intentionally injecting errors.  Therefore if the traffic generator
said it sent x packets with no internal errors or late-collisions, then I
believe it.  My assumption is that the cited clause applies to errors that
the transmitter can sense.

However, we can not say that successfully transmitted and successfully
received are the same things.

Imagine a DUT that has a lot of noise coupling into the PHY receiver.  As
more packets are processed the noise floor increases and causes more inbound
packets to be corrupted.  Even though the test is run at 100% intended load,
the DUT could be corrupting untold quantities of packets.  We would forgive
the poor product design and report the test as passing at 100% intended
load?  The only way to catch that anything suspicious was occurring is by
enforcing the rule that we should be reporting results based upon offered
load -- which would now be something less than 100%.  If our corruption was
only .1%, then removing anything the DUT declares as an error would lead to
a 99.9% throughput score.

Now let's take the same imaginary DUT, that is corrupting 0.1%, and test it
at 90% load.  Maybe the DUT is only corrupting 0.09% at this load rate.
Would we still consider it as passing also?

The entire purpose of throughput testing was to measure the data rate at
which NO Loss occurs.  If we start allowing certain loss conditions, then
where do we draw the line.  I certainly would not advocate allowing the DUT
to declare the pass/fail criteria.

Maybe we have to say that zero loss must actually allow for some defined
allowance for BER (i.e.. anything less than 1e-08 or 1e-10 is considered a
random bit error and who cares why it was lost).  Assuming some allowance
for BER doesn't change the process for measuring offered load, it changes
the process for measuring loss.  Providing a quantified BER acceptance
clause is a lot different than saying, look at the DUT and forgive anything
that appears to be a physical or data link error.

I go back to my original proposition; that I don't believe the RFCs have a
done a good job of defining the process to measure offered load, and at some
point measurements requiring absolutely no loss will become a practical
impossibility.

John

-----Original Message-----
From: Perser, Jerry [mailto:[email-masked]] 
Sent: Thursday, March 06, 2003 11:21 AM
To: Dawson, John [BLVWA:382:EXCH]; [email-masked]
Subject: RE: [bmwg] dsmterm-05 - Forwarding Congestion

John,

Jerry.

-----Original Message-----
From: John Dawson [mailto:[email-masked]]
Sent: Thursday, March 06, 2003 10:42 AM
To: [email-masked]
Subject: RE: [bmwg] dsmterm-05 - Forwarding Congestion

John 
-----Original Message----- 
From: Perser, Jerry [mailto:[email-masked]] 
Sent: Thursday, March 06, 2003 9:45 AM 
To: Newman, David; Barbara Denny 
Cc: [email-masked] 
Subject: RE: [bmwg] dsmterm-05 - Forwarding Congestion 

Barbara,

---

From: Gabriel Heininger <[email-masked]>
Date: Thu, 6 Mar 2003 13:51:14 -0800

> In

Yup. Those runts really muck up the SMTP authentication in the RFC 2554
tests.
_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: Gabriel Heininger <[email-masked]>
Date: Thu, 6 Mar 2003 13:51:14 -0800