Subject: RE: [bmwg] ALT/ADT activities

From: [email-masked]
Date: Sat, 24 Jun 2006 07:11:13 -0400

Hi Rivalino,

Thank you very much for your review and comments.  Please see my points
below.  Further discussion is welcome.

Scott

-----Original Message-----
From: Rivalino Matias Jr. [mailto:[email-masked]]
Sent: Friday, June 23, 2006 4:29 PM
To: [email-masked]
Subject: RES: [bmwg] ALT/ADT activities

Hi List,

I read the material [1, 2] and bellow I'm sending my doubts and suggestions.
Also, I would like to make clear that some misinterpretation from my side is
possible due to the language, technical vocabulary, others.

a)	In [1], the term "Accelerated Life Benchmark" was used. I found it
more
three or four times in the same text. What is the difference between
"Accelerated Stress Benchmark" and "Accelerated Life Benchmark" ? Also,
another related term was used in the same document: "Accelerated Stress
 Test". Are those synonymous with each other?

Scott>  Great catch.  I have updated both the terminology and methodology
documents to use only "Accelerated Stres Benchmark".

b)	In [1], Appendix 1, the term "Memory" could be changed to "principal
memory" in order to avoid any doubts related to others types of memories,
such as flash memory.

Scott> The BMWG charter is specific to black-box measurements.
The appendix provides white box benchmarks for informational purposes only,
as it was agreed by the working group that this would be useful.  All system
memory (or memories) are applicable.  Granted there should be no change to
flash memory during the test, but if there were then that would be an
interesting data point.  Since this is a white-box measurement, the
methodology will not address how and which memory to monitor.   I like this
apporach because memories are so implementation specific and developing a
benchmarking standard could result in infinite test cases.  In short, I
would prefer to leave the appendix with 'memory' and not discuss which
memory.  Any thoughts from other folks on the mailing list?

c)	In the Abstract sections from [1] and [2], I got a little bit
confused
with the following fragments:

"... This document provides the Terminology for performing Stress
   Benchmarking of networking devices.

The title of the document is: "Terminology for Accelerated Stress
Benchmarking". Here is not clear the difference between "accelerated stress
benchmarking" and "stress benchmarking". Is there difference in this
context?

Scott>  No difference.  I have added the word 'accelerated' to the abstracts
to eliminate potential confusion.

d)	In [2], section 3.3, I suggest to substitute the term
"repeatability" to
replication ("... for test replication when ..."). IMHO, "replication" is
more
used by the test literature (DOE, ALT, AST, etc.).

Scott> 'Replication' and 'Repeatability' mean two very different things.
According to Webster's English dictionary 'replication' means 'a copy'.  An
engineer can replicate a test topology.  Repeatability means that a result
can be obtained again.  The engineer measured repeatable data.  In the
context of this document, 'repeatability' is the correct word choice.
Repeability is a very commonly used term within the networking industry and
is used in numerous web sites, including those of the IP test equipment
vendors.  Within the BMWG we frequently use the 'repeatable' and its
variants. 

e)	In [2], regarding the term "degraded", I have seen "degradated" in
the
accelerated test literature. However, my native language is Portuguese and I
'm not sure about the best alternative.

Scott>  According to Webster's English dictionary there is no such word as
'degradated'.  According to Webster, 'degrade' is the verb form of noun
'degradation'.  Past tense of the verb is 'degraded'.

f)	Basically, there are two purposes to do an accelerated life test
that is
to conduct a qualitative or quantitative analysis. Normally, accelerated
stress test (AST) is focused on the qualitative analysis. Although it is
implicit when applying AST, I think it should be clear in the documents.
Also, I think that could be a good idea to extend the methodology to
consider quantitative analysis too. But, in this case a new approach will be
necessary, because AST is not adequate. ALT or ADT techniques are good
candidates.

Scott>  The Charter of the BMWG is to produce standards for measurement of
performance benchmarks for a single networking device.  This must be
quantitative data.  The BMWG has been concentrating on this accelerated
stress benchmarking standard to ensure repeatable quantitative results are
obtained.  

I still reading the material in order to do new contributions. However, I
would like to listen from this list if I misunderstood something in order to
align my viewpoint with this group.

Best regards,

Rivalino

-----Mensagem original-----
De: Al Morton [mailto:[email-masked]]
Enviada em: sexta-feira, 26 de maio de 2006 12:59
Para: [email-masked]; [email-masked]
Assunto: Re: [bmwg] ALT/ADT activities

At 10:07 AM 5/26/2006, Rivalino Matias Jr. wrote:
>In the context of this workgroup is there some activity focused on
>accelerated tests ?

Yes, we have
Terminology for Accelerated Stress Benchmarking
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-acc-bench-term-08.txt
and a methodology draft that recently expired:
http://tools.ietf.org/wg/bmwg/draft-ietf-bmwg-acc-bench-meth/draft-ietf-bmwg
-acc-bench-meth-04.txt

>I´m working on ALT (accelerated life tests) and ADT (accelerated
degradation
>tests) applied to software components, mainly DUT/SUT that is based on
>hardware and software embedded (e.g. switches, routers, firewalls, load
>balancers, etc.).

Great.  You've already joined the group (welcome!)
You can contribute by reading the drafts above and
providing your comments on the mailing list.

Al
bmwg co-chair

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: "Rivalino Matias Jr." <[email-masked]>
Date: Sat, 24 Jun 2006 12:06:02 -0300

Hi List,

Scott, Thank you for the review. Below a write some comments:

-----Mensagem original-----
De: [email-masked] [mailto:[email-masked]]
Enviada em: sábado, 24 de junho de 2006 08:11
Para: [email-masked]; [email-masked]
Assunto: RE: [bmwg] ALT/ADT activities

Hi Rivalino,

Scott

-----Original Message-----
From: Rivalino Matias Jr. [mailto:[email-masked]]
Sent: Friday, June 23, 2006 4:29 PM
To: [email-masked]
Subject: RES: [bmwg] ALT/ADT activities

...

Rivailno> OK. I did this suggestion based on the DOE (design of experiment)
literature (e.g. [*] and [**]). DOE is strongly used as a framework to
conduce accelerated tests. In the DOE terminology an experiment repetition
is usually called “experiment replication”. To execute n times the same
experiment, with the same experimental conditions, is an essential step in
any experimental test methodology. Anyway, to me both terms are adequate.

[*] – D.C. Montgomery, “Design and Analysis of Experiments”, 6th edition,
John Wiley & Sons, 2005.
[**] – R. Jain, “Art of Computer Systems Performance Analysis Techniques for
Experimental Design, Measurement, Simulation, and Modeling”, John Wiley &
Sons, 1991.

Rivalino> Great ! living and learning :).

Rivalino>  Ok Scott. Let me describe what I’m considering as qualitative and
quantitative utilizing a fictitious scenario. I’m evaluating three routers
in order to know what is the best choice to use in my network project. One
of my test cases is to know if the routers are robust to support 100
gigabytes of data transfer without reboot them. Considering that my typical
workload is 100 megabytes per hour, a test in this typical condition could
require approximately 1024 hours. However, using an accelerated stress test
I can reaches the same result in one hour. The result is: Router A and B
(Passed) and C (Failed).This is a qualitative test.
Another situation is my second test. I want to know what is the MTBF of each
router. Here I have a quantitative necessitya – the MTBF value.
Like the above situation, it is not interesting to test the routers in the
typical workload, because its time to fail is too long. So, I will use an
accelerated stress test in order to reduce the test duration. To do that, I
need to plan how is possible to accelerate the router lifetime in order to
complete the test in a period shortly than the test based on the typical
workload. But, I don’t want to know the MTBF using the stress condition -
this is not my reality ! I want to know the MTBF in typical condition. Then,
I get the MTBF using a stress condition and after I need to calculate the
MBTF considering the typical condition. The final result is that I reduced
the test duration in some order of magnitude and I got the MTBF for the
typical condition. In this situation I have a quantitative approach, because
I have the MTTF value. The above case I had a qualitative approach (Passed
or Failed).
Scott and other guys, please make your comments that will help me to
understand well the group view.

Best regads,

Rivalino

-----Mensagem original-----
De: Al Morton [mailto:[email-masked]]
Enviada em: sexta-feira, 26 de maio de 2006 12:59
Para: [email-masked]; [email-masked]
Assunto: Re: [bmwg] ALT/ADT activities

Yes, we have

degradation

Al
bmwg co-chair