Subject: Re: [bmwg] RFC2544 Statement

From: Barry Constantine <[email-masked]>
Date: Tue, 2 Aug 2011 04:25:13 -0700

Hi Al,

I reread the draft closely and see that your points are well stated.

I have no further comments at this time.

Thanks,
Barry

________________________________
From: Al Morton [[email-masked]]
Sent: Friday, July 29, 2011 8:07 PM
To: Barry Constantine; [email-masked]
Subject: Re: [bmwg] RFC2544 Statement

Hi Barry,

I think you've pointed-out two of the problems
with industry references to RFC 2544 in your message below:

Few, if any, vendors say that their tests are "derived from"
RFC 2544, because then they would clearly state what parts of
the procedures they changed, what new metrics they defined, etc.

Also, few if any service providers use the exact RFC 2544 metrics
in their SLAs. Various metrics like Committed Info Rate and Delay Variation
need to be measured and validated, but these metrics do not appear in
RFC 2544.

Speaking for my co-authors, I think we would prefer that network
providers apply appropriate standards for real-world service activation and
trouble-shooting where they exist (ITU-T Rec Y.1564 is one example),
and develop new standards where they don't. Standards Development Orgs
should include RFC 2544 as an informative reference if they believe
those methods can be used as a starting point for live
network test procedures and trouble-shooting.

Al
(as co-author/participant)

At 06:06 PM 7/28/2011, Barry Constantine wrote:
Content-Language: en-US
Content-Type: multipart/alternative;
         boundary="_000_94DEE80C63F7D34F9DC9FE69E39436BE3A0BCB7BB3MILEXCH1dsjds_"

Hi All,

I think this is a good effort with the right intentions.

There are two (2) scenarios that network providers use RFC2544 (or derivatives) today:

1. "Turn-up" testing which is conducted on a business class, production network before end-customer traffic is commissioned.  This is generally considered the verification of the end-customer SLA.

2. Troubleshooting.  This is performed after a end-customer has been commissioned and the SLA is disputed (poor application performance, etc.).  The network provider coordinates a time to swtich the end-customer to a back-up link and then re-conducts RFC2544 type testing on the production link.

I think this statement will be taken more credibly by network providers if it is acknowledged that these types of scenarios exist today and that caution should be used when performing RFC2544 (especially in case #2, troubleshooting).

Al, I think you mentioned there was a section related to exceptions (?), if these points are agreeable, then I think this section should be moved closer to the front so that a reader will grasp the overall message.

My two cents,
Barry
_______________________________________________
bmwg mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/bmwg

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 02 Aug 2011 11:21:03 -0400

In message <[email-masked]>
Barry Constantine writes:
> 1. "Turn-up" testing which is conducted on a business class,
>    production network before end-customer traffic is commissioned.
>    This is generally considered the verification of the end-customer
>    SLA.
>  
> 2. Troubleshooting.  This is performed after a end-customer has been
>    commissioned and the SLA is disputed (poor application performance,
>    etc.).  The network provider coordinates a time to swtich the
>    end-customer to a back-up link and then re-conducts RFC2544 type
>    testing on the production link.
>  
> I think this statement will be taken more credibly by network
> providers if it is acknowledged that these types of scenarios exist
> today and that caution should be used when performing RFC2544
> (especially in case #2, troubleshooting).
>  
> Al, I think you mentioned there was a section related to exceptions
> (?), if these points are agreeable, then I think this section should
> be moved closer to the front so that a reader will grasp the overall
> message.
>  
> My two cents,
> Barry

Barry,

The intent of RFC2544 is as a methodology for "benchmarking", not for
SLA verification and trouble shooting.  This should be very clear
after reading the first paragraph of the introduction.

  1. Introduction

   Vendors often engage in "specsmanship" in an attempt to give their
   products a better position in the marketplace.  This often involves
   "smoke & mirrors" to confuse the potential users of the products.
 
This is for evaluating equipment in the lab before you make a huge
purchase of equipment for a large network deployment.

Any provider that is going to spend many millios of dollars on a
deployment will hire staff specifically for testing, use the best
available off-the-shelf testing availalbe, and do way more testing
than what is described in RFC2544 before buying anything (often
borrowing the equipment used in an initial evaluation from the vendor,
then only buying from first round qualifiers for extended testing).

Of course, for small purchases far less testing rigor is applied.

BMWG has morphed over time, but originally the focus was on testing
for provider networks in the high growth years of the Internet.  When
RFC 2544 was written there was still way too much emphasis in BMWG on
repeatable results and way too little emphasis on exposing fatal flaws
in routers before they were deployed in real networks.

Curtis

BTW- The worst product ever to benchmark well in limited benchmarks
was the Cisco 7000.  In a real network it was terrible because it
dropped packets after any route change.  A network of Cisco 7000 could
easily be made to go unstable when a lot of route change was occurring
(which resulted in a positive feedback and at times massive
instabilities in networks built with these routers).  Lessons were
learned and the 7500, CEF, DCEF, GSR, etc followed and other vendors
emerged, some which learned, others which didn't.

---

From: Kevin Dubray <[email-masked]>
Date: Tue, 2 Aug 2011 12:53:44 -0400

See, Al, that training as a used car salesman comes in handy! :-)

For me, one of the key tenets in this I-D (and the WG moving forward)  is the notion of an ITE.  Are folks happy with the description of ITE in section 3?

-Kevin

On 8/2/2011 7:25 AM, Barry Constantine wrote:
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> *From:* Al Morton [[email-masked]]
> *Sent:* Friday, July 29, 2011 8:07 PM
> *To:* Barry Constantine; [email-masked]
> *Subject:* Re: [bmwg] RFC2544 Statement
...

---

From: Al Morton <[email-masked]>
Date: Tue, 02 Aug 2011 14:19:37 -0400

At 12:53 PM 8/2/2011, Kevin Dubray wrote:
>For me, one of the key tenets in this I-D (and the WG moving 
>forward)  is the notion of an ITE.  Are folks happy with the 
>description of ITE in section 3?
>
>-Kevin

Let's find-out, Kevin.
For those who are about to read the draft,
ITE = Isolated Test Environment.

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 02 Aug 2011 16:22:03 -0400

> ITE = Isolated Test Environment.

Al,

I assume that the draft is draft-chairs-bmwg-2544-as-00.txt

And yes.  RFC 2544 is intended for ITE only and should not be used in
any other way.

BTW- Serious damage has been done by testing very high end routers
where the lab got accidentally connected to the real network and
misrouted and the lab test equipment, not using just net 10 or other
private addresses proceeded to launch an effective denial of service
attack on a number of networks.  I have seen this happen at both a
network service provider and at a high end equipment vendor.

I read the draft.  Its really short.  If the misuse has occurred, then
you have my support for making this a WG doc and moving it forward
promptly.  No salesmanship or arm twisting required.

If you want you can mention OAM work as well as IPPM.  MPLS-TP loss
measurement (LM) OAM comes to mind for validation of SLA metrics
without adding traffic.  It is a highly accurate passive verification
tool with almost zero overhead.  There is other work going on for
measuring delay and jitter.  Just a suggestion.  Feel free to ignore.

Curtis

---

From: Al Morton <[email-masked]>
Date: Tue, 02 Aug 2011 17:34:59 -0400

>promptly.  No salesmanship or arm twisting required.

Good, thanks.

>measuring delay and jitter.  Just a suggestion.  Feel free to ignore.

I'll take a look at the TP Loss (again) soon.
All passive measurement methods are not created equal...

Al