Subject: [bmwg] URL to BGP covergence methodology issues slides.

From: Kevin Dubray <[email-masked]>
Date: Fri, 07 Dec 2001 11:48:06 -0500

One of the agenda items for the BMWG session at SLC is to
solicit input with regards to issues facing the convergence 
benchmark methodologies.

The principals offer a view of the issues here:

   http://www.lepp.com/IETF/BGPconv.ppt

It would be great if you could bring your input to the
meeting!  (For those unable to attend, your feedback is
welcome right here on this list.)

-Kevin

_______________________________________________
bmwg mailing list
[email-masked]
http://www1.ietf.org/mailman/listinfo/bmwg

---

From: Jim McQuaid <[email-masked]>
Date: Tue, 11 Dec 2001 10:02:43 -0500

Howard wrote:
-----------------------
The concern about number of interfaces/peers, however, gets very 
touchy because we are getting right on the edge of where the 
salesdroids start trying to manipulate. Can there be -- and I 
honestly mean this as a question -- a consensus on this?
-----------------------

My thoughts:

History shows that consensus on "what is normal" in almost any BMWG context
is difficult to impossible to achieve.  One reason is the
salesdroid-proximity effect but another is the genuinely wild nature of
networks.

The traditional, and I think, successful way to deal with this is to have a
methodology which clearly calls out such a dimension as a "MUST report which
size" type of metric.  Of course, publishing a test in which product X looks
great at metric_value=1 and terrible at metric_value = something large and
useful, goes a long way to institutionalizing competition along a dimension
that we believe is meaningful (which is why it's in the methodology in the
first place).

cheers,

Jim McQuaid

NetIQ - Raleigh

-----Original Message-----
From: Howard C. Berkowitz [mailto:[email-masked]]
Sent: Tuesday, December 11, 2001 9:54 AM
To: David Newman; Padma Krishnaswamy
Cc: [email-masked]
Subject: RE: [bmwg] URL to BGP covergence methodology issues slides.

At 8:03 AM -0800 12/8/01, David Newman wrote:
>  > Have you any rules of thumb as to this: ie, if testing core
>>  routers, how many peers would you consider representative?
>
>Of course, the question here is "representative of what?" I don't know that
>there is any one number or range that is meaningful for every user, or even
>for most users. I have a few numbers but they're 2-3 years old and I don't
>know if they're still accurate.
>
>Maybe Randy or other folks with ops experience can comment here.

Do note there are some representative route mixes for various kinds 
of router (core, interprovider, customer, etc.) in Appendix A of the 
methodology document.

The concern about number of interfaces/peers, however, gets very 
touchy because we are getting right on the edge of where the 
salesdroids start trying to manipulate. Can there be -- and I 
honestly mean this as a question -- a consensus on this?

An alternative is simply requiring the number of interfaces and peers 
to be a part of the test report, so the market/customer will demand 
comparable reports on competitive routers.

>
>
>>  One other thing,we're still not including forwarding. The reason
>>  is we want
>>  to clearly define the aspects of  convergence in the routing plane
before
>>  the next step- combining the convergence measurements to include
>>  the data plane-
>>  which could be the forwarding table plus any other ops relegated
>>  to that dimension. The reason is to isolate at least conceptually
>>  the different effects,
>>   so we can define tests which might measure the same phenomenon
>>  in the least
>>  implementation dependent way.
>
>Totally agree. BTW, in previous tests I've done of convergence, forwarding
>*was* the metric. Your approach of isolating the control-plane function is
>better.
>
>I'm also glad to see you do plan to add forwarding measurements once the
>control-plane metrics are nailed down.
>
>dn
>(who regrets not being in SLC because he's in Asia lecturing on this stuff)
>
>
>>
>>  Padma
>>
>>  On Sat, Dec 08, 2001 at 02:22:59AM -0800, David Newman wrote:
>>  >
>>  > nice sound effects :)
>>  >
>>  > Besides the factors mentioned in the slides, some other
>>  possible issues to
>>  > consider in building test tables:
>>  >
>>  > --prefix length distribution?
>>  > --LOCAL_PREF in use?
>>  > --contiguity of prefixes? (aggregation is good in production, bad in
>>  > testing)
>>  > --for flapping, contiguity of prefixes withdrawn or
>>  readvertised (big impact
>>  > on forwarding in some implementations)
>>  >
>>  > Also, in measuring convergence times, obviously the number of
>>  > boxes/interfaces matters a lot in any convergence measurement.
>>  I'm sure the
>>  > presenters will cover this; I just didn't see it in the slides.
>>  >
>>  > dn
>>  >
>>  > > -----Original Message-----
>>  > > From: [email-masked] [mailto:[email-masked]]On
>>  Behalf Of Kevin
>>  > > Dubray
>>  > > Sent: Friday, December 07, 2001 8:48 AM
>>  > > To: [email-masked]
>>  > > Subject: [bmwg] URL to BGP covergence methodology issues slides.
>>  > >
>>  > >