Subject: Re: [bmwg] draft-varlashkin-router-conv-bench-00

From: "Jan Novak (janovak)" <[email-masked]>
Date: Tue, 13 Dec 2011 16:22:19 +0100

Hi,

>This are two main categories who will execute tests. So it looks like
it's >reasonable to assume that testers will have means to execute the
tests >according to relevant scope. Would you agree?

Yes, it was just the wording in the text with just the physical
devices, otherwise yes, it gives the freedom to do what everybody
needs to do.

> What do you (and others in BMWG) think about described approach, would
that address your concerns?

5.4 Explicitly requires a BGP exchange - the non-DUT has to initiate 
a BGP exchange and that may depend on the non-DUT - maybe easiest this
could be circumvented by requiring the other BGP peer is from the same
vendor/same implementation or tested always with the same one
when comparing two DUT implementation ?? Although maybe it is covered
already by the requirement the update is immediate - so as long as this
is considered it is ok.

Jan

The climate of Edinburgh is such that the weak succumb young .... 
and the strong envy them.
                                 Dr. Johnson

-----Original Message-----
From: Ilya Varlashkin [mailto:[email-masked]] 
Sent: 31 October 2011 23:10
To: Jan Novak (janovak)
Cc: IETF BMWG
Subject: Re: [bmwg] draft-varlashkin-router-conv-bench-00

Hi Jan,

thank you for the review! Please see answers inline.

On Oct 27, 2011, at 11:31 , Jan Novak (janovak) wrote:

> Hi Ilya,
> 
> I had a read and would have few comments on it:
> 
> 1) it would benefit from a definitions section prior
> of the test params/procedures ones - of what is being
> tested/measured and the related terms - you use for
> example the various Timers in the text which are explained
> only later on
> 

we intend to do more editorial polishing in subsequent revisions of the
draft, particularly to make it more logically organised.

> 2) The ER1 - ERn device - in the current economical 
> environment :-) test groups might find it difficult to
> dedicate actually standalone physical devices just to
> originate few prefixes each - it could be achieved using dot1q
> subinterfaces/trunks to one device or better with a simulator
> (which I think is available as a function of most modern
> test equipment) to generate ISIS/OSPF link state database
> of any size or similarly a BGP table with variety of
> next hops if that's what you are after here
> 

Although we've indeed assumed that a protocol simulator will be used,
the tests are designed so that they can be performed using either
physical devices or e.g. open-source software running in virtual
machines to act as couple dozen of edge routers (I'll get to whether
that's sufficient shortly), other methods are possible. Note that ER1 -
ERn devices by themselves are barely involved in the test and do not
need to be of high-performance type. 

I understand that finding performance limits may require more ERn
routers than could be comfortably setup without simulator. However,
let's for a moment consider who will perform tests and why. One category
of testers are equipment manufacturers. Their goal is to get as much
details about the device as possible - maximum performance under various
conditions, bottlenecks and so on. For this type of tests simulator is
indispensable. But chances are that this category of testers already
have a simulator, perhaps even more than one. Second category are
service providers. Their goal is likely to verify that equipment is
suitable for their network now and in foreseeable future. But do all
service providers need to have same number of ERn? Small networks have
fewer resources for their tests but they also have only handful of exit
points, for which it's reasonable to setup necessary number of simulated
routers using VM or physical routers. Bigger networks need to use bigger
test topology, but they're also likely to have more resources available
for testing, chances are they even have protocol simulators or can rent
them. This are two main categories who will execute tests. So it looks
like it's reasonable to assume that testers will have means to execute
the tests according to relevant scope. Would you agree?

> 3) The 5.1-3 tests are well defined and 5.4-5 are certainly
> tests of interest in any network but in my opinion they 
> introduce to many dependencies on various things which have
> no relation to the DUT which is under the test - the
> particular topology, the BGP implementation (or how BGP
> update or min advert interval are implemented and how updates
> are propagated through the several boxes before they get to
> the DUT) or link state protocol implementation - how fast
> the LSAs will be flooded and how the SPF will be triggered
> (although that at least is local to the DUT itself) which as
> result might make the measurements of the same DUT incomparable
> if different "surrounding" equipment is used.
> 

It is great that you've brought this up, because this is exactly what
authors of the draft would like to sort out as first priority. This
topic can potentially be quite big and to keep focus let's separate it
into dedicated thread in the follow-ups. Authors of the draft are aiming
to minimise if not completely get rid of dependencies of the methodology
on variables like you've mentioned. 

For IGP we have already mentioned 'D seconds' in section 5.3, but I see
now that we need to tighten loose ends here. We could add extra
auxiliary test to measure IGP flood time in control plane and then
subtract this time from results obtained in 4.2 and 4.3. Forwarding
plane on non-DUT devices appears to be safe from variations due to
selection of the metrics in test topology - routers that expect to be in
the forwarding path are pre-configured to forward along that path always
(i.e. pre-convergence and post-convergence states are the same).

For BGP the draft provides initialisation time (T1) test, which will
take care of initial convergence that may indeed be different depending
on setup of non-DUT devices but will have no influence on the results.
In other words, if in one environment it takes X1 seconds before DUT can
successfully deliver packet to NetA but in another environment (due to
different configuration or different platform of e.g. R3 or R4) it takes
X2 seconds we still expect to see exactly same (modulo accuracy) number
of lost packets and therefore same duration of the outage/convergence.
We probably will need to refine test in 4.1 to ensure that all involved
devices have successfully initialised their control- and data-planes and
that it's safe to start the tests 4.2 & 4.3.

Let's now move to BGP exchange at the time of failure. If we require
that R4 always uses direct connection to NetA as best path, then in the
event of failure where DUT needs to start forwarding over R4 there is no
BGP exchange involved, so it looks like we don't have dependency on
non-DUT here and different configurations or platforms expected to
produce same result for test 4.2. With test 4.3 things appear to be even
more clear - the only BGP session [S] has is the one with DUT, so we
don't expect [S] to send any withdrawal messages that could slow down
pace at which DUT is sending new network info and thus we have no
dependency on MRAI configuration unless it's on the DUT itself.

What do you (and others in BMWG) think about described approach, would
that address your concerns?

Kind regards,
iLya

---

From: "iLya" <[email-masked]>
Date: Tue, 13 Dec 2011 17:04:57 +0100

Hi Jan,

--------------------------------------------------
> devices, otherwise yes, it gives the freedom to do what everybody

If you or someone else have an alternative wording, I'll be happy to 
consider it. Original text was not meant to confuse anyone but it is 
probably result of me not being native english speaker.

> already by the requirement the update is immediate - so as long as this

I don't think it's necessary to require same implementation/vendor as in my 
opinion it does not remove variable but only obfuscates it. If "immediate" 
is not sufficient requirement we could perform yet another auxilliary test 
of device sending the update using control plane (4.3) and measure precisely 
how much time passes between we hit "WITHDRAW NOW" button and WITHDRAW being 
actually sent.

Kind regards,
iLya