Subject: [bmwg] draft-varlashkin-router-conv-bench-00

From: Ilya Varlashkin <[email-masked]>
Date: Mon, 24 Oct 2011 23:56:40 +0200

Hello BMWG,

as previously mentioned during IETF81, there's a new draft on convergence benchmarking. The draft is available at http://tools.ietf.org/html/draft-varlashkin-router-conv-bench-00. Authors would like to solicit discussion about the approach described in the draft.

Kind regards,
iLya

---

From: "Scott Poretsky" <[email-masked]>
Date: Mon, 24 Oct 2011 22:00:53 +0000

Ilya and Rajiv,

Is there an associated Terminology document that can be referenced?  What is defined as a "Contemporary Router"?

Scott

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Ilya Varlashkin
Sent: Monday, October 24, 2011 5:57 PM
To: IETF BMWG
Subject: [bmwg] draft-varlashkin-router-conv-bench-00

Hello BMWG,

Kind regards,
iLya

_______________________________________________
bmwg mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/bmwg
##############################################################################################
This message is intended only for the designated recipient(s).It may contain confidential or proprietary information.
If you are not the designated recipient, you may not review, copy or distribute this message.
If you have mistakenly received this message, please notify the sender by a reply e-mail and delete this message. 
Thank you.
##############################################################################################

---

From: Rajiv Papneja <[email-masked]>
Date: Mon, 24 Oct 2011 22:24:38 +0000

Hi Scott,

RFC4098 is the term document for this draft and draft-papneja-bgp-basic-dp-convergence-02.txt (http://tools.ietf.org/html/draft-papneja-bgp-basic-dp-convergence-02). However, this and other new terms are not defined in the published version, and we have been considering extending the term document in near future to define these. 

-Rajiv 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Scott Poretsky
Sent: Monday, October 24, 2011 6:01 PM
To: Ilya Varlashkin; IETF BMWG
Subject: Re: [bmwg] draft-varlashkin-router-conv-bench-00

Ilya and Rajiv,

Scott

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Ilya Varlashkin
Sent: Monday, October 24, 2011 5:57 PM
To: IETF BMWG
Subject: [bmwg] draft-varlashkin-router-conv-bench-00

Hello BMWG,

Kind regards,
iLya

---

From: Jay Karthik <[email-masked]>
Date: Mon, 24 Oct 2011 20:07:13 -0400

Rajiv, Ilya,

Could you include what is out of scope for this document ? As I read this
initial version, I am reminded of the failure detection methodology, ECMP,
concurrent failures, span failure (A single span cut in the fiber network
can cause multiple link failures at other layers).

Do you agree that the node being benchmarked would behave differently for
local and remote failures ?

Would this methodology be applied to both edge and provider nodes ? If not
can these procedures be generalized ?

Using 2 IGPs does not appear to be a common practice and not sure if this
proposal mimics a real world scenario.

Best,
Jay 

On 10/24/11 6:24 PM, "Rajiv Papneja" <[email-masked]> wrote:

> Hi Scott,
> 
> RFC4098 is the term document for this draft and
> draft-papneja-bgp-basic-dp-convergence-02.txt
> (http://tools.ietf.org/html/draft-papneja-bgp-basic-dp-convergence-02).
> However, this and other new terms are not defined in the published version,
> and we have been considering extending the term document in near future to
> define these. 
> 
> -Rajiv 
> 
> as previously mentioned during IETF81, there's a new draft on convergence
> benchmarking. The draft is available at
> http://tools.ietf.org/html/draft-varlashkin-router-conv-bench-00. Authors
> would like to solicit discussion about the approach described in the draft.
> 
> Kind regards,
> iLya
> 
> 
>

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Mon, 24 Oct 2011 22:10:10 -0500

Hi iLya, authors,

I think this is a very useful approach, one that is relevant to
benchmarking that gets closer to deployments and to customer experience
of failure recovery. As I mentioned earlier, I think this should be
pursued.

Scanning through the document, one initial reaction is that the subject
needs very sharp definition of terms and very well demarked scope. Some
examples:

"Convergence benchmarking on contemporary routers"

Contemporary means now() to the reader -- at the same time. What is
"contemporary", especially when "RFCs are forever"?

   This document specifies methodology for benchmarking convergence of
   routers without making assumptions about relation and dependencies

What is "convergence of routers"? how are routers as pieces of equipment
converging? "Convergence of routes and dataplane programming to allow
for traffic to flow"? Or "dataplane benchmark for dynamically routed IP
networks"?

Again, I think this is useful work, and a good start that would benefit
from sharpening.

Thanks,

-- Carlos.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Ilya Varlashkin
Sent: Monday, October 24, 2011 5:57 PM
To: IETF BMWG
Subject: [bmwg] draft-varlashkin-router-conv-bench-00

Hello BMWG,

as previously mentioned during IETF81, there's a new draft on
convergence benchmarking. The draft is available at
http://tools.ietf.org/html/draft-varlashkin-router-conv-bench-00.
Authors would like to solicit discussion about the approach described in
the draft.

Kind regards,
iLya

---

From: Al Morton <[email-masked]>
Date: Tue, 25 Oct 2011 09:35:13 -0400

At 11:10 PM 10/24/2011, Carlos Pignataro (cpignata) wrote:
>...Again, I think this is useful work, and a good start

big +1

>that would benefit from sharpening.
+1, here's another example:

S1 Intro:
...
    When a failure occurs on the network a router needs to:

    1.  select new best path so that the packets, which already arrived
        to the router, can be forwarded

    2.  let other routers know about new network state so they can find
        new best path from their perspective

since the process really starts after detection, I suggest:

    When a router detects a failure on the network, it now must:

    1.  select the new best path so that packets can be forwarded,
        including those which already arrived, and

    2.  let other routers know about new network state so they can find
        new best path from their perspective

Next paragraph starts:
    How fast a router can perform these two functions characterise
    router's performance with regards to convergence.  Note that in...

This is one of your main points, and could be sharpened as:

    How quickly a router can perform these two functions characterizes
    the router's performance with regard to forwarding restoration
    and route convergence.  Note that in...

hope this helps, spell-check will help, too,
Al
(as a participant)

---

From: Ilya Varlashkin <[email-masked]>
Date: Tue, 25 Oct 2011 21:17:01 +0200

Hi Jay,

thank you for your comments. Please see responses inline.

> can cause multiple link failures at other layers).

There are failure scenarios described in Section 5. The intention was to limit tests to those scenarios because they're clear in regards to what's failing and what reaction to expect. If you believe that some scenarios should be changed or more scenarios need to be added, we could discuss this possibility. Please note that the goal of the document is to produce a methodology that depends only on DUT performance, i.e. performance of other systems (however good or bad) either doesn't affect results at all or can be obtained prior the tests and subtracted from overall performance to obtain DUT performance alone. Also, this document should be treated as complementary to draft-papneja-bgp-basic-dp-convergence-02, which contains more test cases.

> local and remote failures ?

Completely agree with you here. That's why we have multiple failure scenarios to cover at least most common cases. Note that failures of links and nodes not directly attached to DUT are treated like: "either DUT has means to monitor non-direct failures (e.g. by using some form of echo-reply protocol) or it will be told by nodes adjacent to it _immediately_ or within measurable interval when failure occurs". From that perspective (immediate or measurable notification from adjacent node) we do not distinguish between e.g. R2-R3 failure at R2 end and at R3 end. But for things within direct reach of the DUT we of course care about local failure and remote failure. I believe "link down with LoS" and "link down without LoS" address this issue. If you think we need to add an extra test, we can consider it.

> can these procedures be generalized ?

It can be applied to any node. Though I'm not sure how much help it would be to test BGP on a P node. Because we're still in the process of developing test and trying to understand how various components affect each other, in sake of deterministic results the current version explicitly says "Provided methodology is primary intended for testing routers running BGP and some form of link-state IGP with or without MPLS.", which should be interpreted as "it may be applied to other cases but that's not focus at the moment" . We can reconsider this once we test robustness, accuracy and feasibility to apply more generic language. What we do not want is to end up with a document that will take half a year to read and a year to wait for the results of the tests (this is not a joke - in current state with all variables we can easily end up with 1 week of machine-aided test run).

> proposal mimics a real world scenario.

DUT itself doesn't run two IGP and doesn't even know that two IGP exist somewhere. And we're not testing part of the network where we have (if we do!) second IGP, but only DUT itself. Moreover, second IGP is not even required (it's not part of "normative" language of the draft, but of "informative") - it's suggested as a method to ensure that convergence depends only on DUT and on DUT alone. Would you agree that such use is valid for our purpose even if you don't find it in "real world"?

If you feel some parts require more clarification, please let authors know.

Kind regards,
iLya

---

From: Rajiv Papneja <[email-masked]>
Date: Tue, 25 Oct 2011 23:04:39 +0000

Hi Carlos,

Always appreciate your input! 

We agree that terminology is needed and as I noted to Scott, we are planning on extending the RFC4098 to include all the terms that we introduce as part of this work. I also agree on sharpening the scope as well especially in the light of draft-papneja-bgp-basic-dp-convergence-02.

By using the term "contemporary," we mean that we are considering routers that we can find/in-use today (or in near future) in production networks and we are primarily focusing on DUT performance here. 

If you feel that there is something that needs to be added to further clarify the confusion, we would be happy to consider. 

Thanks 
Rajiv

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Carlos Pignataro (cpignata)
Sent: Monday, October 24, 2011 11:10 PM
To: Ilya Varlashkin; IETF BMWG
Subject: Re: [bmwg] draft-varlashkin-router-conv-bench-00

Hi iLya, authors,

Thanks,

-- Carlos.

-----Original Message-----

Hello BMWG,

Kind regards,
iLya

---

From: "Jan Novak (janovak)" <[email-masked]>
Date: Thu, 27 Oct 2011 11:31:28 +0200

Hi Ilya,

I had a read and would have few comments on it:

1) it would benefit from a definitions section prior
of the test params/procedures ones - of what is being
tested/measured and the related terms - you use for
example the various Timers in the text which are explained
only later on

2) The ER1 - ERn device - in the current economical 
environment :-) test groups might find it difficult to
dedicate actually standalone physical devices just to
originate few prefixes each - it could be achieved using dot1q
subinterfaces/trunks to one device or better with a simulator
(which I think is available as a function of most modern
test equipment) to generate ISIS/OSPF link state database
of any size or similarly a BGP table with variety of
next hops if that's what you are after here

3) The 5.1-3 tests are well defined and 5.4-5 are certainly
tests of interest in any network but in my opinion they 
introduce to many dependencies on various things which have
no relation to the DUT which is under the test - the
particular topology, the BGP implementation (or how BGP
update or min advert interval are implemented and how updates
are propagated through the several boxes before they get to
the DUT) or link state protocol implementation - how fast
the LSAs will be flooded and how the SPF will be triggered
(although that at least is local to the DUT itself) which as
result might make the measurements of the same DUT incomparable
if different "surrounding" equipment is used.

Jan

The climate of Edinburgh is such that the weak succumb young .... 
and the strong envy them.
                                 Dr. Johnson

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Ilya Varlashkin
Sent: 24 October 2011 22:57
To: IETF BMWG
Subject: [bmwg] draft-varlashkin-router-conv-bench-00

Hello BMWG,

Kind regards,
iLya

---

From: "Jan Novak (janovak)" <[email-masked]>
Date: Thu, 27 Oct 2011 11:35:59 +0200

P.S.

"they" here means the 5.4 and 5.5 tests, the others are fine

---

From: Ilya Varlashkin <[email-masked]>
Date: Tue, 1 Nov 2011 00:10:00 +0100

Hi Jan,

thank you for the review! Please see answers inline.

> example the various Timers in the text which are explained

we intend to do more editorial polishing in subsequent revisions of the draft, particularly to make it more logically organised.

> next hops if that's what you are after here

Although we've indeed assumed that a protocol simulator will be used, the tests are designed so that they can be performed using either physical devices or e.g. open-source software running in virtual machines to act as couple dozen of edge routers (I'll get to whether that's sufficient shortly), other methods are possible. Note that ER1 - ERn devices by themselves are barely involved in the test and do not need to be of high-performance type. 

I understand that finding performance limits may require more ERn routers than could be comfortably setup without simulator. However, let's for a moment consider who will perform tests and why. One category of testers are equipment manufacturers. Their goal is to get as much details about the device as possible - maximum performance under various conditions, bottlenecks and so on. For this type of tests simulator is indispensable. But chances are that this category of testers already have a simulator, perhaps even more than one. Second category are service providers. Their goal is likely to verify that equipment is suitable for their network now and in foreseeable future. But do all service providers need to have same number of ERn? Small networks have fewer resources for their tests but they also have only handful of exit points, for which it's reasonable to setup necessary number of simulated routers using VM or physical routers. Bigger networks need to use bigger test topology, but they're also likely to have more resources available for testing, chances are they even have protocol simulators or can rent them. This are two main categories who will execute tests. So it looks like it's reasonable to assume that testers will have means to execute the tests according to relevant scope. Would you agree?

> if different "surrounding" equipment is used.

It is great that you've brought this up, because this is exactly what authors of the draft would like to sort out as first priority. This topic can potentially be quite big and to keep focus let's separate it into dedicated thread in the follow-ups. Authors of the draft are aiming to minimise if not completely get rid of dependencies of the methodology on variables like you've mentioned. 

For IGP we have already mentioned 'D seconds' in section 5.3, but I see now that we need to tighten loose ends here. We could add extra auxiliary test to measure IGP flood time in control plane and then subtract this time from results obtained in 4.2 and 4.3. Forwarding plane on non-DUT devices appears to be safe from variations due to selection of the metrics in test topology - routers that expect to be in the forwarding path are pre-configured to forward along that path always (i.e. pre-convergence and post-convergence states are the same).

For BGP the draft provides initialisation time (T1) test, which will take care of initial convergence that may indeed be different depending on setup of non-DUT devices but will have no influence on the results. In other words, if in one environment it takes X1 seconds before DUT can successfully deliver packet to NetA but in another environment (due to different configuration or different platform of e.g. R3 or R4) it takes X2 seconds we still expect to see exactly same (modulo accuracy) number of lost packets and therefore same duration of the outage/convergence. We probably will need to refine test in 4.1 to ensure that all involved devices have successfully initialised their control- and data-planes and that it's safe to start the tests 4.2 & 4.3.

Let's now move to BGP exchange at the time of failure. If we require that R4 always uses direct connection to NetA as best path, then in the event of failure where DUT needs to start forwarding over R4 there is no BGP exchange involved, so it looks like we don't have dependency on non-DUT here and different configurations or platforms expected to produce same result for test 4.2. With test 4.3 things appear to be even more clear - the only BGP session [S] has is the one with DUT, so we don't expect [S] to send any withdrawal messages that could slow down pace at which DUT is sending new network info and thus we have no dependency on MRAI configuration unless it's on the DUT itself.

What do you (and others in BMWG) think about described approach, would that address your concerns?

Kind regards,
iLya