Subject: Re: [bmwg] benchmarking of routing protocols modern routers

From: Al Morton <[email-masked]>
Date: Wed, 13 Jul 2011 12:19:03 -0400

I want to hear from all interested parties, but I'll
add my brief thoughts now:

All our dataplane/black-box measurements measure packet
transfer characteristics and *infer* something from those
measurements, like convergence time.  We try to strengthen
the inference through test design and configuration, yet there
are many actions possible inside a black-box that could restore a
datapath after our procedures provide the intended stimulus.

It appears to me that we should now recognize and own the problem.
There are cases where the inference may break down and something else
was measured, so we could recognize this possibility as a source
of experimental error and give the topic the treatment it deserves,
including some explanation in scope section(s) where applicable.

we'll all keep learning together,
Al
bmwg chair

At 10:10 AM 6/30/2011, Rajiv Papneja wrote:
>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]] On Behalf Of Ilya
>Varlashkin
>Sent: Tuesday, June 28, 2011 6:45 PM
>To: Rajiv Papneja; IETF BMWG; [email-masked]
>Subject: Re: [bmwg] benchmarking of routing protocols modern routers
>
> >> We will try to clarify things in the next version based on AIs from
> >> the last meeting and ML discussions, but question remains that
> >> whether we prioritize control plane benchmarking over data plane or we
>look for a hybrid approach.
> >> May be Al can comment.
> >>
> >
> > I would suggest renaming draft to "control-plane convergence" (i.e.
>removing "BGP" from it) and stating in Intro what exactly are you trying to
>measure.
> >
>
>Correction - I've meant to say "data/forwarding-plane convergence".
>
>[RP]  I believe that we should let the group, and the group leadership
>decide what this work should be called. We had presented the work before we
>submitted the draft in IETF 76 and 77 and in those meetings the discussions
>were limited to the prioritization of control vs. data plane benchmarking.
>AFAIR, the discussion on changing the motivation or not calling this as BGP
>convergence did not arise. We are flexible and would want to produce work
>which is useful to the carrier community, and we do not have any
>reservations against calling it "data/forwarding-plane convergence," but
>would like to hear opinion of other SPs, fellow BMWGers and the chair.
>
>I really thank you for bringing this discussion up. We respect your opinion
>and would like to work with you in getting this where it should be. We have
>a lot of test cases and this set  should produce a great benchmark for
>establishing a fair playing field for different vendors products.
>
>Rgds,
>/Rajiv
>
>/iLya
>
>_______________________________________________
>bmwg mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/bmwg
>
>_______________________________________________
>bmwg mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/bmwg

---

From: Carlos Pignataro <[email-masked]>
Date: Wed, 13 Jul 2011 14:42:08 -0400

Al, all,

My 2¢ say that the point raised by Ilya is quite valid, and a corollary
is that the inference can be qualified (and sometimes bounded). I think
that one of the main points, is that the inference-introduced error is
different for different protocol test cases, and that BGP in particular
is one that is further from the assumption than others.

I also think that it is very important to measure data-plane black-box
scenarios, as that is observable downtime; it is also important in those
methodologies to specify the conditions/requirements, what is being
measured, and what other variables contribute to the delta.

In summary, I agree that BMWG should own the issue. In a way, that's
what we did (in the reverse direction) with RFC5695, measuring
forwarding but describing the effect of dynamic protocols.

Thanks,

-- Carlos.

---

From: Ilya Varlashkin <[email-masked]>
Date: Wed, 13 Jul 2011 22:46:39 +0200

Al,

I had discussion with Rajiv and Bhavani and we came to a conclusion to revise existing BGP benchmarking draft to be more clear regarding what's being tested. It's likely that we won't make update before submission freeze begins but we have a plan regarding future development of the draft and it hopefully will be presented during up coming IETF by either Rajiv or Bhavani (unfortunately I won't make it to Canada but hope to be there in Taiwan).

Recently I was busy measuring convergence for our network and some procedures look quite generic to be applicable for most (if not all) networks. It is clear that modern routers can restore traffic flow before protocols converge, and updated tests will account for that. Can't say more at the moment because that would be speculation. So let's way for short while and we will bring update when there's something solid.

Cheers,
iLya

/iLya

---

From: Al Morton <[email-masked]>
Date: Thu, 14 Jul 2011 07:55:54 -0400

At 04:46 PM 7/13/2011, Ilya Varlashkin wrote:
>I had discussion with Rajiv and Bhavani and we came to a conclusion 
>to revise existing BGP benchmarking draft to be more clear regarding 
>what's being tested.

That's a very useful outcome, thanks to you all.

>It's likely that we won't make update before submission freeze 
>begins but we have a plan regarding future development of the draft 
>and it hopefully will be presented during up coming IETF by either 
>Rajiv or Bhavani (unfortunately I won't make it to Canada but hope 
>to be there in Taiwan).

Note that draft submission opens again on Monday of IETF week,
but the presentation to summarize the discussion on the list
and the proposed path forward will be welcome during BMWG's session.
It will give folks who have followed along another opportunity for input.

regards,
Al
bmwg chair

---

From: "Rajiv Papneja" <[email-masked]>
Date: Thu, 14 Jul 2011 10:10:02 -0400

> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Al Morton
> Sent: Thursday, July 14, 2011 7:56 AM
> To: Ilya Varlashkin
> Cc: [email-masked]; 'IETF BMWG'; Rajiv Papneja
> >I had discussion with Rajiv and Bhavani and we came to a conclusion to
> >revise existing BGP benchmarking draft to be more clear regarding
> >what's being tested.
> 

[RP] Yes it is! Thanks to you as well for encouraging feedback from the
group. We are clarifying items in the next revision.
> 
> >It's likely that we won't make update before submission freeze begins
> >but we have a plan regarding future development of the draft and it
> >hopefully will be presented during up coming IETF by either Rajiv or
> >Bhavani (unfortunately I won't make it to Canada but hope to be there
> >in Taiwan).
> 
> Note that draft submission opens again on Monday of IETF week, but the
> presentation to summarize the discussion on the list and the proposed path
> forward will be welcome during BMWG's session.
[RP] 
[RP] Certainly Al, this is what we planned on doing. We are talking again
next Monday to produce an update for the group and seek additional comments.
We have so far agreed to produce probably two drafts but let us see how it
goes.

Regards,
/Rajiv