Subject: Re: [bmwg] Feedback for Benchmarking Methodology for OpenFlow SDN Controller Performance

From: "Castelli, Brian" <[email-masked]>
Date: Wed, 4 Jun 2014 15:28:42 +0000

I forgot one comment…

Section 6.1.1.1.5 reporting format:

The last sentence of the first paragraph says, “The test report MUST indicate whether learning was enabled or disabled.” What type of learning does this requirement refer to? And, if it’s important, shouldn’t it be a Setup Parameter listed in section 6.1.1.1.2?

From: <Castelli>, Brian Castelli <[email-masked]<mailto:[email-masked]>>
Date: Wednesday, June 4, 2014 at 11:15 AM
To: "Bhuvan (Veryx Technologies)" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Cc: 'Anton Basil' <[email-masked]<mailto:[email-masked]>>, 'Vishwas Manral' <[email-masked]<mailto:[email-masked]>>
Subject: Re: [bmwg] Feedback for Benchmarking Methodology for OpenFlow SDN Controller Performance

Bhuvan,

Thank you for the response and the advice on how to best participate in the discussion. I have a few follow-up comments/questions:

Section 6.1.1.1.3 Procedure:

How is it that we are going to assist the person running the test at creating the proper packet-in messages? What are the characteristics of the packet-ins that will make this test successful?

[Bhuvan] To make the test successful, we need to simulate packet-in messages for each flow. Typically flows are identified using dst.mac and dst.ip.

[Brian] Is this well known to test authors? Should the document provide guidance about how to create packet_in messages that correspond to the desired flows?

  *   In the reactive mode setup, there is no mention of the rate at which packet-ins are sent, but the test result can be skewed by the rate. If I send only one packet-in per second, for example, the FSR can’t be faster than one flow per second. I think we should consider specifying the rate at which the packet-ins are sent to the controller, iterating over packet-in rates, iterating over numbers of switches.

[Bhuvan] We agree with your comment. We are planning to define the minimum number of packet-ins that need to be sent on each OpenFlow connection (typically greater than 1). But the assumption is that we need to send as many packet-in messages as possible on an established OpenFlow connection to measure the FSR.

 [Brian] I think it would be interesting to iterate over different packet_in rates. I would expect controller response time to degrade with increased packet_in load. An iteration would allow us to determine how well the controller scales and how gracefully response time degrades.

  *   Without the iteration I have suggested, above, I don’t understand why the test should be repeated multiple times? Do we not expect the test results to be consistent between test runs?

[Bhuvan] We expect slight variations across test runs. Hence we are repeating the test multiple times to provide tester a reliable metric.

[Brian] Personal experience says that testers do not care about slight variations. They want to run the test and get an answer, not run the test 10 times and find that variation is less than 1% across all tests. It is far more interesting for a tester when there is a variable that changes between iterations, as I have suggested, above. Running the same exact test multiple times is usually a waste of test resources.

Section 6.1.1.1.4 Measurement:

  *   FSRi equation is incorrect. FSR = number of flows/time, not OFR/time.

[Bhuvan] I would like to provide some more clarification on this equation. Here, the number of flows = number of packet-in messages sent to the controller. We have conducted this test with various controllers and observed that the number of OF responses received from the controller is always lesser than the number of packet-in messages sent to the controller due to the TCP window restrictions. Hence we recommend, FSR should be measured based on the OF responses received from the controller. The FSR equation will therefore be OF responses/time.

 [Brian] This is my mistake. I jumped to the conclusion that ‘R’ in OFR was rate. Looking back at section 2, Terminology, OFR = OpenFlow responses. I thought the equation was rate/time. My bad. I withdraw the comment.

E-mail confidentiality.
--------------------------------
This e-mail contains confidential and / or privileged information belonging to Spirent Communications plc, its affiliates and / or subsidiaries. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution and / or the taking of any action based upon reliance on the contents of this transmission is strictly forbidden. If you have received this message in error please notify the sender by return e-mail and delete it from your system.

Spirent Communications plc
Northwood Park, Gatwick Road, Crawley, West Sussex, RH10 9XN, United Kingdom.
Tel No. +44 (0) 1293 767676
Fax No. +44 (0) 1293 767677

Registered in England Number 470893
Registered at Northwood Park, Gatwick Road, Crawley, West Sussex, RH10 9XN, United Kingdom.

Or if within the US,

Spirent Communications,
26750 Agoura Road, Calabasas, CA, 91302, USA.
Tel No. 1-818-676- 2300

---

From: "Bhuvan \(Veryx Technologies\)" <[email-masked]>
Date: Thu, 5 Jun 2014 21:08:15 +0530

Dear Brain,

Please find my responses inline to your comments and queries.

Thanks,

Description: Description: Description: Description: Description:
Description: Description: Description: Description: Veryx-Logo for Webinar

Bhuvan | Cloud and SDN Solutions

Phone: +91 44 4567 2222 x242

Mobile: +91 988 428 6693 

 <http://www.veryxtech.com/> www.veryxtech.com

From: Castelli, Brian [mailto:[email-masked]] 
Sent: Wednesday, June 04, 2014 8:59 PM
To: Castelli, Brian; Bhuvan (Veryx Technologies); [email-masked]
Cc: 'Anton Basil'; 'Vishwas Manral'
Subject: Re: [bmwg] Feedback for Benchmarking Methodology for OpenFlow SDN
Controller Performance

I forgot one comment.

The last sentence of the first paragraph says, "The test report MUST
indicate whether learning was enabled or disabled." What type of learning
does this requirement refer to? And, if it's important, shouldn't it be a
Setup Parameter listed in section 6.1.1.1.2?

[Bhuvan] Learning here indicates MAC/IP address learning. As suggested, this
can be made as setup parameters.

From: <Castelli>, Brian Castelli <[email-masked]>
Date: Wednesday, June 4, 2014 at 11:15 AM
To: "Bhuvan (Veryx Technologies)"
<[email-masked]>, "[email-masked]" <[email-masked]>
Cc: 'Anton Basil' <[email-masked]>, 'Vishwas Manral'
<[email-masked]>
Subject: Re: [bmwg] Feedback for Benchmarking Methodology for OpenFlow SDN
Controller Performance

Bhuvan,

Thank you for the response and the advice on how to best participate in the
discussion. I have a few follow-up comments/questions:

How is it that we are going to assist the person running the test at
creating the proper packet-in messages? What are the characteristics of the
packet-ins that will make this test successful?

[Bhuvan] To make the test successful, we need to simulate packet-in messages
for each flow. Typically flows are identified using dst.mac and dst.ip.

[Brian] Is this well known to test authors? Should the document provide
guidance about how to create packet_in messages that correspond to the
desired flows?

[Bhuvan] We can provide an example on how to create packet_in messages for a
desired flows.

 In the reactive mode setup, there is no mention of the rate at which
packet-ins are sent, but the test result can be skewed by the rate. If I
send only one packet-in per second, for example, the FSR can't be faster
than one flow per second. I think we should consider specifying the rate at
which the packet-ins are sent to the controller, iterating over packet-in
rates, iterating over numbers of switches.

[Bhuvan] We agree with your comment. We are planning to define the minimum
number of packet-ins that need to be sent on each OpenFlow connection
(typically greater than 1). But the assumption is that we need to send as
many packet-in messages as possible on an established OpenFlow connection to
measure the FSR.

 [Brian] I think it would be interesting to iterate over different packet_in
rates. I would expect controller response time to degrade with increased
packet_in load. An iteration would allow us to determine how well the
controller scales and how gracefully response time degrades.

[Bhuvan] We agree with your feedback. We have covered this scenario as part
of section
<http://tools.ietf.org/html/draft-bhuvan-bmwg-of-controller-benchmarking-00#
section-6.1.1.2> 6.1.1.2  - Flow setup rate under incremental network load.

*	Without the iteration I have suggested, above, I don't understand
why the test should be repeated multiple times? Do we not expect the test
results to be consistent between test runs?

[Bhuvan] We expect slight variations across test runs. Hence we are
repeating the test multiple times to provide tester a reliable metric.

[Brian] Personal experience says that testers do not care about slight
variations. They want to run the test and get an answer, not run the test 10
times and find that variation is less than 1% across all tests. It is far
more interesting for a tester when there is a variable that changes between
iterations, as I have suggested, above. Running the same exact test multiple
times is usually a waste of test resources.

[Bhuvan] For testing controller functionality, the obtained results would be
consistent across iterations. But for the performance tests, the behavior of
system would not be consistent and we recommend iterating the tests for
atleast 3 times to provide tester a reliable metric.

*	FSRi equation is incorrect. FSR = number of flows/time, not
OFR/time.

[Bhuvan] I would like to provide some more clarification on this equation.
Here, the number of flows = number of packet-in messages sent to the
controller. We have conducted this test with various controllers and
observed that the number of OF responses received from the controller is
always lesser than the number of packet-in messages sent to the controller
due to the TCP window restrictions. Hence we recommend, FSR should be
measured based on the OF responses received from the controller. The FSR
equation will therefore be OF responses/time.

 [Brian] This is my mistake. I jumped to the conclusion that 'R' in OFR was
rate. Looking back at section 2, Terminology, OFR = OpenFlow responses. I
thought the equation was rate/time. My bad. I withdraw the comment.

E-mail confidentiality.
--------------------------------
This e-mail contains confidential and / or privileged information belonging
to Spirent Communications plc, its affiliates and / or subsidiaries. If you
are not the intended recipient, you are hereby notified that any disclosure,
copying, distribution and / or the taking of any action based upon reliance
on the contents of this transmission is strictly forbidden. If you have
received this message in error please notify the sender by return e-mail and
delete it from your system.

Spirent Communications plc
Northwood Park, Gatwick Road, Crawley, West Sussex, RH10 9XN, United
Kingdom.
Tel No. +44 (0) 1293 767676
Fax No. +44 (0) 1293 767677

Registered in England Number 470893
Registered at Northwood Park, Gatwick Road, Crawley, West Sussex, RH10 9XN,
United Kingdom.

Or if within the US, 

Spirent Communications,

---

From: "Castelli, Brian" <[email-masked]>
Date: Thu, 19 Jun 2014 04:18:19 +0000

I am sorry for the delay in responding. I have many additional comments on the draft, but I’d like to pop up a level before going into the specific details. In this message I will limit my comments to three high-level issues.

First, we must have an agreement on the terms and definitions used by a family of SDN benchmarking specifications. The definitions section of the current document has only two entries, yet there are terms used throughout the document that are defined in place or by context. As we produce a family of benchmarking specifications for SDN, all related documents must draw from a common set of definitions to avoid confusion and duplication. We should, therefore, produce a draft of definitions that can be referred to by this and subsequent benchmark specifications.

Second, we must ensure that the collection of tests in the draft are appropriate. Tests 6.1.1 and 6.1.2 are inverse of one another and therefore redundant. We should consolidate them into a single family of controller-response tests and simplify. Test 6.1.3 does not adequately isolate the controller in the test because the overall response time depends on the response time of the switches. If a tester runs the same test on the same controller with different switches, he or she will get different answers. The goal of a benchmark test is to enable apples-to-apples comparisons between controllers. I think a step back to reconsider the collection of proposed tests is in order.

Third, we must consider the work currently being done by other standards bodies. The Open Networking Foundation Testing & Interop Working Group, for example, is currently developing it’s own OpenFlow Controller benchmarks. I would prefer a combined solution if possible.

The draft under consideration should not go further until these issues are addressed.

From: "Bhuvan (Veryx Technologies)" <[email-masked]<mailto:[email-masked]>>
Date: Thursday, June 5, 2014 at 11:38 AM
To: Brian Castelli <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Cc: 'Anton Basil' <[email-masked]<mailto:[email-masked]>>, 'Vishwas Manral' <[email-masked]<mailto:[email-masked]>>
Subject: RE: [bmwg] Feedback for Benchmarking Methodology for OpenFlow SDN Controller Performance

Dear Brain,

Thanks,

[Description: Description: Description: Description: Description: Description: Description: Description: Description: Veryx-Logo for Webinar]

Bhuvan | Cloud and SDN Solutions
Phone: +91 44 4567 2222 x242
Mobile: +91 988 428 6693
www.veryxtech.com<http://www.veryxtech.com/>

To: Castelli, Brian; Bhuvan (Veryx Technologies); [email-masked]<mailto:[email-masked]>

I forgot one comment…

The last sentence of the first paragraph says, “The test report MUST indicate whether learning was enabled or disabled.” What type of learning does this requirement refer to? And, if it’s important, shouldn’t it be a Setup Parameter listed in section 6.1.1.1.2?
[Bhuvan] Learning here indicates MAC/IP address learning. As suggested, this can be made as setup parameters.

Bhuvan,

[Bhuvan] We can provide an example on how to create packet_in messages for a desired flows.
 In the reactive mode setup, there is no mention of the rate at which packet-ins are sent, but the test result can be skewed by the rate. If I send only one packet-in per second, for example, the FSR can’t be faster than one flow per second. I think we should consider specifying the rate at which the packet-ins are sent to the controller, iterating over packet-in rates, iterating over numbers of switches.
[Bhuvan] We agree with your comment. We are planning to define the minimum number of packet-ins that need to be sent on each OpenFlow connection (typically greater than 1). But the assumption is that we need to send as many packet-in messages as possible on an established OpenFlow connection to measure the FSR.
 [Brian] I think it would be interesting to iterate over different packet_in rates. I would expect controller response time to degrade with increased packet_in load. An iteration would allow us to determine how well the controller scales and how gracefully response time degrades.
[Bhuvan] We agree with your feedback. We have covered this scenario as part of section 6.1.1.2<http://tools.ietf.org/html/draft-bhuvan-bmwg-of-controller-benchmarking-00#section-6.1.1.2>  - Flow setup rate under incremental network load.

[Bhuvan] For testing controller functionality, the obtained results would be consistent across iterations. But for the performance tests, the behavior of system would not be consistent and we recommend iterating the tests for atleast 3 times to provide tester a reliable metric.

E-mail confidentiality.

Or if within the US,

Spirent Communications,

---

From: "Banks, Sarah" <[email-masked]>
Date: Thu, 19 Jun 2014 07:54:19 -0500

Hi Brian,
	Let me address inline.

>
>Second, we must ensure that the collection of tests in the draft are
>appropriate. Tests 6.1.1 and 6.1.2 are inverse of one another and
>therefore redundant. We should consolidate them into a single family of
>controller-response tests and simplify.

SB// I don't know that I agree with this; inverse tests aren't redundant
in my opinion; I'm not sure where the value is in consolidating them into
a family (is it not your position that they're redundant? That would imply
removing one or the other..). I have no issue with them being separate
tests, in the same section, which they are. To my eye, that's perfectly
readable.

>Test
> 6.1.3 does not adequately isolate the controller in the test because the
>overall response time depends on the response time of the switches. If a
>tester runs the same test on the same controller with different switches,
>he or she will get different answers.
> The goal of a benchmark test is to enable apples-to-apples comparisons
>between controllers. I think a step back to reconsider the collection of
>proposed tests is in order.

SB// Perhaps this could be a variable called out; I tend to agree, if the
purpose is to test the controller and not the switches, this should be
documented as part of the topology at the "start of the hour", so that the
only equipment being tested is the controller.

>
>
>
>Third, we must consider the work currently being done by other standards
>bodies. The Open Networking Foundation Testing & Interop Working Group,
>for example, is currently developing it’s own OpenFlow Controller
>benchmarks. I would prefer a combined solution
> if possible.
>

I partially agree, and I believe the authors might as well, in that we
know other bodies are working on OpenFlow, for example. I do think some
understanding of what's happening within those organizations would be an
excellent starting point. However, I don't believe that this issue, with
the 2 aforementioned issues, are a reason to stop proceeding with the
document. If, through the process of working through this, we determine as
a group, or the authors determine on their own, that what other groups are
doing is sufficient, we can make that call at that time.  It's very early
in the draft, and I wouldn't mind seeing where this takes us as a group,
particularly if we can have some form of communication or understanding
with other groups throughout the industry who might be looking at this
problem in some way.

Kind regards,
Sarah

---

From: "Bhuvan \(Veryx Technologies\)" <[email-masked]>
Date: Thu, 19 Jun 2014 18:59:59 +0530

Hi Sarah,

Thanks for your response. I thought of providing similar response to Brain.

Brain,

Hope Sarah's response provides the required clarity. Please get back for additional clarifications and comments.

Best Regards,
Bhuvan

-----Original Message-----
From: Banks, Sarah [mailto:[email-masked]] 
Sent: Thursday, June 19, 2014 6:24 PM
To: Castelli, Brian; Bhuvan (Veryx Technologies); [email-masked]
Cc: 'Anton Basil'; 'Vishwas Manral'
Subject: Re: [bmwg] Feedback for Benchmarking Methodology for OpenFlow SDN Controller Performance

Hi Brian,
	Let me address inline.

>controller-response tests and simplify.

SB// I don't know that I agree with this; inverse tests aren't redundant in my opinion; I'm not sure where the value is in consolidating them into a family (is it not your position that they're redundant? That would imply removing one or the other..). I have no issue with them being separate tests, in the same section, which they are. To my eye, that's perfectly readable.

>Test
> 6.1.3 does not adequately isolate the controller in the test because 
>the overall response time depends on the response time of the switches. 
>If a tester runs the same test on the same controller with different 
>switches, he or she will get different answers.
>between controllers. I think a step back to reconsider the collection 
>of proposed tests is in order.

SB// Perhaps this could be a variable called out; I tend to agree, if the purpose is to test the controller and not the switches, this should be documented as part of the topology at the "start of the hour", so that the only equipment being tested is the controller.

>
>
>
>Third, we must consider the work currently being done by other 
>standards bodies. The Open Networking Foundation Testing & Interop 
>Working Group, for example, is currently developing it’s own OpenFlow 
>Controller benchmarks. I would prefer a combined solution  if possible.
>

I partially agree, and I believe the authors might as well, in that we know other bodies are working on OpenFlow, for example. I do think some understanding of what's happening within those organizations would be an excellent starting point. However, I don't believe that this issue, with the 2 aforementioned issues, are a reason to stop proceeding with the document. If, through the process of working through this, we determine as a group, or the authors determine on their own, that what other groups are doing is sufficient, we can make that call at that time.  It's very early in the draft, and I wouldn't mind seeing where this takes us as a group, particularly if we can have some form of communication or understanding with other groups throughout the industry who might be looking at this problem in some way.

Kind regards,
Sarah

---

From: "Castelli, Brian" <[email-masked]>
Date: Sat, 21 Jun 2014 00:47:34 +0000

Thank you, Sarah.

The specific inverse example that I wrote about is a literal inverse with
considerable overlap between test cases. If I measure the rate, I can do
1/rate to get the time. If I measure the time, I can do 1/time to get the
rate. I see no point in putting a tester through both sets of tests in
their current form. Plus, as I wrote about in my previous set of comments,
there are places in the test cases where the tester is urged to repeat the
same test multiple times without changing test conditions and then plot
the result on a graph. This is an example of extra work that provides
little or no value to the tester. We can accomplish the goals of 6.1.1 and
6.1.2 with a test case or two that use 40% fewer words in the document.

My two main concerns are getting the terminology right and making sure
that the proposed set of tests will achieve our goals. I believe those
goals ought to include a common taxonomy, enabling apples-to-apples
comparisons, and minimizing the time/work required to execute the test
cases.

I am willing to help improve the benchmark. I understand that it is an
early draft. The time is right to make sure that we are developing the
best specification that we can.

How do we continue this process? Via email to this list? I am willing to
help.

On 6/19/14, 8:54 AM, "Banks, Sarah" <[email-masked]> wrote:

>>for example, is currently developing it¹s own OpenFlow Controller

E-mail confidentiality.

Or if within the US,

Spirent Communications,

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Sat, 21 Jun 2014 08:30:15 -0400

Hi Brian, Sarah, Bhuvan, and all,

a couple of quick answers and a comment:

> From: bmwg [mailto:[email-masked]] On Behalf Of Castelli, Brian
...
> comparisons, and minimizing the time/work required to execute the test

Great, that's exactly what we are chartered to do in BMWG.
http://datatracker.ietf.org/wg/bmwg/charter/

> How do we continue this process? Via email to this list? I am willing to

It's principally e-mail (just as you've been doing) and face-to-face meetings
three times a year.  Sarah mentions that our next meeting is in her home town:
http://www.ietf.org/meeting/90/index.html

Make some text proposals for the draft, and we can discuss them on the list.

Having said that, and recognizing that there appears to be a comparable 
activity in ONF that (I assume) is OpenFlow-specific, perhaps a way to 
add value to the industry is to approach the SDN controller problem
more generically, such that different controller designs performing the 
same tasks could be benchmarked as black-boxes.  I realize this approach
has substantial implications for the draft, but the benefit of wider 
applicability.

food for thought,
Al
(as a participant)

---

From: "Bhuvan \(Veryx Technologies\)" <[email-masked]>
Date: Mon, 23 Jun 2014 20:23:18 +0530

Hi Al,

Thanks for your comment. I do agree with you that there needs to be a common
benchmarking mechanism for controller designs performing the same tasks.
But the technology is still relatively immature and the approach has various
dimensions including centralized control vs distributed control (controller
less).
Again the centralized control approach uses different programming methods
(OpenFlow vs Non OpenFlow).

Having said that, for better usability/understanding of the draft and to
avoid misinterpretation, I personally feel it would be good to have separate
benchmarking methodology for each approach (though the metrics remain same).
Also we would be happy to continue the effort to extend the same metrics for
other approaches too.

Please let me know if I've missed something.

Sarah/Brain, please also share your thoughts on this.

Thanks,
Bhuvan

-----Original Message-----
From: MORTON, ALFRED C (AL) [mailto:[email-masked]] 
Sent: Saturday, June 21, 2014 6:00 PM
To: Castelli, Brian; Banks, Sarah; Bhuvan (Veryx Technologies);
[email-masked]
Cc: 'Anton Basil'; 'Vishwas Manral'
Subject: RE: [bmwg] Feedback for Benchmarking Methodology for OpenFlow SDN
Controller Performance

...

> How do we continue this process? Via email to this list? I am willing 
> to help.

It's principally e-mail (just as you've been doing) and face-to-face
meetings three times a year.  Sarah mentions that our next meeting is in her
home town:
http://www.ietf.org/meeting/90/index.html

Having said that, and recognizing that there appears to be a comparable
activity in ONF that (I assume) is OpenFlow-specific, perhaps a way to add
value to the industry is to approach the SDN controller problem more
generically, such that different controller designs performing the same
tasks could be benchmarked as black-boxes.  I realize this approach has
substantial implications for the draft, but the benefit of wider
applicability.

food for thought,
Al
(as a participant)

---

From: "Castelli, Brian" <[email-masked]>
Date: Mon, 23 Jun 2014 15:39:16 +0000

I really like Al¹s suggestion to generalize SDN controller benchmarking
because 1) the ONF effort is OpenFlow specific and 2) OpenFlow does not
define SDN. OpenFlow is one of several southbound protocols that can be
used between a controller function and network node functions. Regardless
of the maturity of current OpenFlow implementations, I believe that we can
define a family of benchmarks that have general applicability to SDN
environments because there are common functions/services that will be
provided independent of protocol. At the end of the day, protocols are
commodities. It¹s the service they provide that adds value.

The current draft contains many of the elements required to sufficiently
define a benchmarking methodology for a controller function in an SDN
environment. We should follow Al¹s suggestion, define the taxonomy used in
the draft, and consolidate wording to make the document easier to read and
apply.

What follows is an attempt to define a higher-level set of SDN controller
test cases. It builds on the work done in the draft. I will provide some
detail on the first test case in order to establish a cadence. Subsequent
test cases will not have the detail, but the ability to iterate over
variables and so on is implied.

The SDN Controller test taxonomy would include:

- Asynchronous Message Response Rate. A test case to measure AMRR would
cover a variety of conditions that are analogous to the following
OpenFlow-specific events:

  o Packet_in received
  o Flow expiration received
  o Link down received

In other words, AMRR would be used to measure the ability of an SDN
controller to respond to a variety of asynchronous (i.e. unexpected)
conditions. One methodology for measurement would be specified by the
benchmark. The tester would then apply that methodology to packet_ins,
flow expiration, link down, and whatever other AMR testing is appropriate.

The AMRR methodology should include iteration variables and goal seeking.
The tester ought to be able to use the methodology to determine, for
example, the maximum packet_in AMRR for a variety of conditions, such as
variations in the number of connected nodes, the complexity of the
packet_ins, and so on. Variables also include negative testing with
invalid packets, for example.

- Synchronous Message Response Rate. A test case for measuring SMRR would
cover a variety of conditions analogous to the following OpenFlow-specific
events:

  o Hello packet exchange
  o Echo request/response exchange

- Node Acquisition Rate. A test case for measuring NAR would be a special
case of SMRR, but it specifically measures the rate at which an SDN
controller can connect to nodes in a network.

- Node Acquisition Maximum. A test case for NAM would measure the maximum
number of network nodes that an SDN controller can connect to without
error.

- FailOver Response Time. There are at least two test cases for measuring
FORT. One would measure a control cluster¹s ability to survive a
controller failure. How long does it take the controller function to be
assumed by another controller element? Another would measure the
controller¹s ability to respond to node failures to do things like
re-route traffic. How long does it take for the controller to send out
commands that handle the error? This is a bit like measuring convergence
time, but care must be taken to keep the node variables out of the
measurement.

- FailBack Response Time. Test cases for FBRT would be used to measure the
response time when the controller comes back up or the node comes back
online. How long does it take the controller to send the commands that
restore service?

>substantial implications for the draft, but the benefit of wider

Spirent Communications E-mail confidentiality.
-----------------------------------------------------------------------
This e-mail contains confidential and / or privileged information belonging to Spirent Communications plc, its affiliates and / or subsidiaries. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution and / or the taking of any action based upon reliance on the contents of this transmission is strictly forbidden. If you have received this message in error please notify the sender by return e-mail and delete it from your system.

Or if within the US,

Spirent Communications,

---

From: "Bhuvan \(Veryx Technologies\)" <[email-masked]>
Date: Wed, 25 Jun 2014 19:28:17 +0530

Hi Brain,

I have a couple of comments which are inline with tag Bhuvan

> commodities. It¹s the service they provide that adds value.

[Bhuvan] I agree with you. We should follow Al's suggestion to make this
draft generic. We should make sure that the draft provides sufficient
clarity so that it can be used on wide range of SDN controllers.

> the draft, and consolidate wording to make the document easier to read and

[Bhuvan] If my understanding is correct, Al's suggestion is to make the
current draft generic enough so that this methodology can be used to
benchmark SDN controllers irrespective of southbound protocols.

> What follows is an attempt to define a higher-level set of SDN controller
test
> cases. It builds on the work done in the draft. I will provide some detail
on
> the first test case in order to establish a cadence. Subsequent test cases
will
> not have the detail, but the ability to iterate over variables and so on
is
> implied.

[Bhuvan] I believe the next set of activities is to make the defined metrics
and methodology generic enough so that it becomes protocol independent.
Secondly the usage of the term testcase in the context of benchmarking is
slightly confusing. The intent of benchmarking is to measure the performance
metric rather than to conclude pass or fail.
 
> - Asynchronous Message Response Rate. A test case to measure AMRR
> would cover a variety of conditions that are analogous to the following
invalid
> packets, for example.
> 
> - Synchronous Message Response Rate. A test case for measuring SMRR
> would cover a variety of conditions analogous to the following OpenFlow-
> specific
> events:
> 
>   o Hello packet exchange
controller
> can connect to nodes in a network.
> 
> - Node Acquisition Maximum. A test case for NAM would measure the
> maximum number of network nodes that an SDN controller can connect to
> without error.
> 
controller
> failure. How long does it take the controller function to be assumed by
> another controller element? Another would measure the controller¹s ability
> to respond to node failures to do things like re-route traffic. How long
does it
> take for the controller to send out commands that handle the error? This
is a
> bit like measuring convergence time, but care must be taken to keep the
> node variables out of the measurement.
> 

> >Thanks for your comment. I do agree with you that there needs to be a
> >common benchmarking mechanism for controller designs performing the
> >same tasks.
> >But the technology is still relatively immature and the approach has
> >various dimensions including centralized control vs distributed control
> >(controller less).
> >Again the centralized control approach uses different programming
> >methods (OpenFlow vs Non OpenFlow).
> >
> >Having said that, for better usability/understanding of the draft and
> >to avoid misinterpretation, I personally feel it would be good to have
> >separate benchmarking methodology for each approach (though the
> metrics
> >remain same).
> >Also we would be happy to continue the effort to extend the same
> >metrics for other approaches too.
> >
> >Subject: RE: [bmwg] Feedback for Benchmarking Methodology for
> OpenFlow
> >SDN Controller Performance
> >
> >> My two main concerns are getting the terminology right and making
> >> sure that the proposed set of tests will achieve our goals. I believe
> >> those goals ought to include a common taxonomy, enabling
> >> apples-to-apples comparisons, and minimizing the time/work required
> >> to execute the test cases.
> >
> >> I am willing to help improve the benchmark. I understand that it is
> >> an early draft. The time is right to make sure that we are developing
> >> the best specification that we can.
> >>
> >meetings three times a year.  Sarah mentions that our next meeting is
> >in her home town:
> >same tasks could be benchmarked as black-boxes.  I realize this
> >approach has substantial implications for the draft, but the benefit of
> >wider applicability.
> >
> >food for thought,
> >Al
> >(as a participant)
> >
> >
> 
> This e-mail contains confidential and / or privileged information
belonging to
> Spirent Communications plc, its affiliates and / or subsidiaries. If you
are not
> the intended recipient, you are hereby notified that any disclosure,
copying,
> distribution and / or the taking of any action based upon reliance on the
> contents of this transmission is strictly forbidden. If you have received
this
> message in error please notify the sender by return e-mail and delete it
from
> your system.