Subject: [bmwg] Benchmarking Methodology for OpenFlow SDN Controller Performance

From: "Bhuvan \(Veryx Technologies\)" <[email-masked]>
Date: Tue, 1 Apr 2014 20:37:58 +0530

Hi folks,

SDN is gaining lot of traction in the industry among vendors and service
providers.

We clearly see a need for test methodologies to measure the performance of
SDN based network.
 
Keeping this viewpoint, we have drafted a benchmarking methodology for
OpenFlow SDN controller performance.
 
http://tools.ietf.org/html/draft-bhuvan-bmwg-of-controller-benchmarking-00

We would love to hear any comments and queries on the same.

Thanks,

Authors

---

From: "Banks, Sarah" <[email-masked]>
Date: Thu, 3 Apr 2014 12:47:44 -0400

Hello BMWG,
	I sent comments to the authors on the thread here privately and forgot to
cc bmwg. The authors have asked me to send these comments to the list.
Here is the original email.

Thanks
Sarah

On 4/1/14, 5:27 PM, "Banks, Sarah" <[email-masked]> wrote:

>Authors,
>
>	Thanks for your draft! SDN is interesting, and has come up in some
>hallway conversations - it's nice to see a draft on it. I've reviewed the
>doc, and I'll try not to take a stance, but rather, address issues that I
>see that are either technically wrong or lead to the potential to be
>wrong, or make the draft more readable. I'll point out that your draft
>could use an Editor's eye - there are grammatical issues throughout. I'm
>happy to help if you need, please let me know.
>
>	I like to review section by section. I hope that approach works for you.
>Here goes. :)
>
>	Overall, I like that your approach is very straight forward, but I find
>that the document suffers from a true introduction. Indeed, your
>Appendices read very nicely, and I found myself thinking that this
>information would have been nice at the top of the hour - at the beginning
>of the document. Please consider relocating this information. Not everyone
>is an SDN guru, and not everyone takes an identical approach - level
>setting where you're coming from and your scope will make consuming the
>rest of the document (hopefully) easier. Does this make sense?
>
>	Let me go section by section now.
>
>In Section 5.1 - You mention that the switches need not be physical
>switches, but emulated/simulated by software or test tools instead - the
>last sentence of this paragraph says that you "MUST indicate the number of
>switches and hosts..." - consider referencing the emulated/simulated
>number too. A quick read might leave open the idea that only physical
>switches used be noted/listed.
>
>Section 5.3 refers to the ability to have differing ways to setup channels
>and connection type, yet only leverages some (as yet undefined) channel
>setup modes. Why? Why are all of the methods/modes not applicable or being
>used?
>
>Section 5.3.1.2: In active-passive mode, you state, "Any asynchronous
>communications from the switch have to be sent on the active channel."
>While I suspect you don't mean this in a normative way, it begs the
>question - why?
>
>Section 5.3.1.3 - why call this out if you aren't going to require that
>their use be noted in the test results?
>
>Section 5.3.2 - What is controller teaming? (OK, I know what it is, but
>you're using a term not yet defined. Please define.)
>
>Section 5.3.2 - Why are you recommending the use of TCP here, over the
>alternatives? Be specific.
>
>In general, you don't take a stance on the number of times a test should
>be run, nor the amount of time to wait/pause between tests. Would you
>consider adding values to your recommendations?
> 
>Your reporting format is repeated over and over - I'm a fan of state it
>once, and modify it when needed. :) Just a thought.
>
>Section 9 - Security Considerations - to my eye, it's unacceptable to
>state that there might be security considerations in the security
>considerations section, and then state that they're out of scope. I
>disagree. They are in scope. What are the issues? It's a north bound
>interface in the lab. What's up?
>
>Is Appendix A (Section 10) a sample test report? There's no information
>about the setup/topology of the setup. If this isn't what App A is
>intended to be, consider adding a sample test report. I'd argue it might
>not live as an appendix, either. :)
>
>In that vein, Appendix A has no quantification about WHAT it is - what is
>it? Why are there Nas? :) Please clarify.

---

From: "Bhuvan \(Veryx Technologies\)" <[email-masked]>
Date: Fri, 4 Apr 2014 14:38:25 +0530

Hello BMWG,

I am forwarding our response that we have provided to Sarah. 

Thanks,

Bhuvan

From: Bhuvan (Veryx Technologies) [mailto:[email-masked]] 
Sent: Thursday, April 03, 2014 12:13 PM
To: Banks, Sarah ([email-masked])
Cc: 'Anton Basil'; vishwas manral ([email-masked])
Subject: RE: [bmwg] Benchmarking Methodology for OpenFlow SDN Controller Performance

Hello Sarah,

We have gone through your comments and they are good.

Please find below our responses to your comments with tag [Authors].

Also I would like to check with you if we can forward your comments to the mailing list to generate more discussion around this.

Best regards,

Bhuvan

-----Original Message-----
From: Bhuvan (Veryx Technologies) [mailto:[email-masked]] 
Sent: Wednesday, April 02, 2014 3:10 PM
To: 'Banks, Sarah'
Cc: [email-masked]'; [email-masked]'
Subject: RE: [bmwg] Benchmarking Methodology for OpenFlow SDN Controller Performance

Hello Sarah,

Thanks for taking time to review this draft and provide your comments.

We will go over all your comments and provide our response shortly.

Best regards,

Bhuvan

-----Original Message-----

From: Banks, Sarah [mailto:[email-masked]]

Sent: Wednesday, April 02, 2014 5:58 AM

To: Bhuvan (Veryx Technologies); [email-masked]; [email-masked]

Subject: Re: [bmwg] Benchmarking Methodology for OpenFlow SDN Controller Performance

Authors,

                Thanks for your draft! SDN is interesting, and has come up in some hallway conversations - it's nice to see a draft on it. I've reviewed the doc, and I'll try not to take a stance, but rather, address issues that I see that are either technically wrong or lead to the potential to be wrong, or make the draft more readable. I'll point out that your draft could use an Editor's eye - there are grammatical issues throughout. I'm happy to help if you need, please let me know.

[Authors] Sure. We would love to take your help in addressing them in the draft.

Here goes. :)

                Overall, I like that your approach is very straight forward, but I find that the document suffers from a true introduction. Indeed, your Appendices read very nicely, and I found myself thinking that this information would have been nice at the top of the hour - at the beginning of the document. Please consider relocating this information. Not everyone is an SDN guru, and not everyone takes an identical approach - level setting where you're coming from and your scope will make consuming the rest of the document (hopefully) easier. Does this make sense?

[Authors] Yes. We do agree your point. We will move the appendix B (OpenFlow Protocol Overview) information after the scope section.

In Section 5.1 - You mention that the switches need not be physical switches, but emulated/simulated by software or test tools instead - the last sentence of this paragraph says that you "MUST indicate the number of switches and hosts..." - consider referencing the emulated/simulated number too. A quick read might leave open the idea that only physical switches used be noted/listed.

[Authors] We will reword this sentence to reflect the emulated/simulated entities too. 

Section 5.3 refers to the ability to have differing ways to setup channels and connection type, yet only leverages some (as yet undefined) channel setup modes. Why? Why are all of the methods/modes not applicable or being used?

[Authors] Though we have described all the methods/modes that are applicable, our intent is to recommend the methods/modes that are supported by controllers available in the market. Performance measurement over other methods/modes is mentioned as optional, depending on the support available on the controller

Section 5.3.1.2: In active-passive mode, you state, "Any asynchronous communications from the switch have to be sent on the active channel."

While I suspect you don't mean this in a normative way, it begs the question - why?

[Authors]  We think we can remove this sentence from the draft, as in one of the test we have mentioned that to initiate communication on the passive channel as well.

Section 5.3.1.3 - why call this out if you aren't going to require that their use be noted in the test results?

[Authors] We did mention their use in Section 5.3.2. So we think of having this section in the draft. Please let us know your thoughts.

Section 5.3.2 - What is controller teaming? (OK, I know what it is, but you're using a term not yet defined. Please define.)

[Authors] Sure. We will define this in the terminology section.

Section 5.3.2 - Why are you recommending the use of TCP here, over the alternatives? Be specific.

[Authors] Since TCP is being supported by all OpenFlow implementations, we have recommended TCP over others.

In general, you don't take a stance on the number of times a test should be run, nor the amount of time to wait/pause between tests. Would you consider adding values to your recommendations?

[Authors] We could certainly look at recommending values for parameters like ‘number of times a test should be run’. But for others, we may not be able to define/recommend values, as these parameters may vary on implementation to implementation.

Your reporting format is repeated over and over - I'm a fan of state it once, and modify it when needed. :) Just a thought.

[Authors] This is a good suggestion. We can remove this portion wherever repeated in this draft.

Section 9 - Security Considerations - to my eye, it's unacceptable to state that there might be security considerations in the security considerations section, and then state that they're out of scope. I disagree. They are in scope. What are the issues? It's a north bound interface in the lab. What's up?

[Authors] Fine. We will review this section and add specific issues.

Is Appendix A (Section 10) a sample test report? There's no information about the setup/topology of the setup. If this isn't what App A is intended to be, consider adding a sample test report. I'd argue it might not live as an appendix, either. :)

In that vein, Appendix A has no quantification about WHAT it is - what is it? Why are there Nas? :) Please clarify.

[Authors] We have missed to provide the required descriptions to this section. We shall add a few lines of description that would clarify the section’s purpose.

Thanks

Sarah

BMWG Co-Chair

On 4/1/14, 8:07 AM, "Bhuvan (Veryx Technologies)"

>SDN is gaining lot of traction in the industry among vendors and 

>service providers.

>We clearly see a need for test methodologies to measure the performance 

>of SDN based network. Keeping this viewpoint, we have drafted a 

>benchmarking methodology for OpenFlow SDN controller performance.

> <http://tools.ietf.org/html/draft-bhuvan-bmwg-of-controller-benchmarking> http://tools.ietf.org/html/draft-bhuvan-bmwg-of-controller-benchmarking

>-00  We would love to hear any comments and queries on the same.