Subject: [bmwg] question on data center benchmarking draft

From: shiyang <[email-masked]>
Date: Tue, 12 Aug 2014 11:15:13 +0800 (CST)

Hi Lucien,

When reading your draft about the methodology of the data center benchmarking（ draft-bmwg-dcbench-methodology-02）, we found it an very interesting and useful document, and we are trying to apply it into our experiments with DC switches.

However, there are some questions:

(1) what does PORT CAPACITY mean in section 2.2 ? 

(2) why latency and jitter are not considered if the traffic generator can not be connected to all ports on the DUT in section 2.2? 

     In our case, we'd like to test the relationship between the frame latency and the switch while we only have a generator with not so many ports.

(3) In Section 3.2, Measure maximum DUT buffer size with many to one ports, what does the equation ((N-1)/port capacity * 99.98)% in the iteration mean? What do you expect from the setting?

(4) In section 6, what is the definition of stateful and stateless flow from the generator side of view? There are only unofficial descriptions like “large and small flows” in the text, but in practice, like how can we use in testing setup to produce  the large and small flow?

(5) There is no clear definition for “flow latency”. Is it the same as the frame latency definition defined in the metric draft (draft-dcbench-def-01)?

Thank you very much!

Best,

Yang Shi

---

From: "Lucien Avramov (lavramov)" <[email-masked]>
Date: Tue, 12 Aug 2014 12:34:47 -0700

Hi Yang Shi,

Thank you for reading our draft.
Please find answers inline.

Lucien

On 8/11/14, 8:15 PM, shiyang wrote:
> Hi Lucien,
>
> When reading your draft about the methodology of the data center
> benchmarking（draft-bmwg-dcbench-methodology-02）, we found it an very
> interesting and useful document, and we are trying to apply it into our
> experiments with DC switches.
>
Port Capacity is the speed capability for this given port; for example 
1GE, 10GE, 40GE, 100GE etc.

>
> (2) why latency and jitter are not considered if the traffic generator
> can not be connected to all ports on the DUT in section 2.2?
Because the idea is to have a standard and comprehensive way to measure 
these two parameters. For example a switch can have an architecture that 
will provide different values depending on what ports are tested 
[multi-stage ASIC architecture for example], and therefore all ports 
should be tested in order to measure with accuracy latency and jitter.
>
>       In our case, we'd like to test the relationship between the frame
> latency and the switch while we only have a generator with not so many
> ports.

You can use also a snake test as depicted in section 2.2 this will move 
traffic accross all ports.

>
> (3) In Section 3.2, Measure maximum DUT buffer size with many to one
> ports, what does the equation ((N-1)/port capacity * 99.98)% in the
> iteration mean? What do you expect from the setting?
The reason this is mentioned is to cover what rate should be send on 
each port in order to not have an oversubcription. For example if you 
test two ports that support 10GE speed [N=2] [port capacity = 10GE], you 
should send 4.999 GE traffic from these two ports going to the third 
destination port [which will make it 9.998GE of traffic on that egress 
port, so it's not naturally oversubcribed]
>
> (4) In section 6, what is the definition of stateful and stateless flow
> from the generator side of view? There are only unofficial descriptions
> like “large and small flows” in the text, but in practice, like how can
> we use in testing setup to produce  the large and small flow?
Stateful traffic is TCP
Stateless traffic is UDP

Idea is that you generate large MTU TCP packets [for example 1500 or 
9216 bytes] and at the same time small UDP packets [100's of bytes]. and 
you would care to measure the latency/jitter of the UDP packets while 
measuring the goodput on the TCP packets.
>
> (5) There is no clear definition for “flow latency”. Is it the same as
> the frame latency definition defined in the metric draft
> (draft-dcbench-def-01)?
A flow is a sequence of frame defining an application. Averaging the 
frame latencies measured for a given application during this test would 
give us the flow latency. That's the concept we are behind.

---

From: shiyang <[email-masked]>
Date: Thu, 14 Aug 2014 10:28:05 +0800 (CST)

Hi Lucine,

Thanks for your reply.
We still have 2 questions:
(1) for the requirement of "traffic generator must be connected to all ports on the DUT", in our case, for instance, we have 4 ports on traffice generator and 12 ports on DUT, do you think it is a possible test standard such like:
iteration1: ingress port 1 and 2 send line rate to egress port 1 and 2 seperately.
iteration2: ingress port 1 and 2 send line rate to egress port 2 and 3 sepearately.
........
iteration11: ingress port 1 and 2 send line rate to egress port 11 and 12 sepearately.
iteration12: ingress port 3 and 4 send line rate to egress port 1 and 2 sepearately.
........
iteration 23: ingress port 3 and 4 send line rate to egress port 11 and 12 sepearately.
And we measure the latency and jitter each iteration?

(2) In Section 3.2, Measure maximum DUT buffer size with many to one ports, we think the equation " sending each [[N-1]/[port capacity]*99.98] % of line rate per port to the N egress port " is not the right way to represent: for one thing, [N-1] should be on the denominator part, for another thing, if you said it's some percentage of line rate, then should there be no [port capacity] in the equation? 
We think the better way to express it is: sending each[(1/[N-1])*99.98%] of line rate per port.....

Thanks a lot!
Best,
Yang