Subject: RE: [bmwg] Meeting Minutes Review: IPsec Terminology

From: Scott Bradner <[email-masked]>
Date: Tue, 28 Oct 2003 18:18:38 -0500 (EST)

fwiw
it is my opinion that doing a perfomance test of a stream of
a mix of packet sizes is almost useless - very small changes in
the mix can produce large changes in the result and any time that
small changes in conditions can produce large changes in output
you get a test that is quite hard to 1/ repeate and 2/ find any meaning in

This is someting that the WG has talked about from time to time, starting
in Vancover a long time ago, and always decided that such a test was
not one that would produce clear enough information to be meaningful 
to do.

I think it is far more important to test multiple sizes one after the other

I can think of one build-to-be-tested case that could be overcome 
by a mix of sizes and that is an auto tuning of buffer sizes based
on packet size but that hardly seems worth dealing with.

Scott

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: Tim Van Herck <[email-masked]>
Date: Tue, 28 Oct 2003 16:00:02 -0800

Scott,

> you get a test that is quite hard to 1/ repeate and 2/ find any meaning in

Useless is maybe too strong of a word here but I 100% agree that it is 
far more difficult the reproduce previously obtained result with traffic 
mixes versus a single packet size stream.

But the reality of the matter is that people, whether you like it or 
not, will use it. Like Brian already mentioned, people want to see a 
single number in a spec and don't usually want to deal with looking at a 
long result table that doesn't fit in the spec layout. IMIX is NOT a 
technical requirement but rather a marketing prop that is used without 
any sound foundation.

Although testing with a single packetsize and just walking through all 
packetsizes, if required, would be much more easier, I hate to admit 
that people still request & report IMIX datapoints.

> I think it is far more important to test multiple sizes one after the other

Correct, IMIX testing will be optional !

> on packet size but that hardly seems worth dealing with.

Sounds too platform specific and we're trying to include all IPSec 
devices, from PDA's to large military style encryption engines.

What if we do something like this :

A prerequisite will be that ingress and egress MTU have to be identical.

Then we send a round robin stream of the following packets :
  - TCP (minimum allowed packetsize on PHY) bytes
  - UDP 128 bytes
  - TCP 256 bytes
  - UDP 512 bytes
  - keep on alternating UDP/TCP, increase by 256 bytes
  - ....
  - UDP (max MTU) bytes

	Tim-

---

From: Scott Bradner <[email-masked]>
Date: Tue, 28 Oct 2003 19:07:09 -0500 (EST)

> single number in a spec 

this was one of the 1st things that BMWG talked about when the WG
ws formed just because a lot of people would like to have such
a number - it is too bad that it may not be possible to come up
with one that gives the buyer any actual information to decide between
vendors on

>   - keep on alternating UDP/TCP, increase by 256 bytes

that is about what I used to do (modulo alternating protocols 
within a packet size, I ran all the sizes then changed protocols but
I do not see any reason to do it one way or another)

but I do not see how that gets one closer to a single magic number
(which the BMWG designated a "millstone" after Dave Mills)

Scott

---

From: Scott Bradner <[email-masked]>
Date: Tue, 28 Oct 2003 19:07:09 -0500 (EST)

Scott

---

From: Tim Van Herck <[email-masked]>
Date: Tue, 28 Oct 2003 16:20:00 -0800

Scott,

> (which the BMWG designated a "millstone" after Dave Mills)

Would you agree that we try a middleground solution where we stay away 
from IMIX and redefine -a- general mix where we only suggest a potential 
mix format but require the mix to be reported in any of the documented 
results.

	Thanks,
	Tim-

---

From: Tim Van Herck <[email-masked]>
Date: Tue, 28 Oct 2003 16:20:00 -0800

Scott,

	Thanks,
	Tim-

---

From: Scott Bradner <[email-masked]>
Date: Tue, 28 Oct 2003 19:22:58 -0500 (EST)

> mix format but require the mix to be reported in any of the documented 

if you mean a concurrent mix of sizes that I think we will just
agree to disagree - if you mean a sequence of sizes then I'm in that camp

Scott

---

From: Tim Van Herck <[email-masked]>
Date: Tue, 28 Oct 2003 16:28:58 -0800

> agree to disagree - if you mean a sequence of sizes then I'm in that camp

Concurrent mix but enter it as an optional datapoint since it is a 
marketing style result. I'd also be more inclined to only go with 
discrete datapoint reporting. And put emphasis in the document that 
testing with mixes produces fluctuating results.

	Tim-

---

From: Tim Van Herck <[email-masked]>
Date: Tue, 28 Oct 2003 16:28:58 -0800

Tim-

---

From: Tim Van Herck <[email-masked]>
Date: Tue, 28 Oct 2003 16:00:02 -0800

Scott,

	Tim-

---

From: Scott Bradner <[email-masked]>
Date: Tue, 28 Oct 2003 19:22:58 -0500 (EST)

Scott