Subject: [bmwg] WGLC on IPv6 Benchmarking

From: Al Morton <[email-masked]>
Date: Sat, 14 Jul 2007 22:35:44 -0400

BMWG:

A WG Last Call period for the Internet-Draft on

"IPv6 Benchmarking Methodology"

<draft-ietf-bmwg-ipv6-meth-02.txt>

will be open from 14 July 2007 through 10 August 2007.

A URL for this Internet-Draft is:

http://www.ietf.org/internet-drafts/draft-ietf-bmwg-ipv6-meth-02.txt

Version 02 incorporates changes suggested during development

prior to the draft becoming a chartered item, and the recent

WGLC comments (closing on 20 March 2007).

The authors believe that all comments from the working group

have been addressed, and have documented all comments and
resolutions

to date:

http://home.comcast.net/%7Eacmacm/BMWG/IPv6-meth-comment-resolution.pdf

They would also like to point out that their mixed traffic
methodology

has been proven with an actual test tool implementation of their draft

recommendations.

This draft is continuing through the BMWG Last Call Process. This is

the second Working Group Last Call. See

http://www1.ietf.org/mail-archive/web/bmwg/current/msg00846.html

Please express your opinion on whether or not this
Internet-Draft

should be submitted to the Area Directors for consideration to

publish the draft as an Informational RFC.  Send your comments

to this list or [email-masked]

Al

bmwg chair

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: David Newman <[email-masked]>
Date: Sun, 15 Jul 2007 13:27:10 -0700

> http://www.ietf.org/internet-drafts/draft-ietf-bmwg-ipv6-meth-02.txt

Question and comment on Sonet frame length:

Per the discussion in section A.2, isn't the shortest possible
IPv6-over-Sonet frame length 47 bytes, not 53 bytes as recommended in
section 5.1.2? (4+40+2+1)

Beyond an old religious bias against 53-byte PDUs, I also believe
benchmark tests should be as stressful as possible. In this context,
stressful means determining the highest possible frame rate a device
will see. A 53-byte frame doesn't do that.

If a device only supports a 4-byte CRC, then 49 should be the minimum.
But in any event I don't believe 53 bytes is sufficiently stressful.

dn

---

From: "Chip Popoviciu \(cpopovic\)" <[email-masked]>
Date: Sun, 15 Jul 2007 22:10:55 -0400

Hello David,

There are two reasons for proposing this larger minimum frame size:
1) We wanted to stay in the spirit of RFC 2544 and allow in the minimum
sized frame room for an upper layer header as well. Here is the
paragraph of RFC 2544 we referred to:

  "Theoretically the minimum size UDP Echo request frame would consist
   of an IP header (minimum length 20 octets), a UDP header (8 octets)
   and whatever MAC level header is required by the media in use."

2) If the minimum size for Ethernet is 64 bytes then the minimum packet
size is 46 bytes. Since we do not have traffic sources which have SONET
access interfaces that would send line rate traffic, and since in IPv6
we do not have fragmentation along the path of the traffic, it could be
considered that the smallest packet size in the network will be 46 bytes
long. On the SONET links then the smallest frame size would be 53 bytes.

What are your thoughts on the above two considerations? We do not have a
strong position in favor of 53 bytes so lets come to an agreement within
the work group on what would we best and we can go with that.

In the end, I think that below 64 bytes it really matters little what
value we select, whether we go for a goal to achieve maximum fps (as you
suggest) or we go for rules of the kind mentioned above (which we
proposed). At these sizes there is no room for packet stamps which allow
for various types of statistics (delay, out of order, etc) measured by
test tools anyway so tests are less rich in terms of results. 

Thank you for your comments!

Best Regards,
Chip  

-----Original Message-----
From: David Newman [mailto:[email-masked]] 
Sent: Sunday, July 15, 2007 4:27 PM
To: [email-masked]
Subject: Re: [bmwg] WGLC on IPv6 Benchmarking

dn

---

From: David Newman <[email-masked]>
Date: Sun, 15 Jul 2007 20:43:03 -0700

>    and whatever MAC level header is required by the media in use."

True, but note all this still fits comfortably inside Ethernet's 64-byte
minimum (with room for a test instrument's signature field to boot).

> size is 46 bytes. 

No, it's 20 (v4) or 40 (v6) bytes, plus padding :-)

Since we do not have traffic sources which have SONET
> proposed). 

If I'm the implementer building the v6 engine, I'd want to know the pps
limit.

If I'm the marketer selling a perhaps-less-than-optimal-for-v6 product,
I'd want to use larger frame sizes and wave my hands around and cry,
"Real world! Real world!"

Can you guess where my loyalties lie?

At these sizes there is no room for packet stamps which allow
> test tools anyway so tests are less rich in terms of results. 

The size of the signature field is a separate issue.

With a signature field we're looking at a much larger minimum length,
somewhere well north of 60 bytes. I agree that 47-vs.-53 isn't a
meaningful distinction, but only for measurements that require a
signature field. The same tools that use signature fields also allow
them to be disabled; I'm raising a more general question of how
stressful a stress test should be.

This issue also will affect the ID's Ethernet lengths. AFAIK no
commercial tool vendor generates a 64-byte Ethernet frame containing an
IPv6 header AND a signature field.

In your comments, you mentioned there is at least one commercial
implementation of the ID already out there. What is its minimum Sonet
length with a signature field enabled?

One possible solution: For both Ethernet and Sonet, define two short
lengths -- one that's just the minimum L2 framing+IPv6 header to
determine throughput and another that includes a signature field to
measure latency and sequencing.

dn

---

From: "Chip Popoviciu \(cpopovic\)" <[email-masked]>
Date: Mon, 16 Jul 2007 00:43:08 -0400

Hi David,

Please see my comments in line. <c></c>

Best Regards,
Chip 

-----Original Message-----
From: David Newman [mailto:[email-masked]] 
Sent: Sunday, July 15, 2007 11:43 PM
Cc: [email-masked]
Subject: Re: [bmwg] WGLC on IPv6 Benchmarking

On 7/15/07 7:10 PM, Chip Popoviciu (cpopovic) wrote:
> 1) We wanted to stay in the spirit of RFC 2544 and allow in the 
> minimum sized frame room for an upper layer header as well. Here is 
> the paragraph of RFC 2544 we referred to:
> 

<c>
Of course (the signature is a different story though) and that might be
another reason why I appreciate the choice that 2544 made instead of
going for the limit case, in my opinion 2544 had a very nice practical
foresight. The point I was making here with that quote was that in the
same spirit, even when going for the limit case of the smallest frame
possible, it might be worth considering leaving room for a UDP header as
well, just as in the case of 2544. 
</c> 

> 2) If the minimum size for Ethernet is 64 bytes then the minimum 
> packet size is 46 bytes.

<c>
An option of course but since we do not make (frame-size, packet-size)
tuple recommendations it seems to me that padding would add too much
variance in the methodology :)
</c>

> we do not have fragmentation along the path of the traffic, it could 
> be considered that the smallest packet size in the network will be 46 
> bytes long. On the SONET links then the smallest frame size would be
53 bytes.
> 
> What are your thoughts on the above two considerations? We do not have

> a strong position in favor of 53 bytes so lets come to an agreement 
> within the work group on what would we best and we can go with that.
> 

<c>
Well, I am assuming that you are more of a tester like me. I think an
implementer will carefully choose his/her battles because after all, no
implementation is perfect and resources are limited. So if an
implementer will have to choose between forwarding line rate 40 byte
packets at the cost of some capabilities that are trully useful in real
life, I would hope he/she would choose the later even though it is not
unheard of implementers taking the other route. A marketeer would of
course push for packets larger than 64 as he/she would bind the argument
to actual applications. As a tester I am interested in pushing the
limits but sometimes practical, operational experience does make its way
into ones perspective :). Of course, sometimes a practical perspective
is just that, a practical perspective. 
</c>

<c>
Lets open the proposal for discussion and see what the WG thinks.
Personaly I am not in favour of it because the two types might become
confusing. We already have a note in the draft about the signature
related issues so I would rather stick with defining a single type of
minimum, be it 47 or 53. I like RFC 2544's clarity and simplicity (as
you probably noticed by my many refrences to it :) so my vote goes for
sticking to its approach but this is an opportunity to take a new
perspective should we all agree it is needed.
</c>

Best Regards,
Chip

dn

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 17 Jul 2007 10:49:07 -0400

> "Real world! Real world!"

Maybe benchmarking is its own little world but the real world is what
matters because thats where the money to support the work comes from.

Just a data point here.  At the last router vendor I worked for 64 was
the magic number for IPv4 from ISP requirements.  If you could do 40,
all the better which meant that vendors all aimed for 40.  That was
IPv4 payload, not including L2 overhead.

The ISPs don't care what BMWG specifies.  They will provide their own
number if they disagree.  The enterprise market does not have that
luxury and may also be more prone to pay attention to meaningless
benchmark figures that marketeers waive around if theirs is better.

Curtis

---

From: "Chip Popoviciu \(cpopovic\)" <[email-masked]>
Date: Tue, 17 Jul 2007 17:56:10 -0400

I think Curtis touches here on a very important point: making sure that
our recommendations are meaningful theoretically but they are also
practically relevant. Ultimately, everyone can make their own selection
but it would be great if it is just a super set of BMWG recommendations.
This was in part the reasoning behind some of the practically oriented
proposals made.

Marketeers will always find the way to put things in a light they see
positive so we don't have to worry about them.

Best Regards,
Chip

-----Original Message-----
From: Curtis Villamizar [mailto:[email-masked]] 
Sent: Tuesday, July 17, 2007 10:49 AM
To: David Newman
Cc: [email-masked]
Subject: Re: [bmwg] WGLC on IPv6 Benchmarking 

In message <[email-masked]>
David Newman writes:
>  
> If I'm the marketer selling a perhaps-less-than-optimal-for-v6 
> product, I'd want to use larger frame sizes and wave my hands around 
> and cry, "Real world! Real world!"

Curtis