Subject: [bmwg] Meeting Minutes Review: IPsec Terminology

From: Michele Bustos <[email-masked]>
Date: Thu, 23 Oct 2003 18:57:32 -0700

To address the issues raised at the last meeting regarding the IPsec Term
draft.

>The current draft defined IMIX (Internet Mix for traffic synthesis),
>but did not include the reference that Michele Bustos provided on the list.
>There was a comment suggesting that this is a good definition to include,
>with the caveat that no one mix of packet sizes represents Internet usage.

We eliminated the NLARR reference from the draft for several reasons.
The reference never actually refers to IMIX, and they never suggest any
frame 
sizes or traffic patterns.

If anyone can provide a reference for IMIX specifically, we would be happy
to investigate it.

Michele Bustos
Ixia
[email-masked]

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: "Talbert, Brian" <[email-masked]>
Date: Tue, 28 Oct 2003 16:02:44 +0000

The definition seems to draw from Agilent's Journal of Internet Test
Methodologies.  That is about the only source where I have seen the term
"Simple IMIX" referenced.   Though your definition here does seem to draw
the Ixia IMIX test script which uses 64, 570, and 1518 as packet sizes
(including Ethernet header).  Agilent sticks to IP (as should probably be
done here) but with some variance in the packet sizes, using 40, 576, and
1500.

In any event, I will again raise caution about trying to define IMIX or at
least to give reference to any particular distribution.  The distribution
provided does not appear to be directly based upon any data studies, but
rather taken from 3rd party sources, be it Agilent or Ixia.  And that may in
itself be fine if so recorded (as examples) but there is concern here as
well.  For one, the data from which these were based (the NLANR data) is
coming up on 3 years old and more importantly it represents generalized
Internet data.  This draft focuses on VPN traffic patterns, for which there
isn't necessarily any correlation at all to the NLANR data.  

=================================================
To echo (quite literally) my previous sentiments:

 Well, I'm not sure that someone must state it in an RFC ... not at
least in so far as it defines a specific pattern.  First, why do we all use
IMIX?  I use it because our sales engineers and marketing folks don't want a
lot of information.  They want a single number that they can quote that
gives a rough idea of the "customer experience".  I suspect the appeal is
similar for vendors, and IMIX is OK for this purpose.  So the question
becomes, would a standardized IMIX fairly represent the "customer
experience".  Well, if that standard includes a specific packet size pattern
then it would be fair assuming that the customer experience closely
resembles the source of the model data.   And we know a few things about the
model data: it is several years old and was only then representative of
general Internet consumption.  So, how well does this data model represent
VPN applications where the usage is quite differnet from general Internet
consumption?   VPNs bring the element of enterprise applications into the
foray and so an IMIX model based on general Internet traffic will loose
correlation significance.   

The alternative would be to define IMIX very generically and not include
specific packet distributions.  Do this looses the advangate of have easy
cross-comparison but it does allow replication.   Also, while this doesn't
standardize everyone on the exact same mix, it would at least allow for you
to stipulate that the IMIX pattern (packet sizes and interleave) be
documented.  This would still go a long way as I see many reports of "IMIX"
testing only to find out later that the pattern had a smallest packet size
of 512 or that max size was reduced to avoid fragmentation effects, etc.

I think you pretty much already do this, though for clarity I would avoid
any specific references in the discussion and instead focus on issues such
as:

1.  General distribution of packet sizes
2.  Interleaving of packets
3.  Correlation to actual data
etc.

If you definately think it is best to go forward with referencing IMIX
in the meth then you should probably have it defined in the term. I don't
think it would be good to rely on Agilent or another outside source for
definition.   This is how I would probably tackle it:

=======================
Mixed Packet Size Stream

Definition:  A stream (per Network Layer Traffic Control Term draft??) of
packets with varying framesizes.

Discussion:  Mixed packet size streams allow testing to be conducted using
traffic patterns that simulate those observed in live networks.  Such
streams can be used to analyze forwarding rates, throughput, and other
metrics in a scenario that best represents the specific deployment
utilization of a device.

The particular distribution of packet sizes and how they are interleaved may
vary depending upon expected operational use of a device.  It is conceivable
that for some purposes a packet mix representing general Internet traffic
(Internet Mix, or IMIX) will be appropriate, while for other purposes a
packet mix representing other characteristics will be more appropriate (for
examplle, a VPN Mix, VoIP Mix, etc.).   In all cases, however, it is
important the the specific packet sizes, thier distribution ratios, and how
they are interleaved be as closely correlated to the operational usage as
possible.   The National Laboratory for Applied Network Research has
collected data that could be used to correlate an Internet MIX with general
Internet traffic.  Other data collection efforts may be necessary to
significantly correlate a mixed packet size stream to actual usage patterns.

Measurment Units:  The units applicable to the underlying test to wich the
mixed packet size stream is being applied.
==========================

This approach to defining a "MIX" allows for the flexiblity for the
individual tester to define a mix that meets thier own needs and alows it to
be applied to a variety of metrics.  You could for instance further define a
"VPN MIX Throughput" or "Internet with G.729 MIX Forwarding Rate".   It also
opens the doors to require in a meth (or here if that seems more
appropriate) that any type of MIX must be defined in results to include:

1.  Packet sizes within the stream
2.  Distribution ratios of the packet sizes
3.  Interleaving characteristics
???4.  Targeted operational use?
???5.  Correlation to targeted use

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Brian Talbert
UUNET Access Engineering
MCI
 
Office:  703-886-5812              Fax:   703-886-0535
Email:   [email-masked]     AIM:   wbriantalbert
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

>>-----Original Message-----
>>From: [email-masked] [mailto:[email-masked]] On 
>>Behalf Of Michele Bustos
>>Sent: Thursday, October 23, 2003 9:58 PM
>>To: [email-masked]'
>>Cc: Ipsec-Term@External. Cisco. Com (E-mail)
>>Subject: [bmwg] Meeting Minutes Review: IPsec Terminology
>>
>>
>>To address the issues raised at the last meeting regarding 
>>the IPsec Term draft.
>>
>>>but did not include the reference that Michele Bustos 
>>provided on the 
>>>list. There was a comment suggesting that this is a good 
>>definition to 
>>>include, with the caveat that no one mix of packet sizes represents 
>>>Internet usage.
>>
>>We eliminated the NLARR reference from the draft for several 
>>reasons. The reference never actually refers to IMIX, and 
>>they never suggest any frame 
>>If anyone can provide a reference for IMIX specifically, we 
>>would be happy to investigate it.
>>
>>Michele Bustos
>>Ixia
>>[email-masked]
>>
>>

---

From: [email-masked]
Date: Tue, 28 Oct 2003 17:17:45 -0500

Also consider that the specified IMIX does not account for 4470 byte
packets.  4470 is the default MTU on POS links.  

I do recommend stating that "a mix of Packet Sizes can be used and MUST be
reported with the results".  That could eliminate the argument about
definition of "IMIX" and allow the document user to select the best "IMIX"
for his/her network.  Is this a good solution?

Scott

-----Original Message-----
From: Talbert, Brian [mailto:[email-masked]]
Sent: Tuesday, October 28, 2003 11:03 AM
To: 'Michele Bustos'; [email-masked]'
Cc: Ipsec-Term@External. Cisco. Com (E-mail)
Subject: RE: [bmwg] Meeting Minutes Review: IPsec Terminology

=======================
Mixed Packet Size Stream

---

From: Merike Kaeo <[email-masked]>
Date: Tue, 28 Oct 2003 14:40:18 -0800

I think some packet sizes should be explicitly stated so that results would 
be comparable if different testing devices were used to test different 
products.  I personally like the idea of using 64-byte, 768-byte, 1518-byte 
and 4470-byte packets if for no other reason that it covers a mix.  Min and 
max mtu is always good to catch sw issues and then some mid-size packets 
give you feel for what performance curve looks like for a device.  The 
768-byte number can be removed or replaced with another size.....just threw 
it out there......somewhy I remember that looking at corporate network 
traffic data from a few years back produced a lot of traffic around that size.

- merike

---

From: Merike Kaeo <[email-masked]>
Date: Tue, 28 Oct 2003 14:40:18 -0800

- merike

---

From: [email-masked]
Date: Tue, 28 Oct 2003 17:57:42 -0500

Merike,

We are going to end up debating Packet Sizes for an "Imix" instead of IPsec
benchmarking.  I would add to your list 40 bytes, 48 bytes, and 512 bytes
packets.  40 and 48 bytes are common POS benchmarks and 512 bytes is a
common packet size on the Internet.  

The Imix definition is not unique to IPsec.  It applies to many BMWG drafts
and RFCs.  One alternative is to reference the phrase "a mix of packet
sizes" in the IPsec draft and have it _defined_ in the General Benchmarking
Terminology draft that Al and Kevin plan to write.  This addresses the
problem globally and enables IPsec draft to make progress by putting the
focus back on IPsec issues. 

BTW, Great talk at NANOG last week.

Scott
-----Original Message-----
From: Merike Kaeo [mailto:[email-masked]]
Sent: Tuesday, October 28, 2003 5:40 PM
To: [email-masked]; [email-masked];
[email-masked]; [email-masked]
Cc: [email-masked]
Subject: RE: [bmwg] Meeting Minutes Review: IPsec Terminology

traffic data from a few years back produced a lot of traffic around that
size.

- merike

---

From: Merike Kaeo <[email-masked]>
Date: Tue, 28 Oct 2003 15:21:00 -0800

>Terminology draft that Al and Kevin plan to write.

ok....I agree having  the mix defined generally is a better solution 
overall.  I would hate for the benchmarking docs for IPsec get hung up on 
test packet size debates :)  One would hope that people are getting smarter 
and asking for what packet sizes were tested when looking at perfomance 
data and comparing numbers.......

- merike

---

From: Merike Kaeo <[email-masked]>
Date: Tue, 28 Oct 2003 15:21:00 -0800

- merike

---

From: Michele Bustos <[email-masked]>
Date: Tue, 28 Oct 2003 15:29:44 -0800

I think that this should be an interface independent discussion.

-----Original Message-----
From: Tim Van Herck [mailto:[email-masked]]
Sent: Tuesday, October 28, 2003 3:20 PM
To: [email-masked]
Cc: [email-masked]; [email-masked]
Subject: Re: [bmwg] Meeting Minutes Review: IPsec Terminology

	Scott,

> packets.  4470 is the default MTU on POS links.  

same argument goes if you enable jumboframes on GE links.

> for his/her network.  Is this a good solution?

Could be a good solution to avoid heated debates about the right 
packetsizes for sure. What we also can do is make the mix more 
futureproof and don't rely anymore on data extracted from studies.
What if we just define a mix that sweeps from the minimum allowed 
packetsize to the MTU of the egress interface and make sure that the 
ranges are reported. On top of getting a comparable result you will also 
  exercise the entire datapath.

So, as far as I can see we have three ways out of this :

1. Imix is a strict defined packetmix
    - poses problems with newly introduced links and also IPv6, it
      will work with old data
    - has the advantage of providing an apples-to-apples comparison
2. Imix is -a- mix, user reports what it is
    - hard to compare results
    - probably will still use the old style IMIX distributions
3. Imix is a packetsweep (after renaming the mix)
    - good chance of comparable results
    - can be adjusted to the needs of the network
    - exercises the full datapath
    - more futureproof

Let the voting begin !

	Tim-

---

From: Michele Bustos <[email-masked]>
Date: Tue, 28 Oct 2003 15:35:19 -0800

Hi Scott, 
I agree.
As long as the mix utilized is included in the reported results, then we are
testing apples to apples.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
Sent: Tuesday, October 28, 2003 2:18 PM
To: [email-masked]; [email-masked]; [email-masked]
Cc: [email-masked]
Subject: RE: [bmwg] Meeting Minutes Review: IPsec Terminology

Scott

-----Original Message-----
From: Talbert, Brian [mailto:[email-masked]]
Sent: Tuesday, October 28, 2003 11:03 AM
To: 'Michele Bustos'; [email-masked]'
Cc: Ipsec-Term@External. Cisco. Com (E-mail)
Subject: RE: [bmwg] Meeting Minutes Review: IPsec Terminology

=======================
Mixed Packet Size Stream

---

From: Michele Bustos <[email-masked]>
Date: Tue, 28 Oct 2003 15:37:54 -0800

Excellent.  Can we agree then that the frame/packet sizes should be defined,
with an escape clause for those who want to define their own mix.  BUT
including a mandate that all tested sizes be reported, ensuring
apples-to-apples.

Thoughts?

-----Original Message-----
From: Merike Kaeo [mailto:[email-masked]]
Sent: Tuesday, October 28, 2003 3:21 PM
To: [email-masked]; [email-masked];
[email-masked]; [email-masked]; [email-masked]
Cc: [email-masked]
Subject: RE: [bmwg] Meeting Minutes Review: IPsec Terminology

- merike

---

From: Michele Bustos <[email-masked]>
Date: Tue, 28 Oct 2003 15:37:54 -0800

Thoughts?

-----Original Message-----
From: Merike Kaeo [mailto:[email-masked]]
Sent: Tuesday, October 28, 2003 3:21 PM
To: [email-masked]; [email-masked];

- merike

---

From: Scott Bradner <[email-masked]>
Date: Tue, 28 Oct 2003 18:18:38 -0500 (EST)

fwiw
it is my opinion that doing a perfomance test of a stream of
a mix of packet sizes is almost useless - very small changes in
the mix can produce large changes in the result and any time that
small changes in conditions can produce large changes in output
you get a test that is quite hard to 1/ repeate and 2/ find any meaning in

This is someting that the WG has talked about from time to time, starting
in Vancover a long time ago, and always decided that such a test was
not one that would produce clear enough information to be meaningful 
to do.

I think it is far more important to test multiple sizes one after the other

I can think of one build-to-be-tested case that could be overcome 
by a mix of sizes and that is an auto tuning of buffer sizes based
on packet size but that hardly seems worth dealing with.

Scott

---

From: Michele Bustos <[email-masked]>
Date: Tue, 28 Oct 2003 15:29:44 -0800

-----Original Message-----
From: Tim Van Herck [mailto:[email-masked]]
Sent: Tuesday, October 28, 2003 3:20 PM
To: [email-masked]
Cc: [email-masked]; [email-masked]
Subject: Re: [bmwg] Meeting Minutes Review: IPsec Terminology

	Scott,

Let the voting begin !

	Tim-

---

From: Michele Bustos <[email-masked]>
Date: Tue, 28 Oct 2003 15:35:19 -0800

Hi Scott, 
I agree.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
Sent: Tuesday, October 28, 2003 2:18 PM
To: [email-masked]; [email-masked]; [email-masked]
Cc: [email-masked]
Subject: RE: [bmwg] Meeting Minutes Review: IPsec Terminology

Scott

-----Original Message-----
From: Talbert, Brian [mailto:[email-masked]]
Sent: Tuesday, October 28, 2003 11:03 AM
To: 'Michele Bustos'; [email-masked]'
Cc: Ipsec-Term@External. Cisco. Com (E-mail)
Subject: RE: [bmwg] Meeting Minutes Review: IPsec Terminology

=======================
Mixed Packet Size Stream

---

From: Sunil Kalidindi <[email-masked]>
Date: Tue, 28 Oct 2003 22:08:54 -0800

We should do # 2.

I agree with Brian that we should be careful about defining a "standard"
mix. I dont think we know what that is - especially with IPSec VPNs. Even if
we know what it is on a *given network*, *today*, I think we all agree that
may not apply in a general sense.

To suggest that there is a "standard" will increase the confusion that
"IMIX" has already caused in the industry.

I run into several people every week who think that they just need to test
with this magical "IMIX" distribution but are not sure what it is.

For the marketing numbers, most people end up using the best case number
anyway.

- Sunil

-----Original Message-----
From: Tim Van Herck
To: [email-masked]
Cc: [email-masked]; [email-masked]
Sent: 10/28/2003 3:19 PM
Subject: Re: [bmwg] Meeting Minutes Review:  IPsec Terminology

	Scott,

Let the voting begin !

	Tim-

---

From: Sunil Kalidindi <[email-masked]>
Date: Tue, 28 Oct 2003 22:08:54 -0800

We should do # 2.

- Sunil

-----Original Message-----
From: Tim Van Herck
To: [email-masked]
Cc: [email-masked]; [email-masked]
Sent: 10/28/2003 3:19 PM
Subject: Re: [bmwg] Meeting Minutes Review:  IPsec Terminology

	Scott,

Let the voting begin !

	Tim-

---

From: "Talbert, Brian" <[email-masked]>
Date: Wed, 29 Oct 2003 14:47:30 +0000

>>Behalf Of Tim Van Herck
>>Sent: Tuesday, October 28, 2003 6:20 PM

...

>>    - has the advantage of providing an apples-to-apples 

This does provide apples-to-apples comparison of device performance but as
has already been suggested, one of the primary drivers for an IMIX number is
for marketing/sales ease of use.  And as such the number is not just used to
compare one vendor to another but also to provide some notion of customer
experience for a given service.  So, the strictly defined mix does start to
become apples-to-orange from the perspective of trying to understand how a
given device will perform for a given application.  Of particular note in
this discussion is for VPN applications, for at least in my experience the
Ipsec devices that we benchmark are used to build VPNs and to forwad data
for enterprise  applications securelty across the Internet and not merely to
send generalized Internet data through an Ipsec tunnel.  

>>    - probably will still use the old style IMIX 

If IMIX must be defined at all this would probably get my vote as it
provides the freedom to create custome mixes that suite the application of
the device.  Now my perspective is that of the service provider who is
characterizing service expectations and it is not the perspective of an
equipment manufacturer, who could conceivably create a mix that is favors
the nature of their device rather than any particular application (we
already see vendors using simple IMIX streams in the VPN space that omit the
1500 byte packet in favor of a 1400 byte packet because it exposes the
inefficiency with which they reassemble fragments).   So this is an admitted
down side but I don't see a more positive approach other than perhaps to
standardize many different mixes suitable for a variety of applications, but
I don't see this as being feasible.

>>    - exercises the full datapath

One of the downsides to this is that you will essentially create a strictly
defined packet mix, with all of it's downsides.  For instance, if as
suggested you sweep from min to max and do so in fixed n-byte increments,
you may end up yielding results that unfairly favor one vendor over another
simply based upon the nature of their internal architecture, which may for
instance use n-byte cells to move data chuncks through the switching fabric.
Modify the sweep to n+1 increments and you get vastly different results.  So
how do you choose n?  There are potential solutions, like randomizing the
packet size, etc., but ....

>>Let the voting begin !

... I still vote for number 2, simply define the general concept of a mixed
packet stream (in a general benchmarking terminology document so that the
definition carries across all BMWG efforts) and require that the stream be
clearly defined to include not only packet size distribution but the
interleaving pattern and any other parameters that may further characterize
the nature of the stream.

Brian Talbert