Subject: Re: Comments on bmwg-secperf-04

From: David Newman <[email-masked]>
Date: Wed, 05 Aug 1998 15:16:15 -0400

Hi Kevin,

Thanks very much for the feedback. Insightful comments, as usual.

Comments inline:

> In section 3.4, Bit Forwarding Rate, you write:  "...this
> measurement counts bits per second rather than frames per
> second.  Per-frame metrics are not meaningful in the context
> of a flow of application data between endpoints."
> 
> Two comments.  First, if you leave the above declaration (i.e.,
> per frame metrics are not meaningful), you may wish to offer a
> explanation as to your rationale. This would help the reader
> build an understanding as to your conclusion.
> 
> Second, if per-frame performance is meaningless in the firewall
> context, (I'm not sure I _totally_ agree), why are you carrying the
> frame overhead in the metric versus some sort of Effective Bit
> Forwarding Rate (bit fwd rate of data payload component only?)
> 
Yes, this text is problematic.

An earlier implementation of our testing tool did exactly what you
suggest--it measured payload only, as reassembled at the application
layer. It was with reassembly in mind that I proposed a metric in bits
or bytes rather than frames per second. We no longer test that way; I'm
much more interested in activity on the wire, since doing so mitigates
the effects of the testing platform's upper-layer software, OS, drivers,
etc. 

Still, we're dealing with applications here and not streams of same-size
packets. Sniffing any ftp or http session will reveal packet sizes all
over the map (and acknowledgments, if we're using half-duplex links).
That makes per-frame metrics difficult. What frame size am I measuring?
Should I average all the frame sizes I see in a given capture? Use a
median? A harmonic mean? 

Bit-based metrics don't raise those questions. I can always measure how
much data I moved per unit of time, regardless of how it got chopped up
along the way.

In terms of fixing this, I'd suggest replacing the last sentence quoted
with:

"Per-frame metrics are not meaningful in the context of a flow of
application data, where data may be segmented into frames of variable
size."

> 
> Section 3.7.  Connection.  I would move this wonderfully defined
> abstration ahead of the connection-related metrics (e.g. concurrent
> connections.)  I had lots of questions about section 3.6, concurrent
> connections, until I read your definition of connection... ;-)
> 

This is just a function of sorting the definitions in alphabetical
order. At the Munich meeting Jim McQuaid noted that the alpha sorting of
RFC 1242 allowed for very fast lookups of any term. That's why I used
it. Not sure there are enough metrics defined in this ID to justify
sorting by type of metric.

> 
> Section 1. Introduction.  "The primary metrics used in this document
> are bit forwarding rate and connections."  A nit.  As you defined
> in section 3.6, a connection is a abstraction, not a metric. Perhaps,
> "... a bit forwarding rate metric and connection-related metrics."

Good catch. That should be corrected.

dn

---

From: "Michael C. Richardson" <[email-masked]>
Date: Wed, 05 Aug 1998 17:13:06 -0400

-----BEGIN PGP SIGNED MESSAGE-----

>>>>> "David" == David Newman <[email-masked]> writes:
    David> Still, we're dealing with applications here and not streams of
    David> same-size packets. Sniffing any ftp or http session will reveal
    David> packet sizes all over the map (and acknowledgments, if we're using

  Huh? If you are doing FTP over ethernet, you will see:
	1500-40 = 1460 bytes of payload per frame

  until the last frame of the transfer. If you see any fragments it will
because you have introduced some part of the network that has a different
MTU, *and* you sender isn't doing path MTU and/or your firewall isn't
capable of passing that MTU information onwards.
  If you want to do a performance test that makes some attempt to show
performance under conditions of fragmentation (or some percentage of
fragmentated packets) then you clearly have to report the type of
fragmentation that the machine under test experienced.

    David> half-duplex links).  That makes per-frame metrics difficult. What
    David> frame size am I measuring?  Should I average all the frame sizes I
    David> see in a given capture? Use a median? A harmonic mean?

  I think a breakdown by size of frame is reasonable.

    David> Bit-based metrics don't raise those questions. I can always
    David> measure how much data I moved per unit of time, regardless of how
    David> it got chopped up along the way.

  This sounds like MIPS == Meaningless Information Passed on by Salesperson.

  I think you have to collect frame-size information during a test so that
you can verify that one system under test was not experiencing a wastely
different load than another. If that occurs that doesn't mean you can't
compare the bit-based numbers: you simply have to qualify things. 

e.g: "Firewall X had three times the throughput of firewall Y, this is probably
due to the fact that firewall X tries to help the senders and receivers use a
reasonable path MTU. Firewall Y, on the other hand, while having a 35% faster
CPU, appears to spend more of its time doing virtual fragment
reassembly. This is evidenced by the fact that we observed almost no
fragmentation on the link with firewall X when we introduced an MTU change
in the border router."

   :!mcr!:            |  Network and security consulting/contract programming
   Michael Richardson |         Firewalls, TCP/IP and Unix administration
 Personal: <A HREF="http://www.sandelman.ottawa.on.ca/People/Michael_Richardson/Bio.html">[email-masked]</A>. PGP key available.
 Corporate: <A HREF="http://www.sandelman.ottawa.on.ca/SSW/">[email-masked]</A>. 
	ON HUMILITY: To err is human, to moo bovine.