Subject: Re: Comments on bmwg-secperf-04

From: David Newman <[email-masked]>
Date: Wed, 05 Aug 1998 19:38:33 -0400

>     David> packet sizes all over the map (and acknowledgments, if we're using
> 
>   Huh? If you are doing FTP over ethernet, you will see:
> 	1500-40 = 1460 bytes of payload per frame
> 

True enough, Michael. But there are a couple of other considerations:

1. You've given an example of one application type, and firewalls handle
*multiple* application types simultaneously--each of which may have a
different packet size distribution. 

This isn't like router or switch testing where offering one packet size
at a time is sufficient. That approach doesn't describe how a firewall
will behave when it has to bobble 20 application types. What's the
forwarding rate then? Should we produce N-dimension graphs showing
multiple forwarding rates, broken down by packet size (e.g., for the
past 30 seconds we measured 6000 pps for 64-128 bytes, 1200 pps for
128-256 bytes, 400 pps for 256-512 bytes, etc.)? And should we further
break down per-packet forwarding rates by application? 

Provided I've been sufficiently rigorous in describing the offered load,
it seems much "cleaner" to me to report forwarding rate (especially
aggegrate forwarding rate) in bits rather than packets.

2. Even with only one application, there are still acknowledgments. At
least on half-duplex circuits, an ftp stream won't consist entirely of
long packets. Again, would you report separate packet rates for 64- and
1,518-byte packets? 

>   If you see any fragments it will
> because you have introduced some part of the network that has a different
> MTU, *and* you sender isn't doing path MTU and/or your firewall isn't
> capable of passing that MTU information onwards.

This is a very good point, especially with regard to firewalls based on
an application proxy architecture. Because AP-based firewalls terminate
and set up new connections on their ingress and egress interfaces, it's
entirely possible there will be different MTUs at work on different
firewall interfaces. 

Our current methodology assumes the use of only one MTU on the test bed.
That's probably not a safe assumption to make. It may be well worth
investigating whether AP-based firewalls do path MTU discovery and
whether firewalls based on stateful packet filtering pass along such
information. However, this strikes me as a methodology issue rather than
definition of metrics. 

>   I think you have to collect frame-size information during a test so that
> you can verify that one system under test was not experiencing a wastely
> different load than another. If that occurs that doesn't mean you can't
> compare the bit-based numbers: you simply have to qualify things.
> 
Again, provided the offered load is rigorously described to begin with,
does the use of per-frame metrics tell me any more about device
performance than bit-based metrics? 

dn

---

From: "Michael C. Richardson" <[email-masked]>
Date: Wed, 05 Aug 1998 21:54:03 -0400

-----BEGIN PGP SIGNED MESSAGE-----

>>>>> "David" == David Newman <[email-masked]> writes:
    >>  Huh? If you are doing FTP over ethernet, you will see: 1500-40 = 1460
    >> bytes of payload per frame
    >> 

    David> True enough, Michael. But there are a couple of other
    David> considerations:

    David> 1. You've given an example of one application type, and firewalls
    David> handle *multiple* application types simultaneously--each of which
    David> may have a different packet size distribution.

  While it is true that different applications will have different *datagram*
distribution sizes, resulting in a distribution of *frame* sizes, it isn't
true that you will have a random distribution of *fragment* sizes for a given
configuration of routers and MTUs. Let's review the terms:
	datagram == an IP payload+IP header
	frame    == a media payload+media header
	fragment == a way of splitting *datagrams* into multiple *frames*

  Now, if we are doing typical firewall applications: DNS, HTTP, FTP, SMTP,
telnet, and other HTTP 1.0-like protocols (request/response things like
whois, finger, identd, etc.) then you are seeing TCP streams. The exception
from above is DNS, but DNS tries hard to say that if it doesn't fit into
an unfragmented UDP packet, then you'd better use TCP instead.

  TCP tries to send with an MSS set so that the datagrams that it produces
are always smaller enough so that they won't fragment. The naive way has
been to assume a non-local MTU of 576 bytes. The smarter way is to do Path
MTU discovery. If a firewall causes PMTU to fail, then the throughput *will*
be lower. Worse, the firewall *load* will be higher for a given offered
network load since it will have to process more, smaller frames.

  Now, you may want to do NFS or something like that through the firewall.
(Not so crazy if you have a VPN!) If you go ahead and do 8k read/writes, then
you will get UDP fragmentation, and you will begin to make your firewall
work. But, I can tell you right now what size fragments you are going to see.

  That is a good, real life question: "what is my HTTP throughput with
background 8k NFS activity going on?"  --- an MIS guy with a mission critical
firewall in front of his web server will want that question answered before  
they turn NFS forwarding on for their firewall. If it screws up their HTTP
server, someone's head will role. "You should have tested that first". This
person may therefore want to get that data from the vendor, or even hire
some testing company to determine this for him.

  I've heard somewhere, and never checked, that NFS v3 may be able to do PMTU
as well. 

    David> This isn't like router or switch testing where offering one packet
    David> size at a time is sufficient. That approach doesn't describe how a

  Exactly. And, unless you report the distribution of frame sizes along
with the bit rate rate, the bit rate is a useless piece of information. 

  Remember, these are *SECURITY* devices. Security devices are useless if
they can be crippled from 100Mb/s "throughput" to 10Mb/s "throughput" by
sending zillions of 21 byte IP packets. Attacks of this kind are far more
prevalent than you might think. With things like QoS, VPNs, RSVP, and
diffserv on the horizon, denial of service will become a major
worry. Further, the most likely place for all of these services to be
initially is at the firewall.

  Well, maybe not useless, but at least they are overpriced. Perhaps the
readership of DataComm magazine won't understand N-dimensional graphs showing
multiple forwarding rates, but they will understand "firewall X is overpriced
given how badly it performs under attack of somewhat small fragments"

    David> firewall will behave when it has to bobble 20 application
    David> types. What's the forwarding rate then? Should we produce

  No, neither does bit rate. That is why bit-rate is a Meaningless
Information Passed on by Salesperson. It fails to answer the question: "is
this device fast enough for my application?"
  The output of bmwg ought, I think, also be useful for the educated 
customers: i.e. a group building a firewall out of smart gigabit NIC
cards.
  The NIC/Chip vendors are aware of the fact that they have to have some
support for dealing with fragments. (I had this conversation today at lunch!!!!)

    David> N-dimension graphs showing multiple forwarding rates, broken down
    David> by packet size (e.g., for the past 30 seconds we measured 6000 pps
    David> for 64-128 bytes, 1200 pps for 128-256 bytes, 400 pps for 256-512
    David> bytes, etc.)? And should we further break down per-packet
    David> forwarding rates by application?

  I would like to see all tests done with variances in as many "other"
parameters as possible. While one can't cover the entire N-dimensional array
in a practical test, if the terms don't *define* the N dimensions nobody will
try. 
  There is *NO* point in providing any forwarding rate (whether bits or
frames) unless you indicate what the load of *rejected* packets is.

[The first DataCom test, for instance, didn't even test to see if all of the
services that was supposed to be acceptable (i.e. finger) was available!!!]

  I brought this up at the working group meeting in Washington. You were
receptive, although I haven't had time to read the new document with that
view in mind. (I will get to it prior to the end of the month...)

    David> Provided I've been sufficiently rigorous in describing the offered
    David> load, it seems much "cleaner" to me to report forwarding rate
    David> (especially aggegrate forwarding rate) in bits rather than
    David> packets.

  That's fine to *report* forwarding rate in terms of bits. But it must 
include a statement about the typical (median?) and range of frame sizes,
and perhaps even a statement about the form (gaussian, bimodal, etc.) of
frame sizes observed.

    David> 2. Even with only one application, there are still
    David> acknowledgments. At least on half-duplex circuits, an ftp stream
    David> won't consist entirely of long packets. Again, would you report
    David> separate packet rates for 64- and 1,518-byte packets?

  1. the stream of acks goes in the opposite direction. 
  2. the distribution should probably be reported. ("bimodal" is the term I
  think) 

  Now, different TCP streams will get mixed together. It may very well be
that this stream of ACKs will interfere with the processing of another
TCP streams' long packets. (or visa-verca)

  Furthermore, in the case of application-layer firewalls, stateful filters
that keep lots of state, or things like TCP-spoofers for satellite links, the
idea of running out of buffer space due to the long packets gumming things
up, and then being unable to process any ACK packets and getting deadlocked
does exist!

  One of the tests I used to do is to open several hundred TCP connections
through a firewall to a server that I had setup on the other size. The client
and server both wrote 128k and then went to sleep without reading anything.

  I then made sure that I could still get *some* performance out of the
firewall given that all the firewall, client and server TCP queues are
probably full.

  The same technic can be applied against a stateful filter that does either
virtual or actual fragment reassembly.

    >> If you see any fragments it will because you have introduced some part
    >> of the network that has a different MTU, *and* you sender isn't doing
    >> path MTU and/or your firewall isn't capable of passing that MTU
    >> information onwards.

    David> This is a very good point, especially with regard to firewalls
    David> based on an application proxy architecture. Because AP-based
    David> firewalls terminate and set up new connections on their ingress
    David> and egress interfaces, it's entirely possible there will be
    David> different MTUs at work on different firewall interfaces.

  No, you have it backwards. It isn't the MTU of the interfaces of the
firewall that matter. If the firewall has differing MTU, then *it* just
generates the ICMP messages needed for PMTU, and since it is the firewall,
it shouldn't have any problems getting these messages to the right
sender.

  Application layer firewalls can do PTMU just fine because they will see
the ICMP messages. Stateful filters *seldom* do PMTU properly, and usually
either let in all ICMP messages (bad), or let in none.

    David> Our current methodology assumes the use of only one MTU on the
    David> test bed.  That's probably not a safe assumption to make. It may

  In that case, you *never* see fragments, and so you need to report that
you fact along with your bit-rate. You failed to test the most painful, and
slowest part of a filter to implement. It is also the hardest part to put
into silicon as well.

    David> be well worth investigating whether AP-based firewalls do path MTU
    David> discovery and whether firewalls based on stateful packet filtering
    David> pass along such information. However, this strikes me as a
    David> methodology issue rather than definition of metrics.

  If the terms aren't defined, then the methodology can't make use of them.

    >> I think you have to collect frame-size information during a test so
    >> that you can verify that one system under test was not experiencing a
    >> wastely different load than another. If that occurs that doesn't mean
    >> you can't compare the bit-based numbers: you simply have to qualify
    >> things.
    >> 
    David> Again, provided the offered load is rigorously described to begin
    David> with, does the use of per-frame metrics tell me any more about
    David> device performance than bit-based metrics?

  No. Both metrics are equally useless. Neither answer the question "will it
work for me?" or "will a cracker still all my bandwidth?"

   :!mcr!:            |  Network and security consulting/contract programming
   Michael Richardson |         Firewalls, TCP/IP and Unix administration
 Personal: <A HREF="http://www.sandelman.ottawa.on.ca/People/Michael_Richardson/Bio.html">[email-masked]</A>. PGP key available.
 Corporate: <A HREF="http://www.sandelman.ottawa.on.ca/SSW/">[email-masked]</A>. 
	ON HUMILITY: To err is human, to moo bovine.