Subject: Re: Comments on bmwg-secperf-04

From: David Newman <[email-masked]>
Date: Thu, 06 Aug 1998 09:52:56 -0400

> David> This isn't like router or switch testing where offering one packet
>     David> size at a time is sufficient. That approach doesn't describe how a
> 
>   Exactly. And, unless you report the distribution of frame sizes along
> with the bit rate rate,

As you mentioned in your earlier message, you already know the packet
sizes of each traffic type you offer--e.g., FTP on Ethernet is always
going to generate 1518-byte packets absent some difference in MTU
somewhere on the test bed. You propose to describe this after the
measurement; I suggest that the tester had better describe that info
beforehand.

What's less clear to me is how to describe the real-world example you
present: What's my http throughput with a background of 8k NFS traffic?
Fair question, but let's be precise about what metrics we use. I presume
you'd want to describe both flows using a per-frame metric. Do we say
that http throughput is, say, 1,024 64-byte pps while NFS traffic is
2,048 1,024-byte pps? That confuses the issue by using the same metric
to define different things. 

I'm not averse to using a per-frame metric here, but we'd better come up
with some precise way of describing it.

>   There is *NO* point in providing any forwarding rate (whether bits or
> frames) unless you indicate what the load of *rejected* packets is.
> 
> [The first DataCom test, for instance, didn't even test to see if all of the
> services that was supposed to be acceptable (i.e. finger) was available!!!]
> 
>   I brought this up at the working group meeting in Washington. You were
> receptive, although I haven't had time to read the new document with that
> view in mind. (I will get to it prior to the end of the month...)

Rather than getting into the old "I haven't read the draft but here's
what I think" bind ;-), please do read draft 04--it does define classes
called illegal and rejected traffic with exactly this purpose in mind. 

Also, please don't confuse Data Comm tests with this draft. For the
record, the '97 and '98 Data Comm tests have focused on security,
management, and performance. Michael is correct in stating that the '95
test didn't verify the presence of some specific services; however, that
was not a goal of the test. 

This draft deals primarily with performance; let's keep focused on that.

dn

---

From: "Michael C. Richardson" <[email-masked]>
Date: Thu, 06 Aug 1998 11:11:51 -0400

-----BEGIN PGP SIGNED MESSAGE-----

  I am not for per-frame metrics, or against per-bit metrics. I claim
instead that neither is useful in isolation.
  Yes, on a *given* homogeneous MTU network you know what the frame
size is. But, on such a network there should be no fragments due to TCP.
  I expect the tester to:
	1. confirm that there were no fragments (and report it)
	2. try the test with a non-homogeneous MTU network

  In particular, I expect the tester of a firewall device to determine
the performance of the firewall while under attack of X% load of worst case
fragmented packets (21 byte packets).

>This draft deals primarily with performance; let's keep focused on that.

  I assume you are involved in this effort because Data Comm wants to use
the result as part of its next set of tests. This isn't sinister or anything,
it is expected. My concern is that the terminology is being overly simplified
on the assumption that the terms defined in the draft can be presented
to the public directly.

  I have now read all of the draft, and offer the following comments.

Section 3.6:
      connections are in fact capable of transferring data. If a data
      cannot be sent over a connection, that connection should not be
      counted toward the number of concurrent connections.

  This might seem to suggest that TCP connections through an application
layer firewall which have been flow controlled should not be counted. I would
assume it is only meant to apply to things like TCP connections in FIN_WAIT_2
or something.

Section 3.8:
	connection establishment rate could apply to "stateful" UDP by noting
	the latency for a packet to arrive on one interface and then leave
	by the other interface. 

  I think that the use of "rate" here is wrong. That implies  X
connections/sec. What is being measured is in fact the connection
establishment latency. The rate may be 1/latency, or it might be much higher
if the firewall can overlap may connection setup times.
  There is no mention here that this may also just be a measure of how
quickly the target host can create a new TCP connection.

  I like 3.9.

  An additional term for the DMZ is sometimes the "service network"
Some literature refers to the space between the firewall and the border
router (physically under you control, but logically not) as the DMZ. I 
would favour calling it the "perimeter" network and giving DMZ as an alias.

3.15:
  Now, you said that there were specific text for illegal traffic and
rejected traffic. Why isn't a measurement unit given? 

3.16:
	if the firewall logs to a different host, and that network connection
	is one of the ones used for data transfer, then I think a measure of
	how much data is logged should be kept. In the event of an attack
	on the outside, the firewall could (and maybe SHOULD if configured
	that way) spend all of its available bandwidth on that interface just
	reporting the attack.  This will affect throughput.

3.17: NAT
	the paragraph on proxy is good, but not very clear. I suggest making
	it a seperate paragraph from the type of NAT.

	"It should be noted that many proxy firewalls always perform NAT as a
	function of their architecture. They are typically configured by
	default to use the aggregation method, but may be configured to 
	do a identity translation (no translation), or some other mapping.
	Since proxy based firewalls do some translation, the tester is
	unlikely to observe a performance increase when NAT is turned off."

	Metric: number of translation rules.
	(this is especially relevant most simpler packet filters)

3.24:
      "The order of rules within the rule set is critical. Firewalls
      generally scan rule sets in a "top down" fashion, which is to say
      that the device compares each packet received with each rule in
      the rule set until it finds a rule that applies to the packet.
      Once the device finds an applicable rule, it applies the actions
      defined in that rule (such as forwarding or rejecting the packet)
      and ignores all subsequent rules. For testing purposes, the rule
      set MUST conclude with a rule denying all access."

  This is awfully Cisco Access Control list-like specific.
  
  the order of rules is *often* critical. However, a number of firewalls
either do not permit one to define this order, provide some way to attach
precedence to rules, or define explicit ways of resolving ambiguities 
between rules.
  The IPsec SPD uses the second method (explicit precedence), something
I personally find a pain because it makes optomization very difficult.

  I expect the rule set to be described. I'd even like its size to be
varried.

3.25: many stateful filters permit their statefulness to be disabled. A 
	test should establish the performance penalty in turning stateful
	filter on.

4:
	It isn't enough to just have the safeguards in place. You must also
	excercise them. If you don't, then you don't know the performance
	impact of an attack. 

   :!mcr!:            |  Network and security consulting/contract programming
   Michael Richardson |         Firewalls, TCP/IP and Unix administration
 Personal: <A HREF="http://www.sandelman.ottawa.on.ca/People/Michael_Richardson/Bio.html">[email-masked]</A>. PGP key available.
 Corporate: <A HREF="http://www.sandelman.ottawa.on.ca/SSW/">[email-masked]</A>. 
	ON HUMILITY: To err is human, to moo bovine.