Subject: Comments on bmwg-secperf-04

From: Kevin Dubray <[email-masked]>
Date: Mon, 03 Aug 1998 18:51:16 -0400

Hi David,

In section 3.4, Bit Forwarding Rate, you write:  "...this
measurement counts bits per second rather than frames per
second.  Per-frame metrics are not meaningful in the context
of a flow of application data between endpoints."

Two comments.  First, if you leave the above declaration (i.e.,
per frame metrics are not meaningful), you may wish to offer a
explanation as to your rationale. This would help the reader
build an understanding as to your conclusion.

Second, if per-frame performance is meaningless in the firewall 
context, (I'm not sure I _totally_ agree), why are you carrying the 
frame overhead in the metric versus some sort of Effective Bit
Forwarding Rate (bit fwd rate of data payload component only?)

Section 3.7.  Connection.  I would move this wonderfully defined
abstration ahead of the connection-related metrics (e.g. concurrent
connections.)  I had lots of questions about section 3.6, concurrent
connections, until I read your definition of connection... ;-)

Section 1. Introduction.  "The primary metrics used in this document
are bit forwarding rate and connections."  A nit.  As you defined
in section 3.6, a connection is a abstraction, not a metric. Perhaps,
"... a bit forwarding rate metric and connection-related metrics."

Otherwise, this draft has come along nicely since LA!

-Kevin

---

From: "Mike Winslow" <[email-masked]>
Date: Mon, 3 Aug 1998 21:02:24 -0400

Hi,  if I may comment:

Frame Relay congestion metrics Burst size and burst time both have bearing
on a per frame basis.

These are used to form the committed information rate.

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]]
Sent: Monday, August 03, 1998 7:28 PM
To: [email-masked]
Cc: [email-masked]
Subject: Comments on bmwg-secperf-04

Sender: [email-masked]
Received: from ironbridgenetworks.com (helios.ironbridgenetworks.com
[146.115.140.2])
	by dub-img-7.compuserve.com (8.8.6/8.8.6/2.12) with SMTP id TAA05672;
	Mon, 3 Aug 1998 19:27:40 -0400 (EDT)
Received: from ironbridgenetworks.com by ironbridgenetworks.com
(SMI-8.6/SMI-SVR4)
	id SAA20643; Mon, 3 Aug 1998 18:52:06 -0400
Message-ID: <[email-masked]>
Date: Mon, 03 Aug 1998 18:51:16 -0400
From: Kevin Dubray <[email-masked]>
Organization: IronBridge Networks
X-Mailer: Mozilla 4.05 [en] (WinNT; I)
MIME-Version: 1.0
To: [email-masked]
CC: [email-masked]
Subject: Comments on bmwg-secperf-04
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

Hi David,

-Kevin

---

From: [email-masked]
Date: Thu, 6 Aug 1998 07:08:39 -0400

David wrote:
------------------------
Again, provided the offered load is rigorously described to begin with,
does the use of per-frame metrics tell me any more about device
performance than bit-based metrics?
------------------------

I think that the frames VS. bits question is the wrong question.  Clearly,
each metric carries somewhat different information about performance.
Early router tests certainly showed that handling high packet rates was the
challenge, since per packet processing load was roughly equal - handling
the same data rates with large packets was much easier.

IMHO the best is clearly to be able to produce both metrics (the underlying
measurement ought to be the same).  The ratio of the two metrics itself
might be a figure of interest.  Obviously it denotes the average frame size
over the test interval, but different DUT/SUT may well prove to have
different sensitivities, some more vulnerable to high frame rates and
others having difficulty with high bit rates, indicating [warning: fact
free example] perhaps a CPU weakness in the first case and a bus transfer
problem in the second.

Jim McQuaid
NetScout Systems

---

From: David Newman <[email-masked]>
Date: Thu, 06 Aug 1998 19:51:28 -0400

> measurement ought to be the same).  

Exactly. . .this is why I'd supposed bit-based metrics would be
sufficient.

The ratio of the two metrics itself
> others having difficulty with high bit rates, 

I'm coming around on the use of frame-based metrics. As I said earlier
to Michael, my concern is that if we use pps metrics, we will need some
means of indicating size distribution in any given load. Just saying
"pps" alone ain't enough if we've got sizes all over the place.

> indicating [warning: fact free example] 
 
;-)

dn

---

From: David Newman <[email-masked]>
Date: Fri, 07 Aug 1998 19:46:54 -0400

Lots of good stuff here. Comments inline.

> Section 3.6:
>       connections are in fact capable of transferring data. If a data
>       cannot be sent over a connection, that connection should not be
>       counted toward the number of concurrent connections.
> 
>   This might seem to suggest that TCP connections through an application
> layer firewall which have been flow controlled should not be counted. I would
> assume it is only meant to apply to things like TCP connections in FIN_WAIT_2
> or something.

Yes, it's intended to apply only to situations where the DUT/SUT itself
is the problem, not because of any external change like another box
changing TCP window size.

It would be a good idea to add a sentence making this point. I don't
think it makes sense to describe specific TCP states since 1) state may
not be discernable to the tester and 2) this metric may be used with
connection types other than TCP (ATM, for example). In any event, this
is a caveat worth adding.

> 
> Section 3.8:
> 	connection establishment rate could apply to "stateful" UDP by noting
> 	the latency for a packet to arrive on one interface and then leave
> 	by the other interface.
> 
>   I think that the use of "rate" here is wrong. That implies  X
> connections/sec. What is being measured is in fact the connection
> establishment latency. The rate may be 1/latency, or it might be much higher
> if the firewall can overlap may connection setup times.
>   There is no mention here that this may also just be a measure of how
> quickly the target host can create a new TCP connection.
> 

X connections/second is exactly the point here. This is analagous to the
MAC address learning rate described in section 3.8.2 of RFC 2285. For
connections, as with MAC addresses, we want two measurements: First,
what's the highest absolute number a DUT/SUT can sustain? Second, what's
the highest *rate* at which these can be established?

In the first test, a relatively high number of connections is attempted
at a relatively low rate. In this test, a relatively low number of
connections is attempted at progressively higher rates. 

The rate test is indeed a measure of how quickly the DUT/SUT can create
new TCP connections.

> 3.15:
>   Now, you said that there were specific text for illegal traffic and
> rejected traffic. Why isn't a measurement unit given?
> 

Because these are definitions of traffic classes, not measurement units.

> 3.16:
> 	if the firewall logs to a different host, and that network connection
> 	is one of the ones used for data transfer, 

How common is this? Chapman and Zwicky recommend moving log data over an
OOB serial link to write-once media, and elsehwere I've heard in-band
log transfers referred to as a Bad Idea from a security standpoint. 

> 
> 3.17: NAT
> 	the paragraph on proxy is good, but not very clear. I suggest making
> 	it a seperate paragraph from the type of NAT.
> 
> 	"It should be noted that many proxy firewalls always perform NAT as a
> 	function of their architecture. They are typically configured by
> 	default to use the aggregation method, but may be configured to
> 	do a identity translation (no translation), or some other mapping.
> 	Since proxy based firewalls do some translation, the tester is
> 	unlikely to observe a performance increase when NAT is turned off."

This isn't always true. There are AP implementations on which NAT cannot
be disabled.

> 
> 3.24:
>       "The order of rules within the rule set is critical. Firewalls
>       generally scan rule sets in a "top down" fashion, which is to say
>       that the device compares each packet received with each rule in
>       the rule set until it finds a rule that applies to the packet.
>       Once the device finds an applicable rule, it applies the actions
>       defined in that rule (such as forwarding or rejecting the packet)
>       and ignores all subsequent rules. For testing purposes, the rule
>       set MUST conclude with a rule denying all access."
> 
>   This is awfully Cisco Access Control list-like specific.
> 
>   the order of rules is *often* critical. However, a number of firewalls
> either do not permit one to define this order, provide some way to attach
> precedence to rules, or define explicit ways of resolving ambiguities
> between rules.
>   The IPsec SPD uses the second method (explicit precedence), something
> I personally find a pain because it makes optomization very difficult.
> 
>   I expect the rule set to be described. I'd even like its size to be
> varried.
> 
Completely agree. The contents of the rule set is going to be a major
component (maybe *the* major component) of the methodology. Based on
your comments, I'd change this to make clear that the rule set contents
MUST be described. Size of the rule set is an issue for the methodology.

You also make a good point about the global deny not necessarily being
the last rule. This should be changed to say that such a rule MUST be
included without specifying where.

> 3.25: many stateful filters permit their statefulness to be disabled. A
> 	test should establish the performance penalty in turning stateful
> 	filter on.
> 
> 4:
> 	It isn't enough to just have the safeguards in place. You must also
> 	excercise them. If you don't, then you don't know the performance
> 	impact of an attack.
> 

Clarification needed here. Are you saying that performance should be
benchmarked with and without a given safeguard in place or (more likely)
that performance measurements should be taken with and without an attack
in progress?

If it's the former, that's not a bad idea but bear in mind that some
implementations don't allow safeguards to be disabled.

The latter idea is very interesting, with a couple of caveats:

1. Some attacks have no performance penalty (IP spoofing, for example),
and that raises the question of what attacks should and shouldn't be
measured. Defining which safeguards to use is going to be one of the
thorniest challenges of the methodology; defining which subset to use
for performance benchmarks is an even bigger can of worms. Even once a
set of attacks is agreed upon, there are then questions about the
offered load and the number of sources from which the attack is mounted.

2. This will exponentially increase the number of measurements. Assume
at least one measurement per type of attack (and more, if we experiment
with different traffic orientations and rule sets of various sizes).
Commercial scanners today mount anywhere from 200 to nearly 2,000
different forms of attack (with more added almost daily), so this is
potentially a very large number of data points. 

That said, I agree that benchmarking the performance impact of an attack
is a useful thing to do. The challenge is to describe this in a generic
enough way so as not to burden the tester with huge numbers of data
points, only a small fraction of which will be useful. I'll try to have
a working definition for this ready by Chicago.

dn