Subject: Re: [bmwg] First draft for next-gen firewall (NGFW) performance benchmarking uploaded

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Tue, 9 Jan 2018 02:09:14 +0000

Hi Carsten and Bala,

Thanks for preparing a clear draft that describes
the work you propose.

I read your draft today, and have a few comments
(appended at the end of this message).

I’ll also try to answer your questions in-line, below.
regards,
Al
(as a participant)

From: bmwg [mailto:[email-masked]] On Behalf Of Carsten Rossenhoevel
Sent: Monday, December 18, 2017 10:36 AM
To: [email-masked]
Subject: [bmwg] First draft for next-gen firewall (NGFW) performance benchmarking uploaded

Dear Benchmarking Methodology WG,
My colleague Bala Balarajah has uploaded the first draft of the next-generation firewall (NGFW) benchmarking methodology for your review: draft-balarajah-bmwg-ngfw-performance-00<https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dbalarajah-2Dbmwg-2Dngfw-2Dperformance-2D00&d=DwMDaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=NSrvKLCc3BVODl6aPsYTR-ouHh9nc1inoRHhJLeNU60&s=O94wjso8epI3iVSQ2FxOKIWhFA66zEp4vWB04dfKQE8&e=>.

Currently the document contains sections for the test setup, test bed preparation, reporting guidelines and test cases (complete TOC below).  Please let us know specifically:

- How do you assess the first test case in section 7?  Its format and level of details is meant to serve as a blueprint for additional test cases.
[ACM]
The use of many different dimensions of performance criteria in
7.1.3.4.  Test Results Acceptance Criteria
such as:
   a.  Number of failed Application transaction MUST be 0.01%.

   b.  Number of Terminated TCP connection due to unexpected TCP RST
       sent by DUT/SUT MUST be less than 0.01%
seems very close to PASS/FAIL criteria, which BMWG’s charter precludes:
https://datatracker.ietf.org/wg/bmwg/about/
“Because the demands of a particular technology may vary from deployment
to deployment, a specific non-goal of the Working Group is to define
acceptance criteria or performance requirements.”

However, reporting the number|percent of failed Application transactions
(for example) along with a benchmark like Throughput would be seen
as providing reasonable auxiliary information about the measurement.
These Aux measurements seem to be referred to as KPIs in the draft.

- What do you think about the test equipment configuration section, specifically the traffic load profile and flows in section 4?
[ACM]
See below.

- Section 5 is a bit unusual as it defines test bed requirements for minimum performance.  These were usually taken for granted in the past; for virtualized test solutions they need to be made explicit we (Bala and I) feel.
[ACM]
I agree.

Any feedback and comments are very welcome!  Bala and I will process them swiftly - either before Dec 22 or in the first week of January.

Best regards, Carsten

[ACM]
General:
it seems that a lot of “v” characters have been replaced with “^”.

Section 4.1
I think Figure 1 may not need the Emulated Router function
in the Test Equipment, since it includes external switch/router:

   +-------------------+      +-----------+      +--------------------+
   |Aggregation Switch/|      |           |      | Aggregation Switch/|
   | Router            +------+  DUT/SUT  +------+ Router             |
   |                   |      |           |      |                    |
   +----------+--------+      +-----------+      +----------+---------+
              |                                             |
              |                                             |
  +-----------+-----------+                    +------------+----------+
  |                       |                    |                       |
  | +-------------------+ |                    | +-------------------+ |
  | | Emulated Router(s)| |                    | | Emulated Router(s)| |
  | |     (Optional)    | |                    | |     (Optional)    | |
  | +-------------------+ |                    | +-------------------+ |
  | +-------------------+ |                    | +-------------------+ |
  | |      Clients      | |                    | |     Servers       | |
  | +-------------------+ |                    | +-------------------+ |
  |                       |                    |                       |
  |    Test Equipment     |                    |    Test Equipment     |
  +-----------------------+                    +-----------------------+

                    Figure 1: Testbed Setup - Option 1

COMMENT: I think the Figures should each indicate which side is
the “protected network”, as done in RFC 3511.

Section 4.2 DUT/SUT Config
I like the Table 1: DUT/SUT Feature List, it clearly shows
current coverage and future plans.

Section 4.3 Test Equipment Config
I suggest:
OLD
   This document attempts to explicitly specify which test equipment
   parameters SHOULD be configurable, any such parameter(s) MUST be
   noted in the test report.
NEW
   One goal of this document is to specify the set of test equipment
   parameters that SHOULD be configurable. All such configurable
   parameter(s) MUST be noted in the test report.

4.3.1.  Client Configuration
OLD
   This section specifies which parameters SHOULD be considerable while
   configuring emulated clients using test equipment.  Also this section
NEW
   This section specifies which parameters SHOULD be configurable when
   using emulated clients in the test equipment.  Also this section

4.3.1.1.  TCP Stack Attributes
OLD
   ... Up to 3 retries
   SHOULD be allowed before a timeout event is declared.
NEW
   ...Up to 3 retransmissions
   SHOULD be sent before a timeout event is declared.

COMMENT: TCP RTO isn’t actually calculated for these connections?

4.3.1.2.  Client IP Address Space
...
   The IPv4 ToS byte should be set to '00'.
COMMENT: the TOS byte is obsolete terminology, now DS Field, etc.
for v4 and v6.
...
   The following equation can be used to determine the required total
   number of client IP address.

   Desired total number of client IP = Target throughput [Mbit/s] /
   Throughput per IP address [Mbit/s]
COMMENT: This equation might work with CBR UDP senders, but using many
         parallel TCP connections results in less predictable bit
         rate per connection.

4.3.1.3.  Emulated Web Browser Attributes
COMMENT: Reacting to
“each browser will open up to 6 TCP connections per Server endpoint IP”
The performance variation that results from interactions between
independent TCP connection flow-control is now understood to be
significant, and may affect the results repeatability. See
https://tools.ietf.org/html/draft-ietf-ippm-model-based-metrics-13#section-4
(which is approved, and awaiting publication).
There is much to discuss here that requires full appreciation of
TCP operation in some detail. One major risk is that different
test equipment cannot produce equivalent results when testing
the same DUT with the same configurations.

It is also worthwhile to consider some of the techniques for
stateful TCP described in RFC 7640:
https://tools.ietf.org/html/rfc7640#section-5.2

4.3.1.4.  Client Emulated Web Browser SSL/TLS Layer Attributes
COMMENT:  SSL Inspection seems to be for future work in Table 1.  ??

4.3.2.  Backend Server Configuration
4.3.2.1.  TCP Stack Attributes
...
“... Up to 2 retries SHOULD be allowed
   before a timeout event is declared.
COMMENT: why is the number of retries lower for backend servers
than for clients?  (see 4.3.1.1, similar wording changes suggested here)

4.3.2.2.  Server Endpoint IP Addressing
>>> same comment on TOS byte

4.3.3.1.  Description of Intra-Client Behavior
COMMENT: This is likely another critical specification to
support repeatable results. Also:
s/conical/canonical/  (I think)

4.3.4.  Traffic Load Profile
COMMENT: It seems that attaining “sustain” phase, where:
“  ... the test equipment should keep to generate
   traffic at constant rate for a constant number of active client IPs.
   The recommended time duration for sustain phase is 600 seconds.  This
   is the phase where measurements occur.”
requires some special control over TCP connections, which are generally
not considered to be constant rate.  Maybe this is an Average rate over
some averaging time?

5.  Test Bed Considerations
OLD
   Once the desired maximum performance goals for the system under test
   have been identified, a safety margin of 10 % SHOULD be added for
   throughput and subtracted for maximum latency and maximum packet
   loss.
Suggest:
   Once the desired maximum performance capabilities of the Test Bed
   have been identified, a safety margin of 10 % SHOULD be subtracted for
   throughput and added for maximum latency and maximum packet
   loss. These values MUST exceed the performance goals of the SUT/DUT.

OLD
   Test bed preparation can be performed either by configuring the DUT
   in the most trivial setup (fast forwarding) or without presence of
   DUT.
Suggest:
   Test bed preparation can be performed either by configuring the DUT
   in the most trivial setup (fast forwarding) or without presence of
   DUT (and adding emulated delay to match the SUT/DUT when under test.

Skipping Section 6 for now.
Section 7 comments above.

10.  Security Considerations
The BMWG has a “standard” Security Considerations section text
that you can use as a starting point.

12 Normative references.
RFC 3511 should be prominent in this list (after considering its content).