Subject: Re: [bmwg] WLAN switch controller I-Ds for BMWG to consider

From: David Newman <[email-masked]>
Date: Tue, 10 Jul 2007 21:33:04 -0700

On 6/8/07 1:49 PM, Tom Alexander wrote:
> BMWGers,
> 
> Back at the 67th IETF last November, there was a presentation discussing a
> proposal on CAPWAP (WLAN) switch controller benchmarking. The presentation
> ended by threatening to solicit some help and submit some drafts.
> 
> The help was solicited, and two drafts have been submitted to the IETF I-D
> database:
> 
> draft-alexander-bmwg-wlan-switch-term-00.txt  -- terminology
> draft-alexander-bmwg-wlan-switch-meth-00.txt -- methodology
> 
> Review and comments on these drafts would be much appreciated.

I support the bmwg undertaking these IDs as work items, and congratulate
the authors for posting strong draft zeros.

There are several areas where I believe more specifics would help:

1. In any environment where contention for the medium exists -- and
802.11 WLANs are a poster child for this -- the device that puts a frame
on the medium first wins. This has led to a sort of arms race among
vendors for control of the airwaves.

The methodology makes no mention of three contention mechanisms that can
heavily influence test results. In my experience these mechanisms have
been the main factors in differentiating performance from different
vendors' systems and thus should be included in any test methodology:

The mechanisms are:

a. Interframe space type. It is possible and standards-compliant to use
SIFS instead of DIFS interframe space) under some circumstances. At a
minimum, the type of IFS in use should be declared. Better still would
be a test to measure the IFS in use.

Early Ethernet switches cheated on IFS to gain an advantage in
half-duplex environments. While I am not aware of IFS cheating in WLAN
equipment, it is possible and desirable to measure the spaces between
frames.

b. Coordination function type. Some WLAN access points implement PCF
(point coordination function) instead of the more common DCF
(distributed coordination function). This can be highly desirable in
situations where the flow of traffic is primarily downstream from WTP to
end-station. It gives the WTP far more opportunities for access to the
medium than end-stations.

Again, the CF type in use should be declared at a minimum. More
preferable would some externally observed measurement of the CF type.

c. Client contention behavior. Most but not all WLAN stations use a
exponential backoff algorithm to deal with contention for the medium.
Obviously, a end-station or WTP that fudges this a bit will get its
frame on the medium first, potentially "winning" every collision battle.
(This is a crude example; more sophisticated variations of this might
involve winning only some percentage of the time and/or altering the
winning percentage in response to RF or other conditions.)

It is possible and desirable to measure client contention behavior. I
encourage development of tests in this area.

2. In section 3.3.3.1 of the test methodology ID, the description of the
open-air test bed is seriously underspecified, which almost certainly
will lead to a lack of reproducibility of test results in different labs.

Rather than adding specificity, I would rather see open-air testing
discouraged altogether in the context of this document. (Note: I have
nothing against open-air testing in a *network* context, specific to one
operators' network; in a *lab* context, however, there's just too much
variability.)

3. Just a wording nit in both documents:

s/best efforts/best effort/

4. Also a wording nit, but again in both documents there are several
instances of the term "etc." Specific enumeration would be better.

dn

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: "Tom Alexander" <[email-masked]>
Date: Thu, 12 Jul 2007 18:15:49 -0700

David,

Thanks for your support, and for your detailed review and feedback. Let me
address each of your points in turn.

Item 1. Generally agreed. I've found several vendors (no names, please! ;-)
playing games with MAC protocol parameters to gain preferential access to
the medium. I was trying to avoid the notion of running a functional test on
the SUT, but you are quite correct that we need some way of either detecting
or, at a minimum, reporting these games. (If anyone on this list can point
us at an I-D or RFC to serve as an example, that would be great.)

Item 2. Also generally agreed. This is an area that has to be cleaned up.
There is a distinction between "open-air testing" (which should be heavily
discouraged or outlawed for serious benchmarking efforts) versus
"over-the-air testing" (which is done in calibrated anechoic chambers and is
therefore both predictable and reproducible). The document fuzzes over the
two. This needs to be fixed.

Item 3. Thanks for pointing this out - the next revision will make this
change.

Item 4. I confess that my professorial instinct to say "... and this is left
as an exercise for the reader" is showing through. :-) I will clean up all
the casual "etc"s in the next revision.

Again, I appreciate the review.

Best regards,

- Tom

-----Original Message-----
From: David Newman [mailto:[email-masked]] 
Sent: Tuesday, July 10, 2007 9:33 PM
To: [email-masked]
Cc: 'Muninder Sambi (msambi)'; 'Jerry Perser'
Subject: Re: [bmwg] WLAN switch controller I-Ds for BMWG to consider

The mechanisms are:

dn

---

From: David Newman <[email-masked]>
Date: Thu, 12 Jul 2007 20:06:11 -0700

> us at an I-D or RFC to serve as an example, that would be great.)

Glad we're in agreement!

Functional testing *is* performance testing here, or at least a heavy
influence. Games vendors play with IFS, CF type, and client contention
algorithms can (and do, in my experience) alter the outcome of most
tests described in the methodology ID.

The forward pressure test in RFC 2889, section 5.6, began life as a way
of catching out Ethernet switches playing similar games. There may be
other examples; anyone else know any?

dn

---

From: "Tom Alexander" <[email-masked]>
Date: Wed, 18 Jul 2007 09:18:23 -0700

BMWGers,

Just a reminder that we'd posted a couple of BMWG-related WLAN controller
testing I-Ds a while back, and this is a final poke to the list for
comments. If you haven't done so already (thanks, David!) please review the
documents and send your comments in before the Chicago meeting.

If we don't get any further comments we have to assume that fixing the
issues David pointed out would make them perfect! :-)

Best regards,

- Tom A. (on behalf of the other I-D authors)

---

From: Curtis Villamizar <[email-masked]>
Date: Wed, 18 Jul 2007 14:13:18 -0400

In message <001601c7c957$458d14c0$dc0e10ac@tomd600>
> - Tom A. (on behalf of the other I-D authors)

Or lack or interest.

Why is the IETF BMWG proposing benchmarks for a purely IEEE L2 spec?
What next, database systems?

Curtis

---

From: Al Morton <[email-masked]>
Date: Wed, 18 Jul 2007 14:45:40 -0400

>Why is the IETF BMWG proposing benchmarks for a purely IEEE L2 spec?

Our scope is historically wider than IETF-developed protocols alone.
See the RFCs on LAN Switches, ATM and Frame Relay on our charter page.

Al
bmwg chair

---

From: "Tom Alexander" <[email-masked]>
Date: Wed, 18 Jul 2007 11:51:10 -0700

Hi, Curtis,

Actually these I-Ds have virtually nothing to do with the IEEE 802.11 L2
specifications. Instead, they are aimed at testing WLAN controllers,
protocols for which are being worked on by the CAPWAP group in IETF. The
enclosed presentation (from the November meeting) gives the rationale.

Basically: if the IETF believes that WLAN controllers fall within their
scope and has chartered the CAPWAP group to standardize them, shouldn't BMWG
likewise consider that benchmarking these WLAN controllers is within scope
as well?

On a technical note, CAPWAP-based WLAN controllers have very little of the
IEEE 802.11 standards implemented in them, and certainly have no wireless
interfaces at all. Instead, they are purely Ethernet based, act as
switches/routers for IP traffic, implement IETF-defined authentication and
security protocols, support IETF-defined QoS functions such as DiffServ, and
so on.

The posted I-Ds have a bit of wireless content simply because the easiest
way to gain access to the interfaces of a CAPWAP-based WLAN controller may
be through their wireless access points. Also in some cases it makes sense
to test the controller and APs as a SUT. However, the I-Ds specifically
provide guidelines on how the 802.11 MAC and PHY can be taken out of the
equation so that they do not affect the results.

The purpose of the I-Ds is to test the IETF-defined content of the WLAN
controller, not the IEEE-defined content of the APs. There is already a
standards document (802.11.2) that explains how to test the MAC/PHY of the
APs.

Best regards,

- Tom

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Wednesday, July 18, 2007 11:13 AM
To: Tom Alexander
Cc: [email-masked]
Subject: Re: [bmwg] WLAN switch controller I-Ds for BMWG to consider 

Or lack or interest.

Curtis

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Wed, 18 Jul 2007 21:32:03 +0200

Let me complement Al's phrase: 

... to the extent that these protocols are part of IP-based
internetworking solutions.

Which seems to be the case here. 

Dan
(speaking as a contributor)

> From: Al Morton [mailto:[email-masked]] 
> Sent: Wednesday, July 18, 2007 9:46 PM
> To: [email-masked]; Tom Alexander
> Cc: [email-masked]
> > > If we don't get any further comments we have to assume 
> that fixing 
> > > the issues David pointed out would make them perfect! :-)
> > >
> > > Best regards,
> > >

---

From: Curtis Villamizar <[email-masked]>
Date: Wed, 18 Jul 2007 15:40:58 -0400

In message <000101c7c96c$9cee43a0$dc0e10ac@tomd600>
> standards document (802.11.2) that explains how to test the MAC/PHY of the

OK.  Thanks.

Curtis

ps - PDF please, not PPT.

---

From: "Tom Alexander" <[email-masked]>
Date: Wed, 18 Jul 2007 14:16:46 -0700

BMWGers (and Curtis),

Sorry for shipping out the PPT instead of the PDF in my previous message. I
was much too quick on the "send" button and had forgotten that the
presentation was in the wrong format.

Actually there was no need to include the presentation file at all. As Al
gently reminded me, it's on line in the IETF proceedings. Per Al's request
to me, here is a resubmitted message to the reflector, with the link:

---------------------------------
Actually these I-Ds have virtually nothing to do with the IEEE 802.11 L2
specifications. Instead, they are aimed at testing WLAN controllers,
protocols for which are being worked on by the CAPWAP group in IETF. A
presentation from the November meeting - see
http://www3.ietf.org/proceedings/06nov/slides/bmwg-4/sld1.htm - gives the
rationale.

The purpose of the I-Ds is to test the IETF-defined content of the WLAN
controller, not the IEEE-defined content of the APs. There is already a
standards document (802.11.2) that explains how to test the MAC/PHY of the
APs.
---------------------------------------

Best regards,

- Tom