Subject: Re: [bmwg] WLAN switch controller I-Ds for BMWG to consider

From: Curtis Villamizar <[email-masked]>
Date: Wed, 18 Jul 2007 15:37:22 -0400

In message <[email-masked]>
Al Morton writes:
>  
> At 02:13 PM 7/18/2007, Curtis Villamizar wrote:
> > > If we don't get any further comments we have to assume that fixing the
> > > issues David pointed out would make them perfect! :-)
> > >
> > > Best regards,
> > >
> > > - Tom A. (on behalf of the other I-D authors)
> >
> >
> >Or lack or interest.
> >
> >Why is the IETF BMWG proposing benchmarks for a purely IEEE L2 spec?
>  
> Our scope is historically wider than IETF-developed protocols alone.
> See the RFCs on LAN Switches, ATM and Frame Relay on our charter page.
>  
> Al
> bmwg chair

At least for FR there were issues regarding IP that FR Forum and ATM
Forum would not address (largely due to being in denial about there
being any issues).  For LAN switches HOL blocking issues could be
identified and these were being used in public exchanges.  This was in
part motivated by experiences with HOL blocking with DEC switches in
these public exchanges.  The focus was not on 10baseT hubs or 5 port
10/100 switches.  If anything the LAN switch thing is a disservice for
ignoring jumbo frames which remains an ISP absolute requirement.

802.11 is more of a desktop, home, enterprise, well beyond the
Internet provider edge sort of thing.  Except maybe ptp in some edges.
I think that is a difference worth considering.

Curtis

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: "Tom Alexander" <[email-masked]>
Date: Wed, 18 Jul 2007 14:39:20 -0700

This is an interesting point. Is benchmarking for enterprise equipment
outside the BMWG charter? It does not seem apparent from the official BMWG
web page, but I am comparatively new to the group and quite likely have
missed something.

IEEE 802.11 looked at WLAN controller benchmarking and decided that it is
out of their scope. They in fact sent a letter over to BMWG last year saying
so. Their view was that WLAN controllers operate well above the MAC layer
and therefore have nothing to do with a MAC/PHY group such as 802.11. Plus
the fact that the standards body for WLAN controllers is in IETF was
significant.

Modern 802.11 controllers are following the historical trend of Ethernet LAN
switches and are migrating out of desktops/homes and into service provider
networks. There's quite a bit of activity in metropolitan applications for
WLANs today. My colleagues and I have seen interest from access service
providers and their vendors for usable performance metrics for metro WLAN
equipment.

Best regards,

- Tom

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 19 Jul 2007 10:08:15 -0400

In message <003d01c7c984$1a5f2ea0$dc0e10ac@tomd600>
> web page, but I am comparatively new to the group and quite likely have

Might be a personal prejudice on my part seeing as the work I've done
is very IP core centric.  Historically the BMWG started out with
simple PPS tests on high end routers and focused more on high end.
There was a push long ago to sacrifice repeatability where necessary
to acheive results that were more real world relevant (than the PPS to
one destination at fixed rate for example).  I don't remember an
explicit change in scope toward enterprise and consumer but I also
don't know that the scope was ever explicitly defined to exclude it.

> the fact that the standards body for WLAN controllers is in IETF was

That is beyond provider edge.  BMWG is in OAM area which started out
as operations of the Internet infrastructure, not so much the wireless
hub in the coffee shop, the office, or even the college campus.

> providers and their vendors for usable performance metrics for metro WLAN

I don't know if this is in scope or out of scope for the WG.

Curtis

---

From: "Jerry Perser" <[email-masked]>
Date: Thu, 19 Jul 2007 11:53:36 -0700

Curtis,

You comment's have me puzzled.  They don't match my experiences with the
BMWG over the last several years.

The BMWG started out testing LAN devices.  RFC 1242 defines the difference
between bridges and routers.  RFC 1242 section 3.8 describes a "bit
forwarding device" that I have never seen on any high end routers.  RFC 2285
has the word "LAN Switching Devices" right in its title.  RFC 2647 describes
testing a device that connects to a high end router.  And even the last RFC
(RFC4814) is not centered on only high end routers, but covers all
networking devices.

The reason you don't remember an "explicit change in scope toward enterprise
and consumer" is because there was none.  BMWG was already benchmarking
enterprise devices.  Kevin and Al have done an excellent job keeping the
BMWG vendor neutral.  But certain vendors are more active than others, so it
seems to distort the charter.

I am partially to blame for this.  When I was working on RFC2889, my
employer was making money testing with LAN devices.  When my employer
realized that they can make more money testing "high end routers", my focus
was moved toward RFC4689.  This had nothing to do with the BMWG charter
since both projects fell within the charter.  I just needed a believable
reason for employer to sign my IETF expense reports.

Before you condemn wireless to coffee shops, please consider these points:

1. The IETG's Operations and Management Area has a working group (not a BOF)
on "Control And Provisioning of Wireless Access Points".  This WG has
already produced four RFCs.  They are actively moving the wireless
management out if the hands of the "coffee girl" and into the hands of a
trained professional.

2. The internetworking devices that we benchmark are now getting wireless
added to them.  These devices started out as simple LAN devices.  But then
technologies were added to them to do multicast (2432/3918), firewalls
(2647/3511), then actual routing (3222/4098/4061/4062/4063).  These devices
are being upgraded to the CAPWAP Protocol.

3. When you're sitting in the BMWG meeting in Chicago and you notice that
the internet is slow on your laptop, should we blame the core?  Or should we
characterize each hop using simulated stimuli in a laboratory environment?

Jerry Perser

> From: Curtis Villamizar [mailto:[email-masked]]
> Sent: Thursday, July 19, 2007 7:08 AM
> To: Tom Alexander
> Cc: 'Al Morton'; [email-masked]
> Subject: Re: [bmwg] WLAN switch controller I-Ds for BMWG to consider
> 
> 
> No virus found in this incoming message.
> Checked by AVG Free Edition.
> Version: 7.5.476 / Virus Database: 269.10.9/907 - Release Date: 7/18/2007
> 3:30 PM
> 

No virus found in this outgoing message.
Checked by AVG Free Edition. 
Version: 7.5.476 / Virus Database: 269.10.9/907 - Release Date: 7/18/2007
3:30 PM

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 19 Jul 2007 23:11:35 -0400

In message <[email-masked]>
> BMWG over the last several years.

I remember arguing with Scott Bradner circa 1992/1993 regarding
"Bradner Test consdiered harmfull" because Cisco was using the Bradner
test to claim "fastest router in the world" at a time when Cisco
routers fell over - literally crashed - under the loads that the
Internet provided.  The Cisco 7000 with its "route cache" design was
the height of this folly where Cisco designed specifically to get high
benchmark numbers on less than worthless benchmarks.

When routes changed the Cisco 7000 dropped packets like crazy due to
the route cache design.  At high enough loads BGP and IGP adjacencies
dropped and a network could become unstable.  At even higher loads the
router would crash.  But in the Bradner test the routes never changed.
This was the problem with high repeatability but no relevance to real
world traffic forwarding capability.

maybe it was just that some of the ISP people tried to push the focus
in the direction of high end but it never really was there.

Curtis

---

From: "Jerry Perser" <[email-masked]>
Date: Fri, 20 Jul 2007 12:20:51 -0700

Instead of arguing with Scott Bradner, would you mind reviewing the two WLAN
Switch Benchmarking drafts and argue with us?  One of the reasons Tom
Alexander is bringing the draft to the BMWG is to avoid these pitfalls.
Your comments on the draft will help us make sure we have repeatable
benchmarks, while keeping sight of real world traffic.

Claims and crashes are not limited to Cisco.  Last year I worked with the
esteemed David Newman on a public "Enterprise Wi-Fi Test" for Network World.
When running WLAN Switch Benchmarking Methodology section 5.1.1, one of the
vendors crashed repeatedly.  Before you start faulting the vendor, realize
that they did not have a copy of WLAN Switch Benchmarking Methodology when
verifying their beta code.

Enterprise Wi-Fi needs to be tested.  Duke University is currently having
problems with iPhones on their network
(http://www.networkworld.com/news/2007/071607-duke-iphone.html).  My focus
is to find these problems with a simulated stimuli in a laboratory
environment and to avoid follies like you mentioned.

And everyone else on the BMWG reflector is invited to comment on the WLAN
Switch Benchmarking drafts.  They can be found at:

http://www.ietf.org/internet-drafts/draft-alexander-bmwg-wlan-switch-term-00
.txt

http://www.ietf.org/internet-drafts/draft-alexander-bmwg-wlan-switch-meth-00
.txt

Jerry Perser

> Sent: Thursday, July 19, 2007 8:12 PM
> To: Jerry Perser
> Cc: [email-masked]
> Version: 7.5.476 / Virus Database: 269.10.11/909 - Release Date: 7/20/2007
> 4:39 PM
> 

No virus found in this outgoing message.
Checked by AVG Free Edition. 
Version: 7.5.476 / Virus Database: 269.10.11/909 - Release Date: 7/20/2007
4:39 PM

---

From: Curtis Villamizar <[email-masked]>
Date: Sun, 22 Jul 2007 10:16:25 -0400

In message <[email-masked]>
> benchmarks, while keeping sight of real world traffic.

Maybe you missed the context.  I was arguing with Scott Bradner circe
1993 to mid to late 1990 about highly repeatable benchmarks vs real
world relevant benchmarks.

> verifying their beta code.

The router crashes I was talking about were in gear that was tested in
1993 to about 1997.  Interesting to note that there is gear out there
that still crashes on relatively simple benchmarks.

> environment and to avoid follies like you mentioned.

Thanks for bringing me up to date on this.  As I mentioned earlier, my
focus has been on IP provider core and this is beyond the edges of the
IP provider network.

> http://www.ietf.org/internet-drafts/draft-alexander-bmwg-wlan-switch-meth-00

OK.  Thanks.  You've convinced me that this is in scope for the WG.

Curtis

---

From: "Jerry Perser" <[email-masked]>
Date: Tue, 24 Jul 2007 09:30:39 -0700

> that still crashes on relatively simple benchmarks.

Due to NDAs and desire not to be beaten to a pulp by my CEO, I can not
disclose who is crashing.  Working for a company that makes test equipment
for an industry that doesn't do a lot of the BMWG benchmarks, every week we
have someone crashing.  I just references two instances that were announced
by a publication. 

In fact I almost couldn't get registered for the social because the
"wireless was acting up".  Even the experts who put the 69th IETF Meeting's
network together are having problems.

Jerry Perser