Subject: [87attendees] IETF wireless

From: Tim Chown <[email-masked]>
Date: Mon, 5 Aug 2013 10:48:25 +0100

Hi,

Others have given the wireless network at this IETF due praise.  Indeed the wireless has been excellent for many IETFs now, such that we pretty much take it for granted.  Well done to everyone involved.

Is there any information available on the solution that was delivered?  From memory the APs were Cisco Aironet, but it would be interesting to know about density, controllers, etc.  Usage info, especially for IPv6, would be interesting too.

If the info isn't available, so be it, but it might be of interest to many...

Tim

---

From: Ole Jacobsen <[email-masked]>
Date: Mon, 5 Aug 2013 04:40:02 -0700 (PDT)

Check out the NOC report which was referenced (but not delivered) 
during the plenary on Wednesday. The info you ask for is usually
contained therein.

Ole

Ole J. Jacobsen
Editor and Publisher,  The Internet Protocol Journal
Cisco Systems
Tel: +1 408-527-8972   Mobile: +1 415-370-4628
E-mail: [email-masked]  URL: http://www.cisco.com/ipj
Skype: organdemo

On Mon, 5 Aug 2013, Tim Chown wrote:

> Hi,
> 
> Others have given the wireless network at this IETF due praise.  
> Indeed the wireless has been excellent for many IETFs now, such that 
> we pretty much take it for granted.  Well done to everyone involved.
> 
> Is there any information available on the solution that was 
> delivered?  From memory the APs were Cisco Aironet, but it would be 
> interesting to know about density, controllers, etc.  Usage info, 
> especially for IPv6, would be interesting too.
> 
> _______________________________________________
> 87attendees mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/87attendees
>

---

From: Warren Kumari <[email-masked]>
Date: Mon, 5 Aug 2013 14:54:53 +0200

On Aug 5, 2013, at 1:40 PM, Ole Jacobsen <[email-masked]> wrote:

> during the plenary on Wednesday.

Link for easy clickin': http://www.ietf.org/proceedings/87/slides/slides-87-iesg-opsplenary-7.pptx

W 

> The info you ask for is usually
> contained therein.
> 
> Ole
> 
> 
> Ole J. Jacobsen

-- 
Never criticize a man till you've walked a mile in his shoes.  Then if he didn't like what you've said, he's a mile away and barefoot.

---

From: Tim Chown <[email-masked]>
Date: Mon, 5 Aug 2013 14:50:42 +0100

> Link for easy clickin': http://www.ietf.org/proceedings/87/slides/slides-87-iesg-opsplenary-7.pptx

Thanks - that's great!

The one interesting thing we did see was some people seeing geo-location information stretching back more than one meeting. It would be interesting to see where that information is "tagged", and which apps/services cache it. 

Tim

---

From: Bob Hinden <[email-masked]>
Date: Mon, 5 Aug 2013 18:47:23 +0200

Tim,

> The one interesting thing we did see was some people seeing geo-location information stretching back more than one meeting. It would be interesting to see where that information is "tagged", and wh…

It's too bad we can set it up so the location varies by individual response to every location where we have had an IETF meeting network!  Sort of like location balancing :-)

Bob

---

From: Ray Pelletier <[email-masked]>
Date: Mon, 5 Aug 2013 16:24:34 -0400

> If the info isn't available, so be it, but it might be of interest to many...

The NOC report is at the Proceedings already:  https://datatracker.ietf.org/meeting/87/materials.html

Ray

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 8 Aug 2013 07:57:21 +0200 (CEST)

> Link for easy clickin': http://www.ietf.org/proceedings/87/slides/slides-87-iesg-opsplenary-7.pptx

Scary how many are using the unencrypted wifi.

A suggestion for next IETF would be to call the 802.1x wifi "IETF" and the 
open one "IETF_UNENCRYPTED".

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Stefan Winter <[email-masked]>
Date: Thu, 08 Aug 2013 08:35:33 +0200

Hi,

> Scary how many are using the unencrypted wifi.

Don't only count the ietf.1x network. The IETF also provides eduroam,
which is WPA2/AES with 802.1X and - sorry, NOC: actual server
certificate verification, not a "don't verify ours, and we won't verify
yours".

The Max numbers in the NOC report are

the 3 "secure" ones: 25+27+37 = 89
the two "edu" ones: 38 53 = 91

Which makes for a total of 180 max simultaneous encrypted connections.

Sure, compared to approx. 700 on the open networks it's not so really
great, but still quite a significant portion.

> A suggestion for next IETF would be to call the 802.1x wifi "IETF" and
> the open one "IETF_UNENCRYPTED".

Without verifying the server cert of the 802.1x network, you may well
talk to the network encryptedly, but the network might be a MitM who can
decrypt and snoop out your traffic.

The ietf.1x network is just not as good as it could be - maybe more
people would be inclined to use it if it provides the extra benefit of
being able to identify itself as *the genuine* IETF network. Neither the
open network nor the ietf.1x currently do that.

The fix would be surprisingly simple: in the Network Information page,
actually *publish* the CA cert and its fingerprint that signs the auth
server, and also publish the server's subject/subjectAltNames. This
allows every user to check whether he's on the real thing when connecting.

There are also provisioning tools that can install the CA certs and EAP
settings onto client devices. It's not rocket science.

Greetings,

Stefan Winter

-- 
Stefan WINTER
Ingenieur de Recherche
Fondation RESTENA - Réseau Téléinformatique de l'Education Nationale et
de la Recherche
6, rue Richard Coudenhove-Kalergi
L-1359 Luxembourg

Tel: +352 424409 1
Fax: +352 422473

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 8 Aug 2013 08:42:09 +0200 (CEST)

> decrypt and snoop out your traffic.

Absolutely, but that's harder than just setting your wifi card in promisc 
mode and listening to all the traffic.

On these kinds of events I usually connect to encrypted network that might 
be available, and then try to send as much traffic as possible through ssh 
tunnel.

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 8 Aug 2013 11:34:35 +0200

On Aug 8, 2013, at 08:35, Stefan Winter <[email-masked]> wrote:

> The IETF also provides eduroam

And tons of kudos go to whoever started this.
(Finally, a feature where the IETF is providing *better* service to the academics than to everyone else...)

Grüße, Carsten

PS.: The eduroam number also provide a nice estimate of the number of academics attending.
Of course, there may be a couple people who laboriously configure the WiFi for IETF before noticing they already had been connected, and there may still be a few academics coming from eduroam-challenged institutions.  But overall, this is a nice indicator.

---

From: grenville armitage <[email-masked]>
Date: Thu, 08 Aug 2013 19:46:40 +1000

> And tons of kudos go to whoever started this.

+1

(When I flipped open my laptop on the first day it automagically associated with eduroam. I initially thought this was some delayed holdover from when I'd last used WiFi at my home university. But lo and behold, eduroam in the IC was real, and worked well all week!)

cheers,
gja

---

From: Tim Chown <[email-masked]>
Date: Thu, 8 Aug 2013 11:56:20 +0100

> (When I flipped open my laptop on the first day it automagically associated with eduroam. I initially thought this was some delayed holdover from when I'd last used WiFi at my home university. But…

eduroam's been at the IETF a while.  Very welcome!

Tim

---

From: Klaas Wierenga <[email-masked]>
Date: Thu, 8 Aug 2013 13:10:25 +0200

Carsten, all,

> (Finally, a feature where the IETF is providing *better* service to the academics than to everyone else...)

If I am not mistaken, kudos go to Stefan Winter, the SURFnet team that made it happen at the IETF in Maastricht and IETFs own team that on top of their busy schedule took the time to enable eduroam at the venue as well.

Klaas

P.S. now that we are on the subject of eduroam, can I invite you all to have a look at: http://datatracker.ietf.org/doc/draft-wierenga-ietf-eduroam/ and give us your comments

---

From: Chris Elliott <[email-masked]>
Date: Thu, 8 Aug 2013 07:27:19 -0400

All,

The eduroam ssid and associated authentication back end has been provided
at the IETF since IETF 78 in Maastricht. Paul Dekkers of SURFnet first
approached us with the idea of supporting this authentication system at
IETF meetings (thanks Geert Jan de Groot for bringing this to us!). Many of
the core network volunteer team offered suggestions and insights into how
to best support this. The eduroam folks that have most recently been
involved include Stefan Winter and Dubravko Penezic. Currently, Karen
O'Donoghue is our main contact with the eduroam folks as well as with
Radiator (our authentication server), Bjoern Zeeb handles the
authentication server side of things, and I maintain the AP and switch
configs.

Some of the interesting challenges we've encountered include authentication
timeouts as the eduroam Radius server "tree" is extensive and can result in
many Radius proxies before reaching the end (authoritative?) authentication
server. We're using a couple of techniques to improve reliability and
security and speed up authentication: Radius to the eduroam servers via
TLS/TCP, and supporting DNSRoam to shorten authentication times.

Thanks for the kind words. It's a pleasure to be able to help provide a
service that just works (most of the time!)

Chris.

On Thu, Aug 8, 2013 at 6:56 AM, Tim Chown <[email-masked]> wrote:

> > (When I flipped open my laptop on the first day it automagically
> associated with eduroam. I initially thought this was some delayed holdover
> from when I'd last used WiFi at my home university. But lo and behold,
> eduroam in the IC was real, and worked well all week!)
>

-- 
Chris Elliott
[email-masked]

---

From: Chris Elliott <[email-masked]>
Date: Thu, 8 Aug 2013 07:46:43 -0400

Stefan,

The cert we are using for the IETF SSIDs authenticated via 802.1X (not
including eduroam) is from the same CA that is providing the certs for the
rest of the IETF (including web, email, dnssec, and more, I believe). This
CA (Starfield Secure Certification Authority) is in most or all recent OSes
and browsers root certificate store. Which means most 802.1X supplicants
/can/ verify the server cert we supply. It's up to the user to not un-check
the "Check server cert" box! Our thought was that that was sufficient.

However, we certainly can look into doing as you suggest and publishing
more information. Of course, the page we post this to will probably also be
verified by this same CA... ;-)

Chris.

-- 
Chris Elliott
[email-masked]

---

From: Stefan Winter <[email-masked]>
Date: Thu, 08 Aug 2013 14:10:39 +0200

Hi,

> including eduroam) is from the same CA that is providing the certs for
> the rest of the IETF (including web, email, dnssec, and more, I
> believe). This CA (Starfield Secure Certification Authority) is in most
> or all recent OSes and browsers root certificate store. Which means most
> 802.1X supplicants /can/ verify the server cert we supply. It's up to
> the user to not un-check the "Check server cert" box! Our thought was
> that that was sufficient.

No! This is a common misconception. The "browser trust" and EAP trust
are different, and it is insignificant if the CA used is in the browser
trust store or not.

It may save you the step of installing the PEM file itself, sure, but
the crucial part is that you need to be able to tell your EAP supplicant
on the device that the expected server cert comes from *exactly one* CA.

Further to this, contrary to the web browser case, you need to manually
configure which server name to expect; in the browser, this is
explicitly input by the user in his URL bar - but no equivalent of that
exists for EAP.

We have fought for years to get that into our Identity Providers' heads.
Our official eduroam documentation has an extensive section about all
the concepts and nits to consider:

https://confluence.terena.org/display/H2eduroam/EAP+Server+Certificate+considerations

If you don't pin the exact CA *and* server name in config, you make
yourself susceptible to MitM attacks still:

- if you trust all CAs in the "browser trust" store: the MitM gets
himself an arbitrary certificate from any CA in the trust store and uses
it for his fraudulent EAP server - your EAP supplicant would trust it,
even if it has nothing to do with your EAP server.

- if you don't verify the name (but the exact CA): the MitM tries to
connect to the genuine network once, sees which CA is expected, goes to
the same CA and gets himself a certificate for an unrelated name (to
which he is entitled to by say having a corresponding domain name). He
sets up a fraudulent EAP server with that certificate - your EAP
supplicant will accept it even if he does the extra step of verifying
the CA.

> more information. Of course, the page we post this to will probably also
> be verified by this same CA... ;-)

That's the default CA bootstrap problem. You can put the CA's
fingerprint in a text file, and PGP-sign the file with IETF brass PGP
keys if you like :-)

In eduroam, we *require* IdPs to "publish all information needed to
uniquely identify the authentication server"; and we even have an
automated provisioning tool which sets up supplicants with all the
correct and secure settings - including the CA installation.

It works everywhere except on Karen O'Donoghue's Mac. ;-)

Greetings,

Stefan Winter

> 
> Chris.
> 
> 
> On Thu, Aug 8, 2013 at 2:35 AM, Stefan Winter <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
>     Hi,
> 
>     Tel: +352 424409 1 <tel:%2B352%20424409%201>
>     Fax: +352 422473 <tel:%2B352%20422473>
> 
> 
>     [email-masked] <mailto:[email-masked]>
> [email-masked] <mailto:[email-masked]>

Tel: +352 424409 1
Fax: +352 422473

---

From: Chris Elliott <[email-masked]>
Date: Thu, 8 Aug 2013 08:44:17 -0400

Stefan,

Thanks for your feedback. Note that we have little control over attendee's
client configs, unlike universities or companies. However, we can certainly
provide more information and automated tools like the eduroam community
does.

As none of us are as expert in this field, doing it part time three times a
year on a volunteer basis, we may need further expert advise. The eduroam
docs are great, but if we need more we'll contact you. We'll follow up with
you on this once the VMs are spun up at their between meetings home and I'm
back from vacation--early to mid September.

Chris.

On Thursday, August 8, 2013, Stefan Winter wrote:

> > On Thu, Aug 8, 2013 at 2:35 AM, Stefan Winter <[email-masked]<javascript:;>
> > <mailto:[email-masked] <javascript:;>>> wrote:
> >
> >     Hi,
> >
> >     [email-masked] <javascript:;> <mailto:[email-masked]<javascript:;>
> >
> > [email-masked] <javascript:;> <mailto:[email-masked]<javascript:;>
> >
>
>
> --
> Stefan WINTER
> Ingenieur de Recherche

-- 
Chris Elliott
[email-masked]

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 8 Aug 2013 15:03:43 +0200

On Aug 8, 2013, at 14:44, Chris Elliott <[email-masked]> wrote:

> little control over attendee's client configs, unlike universities or companies

Ha!  I'd certainly wish I had any of that control at my university.

But eduroam offers enough incentives that people do load those .mobileconfig files.
(You could provide one for ietf.1x, too.  After your vacation...)

Grüße, Carsten

---

From: Keith Mitchell <[email-masked]>
Date: Thu, 08 Aug 2013 10:09:27 -0400

On 08/08/2013 01:57 AM, Mikael Abrahamsson wrote:

>>> Check out the NOC report which was referenced (but not
>>> delivered) during the plenary on Wednesday.
> 

Probably I'm being dumb, but I find it too much hassle to configure for
Linux - my network config tool (knetworkmanager *) always asks for more
WPA/802.1x parameter options than are ever published. I could probably
be less lazy and figure them out or ask around, but a sample config or
template file would sure help.

Not that I don't use SSH. Seems to work just great for Android.

Keith

(*) you can take your own view whether this is a developer UI fail or a
protocol feature overload fail...

---

From: Dan Harkins <[email-masked]>
Date: Thu, 8 Aug 2013 16:06:19 +0000

>fingerprint in a text file, and PGP-sign the file with IETF brass PGP

  Trust stores... CA bootstrapping.... Maybe we should get some
authentication scheme on the ietf.1x network that doesn't require using
certificates. Maybe RFC 5931 :-) There's none of that "inner method"
nonsense, no need to select a CA, no real identity versus anonymous
identity issues.

  I certainly would not mind eating that brand of dog food.

  Dan.

---

From: Andrey Lukyanenko <[email-masked]>
Date: Thu, 8 Aug 2013 22:57:33 +0300

All,

I have access to eduroam, but almost always use ietf ssid. Eduroam
periodically asks for login and password, ietf connects right away. Many
academics do not even bother to setup their devices for eduroam (sometimes
it is tricky for Android or WP, do not remember which device exactly caused
me trouble). But it is cool to have eduroam (especcially when there are no
alternative open networks).

Best regards,
Andrey Lukyanenko.

On Thu, Aug 8, 2013 at 12:34 PM, Carsten Bormann <[email-masked]> wrote:

> Of course, there may be a couple people who laboriously configure the WiFi
> for IETF before noticing they already had been connected, and there may
> still be a few academics coming from eduroam-challenged institutions.  But
> overall, this is a nice indicator.
>

---

From: Stefan Winter <[email-masked]>
Date: Fri, 09 Aug 2013 07:57:42 +0200

Hi,

> academics do not even bother to setup their devices for eduroam
> (sometimes it is tricky for Android or WP, do not remember which device
> exactly caused me trouble).

Android < 4.3 and Windows Phone have broken / highly deficient EAP
supplicants on board. There is little we can do about that (except for
lobbying).

Android 4.3 made *significant* leaps towards a good EAP supplicant;
likely becaue of the many complaints of eduroam users, just google for
it. WP8 improved its situation, but it's still far from perfect.
Luckily, Windows Phone is - and likely continues to be - a niche product.

> But it is cool to have eduroam (especcially
> when there are no alternative open networks).

Open networks expose your data payload to everyone. Sure, they won't ask
you for a username/password, and you could consider that an upside if
you don't care about security.

BTW, yesterday we had 1 million national roaming, and 100.000
international roaming logins; and it's semester break period. That's
IMHO a pretty nice figure for a 24h period and I dare challenge your
"many academics don't bother".

Stefan

> 
> 
> Best regards,
> Andrey Lukyanenko.
> 
> 
> On Thu, Aug 8, 2013 at 12:34 PM, Carsten Bormann <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
>     On Aug 8, 2013, at 08:35, Stefan Winter <[email-masked]
>     Of course, there may be a couple people who laboriously configure
>     the WiFi for IETF before noticing they already had been connected,
>     and there may still be a few academics coming from
>     eduroam-challenged institutions.  But overall, this is a nice indicator.
> 

Tel: +352 424409 1
Fax: +352 422473

---

From: Stefan Winter <[email-masked]>
Date: Fri, 09 Aug 2013 08:03:00 +0200

Hi,

>   I certainly would not mind eating that brand of dog food.

Nice advertising ;-)

Sure, EAP-pwd is a very interesting EAP method because it's simple AND
secure. We've made our eduroam IdP admins aware of it, and hope to see
it being used in eduroam, too.
There are some downsides in real deployment; e.g. if you have a
Microsoft AD, EAP-pwd won't get a representation of the password and
can't operate. So it's not for everybody.

But everybody who *can* make use of it should seriously consider it!

Thinking of this (sorry to ruin your ad): for the IETF it would not be a
clear winner: the mutual authentication is based on only you and your
auth server knowing your password.

For ietf.1x, I believe in most meeting incarnations, the username is
"ietf" and password "ietf"? In that case, any MitM can pretend to be
your EAP server. The server-side auth sort of breaks down if passwords
aren't secret. Server-side auth with CA and CN still survives in such
situations.

Greetings,

Stefan

Tel: +352 424409 1
Fax: +352 422473

---

From: Stefan Winter <[email-masked]>
Date: Fri, 09 Aug 2013 08:07:38 +0200

Hi,

> WPA/802.1x parameter options than are ever published.

THat's why NOC should publish them :-)

> I could probably

For our users (our settings), this is on our setup isntructions page:

http://www.restena.lu/restena/en/support/EN-eduroam-setup-knetworkmanager.html

Obviously, change the CA to the one used by IETF NOC, and the server
name as well, and use the ietf access password. All the rest is pretty
standard.

HTH,

Stefan

Tel: +352 424409 1
Fax: +352 422473

---

From: Jim Gettys <[email-masked]>
Date: Fri, 9 Aug 2013 11:18:27 -0400

On Mon, Aug 5, 2013 at 5:48 AM, Tim Chown <[email-masked]> wrote:

> Hi,
>
> Others have given the wireless network at this IETF due praise.  Indeed
> the wireless has been excellent for many IETFs now, such that we pretty
> much take it for granted.  Well done to everyone involved.
>
> Is there any information available on the solution that was delivered?
>  From memory the APs were Cisco Aironet, but it would be interesting to
> know about density, controllers, etc.  Usage info, especially for IPv6,
> would be interesting too.
>

Bufferbloat kills you in many/most conference venues; IETF is one of the
few where it does not.   I first noticed at the Denver NANOG a working
wireless network; but the same people are involved in both NANOG and
IETF.So I poked into the topic to understand why. I was amazed to see a
room of 500 heavy duty network people and observe latencies from Denver to
California consistently under 50ms under load, and investigated why the
network was usable.

At past IETF's I've urged the operation teams that they should document
what they do.  I did not have time in Berlin to repeat the request again.

Besides the layout of AP's that other good operators do, there are a
number of "tricks" that they play:

1) the AP's have minimal internal buffering, and forward their packets back
to a big router.
2) the big router, having lots of aggregated traffic, is able/does run
(W)RED to keep the wireless traffic queues from being too much of an issue.
 This is hopeless in a home environment where you have little aggregation
of traffic (but there, you can now get help from fq_codel if you run
OpenWrt/CeroWrt on your home router; but to fix the situation properly will
still take a lot of work in Linux's WiFi stack).
3) There is some QOS/Diffserv tricks being played to prioritize certain
operations (e.g. DNS lookup, TCP opens, acks, and so on).   An early
version of the marking they do was documented at a NANOG meeting; but I
gather some later rules have been added since then. Those of you who have
looked at fq_codel in detail might recognize that fq_codel happens to have
similar behavior (without needing the hair of such explicit
marking/classification rules).

Anyone setting up an large event today should take a careful look at what
is being done, and it would be good to get the knowledge captured in enough
detail for others to reproduce their work, as it can be done today.
                             - Jim

---

From: Chris Elliott <[email-masked]>
Date: Fri, 9 Aug 2013 12:37:54 -0400

On Friday, August 9, 2013, Jim Gettys wrote:
>
> On Mon, Aug 5, 2013 at 5:48 AM, Tim Chown <[email-masked]<javascript:_e({}, 'cvml', [email-masked]');>
> > wrote:
>
>> Hi,
>
> ...

> If the info isn't available, so be it, but it might be of interest to
We're in active discussions about documenting this, especially why we make
certain decisions in light of the rather special conditions of the IETF
meetings: relatively high network usage per attendee vs. many other
meetings like IEEE and access to donated high speed internet connectivity,
being just two of the parameters.

Also, site selection is a huge part--the network team is part of the site
evaluation and the technical recommendations on the suitability of venues
to host the network you all know and love is taken very seriously, and has
nixed many venues.

> wireless network; but the same people are involved in both NANOG and IETF.
>

I haven't been involved with the NANOG network on years. My understanding
is that our philosophy and design differ fundamentally. We over-provision,
they tune, is an over-simplification...

> So I poked into the topic to understand why. I was amazed to see a room of
> 500 heavy duty network people and observe latencies from Denver to

Part of it is, we're volunteers. We love doing it. And many of us are quite
active in standards, not just the network. At the end of two intense weeks,
we're ready to get back to our "real" lives.

And, as noted above, we hear you, and are working on documenting things.

> 1) the AP's have minimal internal buffering, and forward their packets
> back to a big router.
>

Agreed.

> still take a lot of work in Linux's WiFi stack).

I agree that bufferbloat is a huge issue in many environments. Addressing
it fixed my son's company's network, with >20 people sharing a 35M down/5M
up link, and is just one datum in that story.

However, at the IETF, we have the luxury of "simply" over provisioning and
relying on the enterprise routers to do the right thing without much tuning
at all. Most of our traffic transits the Internet. The internal traffic is
small and not an issue. I address wireless at the end briefly...

Our traffic peaks on the external links hit a max of 200Mbps this meeting.
The highest we've seen was in Hiroshima, where WIDE did the network and we
basically were advisors without much to do, hit 300Mbps.

Now, consider that we had two 1Gbps connections to a high-tier service
provider (DT, in this case) with very low latency, and DT's connectivity to
the rest of the Internet is very good.

To way over-simplify, and maybe trolling a bit, who needs queueing or
QoS when any buffers you have are empty? 1500 byte packets just don't take
enough time to transmit on a 1G link to affect other traffic, including,
for example, small voice packets, especially when they're only carrying
200Mbps peak.

And we often don't try to share the load across our two links--it's
reliability we're after, not more bandwidth.

> marking/classification rules).

Note that we don't do any QoS/Diffserv outside of what would have to be
explicitly turned off in the gear we use.

Once again, I'm not dismissing bufferbloat as a real issue, just that I
don't need to address it in the IETF network.

> detail for others to reproduce their work, as it can be done today.

Agreed.

I think a large part of our success has been from how we build the wireless
side of things. This is almost always the only constrained part of the net.
More details later, but we've actively worked to encourage folks to use the
5Ghz spectrum, and we build the wireless net to maximize the available
bandwidth while managing a low noise level.

And, all equipment, from the Juniper and Cisco gear and other
enterprise/carrier grade equipment, to the wireless drivers in everyone's
client devices, have improved markedly over the past few years. I
especially see the improvement in wireless.

Wireless clients rarely act like the burro between two (or 100...) bales of
hay
these days. Clients used to roam frequently to a slightly "better" AP
(BSSID). Now they stick much better, which leads to other, much easier to
deal with, issues.

I heard a story at this meeting that one of our own directly affected this
(years ago) by simply moving around in front of the desk of a
developer as they were tuning
their code, and wondering why performance sucked and their code kept
triggering roaming...they got the point, and fixed the code.

Thanks for the kind words about the net, and your continued encouragement
for us to document what we do.

Chris.

>> [email-masked] <javascript:_e({}, 'cvml', [email-masked]');>

-- 
Chris Elliott
[email-masked]

---

From: Jim Gettys <[email-masked]>
Date: Fri, 9 Aug 2013 13:06:20 -0400

On Fri, Aug 9, 2013 at 12:37 PM, Chris Elliott <[email-masked]> wrote:

>> So I poked into the topic to understand why. I was amazed to see a room
>> of 500 heavy duty network people and observe latencies from Denver to
> Part of it is, we're volunteers. We love doing it. And many of us are
> quite active in standards, not just the network. At the end of two intense
> weeks, we're ready to get back to our "real" lives.
>

> for example, small voice packets, especially when they're only carrying

I was only referring to the AP/router hop: beyond the IETF router,
classification is pretty much irrelevant given overprovisioned uplinks:
your bottlenecks will typically be in the WiFi hop, not in the back haul.

Remember, a single modern TCP stack and only a few wireless sessions can
fill even a 200-300Mbps pipe.  This wasn't true in the days of Windows XP
(which did not have TCP window scaling turned on, and crawled).

Over provisioning really just cuts the effective size (in time) of buffers
that you have.  Filling any buffers at a bottleneck is what TCP loves to
do. So it's easy to saturate the 802.11 hop, and on the IETF network, I
betcha it's often there, not in the backhaul.

Again, the AP's you are using don't sport giant buffers.

So it can/does really matter to run AQM (on the wireless links), even so
(when people download big files).  And if you aren't running any these
days, you are winning due to the lack of buffering in the AP's.

> explicitly turned off in the gear we use.

At one point, I tracked down the rules/marking in use on the routers/AP's
when I looked into it.  It was all sensible stuff.

> don't need to address it in the IETF network.

And the AP's aren't horribly over-buffered in the first place.

It's people who set up conference networks using consumer devices that
really get bit hard by bufferbloat (or many cheap motels that do likewise).
 I've seen this on many, many occasions.

> I think a large part of our success has been from how we build the
> wireless side of things. This is almost always the only constrained part of
> the net. More details later, but we've actively worked to encourage folks
> to use the 5Ghz spectrum, and we build the wireless net to maximize the
> available bandwidth while managing a low noise level.
>
> Wireless clients rarely act like the burro between two (or 100...) bales
> of hay
> (years ago) by simply moving around in front of the desk of a developer as they were tuning

Yeah, and bufferbloat encourages this behavior: when your throughput drops
(either due to physical environment, or competing traffic), your latency
can go through the roof if the network (and host!) has lots of unmanaged
buffers, and you either die a slow death or move to a different ESSID. And
IETF wireless network doesn't have a pile of unmanged buffers, so you win
(in the downstream).  Saturating the wireless link in the upstream
direction you'll still have problems in your devices; bloat is in our
operating system stacks and device drivers...

> for us to document what we do.

It's excellent!  Keep up the great work.

Jim

---

From: joel jaeggli <[email-masked]>
Date: Fri, 09 Aug 2013 10:11:42 -0700

On 8/9/13 8:18 AM, Jim Gettys wrote:
> 
> 
> 
> On Mon, Aug 5, 2013 at 5:48 AM, Tim Chown <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
>     Hi,
> 
>      Indeed the wireless has been excellent for many IETFs now, such
>     that we pretty much take it for granted.  Well done to everyone
>     involved.
> 
> few where it does not. 

unicast packets are delivered "reliably" on 802.11 networks. really high
latency on the first hop is generally indicative of high noise
floor/lots of contention/pdu's getting clobbered. limiting
retransmission there (equivalent to a shallow buffer) may help you but
it just pushes the cost of retransmission (due to packet loss) to the
end-points rather than eliminating the source of restransmission.

there are performance tweaks you can make to the RF domain, and the link
layer that can help you there. dispensing with low speed rates, backward
compatibility, increasing the multicast rate and so forth. as a
practical matter if you want to keep the 2.4ghz band usable pushing as
many clients as possible to 802.11a/n.

while it's probably a tautology not having crappy wireless is in fact
the first step in not having a crappy wireless network.

> I first noticed at the Denver NANOG a working
> IETF.

the contractor is the same

> So I poked into the topic to understand why. I was amazed to see a
> room of 500 heavy duty network people and observe latencies from Denver
> to California consistently under 50ms under load, and investigated why
> the network was usable.
> 

chris will chime in here I imagine. the APs are cisco 1142s which are
nice high-end dual radio APs but not in an exceptional sense.

the big router in this case is a gibson arch juniper so it's also not
exceptional or particularly high capacity, nor is it doing a lot of
shaping of any variety becuase none of the interfaces are terribly
heavily utilized.

in both cases overprovisioning works rather well.

> (W)RED to keep the wireless traffic queues from being too much of an
> issue.  This is hopeless in a home environment where you have little
> aggregation of traffic (but there, you can now get help from fq_codel if
> you run OpenWrt/CeroWrt on your home router; but to fix the situation
> properly will still take a lot of work in Linux's WiFi stack).
> gather some later rules have been added since then. Those of you who
> have looked at fq_codel in detail might recognize that fq_codel happens
> to have similar behavior (without needing the hair of such explicit

Anton Capella was responsible for this in it's initial iterations. It's
not that uncommon for wireless carriers to employ a similar bag of tricks

afaik we don't do any qos, since we don't have any queues to speak off.

most traffic in the IETF network is north-south due to the hosts on the
network having little to do with each other. That's not really the case
in a homenet. Where some relatively huge flows may be between devices on
the same wireless lan e.g. timemachine backups occuring while you're
trying to stream video or maintain the responsiveness of your ssh session.

> 
> Anyone setting up an large event today should take a careful look at
> what is being done, and it would be good to get the knowledge captured
> in enough detail for others to reproduce their work, as it can be done
> today.
>                              - Jim
> 
> 
> 
>     Tim

---

From: Jim Gettys <[email-masked]>
Date: Fri, 9 Aug 2013 13:35:20 -0400

> end-points rather than eliminating the source of restransmission.

Actually, there is more to this than first appears; see below.

> many clients as possible to 802.11a/n.

Yup.  Very important.  A little broadcast/multicast can ruin your whole
day, along with low speed hosts at the edge of an area.  So by turning them
off, you are preserving the performance of the WiFi hop (at the cost of
usable range; a very good tradeoff in a conference venue, but less so
elsewhere).

> the first step in not having a crappy wireless network.

Actually, in Dave Taht and Felix Fietkau digging into the Linux Atheros
driver, there are
a bunch of other issues.  I would expect that commercial AP vendors have
been similarly digging and fixing busted implementations.

If you have a noisy environment, damaged PDU's can easily cause a form of
input bufferbloat.

TCP currently does not like out of order delivery of packets.  So when a
damaged aggregate arrives, the Linux wifi stack does not forward packets
successfully received after the damaged packet immediately. You get a
number of attempts to fix the damaged packet in that transmit opportunity
(Andrew McGregor can explain in detail).  Once the damaged packet is
successfully received, all the queued packets can be released (until the
next damaged packet).

One of the changes Felix and Dave made in the Linux ath9k driver is to
minimize this receive buffering (it was much larger than it needed to be),
but also fix an infinite loop bug (also present in other Linux drivers), so
that a single damaged packet can't cause very long delays and ensure the
damaged packet gets dropped in a somewhat timely fashion.  At one point,
Dave Taht had seen > 1 second delay on simple ping packets.

So it isn't just a crappy wireless network; it's also been crappy
software/firmware.

Ultimately, lots more work needs to be done in the Linux WiFi stack to
rework its queuing entirely; that work is just underway.

I note this because all the bufferbloat issues apply equally to hosts and
home routers (most of which run Linux).  While you may not be able to
control your hosts (unless you are a Linux user), you can do something
about your home router...  And do so immediately, if you want to bother.

> not that uncommon for wireless carriers to employ a similar bag of tricks

Yup; I could not remember Anton's name.

> afaik we don't do any qos, since we don't have any queues to speak off.

IIRC, it was mostly diffserv marking to help certain packet types along.

> trying to stream video or maintain the responsiveness of your ssh session.

Yup.  Homes are a great place to suffer from bufferbloat.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 9 Aug 2013 20:20:30 +0200 (CEST)

On Fri, 9 Aug 2013, Chris Elliott wrote:

> evaluation and the technical recommendations on the suitability of 
> venues to host the network you all know and love is taken very 
> seriously, and has nixed many venues.

How do the venues react when they hear that they're being rejected on 
these grounds?

I'm still hoping network connectivity is going to not just be a cost 
center for venues going forward, but so far it seems it's still a checkbox 
feature that isn't taken very seriously.

---

From: Dave Cottlehuber <[email-masked]>
Date: Sat, 10 Aug 2013 23:58:47 +0200

+1

As a first time attendee here was my response: "easily best internet
ever, anywhere."[1].

Not just well done, not just world class, but setting the standard for
others to follow. I was surprised at how many apps and tools simply
didn't handle IPv6 at all, but you can't ask a conference to fix that
:-).

[1]: https://twitter.com/dch__/status/362458818329853952

---

From: joel jaeggli <[email-masked]>
Date: Mon, 12 Aug 2013 08:29:23 -0700

> How do the venues react when they hear that they're being rejected on

I prefer to think of this as a positive choice.

Given a options of between multiple venues with various features and
attendant compromises it seems wise to pick the one with the best
overall set of features.

having no network infrastructure to speak of or a totally inadequate
wiring plant is both an increasing rarity and pretty early survey item.

> center for venues going forward, but so far it seems it's still a
> checkbox feature that isn't taken very seriously.

Some problems are hard to solve... we'd like a lot of attenuation
between meeting rooms, which is hard if they're all part of one common
space seperated by airwalls and it's less hard if the building is a
concrete bunker.