Subject: reflections from the trenches of ietf62 wireless

From: "Odonoghue, Karen F CIV B35-Branch" <[email-masked]>
Date: Tue, 15 Mar 2005 11:08:51 -0600

Folks, 

After a few days of decompressing, I have been considering 
what to say that is helpful without unnecessarily prolonging 
this conversation. I have been involved in the delivery of 
wireless for six IETF meetings (#s 46, 56, 58, 60, 61, and 62), 
some less painful than others and four without hosts. For those 
commenting on how a familiar venue should help and wasn't it 
better last time we were in Minneapolis, I distinctly remember 
sitting in the health club hot tub at the end of IETF58 swearing 
I would never do the wireless again. Believe me, it wasn't 
better last time. For whatever reason, Minneapolis hasn't been 
kind to IETF wireless recently. 

As I wrote this it got longer and longer, so the abridged version is:
-  We had problems on Monday, but we believed the wlan to 
be operational (albeit without IPv6) with a few obscure 
problem reports from Tuesday onward. If people were 
indeed experiencing debilitating problems all week, then it 
is unfortunate that we were not aware of the issues. 
-  We can document lessons learned and recommendations 
for future hosts, but I believe the current model for 
providing wireless to attendees is broken and needs to be 
fixed. I would be happy to participate in the discussion on 
how to fix this. These discussions should probably take place
off this already overloaded list. 

So, unless you want gory details and rambling... you can stop 
reading here...

Technical Summary/Issues:
-  We had no wireless hardware one week before we were 
scheduled to install the wireless. We twisted arms to get 
hardware and support from two companies who really 
stepped up at the last moment. We started the wireless 
install on Friday, March 4th.  
-  We did not deploy anything new or experimental. We 
deployed what we had available. In the case of the 
wireless, the alternative was about a dozen Cisco 350s 
the secretariat had stashed away in case of emergency. 
We did what we have done for the past two IETF 
meetings - only with a different combination of 
equipment in a different venue. The addition of 802.11a 
did not add complexity and if anything improved the 
situation by moving some of our wireless users out of 
the b/g range. I would agree that we could drop the wep 
and .1x portions, but again, this worked fine for the 
previous two events. Believe me, we are very risk 
adverse. 
-  After a surprisingly easy install, we had a meltdown 
Monday morning at the beginning of the first session. 
This meltdown and the following shockwaves on 
Monday resulted from some less than optimal 
deployment decisions, a configuration issue, a bug in 
the deployed code, and some unforeseen interactions 
with the infrastructure. In an attempt to stabilize things, 
we shed a number of capabilities culminating in a 
downgrade of controller code on Monday night. I 
would like to say that we did this in a careful and 
reasoned fashion, but I will admit there was a fair 
amount of chaos. 
-  Tuesday morning we wandered around trying to see 
how things were going. Most people we talked to 
seemed happy enough at the time but willing to tell us 
war stories from Monday. (Thank you, but we already 
knew Monday was bad.) We were getting sporadic 
reports of IP connectivity problems, but we couldn't 
seem to catch anyone actively experiencing the 
problem. I sent out email soliciting input from people 
experiencing or having experienced the problem 
sometime after Monday. I received a total of four 
responses over the next two days. 
-  At this point, we considered moving to more current 
code, but the reports we were getting indicated that 
things were working. Because our perception at that 
point was that things were working, we made a decision 
on Tuesday evening not to upgrade. 
-  We don't really have a good way to measure the user 
experience. In this case, we thought the wireless was 
mostly working. I asked for gentle feedback during the 
meeting. With the exception of Steve Casner who 
patiently reported back to us, we received basically 
nothing. The helpdesk was also tracking problems for 
us and again after Monday received very few reports. 
-  Until the flood of email started on Friday, we believed 
that we had delivered a working wireless network (after 
Monday) and with an occasional problem that impacted 
a small subset of users but was not debilitating. 

Structural/Administrative Issues: 
-  I do not see much motivation for hosts (or vendors for a 
meeting without a host) to support the IETF network. 
The risk to benefit ratio is just too high. They can get 
much better exposure in environments that are less 
stressful and that they have more control over. 
-  Advanced staging helps to reduce configuration issues 
and allows more time for operational troubleshooting 
onsite. This can't happen when you don't have a host or 
contract out the service. 

So where do we go from here...Well, we have been asked to 
document our lessons learned. While we can do this, it seems 
to me that there are new issues that bite us each time (e.g. one 
time we had APs that rebooted every time a certain threshold 
of clients was reached. The code was eventually fixed. The 
problem doesn't exist anymore.) We can provide guidelines 
and experiences, but war stories from the world of IETF 
wireless deployments don't seem that useful. I would be happy 
to work to document the basic guidelines that we use, but 
generally there is a set of constraints that complicate things - 
like having no hardware. The key to improving the reliability is 
continuity between meetings and the current model does not 
support that. 

Given the trouble the IETF has with getting sponsors for the 
meetings, perhaps it is time to revisit our model of operation. If 
we want the network (including wireless) as a production 
service, then perhaps we should contract out that service to an 
entity that would be responsible for it and could sustain more 
continuity than the current model allows. This naturally costs 
money. There are people out there that will do this for the right 
price. I would be happy to hand over my green dot to someone 
properly resourced to do the job. 

Karen O'Donoghue
_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: JORDI PALET MARTINEZ <[email-masked]>
Date: Tue, 15 Mar 2005 19:05:24 +0100

Hi Karen,

Thanks for your feedback. I was one of those having problems and not
reporting it, but I didn¹t saw any email asking for more feedback after
Monday, so I assumed the ³fixing² work was on-going. Actually I¹d problems
to read my emails, and I believe even lost some emails for some strange
circumstance, which I tend to associate to the IETF connectivity, because
not having changed anything in my server, neither my laptop, when I come
back to Madrid, everything was working fine (and as said, NOTHING changed).

The sad thing is that for me everything was working fine on Monday, because
even when IPv4 was not working very fine, I was using IPv6 and worked very
well :-))) So from my point of view a big mistake disabling it, because
didn¹t solved the problems, as we have learned afterwards.

In *short* I will say that my conclusion to your exposition is that we lack
for a proper planning, something that we know already for the meeting
arrangements, but which is more critical in terms of ensuring a proper
network deployment (you need time to think about what can be wrong, what
went wrong already in previous occasions, and try to avoid it as much as
possible).

May be will be interesting to setup a mail exploder for taking care of the
meetings planning and preparation ? It can be not only technical but also
about the logistics of the meetings (or two different mail exploders).

Regards,
Jordi

De: "Odonoghue, Karen F CIV B35-Branch" <[email-masked]>
Responder a: <[email-masked]>
Fecha: Tue, 15 Mar 2005 11:08:51 -0600
Para: <[email-masked]>
Asunto: reflections from the trenches of ietf62 wireless

Folks, 

So, unless you want gory details and ramblingŠ you can stop
reading hereŠ 

meetings  only with a different combination of

So where do we go from hereŠWell, we have been asked to
generally there is a set of constraints that complicate things 

Karen O'Donoghue

---

From: Bill Manning <[email-masked]>
Date: Tue, 15 Mar 2005 10:55:00 -0800

On Mar 15, 2005, at 9:08, Odonoghue, Karen F CIV B35-Branch wrote:

>  I distinctly remember
> I would never do the wireless again.

	well... call me old fashioned - I appreciate those good
	folks who go forth and try and give us an environment
	that lets us do good things.  Since we tend to demand
	at least current stuff, we will get new loads, new features,
	-every time- and the presures of pulling this together
	on short notice is fraught w/ soft-spots.

	i'd like to speak a bit wrt "production" and "experimental"
	in the context of network stability.   I'd encourage folks
	who need "production" levels of service should buy the
	same from the venue.  Perhaps I am showing my age,
	but i remember that there are/were only three IETFs per
	year because the fourth slot was ear-marked for live
	testing (Dan Lynch's Interops) ...  and over the years
	we have had new/experimental code/features crop up
	on the IETF network as a matter of course.

	) multicast
	) ipsec
	) vpns
	) OE (thanks Michael)
	
	and others that i am forgetting right now.   that said, i'd
	really like to tweek the IETF network in the next couple
	of mtgs to allow a subset of us to test out our expectations
	and implementations while we are f2f.  And having a 	
	supported and appreciated NOC/network staff is critical
	for this to work.

	so - again - thanks for your hard work.  we could not
	do this w/o you.

--bill

---

From: "Marshall Eubanks" <[email-masked]>
Date: Tue, 15 Mar 2005 14:11:33 -0500

One thing I have heard from the trenches is that the WLAN would be a
lot easier to set up and debug if

- the IETF owned its own base stations
- and these were selected for their functionality for supporting large meetings with
lots of RFI (i.e., as opposed to having to take whatever the sponsors donate).

Do you think that this would solve the problem ? 

My back of the envelope says that this would represent 
a $ 50 to $100 K initial investment, plus shipping costs.

Regards
Marshall Eubanks

On Tue, 15 Mar 2005 11:08:51 -0600

---

From: Leif Johansson <[email-masked]>
Date: Tue, 15 Mar 2005 20:12:42 +0100

> problem reports from Tuesday onward. If people were

I mostly had no problems from Tuesday onwards with one exception: I had
to reload my kernel driver after each sleep - possibly this causes a
full reset of the hardware or something. This is something I seldom (but
not never) have to do.

I'd be willing to lay odds that the people who saw problems from Tuesday
share some hardware characteristics: for instance the same or a small
number of chipsets. My guess is that all AP-chipset combos are not
created equal.

	MVH leifj

---

From: Clint Chaplin <[email-masked]>
Date: Wed, 16 Mar 2005 06:34:23 -0800

I did have a problem Friday morning before the session start.  I got
an IP address, but couldn't even ping the router.  The person next to
me was fine.  Reboot and release/renew didn't help.  I finally walked
to a different place to force a roam, and everything worked fine.

One data point.

-- 
Clint (JOATMON) Chaplin
Wireless Security Technologist
Wireless Standards Manager

---

From: "Tony Hain" <[email-masked]>
Date: Thu, 17 Mar 2005 07:51:27 -0800

Odonoghue, Karen F wrote:

FWIW: Those 350s were provided after a similar bad experience specifically
so there would be consistent and stable equipment available. The fact that
they are considered 'emergency' at this point shows that in fact people do
expect new features from the equipment, overly a fairly short period of
time. Any discussion about acquiring new equipment needs to consider the
replacement costs to keep it up with expectations. Given the state of
evolution for 802.11 equipment this means annual refresh until things settle
down. 

I do appreciate the effort of the volunteer team. I was a mostly happy user
of the A channels, with the only issue being that I had to tunnel IPv6. 

Tony

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Thu, 17 Mar 2005 13:40:23 -0500

On Thursday, March 17, 2005 07:51:27 AM -0800 Tony Hain <[email-masked]> 
wrote:

> expect new features from the equipment

On the contrary, I think it reflects the realization that a dozen AP's is 
not enough to provide service for O(1000) people over a dozen meeting rooms 
plus lobbies, bars, restaurants, etc.  Had the volunteers who built the 
network been unable to get wireless hardware in time, they could have 
broken out the emergency AP's and provided worse coverage to fewer areas 
than we had last week.  It would have been better than nothing, but it 
would _not_ have been better than what we got.  And you'd still all be 
whining.

-- Jeffrey T. Hutzelman (N3NHS) <[email-masked]>
   Sr. Research Systems Programmer
   School of Computer Science - Research Computing Facility
   Carnegie Mellon University - Pittsburgh, PA

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 17 Mar 2005 20:19:55 +0100

On Mar 17 2005, at 19:40 Uhr, Jeffrey Hutzelman wrote:

> they could have broken out the emergency AP's and provided worse 
> coverage to fewer areas than we had last week.  It would have been 
> better than nothing, but it would _not_ have been better than what we 
> got.

Actually, for those of us who had perfect wireless connectivity but 
couldn't get IP packets through for hours on end during most of the 
week, having some of the low-tech APs in addition to the "managed" 
infrastructure would have been nice.

The fun thing about the bugs of the infrastructure was that it looked 
different to everybody, so in hindsight it's clear that no consistent 
picture arrived at the operations team.  I'll do better reporting next 
time...  (On the other hand, some more logging by the operations people 
what they changed would have helped triggering renewed reports.)

Water under the bridge.

I still haven't heard a diagnosis that I could take home and explain to 
other people, but I certainly have a lot of FUD about "intelligent" 
wireless infrastructures in my blood after the last couple of IETFs.  
It might be a good idea for the purveyors of these boxes to come up 
with some research about what went wrong, because I'm not the only 
IETFer who influences major WLAN purchasing decisions at home (we 
continue to believe we have the second largest WLAN in all of eduroam, 
and other institutions in Germany are occasionally looking at what we 
do and why).

Gruesse, Carsten

---

From: Dave Crocker <[email-masked]>
Date: Thu, 17 Mar 2005 13:20:40 -0800

On Thu, 17 Mar 2005 07:51:27 -0800, Tony Hain wrote:
>  The fact that
>  expect new features from the equipment,

it shows that SOME people decided to add new features.  

it does not show that the network really needed them.

and the difference between these two points is exactly where open discussion and agreement among the ietf meeting participants could be helpful.

right now, the folks doing the choosing pretty much have to guess what the folks doing the using want/need.  open discussion could eliminate the guessing.

if, as some have voiced, the community of attendees feel it is essential that we eat our own dogfood of new features, on our meeting production network, then we will have agreed to the consequence, assured lack of stability.

if instead the community feels that reliability for a core set of functions is paramount, then new features can only be added after they are viewed as stable and low-risk.

  d/
  ---
  Dave Crocker
  Brandenburg InternetWorking
  +1.408.246.8253
  dcrocker  a t ...
  WE'VE MOVED to:  www.bbiw.net

---

From: "Lucy E. Lynch" <[email-masked]>
Date: Thu, 17 Mar 2005 16:08:08 -0800 (PST)

> right now, the folks doing the choosing pretty much have to guess what
> the folks doing the using want/need.  open discussion could eliminate
> the guessing.

> if instead the community feels that reliability for a core set of
> functions is paramount, then new features can only be added after they
> are viewed as stable and low-risk.

As if the community speaks with one mind...

After the *less than great* wireless deployment for IETF 55 the NOC
team did an internal crit/self crit which you may find interesting -
note the the first thing we talk about is the sometimes conflicting
requirements of the communities of interest:

*disclaimer* these notes are very old - and as all Marlowe fans know
"that was in another country, And besides, the wench is dead."

NOC55 crit/self-crit

Human Factors

 - 4 groups of players/ 4 sets of needs

   - Host
     - responsible for providing the network, user connectivity (wireless
       & TR), t-shirts & any toys, & hosting a Social if there is one.
     - Expect to pay in excess of 100K and order local loop early!
     - Needs:
       - since the host name is attached to network performance, that
         performance needs to be exceptional
       - Given the amount of money & staff time needed, the host should
         expect to be treated like a major donor and given enough
         cooperation to create a positive outcome for everyone.

   - Secretariat/Foretec
     - responsible for booking hotel & arranging room access
     - responsible for power distribution in meeting rooms and
       deployment of power strips for end users
     - hire AV staff & gear
     - on site registration & temp staff
     - all catering for reception/breaks/etc.
.    - manage meeting schedule (agenda & rooms) - communicate needs
       to the host
     - Needs:
       - cost containment?
       - agenda & meeting requirements info from WG's
       - direction from IETF on required services

   - IESG/WG/etc.
     - determine wg agendas & room needs (size, multicast, time frames
       etc.)
     - convey meeting requirements (services, network access, etc.) to
       Foretec
     - additional meetings (ISOC/IAB/IESG)
     - Needs:
       - infrastructure to support the work of the IETF
         (network, power, meeting rooms, AV, lists, multicast, etc.)
       - Support for admin functions (editor, meeting planners, etc.)
       - accurate accounting (revenue generated, admin costs & meeting
         expense) information for decision making.

   - End Users
     - participate in WGs
     - network & TR users require support, need to provide USEFUL feed
       back
     - Needs:
       - infrastructure
       - enough meeting & hotel space
     - Wants:
       - ubiquitous network & power
       - toys & food???

The interests & duties of the four groups overlap and there are some
tensions where those interests compete (i.e.. if the host wants to deploy
a well tested network, the Secretariat wants to keep hotel costs low there
will be scheduling problems that preclude early in-building deployment)

Constraints (55)

 - non-local host
   - no near site space to bench test network elements
 - short time frames for deployment (room access)
   - in building access to the NOC & TR on Friday (network required SAT)
     - early access (Thurs.) to telco space to rack routers & core servers
     - late access to meeting rooms (in some cases just hours before a
       meeting)
http://darkwing.uoregon.edu/~llynch/noc55/mech/11-11-55thMeetingRoomSchedule.html
 - AP selection
   - based on internal factors & lack of support for some Nokia wireless
     cards in the Foretec owned cisco APs. The (16) cisco APs were carried
     as back ups.
   - marginal vendor support for reported problems.
 - Operational "dogfood" (worthy but experimental)
   - dynamic DNS
   - Bro/IDS

Problem Areas

 - Host/Sec communication
 - Noc/End user communication
 - Loss of core staff (Randy/Fenner & then Rob to other duties)

Technical Problems

 - Wireless
   - long story - Joel & Bill?
 - In building infrastructure/switch failure
   - large room wireless failure
   - first mile multicast problems
   Marquis I-IV & the multicast sub-net both ran off two cat5 runs from the
   basement electrical level to the Ballroom control booth on the lobby
   level (3 floors). 2 [8 port] switches were then deployed in the control
   booth and in building cat5 for Marquis was patched. Under load (Mon.
   AM) these switches failed (too long a run?) They were replaced with 24
   port cisco switch (managed) and the problem did not reoccur.
   (Note: heas says "managed is always better")
 - Juniper/DHCP relay issues
   - bug reported, DHCP relay moved to server
 - ARP white noise
   - large amount of arp traffic seen through the routers from turn-up
     never fully diagnosed, but didn't seem to effect performance

Strengths

 - Noc Team depth -
   - Enough folks to cover & second all key roles
 - Fenner & co. (wireless debugging/monitoring)
 - Joel/Cisco AP deployment under stress
 - TR staff
 - excellent volunteer participation (GA Tech & Nokia!)

Lessons Learned

 - choose the best technical solution w/out regard to politics
   - or be prepared to punt if things get messy
 - understand & advertise that running experimental services will lead to
experimental
   NOT production style service.
 - test all existing in building infrastructure under load
   - managed is better at the edge when using other peoples stuff
     (managed is just better)
 - Don't let outside considerations distract from inside problems
   - multicast loss problems lost in the multicast deploy "noise"
 - provide easily accessible data on network status
 - Provide a direct channel for wireless users to help sort info

Nits

 - more communications tools may not be better...
   (list/web/RT/radios/cell phones etc.)
 - NOC needs faster access to changes on the host page, or a nested NOC
   page (user updates, publicizing monitoring data, etc.)

Needed in Future:

 - better information about how & when wireless worked at past IETFs
   (I suspect it worked when there were many fewer users)
 - A "WiNUrs" BOF (Wireless Network Users) at either NANOG or IETF
   - w/ a clinic to demonstrate dense deployment issues
     (attractive nodes/sticky nodes/adhoc users/overloading failures/
      association refused/etc.)
   - & user education & tools
     - meetings need a quick report mechanism that reports:
       user mac address|OS (version)|card type & driver|settings
       & an FAQ of known problems.
     - users continue to send clear text passwords (Fenner data)

Lucy E. Lynch 				Academic User Services
Computing Center			University of Oregon
llynch  @darkwing.uoregon.edu		(541) 346-1774

---

From: Dave Crocker <[email-masked]>
Date: Thu, 17 Mar 2005 16:18:26 -0800

>  As if the community speaks with one mind...

there is a reason we call our consensus process 'rough'...

on the other hand, we supposedly have some practise at reaching it.

>  "that was in another country, And besides, the wench is dead."

in this case, perhaps not entirely dead.

good stuff. thanks!

  d/
  ---
  Dave Crocker

---

From: Keith Moore <[email-masked]>
Date: Fri, 18 Mar 2005 12:33:15 -0500

I find myself thinking that these are the most important things for
wireless:

1. advertise up front that wireless is an experimental, not production, 
   service.

2. make the wired network production quality.  make sure that there are
   wires to all essential services (e.g. audiocast, chairs, presenter,
   jabber scribe), and make sure those wires are reserved for their 
   intended uses.  if the wireless network works well, so much the
   better.

3. NOC team really needs a visible bidirectional communications path to
   the users that doesn't rely on the network.  I'm thinking in terms of
   a live person and a large white board in a highly visible location - 
   say, near the message board.  The white board should report what the
   NOC team thinks is the status of the network, summarize what is
   reliably known about the nature of the problems, and list known 
   workarounds.  The person needs to write down reports of problems and 
   interact with users to try to precisely identify the conditions that 
   produced the failure.

   The reason that this is needed is that rumor and exaggeration and
   idle speculation on the part of IETF attendees seem at least as
   damaging to the repuation of the network (and the host) as any 
   equipment problems.   Such rumor and exaggeration and idle speculation 
   may even have an adverse operational impact, as users try to route 
   around their perceived (and sometimes incorrect) perceptions of damage.

---

From: "Brett Thorson" <[email-masked]>
Date: Fri, 18 Mar 2005 13:57:07 -0500 (EST)

Mr. Crocker & Mr. Hain,

I use this message only because it was available, and it somewhat sums up
the assumptions and conclusions reached previously many times.  I use it
as an outline only.  Please do not view this as any kind of comment on you
or your thoughts, words or expressions directly.

To the IETF @ Large;

>  On Thu, 17 Mar 2005 07:51:27 -0800, Tony Hain wrote:
Actually are considered 'emergency' at this point shows that in fact
people
>> do

Actaully not.  It shows that new features came about because of the
available hardware.  Why deploy 15 Cisco 340's that all need firmware
upgrades when you get an offer by "Company X" to give you new, easier to
manage hardware, that has added value that will make it easier to find
problems before IETF-ers-at-large start grumbling under their breath about
it?  Maybe you have an answer.  But have you been on even one of the NOC
teams for IETF 58 through 62?  If no -> Mod down 20 points, then
re-evaluate.

> it does not show that the network really needed them.

Scenario:  Ad-Hoc networks.
discussion
Before:  There is an ad-hoc network somewhere on the "x" floor.  They are
sucking people into them.  People are complaining.  Person from company X
walks by:  "Hey I have a product that will pin point that person so you
can show them the light"

After:  Hmm, not as many ad-hoc networks.  We can exclude those people
from the net so we isolate them.  Extra features that come along for the
ride, maybe not as proven as previous, but those new features do help the
ALL VOLUNTEER NOC TEAM to isolate some of those problems.

> and the difference between these two points is exactly where open
discussion and agreement among the ietf meeting participants could be
helpful.

Open discussion would be great.  However, on this list, probably not the
best.  I think IETF people are great, and very smart people.  But from
watching the re-org procedure, there are many want-to-be lawyers,
accountants, business executives, etc.  I'm sure there were many people
commenting on those topics that actually were official members of those
professions.  The point I am making is that running an IETF NOC is so
different that at some point in the interest of the signal to noise ratio
we have to let the bodies in the trenches fight the war and come to that
table with their comments and quite possibly tone down the comments of
IETF meeting participants.  While their comments should be welcome, they
simply don't have the view from both sides of the switches & APs.

I'm all for dicussion, but if your name hasn't been on at least one thank
you list for "NOC team IETF now minus five inclusive" then chances are
your opinions are welcome but possibly not up to date.

Scenario:  Wired drops for the presenter, jabber scribe, etc.
This would be great.  However, this of course adds hardware.  Quite a bit
of it.  I put a switch in a central like location and secure it, POE the
AP's and run 4-6 AP's in multiple rooms with 4-6 CAT5 cables.  Now, I need
to put breakout switches in each room, secure them, tape down the cables,
possibly pull the cables and put them back down if the meeting rooms get
used by other groups.  I need to ship the cables, and the extra switches,
along with the security cables.  These issues continue.

Result:  This is no reason that this cannot be accomplished.  Why is it
difficult?  Security, time to deploy & tape down cat5 cable, shipping and
acquiring extra hardware.

Consequence:  It is easy to say, this should be done.  It's great to talk
about it and come up with the 7 ways it can be accomplished.  It would be
perfect if you could pay for it too.  Until then, you are back to getting
what you pay for.

> right now, the folks doing the choosing pretty much have to guess what
the
> folks doing the using want/need.  open discussion could eliminate the
guessing.

Choosing would be an excellent gift to have when doing a non-hosted
terminal room.  When there isn't a host, you take whatever you can get. 
When I did my first terminal room in San Francisco, Cisco stepped up to
the plate with 2 pallets of gear.  I still had to worry about: Terminals,
wiring, placement, extra routers, extra switches, fiber, this list
continues ad nauseum.  Choice you say?  My choice was either Cisco (which
worked fabulously I might add) or to be incredibly delusional and wait for
something else.

> if, as some have voiced, the community of attendees feel it is essential
that we eat our own dogfood of new features, on our meeting production
network, then we will have agreed to the consequence, assured lack of
stability.

are
> viewed as stable and low-risk.

Which all goes out the window when you settle for the "You get what you
don't pay for" scenario.  Unless you a throwing money at the problem, you
don't get to dictate which solution to chose.  Please, PLEASE don't quote
the meeting fees, etc. at me.  I've done the budgets, I've seen the
costs.Internet
 I've worked for that company.  I've run that NOC.  You take what you can
get.  Even then in some instances, you get what you can take.

Are you paying for the hardware at a hosted IETF?  No.
Are you paying for the hardware at a non-hosted IETF?  No.
Are you getting what you pay for?  Yes, and A LOT MORE!
consistencypersistence
In Summary:  The fees that attendees pay don't come CLOSE to paying for
what is required to run a terminal room.  People, product and sanity.  The
numbers to design/engineer, create, ship, deploy, test, maintain, run and
take down a NOC are not fictious unknown numbers, these can be calculated.
 I've done it before.

No host?  No Sponsor?  Volunteers?  Keep in mind that except for me, the
people who ran IETF NOCS for un-hosted meetings were there to actually
attend meetings.  I can tell you, they didn't make too many of them.

Hardware?  You take what you can get.  If someone wants to donate
hardware, who do they donate it to?  Foretec?  CNRI?  The IETF doesn't
exist as an entity to donate to (at least that's the line I got when I
asked that question).  Then when it breaks, who fixes, it?  Who pays for
the flight cases to ship the stuff so it won't get destroyed?  If hosts
ask to use it, do we let them?  Who pays for customs on that?

Donating hardware or sponsoring.  I've seen a couple of IETFs that I think
went pretty well.  IETF Austria and IETF Yokohama (this list is larger,
but let's use these two as an example).  I saw that Austria had a great
team of employees working on the network for over a year, and I also saw
that in Yokohama there were people sleeping behind server racks.  If
something goes wrong, heaven help you.  You now have 1500 IETF-ers that
are VERY vocal willing to tell you what you did wrong and exactly how
stupid you are.  If you are nice, they MIGHT tell you how to fix it, or
they may have to run off to a meeting (which is after all what they are
there for!).  Note: There are IETFers that are patient, polite and always
willing to help.  You know who you are and behalf of IETFers, NOCers, and
anyone who has ever used an internet protocol, thanks.

Essentially, it is a power company situation.  If you do great, no one
says anything.  If you don't do so hot, sell your stock, it's going to get
ugly.

So we've gone from people sleeping next to racks, and preparing for a year
ahead of time to no sponsors and a volunteer crew that is already taking 1
week off to go to a meeting.  Plus 1 week off to assemble the network. 
Plus 1 week off to coordinate the hardware.  What do they get?  They hope
to stay beneath the heavy and quick hand of that one vocal IPv6'er that
can't ping McMurdo Station over their Sat phone IPv4 VPN on their Timex
Sinclair.

This is not an impossible situation.  However, like many of the other
notes on this topic have stated, you need consistancy, and persistance of
knowledge.  The terminal room document is dead.  There needs to be a new
one.  But after a crew gets done doing a terminal room (donating their 3
weeks and having 6 months sucked from their soul) they probably have more
important things to do like IDs and RFCs that pertain to protocols on the
Internet, which is what they were all there to do in the first place, or
go back to their place of employment and do what they get paid to do.

Full disclosure:  I wasn't in Minneapolis this time.  I don't currently
work for the secretariat.  I'm not getting paid by anybody related to
IETF, I'm just here because I think the IETF is a cool group, and I think
I might be able to help them pursue their goals.

Cheers, good luck, and I cheerfully step back into 3rd party observer mode
until called upon.

--Brett M. Thorson

---

From: Pekka Savola <[email-masked]>
Date: Fri, 18 Mar 2005 22:13:52 +0200 (EET)

On Fri, 18 Mar 2005, Brett Thorson wrote:
>Scenario:  Ad-Hoc networks. discussion Before:  There is an ad-hoc 
>network somewhere on the "x" floor.  They are sucking people into 
>them.  People are complaining.  Person from company X walks by:  "Hey 
>I have a product that will pin point that person so you can show them 
>the light"
>
>After:  Hmm, not as many ad-hoc networks.  We can exclude those 
>people from the net so we isolate them.  Extra features that come 
>along for the ride, maybe not as proven as previous, but those new 
>features do help the ALL VOLUNTEER NOC TEAM to isolate some of those 
>problems.

My question here is -- how did the NOC folks manage with ad-hoc 
networks _before_ the new fancy (and often, it seems, brittle) 
software came in?

Maybe by documenting the procedures used "before" would help others in 
the similar situation, and would allow non-NOC folks (but those which 
might have dealt with similar issues in their local organizations) to 
improve those procedures.  And you never know, maybe even more people 
would then be able to spot the ad-hoc network infidels, leading to 
less work for NOC.

> acquiring extra hardware.

I agree that the wired drops in each room are probably more trouble 
than they're worth, especially considering the fact that the jabber 
scribes and other important people (excluding the chairs) are 
typically littered all over the room.

> worked fabulously I might add) or to be incredibly delusional and wait for
[...]
> weeks and having 6 months sucked from their soul)

It is not clear from your text whether you're referring to the whole 
network (including the backend required for wireless, like DNS and 
DHCP servers etc.) when you say "terminal room", or only those systems 
which are provided for free to the IETFers ("open terminals in the 
terminal room").  I'm assuming the latter.

A constructive suggestion:

We should only provide power sockets and a switch port in the terminal 
room.  Scrap everything else.  No computers, no printers, nothing. 
Simple and easy.  No need for a guard to stand by.

Why do we need anything more?  Just so that the IETF tourists can read 
their email?  Doesn't seem worth the trouble.

When the meeting is hosted by someone, let's not expect them to 
provide the terminal room.  Let's rather try to channel the resources 
to something more constructive (like, the equipment or T-shirts ;-)

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

---

From: John C Klensin <[email-masked]>
Date: Sat, 19 Mar 2005 11:14:33 -0500

--On Friday, 18 March, 2005 22:13 +0200 Pekka Savola
<[email-masked]> wrote:

>...
> We should only provide power sockets and a switch port in the
> terminal room.  Scrap everything else.  No computers, no
> printers, nothing. Simple and easy.  No need for a guard to
> stand by.
> 
> Why do we need anything more?  Just so that the IETF tourists
> can read their email?  Doesn't seem worth the trouble.

Pekka,

While I've suggested scrapping the terminals in the terminal
room on and off for years, I'd be reluctant to give up on the
printers.  Perhaps it is a sign of advancing age, but, when a
document gets above some length or level of complexity, I simply
have to have it on paper that I can flip through and mark up in
order to make sense of it.  I don't use the printers very much
-- I print and bring documents that are available significantly
before IETF with me-- but, when I need them for, e.g., a
document generated during the meeting, it is extremely useful,
indeed necessary, to have them there.  And, yes, I could drag a
portable printer to IETF meetings if I had to, but that seems,
at best, inefficient.

It is also important to have the wired Ethernet there unless we
have a 100% guarantee that the wireless will work smoothly and
efficiently, all of the time.  We aren't there yet, even at
"good" meetings.  And, while we could think about security
enclosures (themselves neither cheap nor light to ship around),
we probably need the guard in most locations as soon as we start
providing either power strips or Ethernet switches.   Sad, but I
think we have had experience that confirms that impression.

     john

---

From: "Stephen Sprunk" <[email-masked]>
Date: Sat, 19 Mar 2005 11:41:35 -0600

Thus spake "Pekka Savola" <[email-masked]>
> typically littered all over the room.

If wired drops are to be eliminated (and I don't oppose that), then the
wireless network reliability needs to be improved to the level of wired
connections.

As a strawman I would propose:

IPv4 unicast coverage of the meeting rooms is mandatory.  All other services
(IPv6, multicast, coverage of restaurants and bars, etc.) are highly
desirable but can be disabled if needed to preserve the mandatory service.
During setup, a rock-solid "mandatory service" network shall be deployed
first so that roll-back can be achieved quickly if necessary; only after
that is working perfectly should the network staff attempt to activate
"desirable" services.

Having separate networks for "mandatory" and "desirable" services doesn't
seem necessary yet, but future experience may prove me wrong.  I hope it
doens't come to that; if so we really need to look at the deployability of
the protocols we're developing.

> Simple and easy.  No need for a guard to stand by.

Printers are needed; as John Klensin noted, being able to print drafts and
other documents helps comprehension of the material.  It also keeps people
productive when the network is unavailable...

> their email?  Doesn't seem worth the trouble.

Let's not forget that the (vast?) majority of IETF attendees are there at
the expense of their employers; access to email keeps them in touch and
reduces the impact of their absence.  Checking in at night from a hotel room
isn't enough.

> provide the terminal room.

I don't think the host should be providing PCs, but IMHO a place to sit with
a wireless laptop (and recharge it) and print out the occasional document is
underrated.  Wired connections for laptops are debatable; perhaps a few
tables near the network equipment, but certainly not the entire room.

S

Stephen Sprunk        "Stupid people surround themselves with smart
CCIE #3723           people.  Smart people surround themselves with
K5SSS         smart people who disagree with them."  --Aaron Sorkin

---

From: Scott W Brim <[email-masked]>
Date: Sat, 19 Mar 2005 13:01:48 -0500

On Sat, Mar 19, 2005 11:14:33AM -0500, John C Klensin allegedly wrote:
> order to make sense of it.  

plus for printing boarding passes

---

From: Pekka Savola <[email-masked]>
Date: Sat, 19 Mar 2005 20:04:43 +0200 (EET)

Mmm.. My message appears to have bene misunderstood.  Perhaps I was 
too terse.. Inline.

> wireless network reliability needs to be improved to the level of wired

I was speaking of the suggested wired drops at the meeting rooms 
(which we have never had), not in the terminal room, so this should 
not be an issue.

> productive when the network is unavailable...

I've also used the printers occasionally, to print some reading flight 
on the return flight (if I didn't bring enough with me).  But 
nevertheless, I think they're more trouble than they're worth.

> reduces the impact of their absence.  Checking in at night from a hotel room

Sure -- but the real questions are:
  1) how many IETF attendees don't have a laptop with them?  Obviously, 
those who do, don't need the open terminals.
  2) how many IETF attendees have a laptop but no wireless card?  Close 
to zero, so for most people, the wired drops in an otherwise empty 
terminal room only serve as a stable connectivity (i.e., stabler than 
the wireless) or for emergencies.

> I don't think the host should be providing PCs,

Good!

> but IMHO a place to sit with

But if we provide printers, there must be folks to clear the paper 
jams and other nastiness (most people will have wrong A4/letter 
settings in any case), provide the thousands of pages of papers which 
folks use to print out 100-page RFCs and PhD theses, etc.

Maybe it would be interested to have some rough stats from the NOC 
whether printing is really causing any issues, or whether it's just my 
imagination and attempt to cut off functions which we absolutely don't 
need.

---

From: Joe Abley <[email-masked]>
Date: Sat, 19 Mar 2005 22:47:07 -0500

> 1. advertise up front that wireless is an experimental, not production,

For people with operational responsibilities, this (above) presents a 
problem. But, see below.

Dislcaimer: I didn't attend the whole week; I arrived on Wednesday, and 
left on Friday, and I obviously can't comment on the quality of the 
network when I was not there.

Using my powerbook with the b/g apple adapter, the wireless network in 
the bar and in most of the corridors was good enough to be able to read 
mail, use jabber and ssh to routers to fix things. This was good. There 
were times when it dropped out, but they weren't too frequent.

The quality of the wireless network in the meeting rooms that I sat in 
was much worse, and was frequently not good enough to fix routers. 
That's largely ok, though; if I'm paying sufficiently little attention 
in the meeting room that I am tempted to fix routers, then I should 
quite probably go and sit somewhere else instead.

I think I've seen someone suggest that the terminal room (with tethered 
ethernet connections) is a reasonable fall-back to the wireless, which 
is true a lot of the time. It is notably not the case, however, if:

(a) you need to be able to talk to someone on a cellphone about a 
problem you are trying to fix; I didn't try, but I got the distinct 
impression that talking into a cellphone for half an hour in the 
terminal room would make be unpopular with everybody else in there.

(b) you need to be able to make or receive calls over IP. Making 
hour-long calls to people in different countries gets really expensive 
when you're forced to fall back to roaming GSM phones (and the quality 
is frequently bad, too; 1900 and 850MHz GSM coverage in the hotel was 
patchy in many places where the wifi was tolerable).

I get the impression that most IETF meeting participants don't have 
operational responsibilities, and if I've got that right, it does not 
seem reasonable that the requirements of the network should necessarily 
be elevated to suit the small proportion of people who do.

However, if there was some place that I could have simultaneously made 
phone calls and got reliable v4 transit (whether wireless or tethered), 
that would have been a fine and useful thing for me.

Joe

---

From: "Tom Petch" <[email-masked]>
Date: Mon, 21 Mar 2005 21:01:55 +0100

<snip>
> those who do, don't need the open terminals.

I don't have a laptop with me and don't want to; I want to be as much part of
the IETF meeting as I can be and lugging a laptop, hunting a power drop with the
right holes, getting distracted by other cyberspace worlds, gets in the way.

In the larger sessions, I look around to see how many people have laptops; my
estimate is around 90%, ie around 10% do not.

Tom Petch