Subject: Re: [81attendees] sucky Delta hotel network (and bufferbloat)

From: Jim Gettys <[email-masked]>
Date: Mon, 01 Aug 2011 16:58:24 -0400

Hotels are probably the best (worst) place to see bufferbloat (and many
other problems) in all their glory, particularly when a pile of IETF
folks arrive.  Fred Baker has been studying this slow motion disaster
for many years; he has some very entertaining plots and data
demonstrating various diseases he's found including bufferbloat.  Might
make a good transport plenary session sometime.

Wireless drivers in the operating systems have lots of issues, such as
trying to retransmit packets until the cows come home in the face of
errors, compounding problems; not all of what you see in sick wireless
networks is bufferbloat; lots of other diseases abound.  If you want to
see (and maybe help fix) such an example, look at:
https://www.bufferbloat.net/issues/216 where we are fighting such a
problem in the Linux stack, specifically the Ath9k driver.

In the hotel case, you have many places to suffer from bufferbloat,
depending on the bottleneck link:

    1) in your host operating systems you'll get bufferbloat, if the
transfer rate you are getting via wireless is below that of the
broadband link, common due to both running through multiple walls and
the sharing of the link with others.  A common ring buffer size in
drivers is 200-300 *packets* on modern hardware.

    2) In many hotels, these access points are home routers rather than
enterprise grade access points, so you get into all the trouble in those
I've noted in my talk and blog.

    3) then you get into the bloat in the back haul from the hotels. 
Many/most hotels are now (under) provisioned over broadband connections,
so that hop, anytime there are a lot of users, saturates and stays
saturated during busy hours, particularly if IETF is in town, so you get
latency and bad packet loss there.  Similar pheonomena occur in venues
like libraries.

    4) particular ISP's who specialised in connecting hotels to the
Internet appear to have been (to be) particularly clueless and do not
run any form of AQM on their networks, even though parts of those
networks are often saturated.  Those ISP's seem to have been merged into
bigger ISP's, but the same clueless operators seem still in charge of
the remains of those networks. I've seen bad problems 5 hops into such
networks.  This is classic http://www.faqs.org/rfcs/rfc2309.html mistake
of the 1990's. I won't name any names, but you can figure out who on
your own easily enough.

If you run Windows, pingplotter is probably the best tool to observe
problems; on Linux, mtr can be useful.  I've done my own work mostly
with vanilla ping and scp (to saturate the path).

And anywhere I go, I *always* run netalyzr first, as it warns me of all
sorts of illness (and tries to test for bufferbloat too).
I recommend everyone do so.  See: http://netalyzr.icsi.berkeley.edu/

So hotels and conferences are particularly fruitful places to see
Internet illnesses of all sorts.  The one exception to the conference
problems I've seen is NANOG/IETF, where cluefull people have done useful
classification and AQM setup and where there is lots of traffic via
enterprise access points that aren't over buffered, so that even RED93
algorithms can succeed at controlling the queues.  Would that someone
would write up that configuration and publish it as a BCP: it would at
least help other conferences and enterprises to do much better, while we
sort out the home router mess which needs other AQM algorithms, as yet
untested.
                        - Jim

---

From: "JOHNSON, ALASTAIR (ALASTAIR)" <[email-masked]>
Date: Tue, 2 Aug 2011 05:11:42 +0800

Jim Gettys wrote:

> problems I've seen is NANOG/IETF, where cluefull people have done 
> useful classification and AQM setup and where there is lots of 
> traffic via enterprise access points that aren't over buffered, so 
> that even RED93 algorithms can succeed at controlling the queues.  
> Would that someone would write up that configuration and publish it as 
> a BCP: it would at least help other conferences and enterprises to do 
> much better, while we sort out the home router mess which needs other 
> AQM algorithms, as yet untested.

Anton Kapela gave a lightning talk at NANOG38 on improving the wireless 
performance. Available at:
http://www.nanog.org/meetings/nanog38/presentations/kapela.pdf

regards,
aj

---

From: Maciek Konstantynowicz <[email-masked]>
Date: Mon, 1 Aug 2011 22:18:31 +0100

Thanks for the pointers, will digest.

The hotel manager I spoke to, didn't even know what IETF is about, and didn't expect any excessive WiFi demands breaking hotel's Internet access service.
Maybe as a community, going forward we could provide more candid feedback re Internet/WiFi performance during IETF gatherings.
Or better give hotel mgmt heads-up about coming tsunami of Internet users... so that hotel could address it pro-actively..

The business incentive for the hotel would be to avoid giving 20% discounts if things don't work out as per my case :)
+ customer satisfaction.

Thanks,
Maciek.

---

From: John C Klensin <[email-masked]>
Date: Mon, 01 Aug 2011 17:37:45 -0400

--On Monday, August 01, 2011 22:18 +0100 Maciek Konstantynowicz
<[email-masked]> wrote:

> The hotel manager I spoke to, didn't even know what IETF is
> about, and didn't expect any excessive WiFi demands breaking
> hotel's Internet access service. Maybe as a community, going
> forward we could provide more candid feedback re Internet/WiFi
> performance during IETF gatherings. Or better give hotel mgmt
> heads-up about coming tsunami of Internet users... so that
> hotel could address it pro-actively..
> 
> The business incentive for the hotel would be to avoid giving
> 20% discounts if things don't work out as per my case :) +
> customer satisfaction.

Sigh.
I used the network in the Delta quite a lot and didn't
experience much of a problem, but kept my expectations set for
"hotel" rather than "IETF".  I also mostly used the wired
Ethernet -- it may be just anecdotal, but I've generally found
that wired Ethernet is much less likely to be problematic than
WiFi if neither is set up by experts.

While warning hotels that we are a collection of heavy Internet
users and that it is a good idea to make sure that their IP
vendors know that they need to tune things up to avoid
embarrassment, the bottom line is that the only times folks seem
to have zero complaints about hotel networks are where the hotel
infrastructure starts out above average and is then hooked into
the IETF network.  It simply isn't plausible to believe that
interconnection will occur with other than the primary hotel.
It would be good to know in advance how likely it is even with
the primary hotel but those opting for other hotels should
arrive with the assumption that they will be dealing with a
generic hotel network and What You Get is What You Get quality
(whether it is free and worth what one pays for it or not).

   john

---

From: John C Klensin <[email-masked]>
Date: Mon, 01 Aug 2011 17:44:56 -0400

--On Monday, August 01, 2011 16:58 -0400 Jim Gettys
<[email-masked]> wrote:

>...
>   Would that someone
> would write up that configuration and publish it as a BCP: it
> would at least help other conferences and enterprises to do
> much better, while we sort out the home router mess which
> needs other AQM algorithms, as yet untested.

Looking on the bright side, we have an important window of
opportunity here.  Most of those el-cheapo home routers/beacons
are going to need forklift upgrades to run IPv6.  If, as Jim
suggests, we agree on what it takes to get things right and can
get the word out soon, we might just see a generation of
IPv6-capable devices with competent buffer and congestion
management.

   john

---

From: Keith Moore <[email-masked]>
Date: Mon, 1 Aug 2011 17:48:36 -0400

> IPv6-capable devices with competent buffer and congestion

It might help even more if someone wanted to take on the task of writing a document called "Recommendations for Buffer and Congestion Management in Consumer-Grade Routers"

(omit "Consumer-Grade" if appropriate)

Keith

---

From: Monty Montgomery <[email-masked]>
Date: Mon, 1 Aug 2011 17:49:45 -0400

> WiFi if neither is set up by experts.

I will note that for four of the six nights I was there, the wired
ethernet in all three of our groups' rooms at the Delta (on three
different floors) was unusable, with ping indicating packet loss over
50% and latencies in the seconds range during the periods where there
was any connectivity at all.  Wireless obviously didn't work either.
I camped at other hotels and various coffeeshops to read mail (we had
a fourth at the Palace Royal; wireless there was fine).

Several folks have repeated the 'just use the wired ethernet' advice,
and I'm sure it ~worked for some.  For the Xiph contingent at the
Delta it was somewhere between badly impacted and nonfunctional for
the entire stay.

Monty

---

From: Ole Jacobsen <[email-masked]>
Date: Mon, 1 Aug 2011 15:01:55 -0700 (PDT)

"Expectations" are indeed important.

Equally important is "common sense and common courtesy" which implies 
a number of do's and don'ts on the hotel network. I am not going to 
attempt to create that list here, but I can well imagine some things 
that I should NOT be doing while on the hotel network, in 
consideration of my fellow IETFers. (Large downloads, live video...)

Ole

Ole J. Jacobsen
Editor and Publisher,  The Internet Protocol Journal
Cisco Systems
Tel: +1 408-527-8972   Mobile: +1 415-370-4628
E-mail: [email-masked]  URL: http://www.cisco.com/ipj
Skype: organdemo

---

From: Jim Gettys <[email-masked]>
Date: Mon, 01 Aug 2011 20:33:48 -0400

> http://www.nanog.org/meetings/nanog38/presentations/kapela.pdf
Yes, this is exactly what I was referring to, though Anton's
configuration has been further tweaked since that date given more
experience.  It's the first time I've seen a loaded wireless network
able to deal with the latencies that VOIP requires when it's under
serious load.

I'm trying to get a hold of those configuration files so we can try to
do the same thing in the home router work Dave Taht and I are up to.

But capturing that information in a BCP, along with the rationale for
the configuration would be a great step forward for the the world.
                    - Jim

---

From: John C Klensin <[email-masked]>
Date: Mon, 01 Aug 2011 20:36:03 -0400

--On Monday, August 01, 2011 15:01 -0700 Ole Jacobsen
<[email-masked]> wrote:

> Equally important is "common sense and common courtesy" which
> implies  a number of do's and don'ts on the hotel network. I
> am not going to  attempt to create that list here, but I can
> well imagine some things  that I should NOT be doing while on
> the hotel network, in  consideration of my fellow IETFers.
> (Large downloads, live video...)

Indeed.  Not just fellow IETFers but anyone unaffiliated with us
who happens to be staying in the hotel at the same time.

    john

---

From: Jim Gettys <[email-masked]>
Date: Mon, 01 Aug 2011 20:47:53 -0400

> (whether it is free and worth what one pays for it or not).

Yes, exactly.

You suffer a lot less on ethernet, as you take one possible point of
bloat out of the path.

Windows, by default, won't saturate a 100Mbps connection (it's tuned to
stay just below 90Mbps, as far as I can see). Mac and Linux will
saturate anything to a gigabit without thinking with even a single TCP
connection, given enough time for them to ramp up.  Even if Mac and
Linux saturate the ethernet link and fill those queues in the
driver/router, at least the amount of time they represent at high
bandwidth is much, much less.

So your queues build in the hotel's router, rather than backing up in
the wireless router and your laptop, at least when plugged in usually.

Wireless, going through multiple walls and also being shared often
guarantees your queues build in that wireless hop, where bloat is often
much worse than the uplink, and shared among others to boot.

So good practical advice until this is sorted out is to plug in to
ethernet whenever possible, and/or carry a high speed router of your own
to pug into that ethernet in your room, to try to guarantee the
bottleneck link is at the hotel backhaul.

Same thing goes at home: the home router/laptop link in your house is
probably more bloated than your broadband connection (and you can
bandwidth shape that to keep those queues small), so if you ensure your
wireless bandwidth is always above your broadband bandwidth you can have
reliable low latency.  But in houses like mine with chimneys, this can
be easier said than done.

There is a lot you can do about bloat once you understand it by moving
the bottleneck link to where it may be less problematic.
                        - Jim

---

From: Dave CROCKER <[email-masked]>
Date: Wed, 03 Aug 2011 09:43:46 -0700

> consideration of my fellow IETFers. (Large downloads, live video...)

Ole,

At various times, such examples have been cited, effectively placing the 
responsibility for poor hotel network performance onto IETF guests at the hotel.

Is this merely a matter of being thorough in listing theoretical possibilities? 
  Or do we have any hard data to support the theory that IETF guests are, in 
fact, abusing the hotel networks?

Is it equally plausible that problem hotels merely do not have adequate 
resources for reasonable use by an Internet-centric block of guests?

What should the site qualifying teams do to assess a hotel's network 
capabilities?  I know what some of that is already being done but it's worth 
making sure that this issue is sufficiently covered by the site qualifying teams.

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: "John R. Levine" <[email-masked]>
Date: 3 Aug 2011 13:15:42 -0400

> resources for reasonable use by an Internet-centric block of guests?

Maybe your experience is different from mine, but that would always be my 
starting assumption unless I had specific reasons to think otherwise.

The most plausible specific reason would be that the IETF was able to feed 
the hotel network from the conference network, as they did at the Quebec 
Hilton.  That was certainly nice, but I suspect the IAOC would have a lot 
of trouble finding a sufficient set of venues and hotels that could all do 
that.

Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. http://jl.ly

---

From: Ole Jacobsen <[email-masked]>
Date: Wed, 3 Aug 2011 10:20:22 -0700 (PDT)

Dave,

It may be possible to do an estimate as follows:

* If/when the IETF network is directly connected to the hotel network 
  the NOC has good measurements on usage. Using these measurements we 
  can probably approximate what the average bandwith requirement per 
  IETF (hotel) user is. Call that Bx.

* A certain number of overflow hotels are visited prior to venue 
  selection. Bandwidth is discussed (if not measured). Let's call that 
  Bh for the total.

* If Bx * (N IETF guest) >> Bh we know there will be problems and we 
  could certainly make some statement about such limits in advance of 
  the meeting.

None of the above is very scientific (except perhaps for the NOC 
measurement part), and sucky networks suffer from more problems than 
just bandwith issues (recall all the port-blocking at a certain hotel 
in Maastricht). While it may be theoretically possible to gather all
this information for all (some) overflow hotels in advance of the 
meeting, at some point this becomes an excercise in diminishing 
returns. Let's not forget that the site visits are done a couple of
years (initially) prior to the actual meeting, even if there are later 
follow-up visits in some cases.

The NOC people can tell us much more about the (depressing) reality
here (if they're still on the list).

Ole

Ole J. Jacobsen

On Wed, 3 Aug 2011, Dave CROCKER wrote:

> Is this merely a matter of being thorough in listing theoretical
> possibilities?  Or do we have any hard data to support the theory that IETF
> guests are, in fact, abusing the hotel networks?
> 
> _______________________________________________
> 81attendees mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/81attendees
> 
>

---

From: Ben Campbell <[email-masked]>
Date: Wed, 3 Aug 2011 12:22:48 -0500

On Aug 3, 2011, at 12:15 PM, John R. Levine wrote:

>> Is it equally plausible that problem hotels merely do not have adequate resources for reasonable use by an Internet-centric block of guests?
> 
> Maybe your experience is different from mine, but that would always be my starting assumption unless I had specific reasons to think otherwise.
> 

More often than not, I find hotels have inadequate resources even when I'm _not_ attending an event that would attract an unusually high proportion of internet-centric guests. 

> The most plausible specific reason would be that the IETF was able to feed the hotel network from the conference network, as they did at the Quebec Hilton.  That was certainly nice, but I suspect the IAOC would have a lot of trouble finding a sufficient set of venues and hotels that could all do that.
> 
> Regards,

---

From: Ole Jacobsen <[email-masked]>
Date: Wed, 3 Aug 2011 11:45:26 -0700 (PDT)

On Wed, 3 Aug 2011, John C Klensin wrote:
>
> Ole,
> 
> Sure, but...
> 
> First of all, while I think it is sensible and prudent for 
> IETF-related guests to avoid, e.g., extensive movie downloads on 
> already-overloaded hotel networks, I don't think we should be 
> spending energy worrying about or casting blame on IETF participants 
> as network abusers.  If, e.g., some movie downloaders are going to 
> cause poor network performance, almost any guest with a laptop can 
> induce the same problem.

I wasn't suggesting otherwise. All I am saying is that we have some 
actual data on typical useage of IETF guests (whether reasonable or
not) and we could maybe use this data to judge if an overflow hotel
is likely or not going to be able to deal with such a load. Again,
nothing scientific and, yes, other "abusers" may exist -- not that
I am calling them that.

> 
> In my experience, many, if not most, hotel networks are woefully
> underprovisioned.  Sometimes that is just cutting costs,
> sometimes it it having installed network arrangements when one
> guest in 20 (or one in 50) used them, only to see that number
> rise to one in five (or, with an IETF group, 49 of 50).
> Sometimes it is just ignorance and/or stupidity.  But the effect
> is the same.  

I agree.

> 
> In addition, while the problems are often with the available
> bandwidth on the hotel's connections to the outside world,
> underprovisioned or congested hotel LANs aren't unusual either.
> I don't think any estimate or report of total available
> bandwidth is going to help.   

Well, it may be a starting point, that's all. And since the underlying 
thread here is "you should have known/told us that the Delta network
was going to suck" we can either try give some indication of this or
we can simply say "you're on your own".

> 
> That brings me back to three things as being potentially useful:
> 
> -- When a block of rooms is negotiated with a particular hotel,
> especially if "free Internet" is part of the negotiation, we
> should exert reasonable efforts to be sure that the hotel
> understands that we tend to all arrive with computers, than we
> tend to be heavy and demanding network users, and that poor or
> sucky Internet service is likely to result in a lot of irritated
> (and probably irritating) customers.

That is already being done.

> 
> -- Independent of how well or poorly a hotel responds to advice
> of that type, we should all understand --with reminder warnings
> as appropriate-- that random-hotel Internet service is often
> going to suck when a much larger-than-usual  percentage of the
> guests are intense Internet users, or Internet users at all.

I agree.

> 
> -- And all of us should be courteous enough to each other, and
> adult enough, to cut back on any bandwidth-hogging activities at
> the first sign of hotel network problems.

Absolutely. 

Ole

---

From: Andrew Sullivan <[email-masked]>
Date: Wed, 3 Aug 2011 15:16:08 -0400

> It might help even more if someone wanted to take on the task of writing a document called "Recommendations for Buffer and Congestion Management in Consumer-Grade Routers"

There's a whole working group with exactly that problem on its plate,
unless I misread the charter of homenet quite incorrectly.  

A

-- 
Andrew Sullivan
[email-masked]

---

From: Keith Moore <[email-masked]>
Date: Wed, 3 Aug 2011 15:34:58 -0400

> unless I misread the charter of homenet quite incorrectly.  

Though this might fit within its charter, homenet's problems are a lot bigger than that.

Keith

---

From: Jim Gettys <[email-masked]>
Date: Wed, 03 Aug 2011 15:45:06 -0400

On 08/03/2011 12:43 PM, Dave CROCKER wrote:
> At various times, such examples have been cited, effectively placing
> the responsibility for poor hotel network performance onto IETF guests
> at the hotel.
>
> possibilities?  Or do we have any hard data to support the theory that
> IETF guests are, in fact, abusing the hotel networks?
>
> Is it equally plausible that problem hotels merely do not have
> adequate resources for reasonable use by an Internet-centric block of
> guests?
>
> capabilities?  I know what some of that is already being done but it's
> worth making sure that this issue is sufficiently covered by the site
> qualifying teams.
>
> d/
>
Hotel networks tend to be under provisioned in the first place; that
just makes network problems (such as bufferbloat, which only is an issue
when saturation occurs) worse.  In my wanderings over the last year,
I've had very bad experiences at most other hotels at peak hours, even
those not occupied by computer geeks, but by typical "users".  I think
there is a lot of video streaming  that has changed the workload
considerably.  The retirement of Windows XP is increasing the fraction
of connections that will saturate any link even with a single TCP
connection; so an increasing fraction of many other activies such as
backup or large file copy will induce saturation.

We may be polite to each other, but try explaining this to the unwashed
masses, who want to watch tonight's episode of "House" or other show as
they do at home.  People aren't just reading email any more.

We can test for whether the hotel's network and back haul network has
problems using existing tools (e.g. netalyzr, the mlabs tools, or even
iperf/netperf + ping) in advance.  I suspect that most of the major
hotels the IETF uses are less likely than most to use broadband
connections for their backhaul as commonly found in smaller
hotels/motels recently connected to the Internet. Resolving problems in
advance may be more of a problem. On a couple occasions, I've tried to
track down clue full people to talk to them, but as an individual patron
of the hotel, had no traction in doing so, and the hotel networks seemed
to be remotely managed by others. And once checked out, I had no
incentive to continue trying.

Wireless at a conference hotel is probably completely infeasible in the
short term: the first wireless networks I've seen working well under
load are those set up at NANOG and IETF, where both commercial grade
AP's and really clue full configuration have managed to succeed.  That's
not even been carefully written down anywhere, though the principles of
it is covered very quickly in Anton Kapela's lightning talk at NANOG 38. 

As soon as your wireless bandwidth is below that of the backhaul, your
problems with bufferbloat will be in your host OS and the hotel's
wireless gear; while we have some test bits for controlling buffering
for Linux (the debloat-testing tree), I have no clue as to when you
might see anything Mac or Windows; in the short term, you really want
the bottleneck some place other than your wireless hop.  Which hop is
the bottleneck link easily varies depending on the ratio of the hotel's
back haul is available to you and how much wireless bandwidth you can get.

In the short term, I'd focus on seeing if the wired hotel network is
provisioned properly under load, it's ISP runs with suitable AQM. At
least then, if you plug into the wired ethernet, it should behave
properly and degrade reasonably under load rather than falling off a cliff.
                                                    - Jim

---

From: "Carlos M. Martinez" <[email-masked]>
Date: Wed, 03 Aug 2011 16:52:41 -0300

Maybe we need to solve the pesky quality-of-service-on-the-internet
thing once and for all :-)

Seriously, giving the fact that we are inventing this PCP thing for
remote-controlling port forwarding, maybe we could do something similar
for traffic class queues on WAN links, so the hotel (or other similarly
under provisioned network) can set up its traffic queues on the remote
side of the WAN link.

The ISP could assign a "pipe" of, say, 5 mbps to Delta Hotel, and a
(hopefully clueful) sys/net admin could set some traffic queues for
video/browsing/email/bit torrent (yes, there are people who do torrents
on hotels, i've seen it with Wireshark) within the 5 mbps pipe.

Can we set the BoF date ?? :)

cheers!

Carlos

-- 
Carlos M. Martinez
LACNIC I+D
PGP KeyID 0xD51507A2
Phone: +598-2604-2222 ext. 4419

---

From: Andrew Sullivan <[email-masked]>
Date: Wed, 3 Aug 2011 16:00:37 -0400

> Though this might fit within its charter, homenet's problems are a lot bigger than that.

Among others, interested participants can't get in the room :)

But yes, I agree.

-- 
Andrew Sullivan
[email-masked]

---

From: Jim Gettys <[email-masked]>
Date: Wed, 03 Aug 2011 16:07:49 -0400

> Can we set the BoF date ?? :)

Note that QOS, useful as it is, just moves what category of service
suffers.  The ultimate solution that our networks don't fall off of
cliffs under load is timely notification of congestion (e.g. packet loss
or ECN), and AQM to do so.  Lying to TCP's servo system gets us into a
world of hurt.  You thought there was TCP congestion avoidance?  Well,
there was once (and I hope will be again...).

Having said that, QOS is very, very useful at preventing out and out
failures and to enforce policies; right now, lots of what we're calling
"ants", those worker packets of DNS, DHCP, RA, etc, are getting lost in
the current mess, with the resulting complete failures of applications. 
So I don't see solving bufferbloat as a AQM versus QOS question at all:
both are useful (though if I have to only pick one, I'd pick working AQM)...
                    - Jim

---

From: John C Klensin <[email-masked]>
Date: Wed, 03 Aug 2011 16:09:27 -0400

> Among others, interested participants can't get in the room :)

I assumed that was just testing a congestion-reduction mechanism
based on a restricted entry model.  New people could get in only
when older ones left.   Not quite token-passing, but close.

Pretty successful experiment, too.  

   john

---

From: Randy Bush <[email-masked]>
Date: Thu, 04 Aug 2011 09:46:22 +0900

> suffers.

bingo!

qos is about whose/which packets to drop.  i am not paid to drop
packets, i am paid to deliver them.

randy

---

From: james woodyatt <[email-masked]>
Date: Wed, 03 Aug 2011 17:53:40 -0700

On Aug 3, 2011, at 17:46 , Randy Bush wrote:
> 
> i am not paid to drop packets, i am paid to deliver them.

Air Canada seems to have taken a similar view with my baggage on the flight home from Québec.  Apparently, I'm to be thankful my baggage was delivered to my home at all.  It's not like a tardy delivery could cost me any money or anything...

At least all the fragments arrived in order.

--
james woodyatt <[email-masked]>
member of technical staff, core os networking

---

From: Dave CROCKER <[email-masked]>
Date: Thu, 04 Aug 2011 06:24:39 -0700

> packets, i am paid to deliver them.

Excellent.

Let's ignore capacity limits and congestion completely.  The knee of the curve 
is a waste of our efforts.

Think of how much simpler life would be if we paid no attention to downsides.

Armies would never give thought to retreat.  Armies are there to advance, not 
retreat.

Carry this further and they would not have to worry about soldiers getting 
injured.  Armies are for hurting the other guys, not getting hurt.

The medical profession would never have to worry about patients dieing.

Investors would never have to worry about the market going down.

Wouldn't it be great if a clever sound bit really did eliminate the need to 
worry about unpleasant realities?

d/

-- 

   Dave Crocker

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 4 Aug 2011 15:30:38 +0000

Dave,

I didn't want to jump on this but I agree with your point.

QoS in providers today is used to prioritize packets for people paying a lot more than others for such things as VPN services.  The rest get usually extremely good "best effort" service as has been the case for most providers who have survived over the last decade or two (survived in some form or another, generous buyouts in the mid to late 1990s not withstanding).

There can be improvement.  Good to hear that there is interest again.  The bufferbloat discussion and work and the mention of transport area activity is a symptom of interest and activity regarding doing things better in the host, CPE, and access/edge devices.

Curtis

> -----Original Message-----
> From: [email-masked] [mailto:81attendees-
> [email-masked]] On Behalf Of Dave CROCKER
> Sent: Thursday, August 04, 2011 6:25 AM
> To: [email-masked]
> Subject: Re: [81attendees] sucky Delta hotel network (and bufferbloat)
> 
> 
>

---

From: "Carlos M. Martinez" <[email-masked]>
Date: Thu, 04 Aug 2011 12:50:23 -0300

I fail to see a better argument for implementing QoS than what happened
to the Delta Hotel in QC. I agree that QoS has been somewhat bastardized
by network operators and used only for extracting a premium for things
that should work anyway.

Arguing that every network should always be overprovisioned encourages
wasteful behaviour on all parts and puts many players in an impossible
position. It might be not so impossible in the US/Canada (although even
there Delta Hotels showed us that it can also happen), but it's
definitely the case in other regions where BW costs are still quite high.

Pretending that congestion does not exist will not make it disappear,
and arguing that the cure for congestion is overprovisioning creates a
situation where networks will always experience periods of poor
perceived quality, as congestion and provisioning happen on different
time scales.

Granted there can be "good" players who always invest ahead of the
curve, but, let's be honest here, they are the minority.

Dropping packets is a bad thing. However, dropping packets blindly (as
in pretending congestion does not exist) is much worse. The network
should avoid dropping packets at all costs, but if it has to drop, then
it should do it smartly and avoid making the situation worse.

I do believe there is a management problem in the same sense as there is
with CGNs and port forwarding, which has lead to the creation of PCP. It
is basically a remote management protocol allowing customer-provider
interaction (ATM UNI anyone ??? :-)))) ). Well, maybe we could do
something for QoS. Phoning your ISP each time you need to forward a port
is every bit as impracticable as phoning the ISP to create a QoS queue.

Warm regards,

Carlos

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 4 Aug 2011 16:02:04 +0000

Even in the most overprovisioned network, congestion occurs with each natural disaster.  The last extremely widespread and dire reminder in the US happened in 2005 as a result of Hurricane Katrina.  However smaller incidents where capacity is temporarily lost (for hours or days) occur almost continuously.

There is an issue with voluntarily marking traffic as real time and cheating that I'd rather not go into on *this* list.

[btw - though interesting, we're drifting off on a tangent.]

Curtis

> [email-masked]] On Behalf Of Carlos M. Martinez
> Sent: Thursday, August 04, 2011 8:50 AM
> To: [email-masked]
> to the Delta Hotel in QC. I agree that QoS has been somewhat
> bastardized by network operators and used only for extracting a premium
> for things that should work anyway.
> 
> I do believe there is a management problem in the same sense as there
> is with CGNs and port forwarding, which has lead to the creation of
> PCP. It is basically a remote management protocol allowing customer-
> provider interaction (ATM UNI anyone ??? :-)))) ). Well, maybe we could
> do something for QoS. Phoning your ISP each time you need to forward a
> port is every bit as impracticable as phoning the ISP to create a QoS
> queue.
> 
> Warm regards,
> 
> Carlos
> 
> > QoS in providers today is used to prioritize packets for people
> paying a lot more than others for such things as VPN services.  The
> rest get usually extremely good "best effort" service as has been the
> case for most providers who have survived over the last decade or two
> (survived in some form or another, generous buyouts in the mid to late
> 1990s not withstanding).
> >
> > There can be improvement.  Good to hear that there is interest again.
> The bufferbloat discussion and work and the mention of transport area
> activity is a symptom of interest and activity regarding doing things
> better in the host, CPE, and access/edge devices.
> >
> > Curtis
> >
> >> Let's ignore capacity limits and congestion completely.  The knee of
> >> the curve is a waste of our efforts.
> >>
> >> Armies would never give thought to retreat.  Armies are there to
> >> advance, not retreat.
> >>
> >> Carry this further and they would not have to worry about soldiers
> >> getting injured.  Armies are for hurting the other guys, not getting
> >> hurt.
> >>
> >> Wouldn't it be great if a clever sound bit really did eliminate the
> >> need to worry about unpleasant realities?
> >>
> >> d/
> >>
> >> --
> >>
> >>    Dave Crocker

---

From: Jim Gettys <[email-masked]>
Date: Thu, 04 Aug 2011 12:05:34 -0400

> perceived quality, as congestion and provisioning happen on different
I don't think that was what Randy was saying though....

I believe he was agreeing with my comment that while QOS is useful, it
won't solve the bufferbloat problem.  QOS just moves who suffers most,
and having buffers perennially full makes them malfunction in general. 
It's why if I have to choose between QOS and working AQM, I'll pick
working AQM all the time.

Modern operating systems hardware and applications will congest any edge
link; congestion in the edge (and a hotel is clearly the edge) is going
to be with us forever.  Handling congestion properly in this environment
is now absolutely essential.

Basically, over CONUS delays, a single TCP connection on WinXP won't do
over 6 MBps (no TCP window scaling).
Later versions of Windows rate limit to below that of a 100Mbps ethernet
switch (saturating most wireless and broadband links). Everything else
will saturate to gigabit or more without blinking twice.

Saturation of the edge is the *normal* situation; we just didn't notice
when BitTorrent  saturated links using many TCP connections on XP years
ago, so we lost years recognising what was happening.

To solve congestion requires *timely* notification of the end points to
slow down rather than tail drop off grossly bloated buffers, and right
now, we only have packet loss to do that signalling.  It's often/usually
so bad that TCP congestion avoidance is no longer functioning. Getting
ECN enabled would give us congestion signalling without necessarily
having packet loss.

>>> advance, not
>>> getting

---

From: "Carlos M. Martinez" <[email-masked]>
Date: Thu, 04 Aug 2011 13:33:21 -0300

For the record, I wasn't trying to argue with Randy :-)))

I just wanted to state my opinion on what I believe is a real problem we
might have to work on.

On 8/4/11 1:05 PM, Jim Gettys wrote:
> and having buffers perennially full makes them malfunction in
> general.  It's why if I have to choose between QOS and working AQM,
> I'll pick working AQM all the time.
>
> Modern operating systems hardware and applications will congest any
> edge link; congestion in the edge (and a hotel is clearly the edge) is
> going to be with us forever.  Handling congestion properly in this
> environment is now absolutely essential.
>
> Basically, over CONUS delays, a single TCP connection on WinXP won't
> do over 6 MBps (no TCP window scaling).
> Later versions of Windows rate limit to below that of a 100Mbps
> ethernet switch (saturating most wireless and broadband links).
> Everything else will saturate to gigabit or more without blinking twice.
>
> Saturation of the edge is the *normal* situation; we just didn't
> notice when BitTorrent  saturated links using many TCP connections on
> XP years ago, so we lost years recognising what was happening.
>
> To solve congestion requires *timely* notification of the end points
> to slow down rather than tail drop off grossly bloated buffers, and
> right now, we only have packet loss to do that signalling.  It's
> often/usually so bad that TCP congestion avoidance is no longer
> functioning. Getting ECN enabled would give us congestion signalling
> without necessarily having packet loss.
>
>

---

From: Roland Bless <[email-masked]>
Date: Thu, 04 Aug 2011 22:35:27 +0200

Hi Carlos,

> to the Delta Hotel in QC. I agree that QoS has been somewhat bastardized

QoS is there for differentiation of packet/flow treatment,
i.e., managed unfairness, while congestion management tries
to provide fairness between users/flows in a best-effort class.
QoS still doesn't solve the problem of trying to share the capacity
between users in the same QoS class.

> position. It might be not so impossible in the US/Canada (although even

Yes, indeed, DDoS flooding, misbehaving flows and unbalanced traffic
matrices may still cause problems for overprovisioned networks.

> it should do it smartly and avoid making the situation worse.

Yes, but this is also true for more clever congestion management schemes.

> is every bit as impracticable as phoning the ISP to create a QoS queue.

There are signaling solutions out there for this, e.g., NSIS
http://tools.ietf.org/wg/nsis/ provided solutions for both
Port management for NAT/Firewalls as well as for QoS signaling.
Giving a tool into the users' hand so that they can actually
decide what is more important and should be treated with better
QoS would be a good thing also from the perspective of net neutrality.
But the complexity of providing everything (control plane + DiffServ
etc.) for QoS support is obviously still a too high effort.

Regards,
 Roland

---

From: "Carlos M. Martinez" <[email-masked]>
Date: Thu, 04 Aug 2011 17:40:54 -0300

Thanks Roland. My main point is that I do believe that we have an
interesting problem at hand, one that goes beyond bashing Delta Hotels
(however fun it can be :-) ).

Funny fact: some of the most interesting IETF-related discussions that
I've had have been on the lists *after* the event itself :-)

cheers!

Carlos