Subject: [Ieprep] Re: some 9/11 stats

From: Ken Carlberg <[email-masked]>
Date: Wed, 09 Jan 2002 11:37:36 +0000

> >- we don't know where along the path the loss occured
> >- the graphs don't show an exact breakdown of how many packets were 
> >lost over what period of time
> >- we don't know the standard deviation of the loss. did some small
> >set of nodes unfairly skewer the overall average?, etc.
> 
>      what exactly can be inferred from the traffic data if it gives the
> above measures (path along the loss,pkt breakdown&SD of the loss) ?

it seems to be widely accepted that congestion in the Internet tends to 
be more prevelant at access points between stub domains and their transit 
providers, and less so towards the core (ie, inter-ISP links, particularly 
amongst Tier-1 providers).  so if the nodes initiating the pings are in
stubs, there is a possibility that the ICMP ping is subject to a higher
probably of experiencing a congestion point at the start of the path
(stub-to-transit) than along the rest of the ISP-to-ISP path.  to me,
it would be nice to distinguish loss from ISP-to-ISP, versus stub-to-ISP.

for the second point, the site indicates that stats are accumulated
every 15 minutes.   its hard to gauge from the graph how many periods
reported significantly higher than normal loss/response.  also, as a
general observation, we don't know how many ICMP packets are sent
within each period (is it 1, 3, 10, 100?).  too many can make the test
unfriendly and can trigger congestion for the 'type' of packet (some
routers have wfq turned on with a relatively small queue size for
ICMP-class packets). 

for the third point we don't know the distribution of loss experienced
amongst the 100+ routers located in North America used for testing.
was it +10% of the routers experiencing 100% loss, and then the rest 
just a slightly higher than normal amount of loss?  or was the loss more
uniformly widespread?

-ken

_______________________________________________
Ieprep mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ieprep

---

From: Sean Donelan <[email-masked]>
Date: Thu, 10 Jan 2002 06:45:50 -0500 (EST)

> it would be nice to distinguish loss from ISP-to-ISP, versus stub-to-ISP.

Congestion in various providers' cores is different than the congestion
on inter-ISP links.  Inter-ISP links are usually an order of magnitude
smaller than their core circuits.  Worse, some providers have an internal
policy of keeping congestion on Inter-ISP links worse than internal
circuits.  These tend to be very high-stress points, and probably the
least capable of successfully implementing any type of priority queuing
without adversly affecting a lot of other stuff.  ISPs won't even
implement security filters on these links because it kills their
equipment.

But, under normal conditions, this doesn't cause problems because most
high traffic, high visibility users are directly connected to multiple
providers.  Their bottlenecks are distributed among multiple access
links with different providers instead of the inter-ISP links.  Therefore
most users experience congestion on the access links instead of the
inter-ISP links.

How does this relate to "emergency" traffic?  I don't know because I
don't know the traffic patterns of emergency users.  Nor do I know a
good source for this data.  Its important because fixing the wrong
bottleneck won't help people stuck in traffic somewhere else.

Which brings back my first, and continuing point.  Where are emergency
users having problems?  After we find out where the problems are, we
can start working on solutions.

We may not be able to predict where the next emergency will happen, but
there are substitutes.  The next time NASA announces a live feed from
Mars, why not put some Voip equipment on either end of the same access
links and measure what happens.  Or give the IT manager at CNN one
of those new Cisco IP phones, and call it during the next breaking
news event.  Or do something else to figure out what the problems
are before fixing them.

> just a slightly higher than normal amount of loss?  or was the loss more

Or was all the loss due to the first router connected to the test
machine because everyone was trying to access the Internet Traffic
Report web site to see how the Internet was performing, and congesting
the same access link ITR uses to take measurements.  Does the graph
reflect the performance of the Internet, or the ITR access link?

See also data collected by CAIDA, RIPE, Inverse, Keynote, Matrix, and
so  forth.  They each collected data from a different viewpoint, and its
not always clear what it all means.  For example, low-level data from ITR
shows increasing packet loss and latency beginning several hours after the
attack, but Keynote web site measurements show the web response was
improving several hours after the attack.

If you look at any 30-day period of Internet Traffic Report data you'll
see periods of high (>25%) packet loss, long (>1000ms) round-trip-times.

What is normal for the Internet?

---

From: senthil ayyasamy <[email-masked]>
Date: Thu, 10 Jan 2002 09:40:52 -0800 (PST)

Hi all, 

some more queries
> can start working on solutions. 
        As mentioned by scott bradner in the minutes that Internet worked normal during Sept 11 and it was only the telephone lines that were loosy....Then how to identify those problems since we have to rely on PSTN networks 

> news event. Or do something else to figure out what 

cant active measurements be taken by injecting live packets(ping) to CNN at such times and verified  ??
> Does the graph 

The method that ITR used for measuring is also necessary ..

> What is normal for the Internet? 

i am very interested in knowing the answer for this question..

-senthil.

SENTHIL KUMAR
MS(COMPUTER NETWORKING)
UMKC,
MO-64112,USA

---------------------------------
Do You Yahoo!?
Send FREE video emails in Yahoo! Mail.

---

From: Steve Blumenthal <[email-masked]>
Date: Thu, 10 Jan 2002 13:36:25 -0500

Sean,

>implement security filters on these links because it kills their

While, I agree with you that inter-provider peering links are an order of 
magnitude lower than our backbone links, I disagree that we go out of our 
way to keep these links congested.  We track traffic on all of our peering 
links on a continuous basis, proactively model traffic growth on these 
links in the same way as any internal link, and work with our peers to 
upgrade Inter-ISP links in advance of congestion.  Our goal is to have 
adequate capacity on these links with some headroom for growth or traffic 
spurts.  These circuits don't have to be as fast as our backbone links 
because with each peer, we have many of these peering inter-connects spread 
across the country, and we are splitting the traffic on our backbone among 
many peers.  Genuity operates a 10Gb/s backbone (which is not too heavily 
loaded today) and, in aggregate today, we have over 25Gb/s of exit capacity 
to our peers.

As the speed of these Inter-ISP links have increased, we have encountered 
delays in getting new links provisioned by the local telco providers and 
this has led to some links becoming temporarily congested.  To get around 
this, the tier-1 ISPs have gotten together to create IPOPs in carrier 
neutral facilities.  The plan would be for the us to bring dark fiber to 
the IPOP location and then interconnect to the other large ISPs via optical 
cross connects in the facility.  The plan calls for a number of these 
facilities around the US with the first ones going into service now.  In 
the long run, I do not think that the Inter-ISP peering circuits will 
continue to be a bottleneck.

We have had some preliminary discussions among the Tier 1 ISPs about 
inter-provider SLAs, prioritization of traffic, and inter-provider 
security.  We, at Genuity, are looking at the secure BGP work coming out of 
BBN.  I am hoping that this IETF effort may accelerate some of these 
inter-ISP discussions along with some other activities going on to create 
an ISP-ISAC.

Steve

---

From: Sean Donelan <[email-masked]>
Date: Sat, 12 Jan 2002 08:54:48 -0500 (EST)

> to CNN at such times and verified  ??

That will tell you how well "ping" performs but not other protocols.
As Fred Baker reported at the IETF BOF, some Internet protocols are
extremely resiliant to packet loss.  They continued to function, for
some value of function, with packet loss in excess of 25%.

Handling SYN packets was an issue in several failures on September 11.
Some firewalls, load balancers, etc collapsed under the flood of SYN
packets, which ended up blocking other users. ICMP was processed by
a different stack, so you could "ping" the site, but not use it.
Although this is an old problem, it is an area which could use some
more work.

This is a classic example of congestive collapse.  In an emergency,
do you impose some type of admission control on SYN packets?  Is this
different than prioritizing SYN packets?  Is this different from good
network management on any other day?

---

From: Fred Baker <[email-masked]>
Date: Sat, 12 Jan 2002 11:05:33 -0800

At 05:54 AM 1/12/2002, Sean Donelan wrote:
>In an emergency, do you impose some type of admission control on SYN 
>packets?  Is this different than prioritizing SYN packets?  Is this 
>different from good network management on any other day?

I'll argue that you might have to, and since there is not much information 
in the SYN (even an emergency device might well get its IP address from 
DHCP, or in an IPv6 world by autoconfiguration), so you can't necessarily 
even make assumptions about that.

I tend to think that the simplest approach under extreme load is to make 
SYN/SYN-ACK processing be lowest priority - place SYN packets into a queue 
which uses only bandwidth which has no other user. Net effect you give fair 
service to those that already got service, and accept new work on a space 
available basis.

And frankly, CNN is a customer of Akamai; the best way for them to unclog 
load to CNN is some combination of putting up effective mirror sites and 
offloading to their offload provider. If I worked at CNN, I would be looked 
*very* hard at how to do a better job of those things.

But CNN is not an "emergency services" scenario, IMHO. Nobody who is 
worried about getting an ambulance to a crash site is checking CNN. They 
are going to the get-me-an-ambulance service, and presumably anyone who is 
going to that service needs an ambulance. Prioritization of SYN requests 
among those-who-get-service and those-who-don't remains to be shown to be a 
requirement.

---

From: Sean Donelan <[email-masked]>
Date: Sat, 12 Jan 2002 16:21:02 -0500 (EST)

> service to those that already got service, and accept new work on a space

I used CAR to rate-limit the SYN packets to a level my servers could
manage.  Essentially treating the "flash-crowd" as a DDOS.  It dropped
the SYN packets over a certain rate, essentially limiting SYN's to a
few percent of the total bandwidth. Picking a rate was trial-and-error.
It stopped collapse, and let me successfully serve more people.  But there
was no cutting to the front of the line.

Is fair service good enough for emergency users as long as they receive
service at some point?  How long of a delay is tolerable before they want
to butt in?

> are going to the get-me-an-ambulance service, and presumably anyone who
> is going to that service needs an ambulance. Prioritization of SYN
> requests among those-who-get-service and those-who-don't remains to be
> shown to be a requirement.

The National Hurricane Center experienced it when a hurricane
hit Florida a few years ago.  At the time, it was a minor inconvience
because NHC emergency users still had non-Internet communications to
access the information.

Essentially any public Internet TCP service is vulnerable to the same
thing.  If emergency users move services onto the Internet, those
services may experience the same crowd effect.

But there are also other solutions, besides a multi-level precedence
pre-emption scheme, that already work on today's Internet.  The simplest
is to give public and emergency users different addresses.  This is the
Publisher's Clearing House approach.  If you have a magazine order, you
use the green envelope with one address for priority processing.  If
you just want to enter the sweepstakes, you use the brown envelope
with a different address.

---

From: Sean Donelan <[email-masked]>
Date: Sun, 13 Jan 2002 14:16:57 -0500 (EST)

> among those-who-get-service and those-who-don't remains to be shown to be a

The National Fire Protection Association has changed its Fire Alarm
Code to include the use of the Internet for fire alarm reporting.
Although this isn't a "get me an ambulance" service, its close.  Because
companies with an existing Internet connection can save up to $50 every
month for each location, I expect this will become a popular feature.

 "October 31, 2001 - Radionics applauds the recent decision of the
 National Fire Protection Association (NFPA) to allow Data Networks,
 both the Internet and private Intranets, to be used for the reporting
 of Fire Alarm and other Life Safety-related signals to central monitoring
 facilities.  The approval of Formal Interpretation 72-99-1 and Tentative
 Interim Amendment 99-3 opens the door for manufacturer's of alarm systems
 to use 21st century communications technology for critical fire alarm
 signaling applications."

Currently fire alarm equipment uses RJ31X telephone lines, which allow the
alarm to "seize" the telephone line and send the alarm.  This effectively
gives the fire alarm the highest priority use of the phone line, even
interrupting a call in progress.  A fire alarm will even preempt a 1-710
call on the line.

By default some VoIP products, such as the Cisco IP Phone, prioritize
voice packets over all other traffic.  I expect this would include the
new NFPA fire alarm signal packets, unless the VoIP installer remembers
to change it.  How will VoIP traffic, fire alarm traffic and "normal"
traffic interact on the net?  Its an interesting problem.  Through the
magic of multiplexing the fire alarm packets may get through even if
VoIP calls have priority.

Do we need to change any Internet Protocols for the NFPA fire alarm
packets?  I don't know.  They are going ahead with IP as it currently
exists, and if there are problems I hope they'll let us know.  I think
that is a good way to proceed.  Solving an identified problem is much
easier than solving all potential problems.

---

From: Ken Carlberg <[email-masked]>
Date: Sun, 13 Jan 2002 23:13:29 +0000

> service at some point?  How long of a delay is tolerable before they want

these are nice questions, but impossible to respond to as a single
answer.  the answer could depend on the type of application, the 
qualities of the app that are offered, the expectations of the user 
of the app, the laws/regulations that are in effect for a given 
region (groan), the contracts (in the form of SLAs) that have been 
established, etc.  

> is to give public and emergency users different addresses.

the above is possible, but not very desireable because of its lack of
portability.  also quite undesireable if this were to be propagated
via BGP with an ensuing increase in routing tables.

-ken

---

From: Fred Baker <[email-masked]>
Date: Mon, 14 Jan 2002 17:06:53 +0100

At 08:16 PM 1/13/2002, Sean Donelan wrote:

>Although this isn't a "get me an ambulance" service, its close.

I'm not certain whether you think we agree or we disagree.

I'm all for them using the Internet to call the fire department. My point 
is that when they call the fire department using the Internet, the packets 
won't go to CNN - they will go to the fire department. And presumably, 
anyone who is sending packets to the fire department will be wanting 
someone to come do something heroic about a fire. The congestion we have 
been discussing is not on the backbone, but on the access link to the site 
being sent packets.

Frankly, during an emergency, if the fire department goes through a 
congestive collapse scenario as CNN etc may have on 9/11, it's not because 
there were a zillion people out browsing CNN to find out what was going on. 
It was because lots of people were calling the fire department.

>easier than solving all potential problems.

yes... I could suggest ECN (RFC 3168) deployment, for various reasons not 
the least of which is the issue we discussed regarding CNN, but (a) I think 
that's a good idea anyway, and (b) I suspect straight IP deployment will 
take them 80% of the way there.

---

From: "Stuart (Stu) Goldman" <[email-masked]>
Date: Thu, 31 Jan 2002 14:33:36 -0700

Folks,

I believe that Publisher's Clearing house does this on purpose to make people
believe they have a better chance if they do buy. ( I believe that the law is
that everybody must have the same chance. Also, the chance of hitting it big is
very, very small.) The point here is that perception is often more important than
reality.

Sean Donelan wrote:

...This is the