Subject: RE: FW: [bmwg] Proposal on Protection Benchmarking

From: "Jay Karthik \(jakarthi\)" <[email-masked]>
Date: Wed, 14 Jun 2006 11:23:04 -0400

Curtis,

We are adding text to suggest SRLG to be out of scope for this draft (at
least the current version). Also adding SRLG to the terminology draft.

This sure opens the door for a new draft.

Cheers,
Jay 

-----Original Message-----
From: Curtis Villamizar [mailto:[email-masked]] 
Sent: Tuesday, June 13, 2006 1:28 PM
To: Jay Karthik (jakarthi)
Cc: [email-masked]; [email-masked]
Subject: Re: FW: [bmwg] Proposal on Protection Benchmarking 

In message
<[email-masked]>
"Jay Karthik \(jakarthi\)" writes:
>  
>  
> Could you please shed more light on "unexpected correlated failures" ?
> If this is significant work, we can have a separate draft for this, 
> that falls under the same proposed work item (Protection
Benchmarking).
> Thanks,
> The author team.

Examples:

  Put fiber on both sides of the Amtrak right of way and have a train
  wreck where fiber on both sides of the track are affected.  (This
  happenned in NJ on the main fiber run from DC and Philly to NYC.)

  Put fiber on both sides of a bidge that falls down in a hurricane.
  (Raritan River bridge, NJ).

  Have multiple fiber cuts due to fiber crossing an earthquake fault
  that gives way.  (This happend in San Diego serving 3 of 4 major
  fiber paths to the rest of the world.  International traffic that
  went by way of Hawaii to San Diego was affected in a addition to
  local traffic.)

  Have some goon dig up the wrong gas line and start breaking a bunch
  of annoying heavy cables that weren't supposed to be their.  (This
  happenned somewhere in the midwest along a gasline right of way in
  unused natural gas pipes).

  Have a rat explode in the power transfer switch at the POP.
  (Happenned in San Jose).

  Have multiple blocks which include POPs of multiple providers shut
  down power including backup power due to a major gas leak.  (This
  happenned in LA).

  Have a flood take out power in the POP and eventually flood the POP
  with 4 feet of water.  (The Mississippi river flooded St. Louis and
  many other cities in the midwest in 1993).

  Power outage while one generator is being repaired and the other one
  fails to start.  (NYC, affected air traffic control on entire East
  Coast for many hours due to outage in FAA private network).

  Etc, etc, etc.

You can read the NANOG for many more example.  Some perhaps even more
ammusing.  This sort of things happens on a fairly regular basis and
increased hurricane activity may make it even more common.

These were all "unexpected" from the standpoint of SRLG configuration or
entirely unexpected.  They were also very well correlated.

No amount of FRR is going to handle these types of outages.  They
require fallback to resignaling the tunnels for MPLS or fallback to IP.
Providers with "BGP free core" do not have fallback to IP as an option.
BGP free core may become quite popular soon making MPLS resignal time
even more important.

Curtis

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: Curtis Villamizar <[email-masked]>
Date: Wed, 14 Jun 2006 12:04:28 -0400

In message <[email-masked]>
> This sure opens the door for a new draft.

That is to say you are ignoring a big part of the real world problem.
At least for now.

Checking that SRLGs function is one issue.  If SRLG are configured,
then the backup path should be selcted to avoid an SRLG that is in
common with the link or node being protected.  That is a functionality
issue, not a benchmark issue.

The second part is regardless of whether SRLG is used or not, a
correlated failure can occur and does occur fairly regularly.  Some
routers recover much more quickly in this case.  The range can be a
few seconds to tens of seconds in real topologies.  Testing should
include restoration time with and without fallback to IP available.
The BGP free core idea is becoming a little more popular with
providers and in that case there is no fallback to IP option.  If you
declare this out of scope, please send me the wording so that there is
a clearly defined problem definition which a later document can
address.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Wed, 14 Jun 2006 13:04:01 -0400

Replying to myself...  I forgot to mention something.

In message <[email-masked]>
> a clearly defined problem definition which a later document can

Some providers are using offline path computation which by their own
admission takes 4-15 minutes to compute a set of optimal paths plus
time to install them.  They mention fallback to dynamicaly routed LSPs
or IP.  See NANOG presentation archives for some examples (Global
Crossing and C&W presentations).  Some providers still do this.

In many of the failures listed below the PSTN service was affected and
took hours, and in a few cases days to be restored.  This is due to
the phone company reliance on transport grooming that was largely a
computer assisted manual process rather than fully automated.  The
most notable might be the San Diego outage.  In that case PSTN
capacity was so degraded that emergency phone service was affected.
The popular press reported that police and fire were recruiting
university staff to communicate with their colleges across the fault
for such things as reporting on the status (safe, hospitalized,
deceased, etc) of families with members who were on the other side of
the fault for the day.

In the San Diego "test" the PSTN did not benchmark very well.  The
purpose of a lab benchmark defined by BMWG would be to determine what
outage times would be with modern equipment and routing techniques.
It would also be useful to determine the level of congestion that
would occur with the resulting traffic layout after a fault.

The best approach for carriers seems to be a two or three level
approach.  FRR can handle isolated faults or correlated faults where
the correlation is anticipated (covered by SRLG).  Dynamic rerouting
can handle those correlated faults where the correlation was
unexpected.  Offline computation can be used to provide optimal
traffic layout.  These three approaches are listed from the quickest
repair to the slowest and from the least optimal traffic layout after
a fault to the most optimal.

With better information on the performance of rerouting fallback
providers who consider offline path computation could have a better
basis on which to decide what approach to use in their network.  It
may also influence (and has in the past) the decision whether to go
with a BGP free core.  For example one provider might decide to
compute primary paths offline and rely on FRR and dynamic routing and
then compute optimal paths for the remaining topology.  The suboptimal
traffic layout would exist for the duration of the offline computation
and configured path install.  Another provider might decide to rely
entirely on FRR and dynamic routing if the traffic layouts resulting
from dynamic routing tend to be very close to optimal.  Knowing how
each of these perform would be useful.  I'm not suggesting we
benchmark offline computation tools, only the dynamic routing that
occurs when "unexpected correlated failures" occur.

Curtis