Subject: RE: FW: [bmwg] Proposal on Protection Benchmarking

From: "Jay Karthik \(jakarthi\)" <[email-masked]>
Date: Wed, 14 Jun 2006 15:51:09 -0400

Curtis,

We completely agree with you on the need for benchmarking the dynamic
routing that occurs following an "unexpected correlated failure". Since
the current focus is on the isolated failures and the various scenarios
for which FRR benchmarking needs to be performed, we like to capture the
importance of "unexpected correlated failures" in our draft that
highlights the need for a separate draft. 

Here is the text we are considering to add (significant portion from
your email with minor edits). Please let us know if this defines the
problem you mentioned and further paves way for a new draft. Better yet,
could you suggest text if any key points are missing.

<Start of additional text>
Regardless of whether SRLG is used or not, a correlated failure can
occur and does occur fairly regularly. Some routers recover much more
quickly in this case. The range can be a few seconds to tens of seconds
depending on the topology.  Benchmarking should include restoration time
with and without fallback to IP available, given that the BGP free core
idea is becoming a little more popular with providers and in this case
there is no fallback to IP option. 

However this draft focuses on FRR failover benchmarking with MPLS TE and
does not propose methodology for correlated failures. 
<End of additional text>

Thanks,
Author Team

-----Original Message-----
From: Curtis Villamizar [mailto:[email-masked]] 
Sent: Wednesday, June 14, 2006 1:04 PM
To: [email-masked]
Cc: Jay Karthik (jakarthi); [email-masked]
Subject: Re: FW: [bmwg] Proposal on Protection Benchmarking 

Replying to myself...  I forgot to mention something.

In message <[email-masked]>
Curtis Villamizar writes:
>  
>  
> In message 
> <[email-masked]>
> "Jay Karthik \(jakarthi\)" writes:
> >  
> > Curtis,
> >  
> > We are adding text to suggest SRLG to be out of scope for this draft

> > (at least the current version). Also adding SRLG to the terminology
draft.
> >  
> > This sure opens the door for a new draft.
> >  
> > Cheers,
> > Jay
>  
>  
> That is to say you are ignoring a big part of the real world problem.
> At least for now.
>  
> Checking that SRLGs function is one issue.  If SRLG are configured, 
> then the backup path should be selcted to avoid an SRLG that is in 
> common with the link or node being protected.  That is a functionality

> issue, not a benchmark issue.
>  
> The second part is regardless of whether SRLG is used or not, a 
> correlated failure can occur and does occur fairly regularly.  Some 
> routers recover much more quickly in this case.  The range can be a 
> few seconds to tens of seconds in real topologies.  Testing should 
> include restoration time with and without fallback to IP available.
> The BGP free core idea is becoming a little more popular with 
> providers and in that case there is no fallback to IP option.  If you 
> declare this out of scope, please send me the wording so that there is

> a clearly defined problem definition which a later document can 
> address.
>  
> Curtis

Some providers are using offline path computation which by their own
admission takes 4-15 minutes to compute a set of optimal paths plus time
to install them.  They mention fallback to dynamicaly routed LSPs or IP.
See NANOG presentation archives for some examples (Global Crossing and
C&W presentations).  Some providers still do this.

In many of the failures listed below the PSTN service was affected and
took hours, and in a few cases days to be restored.  This is due to the
phone company reliance on transport grooming that was largely a computer
assisted manual process rather than fully automated.  The most notable
might be the San Diego outage.  In that case PSTN capacity was so
degraded that emergency phone service was affected.
The popular press reported that police and fire were recruiting
university staff to communicate with their colleges across the fault for
such things as reporting on the status (safe, hospitalized, deceased,
etc) of families with members who were on the other side of the fault
for the day.

In the San Diego "test" the PSTN did not benchmark very well.  The
purpose of a lab benchmark defined by BMWG would be to determine what
outage times would be with modern equipment and routing techniques.
It would also be useful to determine the level of congestion that would
occur with the resulting traffic layout after a fault.

The best approach for carriers seems to be a two or three level
approach.  FRR can handle isolated faults or correlated faults where the
correlation is anticipated (covered by SRLG).  Dynamic rerouting can
handle those correlated faults where the correlation was unexpected.
Offline computation can be used to provide optimal traffic layout.
These three approaches are listed from the quickest repair to the
slowest and from the least optimal traffic layout after a fault to the
most optimal.

With better information on the performance of rerouting fallback
providers who consider offline path computation could have a better
basis on which to decide what approach to use in their network.  It may
also influence (and has in the past) the decision whether to go with a
BGP free core.  For example one provider might decide to compute primary
paths offline and rely on FRR and dynamic routing and then compute
optimal paths for the remaining topology.  The suboptimal traffic layout
would exist for the duration of the offline computation and configured
path install.  Another provider might decide to rely entirely on FRR and
dynamic routing if the traffic layouts resulting from dynamic routing
tend to be very close to optimal.  Knowing how each of these perform
would be useful.  I'm not suggesting we benchmark offline computation
tools, only the dynamic routing that occurs when "unexpected correlated
failures" occur.

Curtis

> > -----Original Message-----
> > From: Curtis Villamizar [mailto:[email-masked]]
> > Sent: Tuesday, June 13, 2006 1:28 PM
> > To: Jay Karthik (jakarthi)
> > Cc: [email-masked]; [email-masked]
> > Subject: Re: FW: [bmwg] Proposal on Protection Benchmarking
> >  
> >  
> > In message
> > <[email-masked]
> > >
> > "Jay Karthik \(jakarthi\)" writes:
> > >  
> > >  
> > > Could you please shed more light on "unexpected correlated
failures" ?
> > > If this is significant work, we can have a separate draft for 
> > > this, that falls under the same proposed work item (Protection
> > Benchmarking).
> > >  
> > > Thanks,
> > > The author team.
> >  
> >  
> > Examples:
> >  
> >   Put fiber on both sides of the Amtrak right of way and have a
train
> >   wreck where fiber on both sides of the track are affected.  (This
> >   happenned in NJ on the main fiber run from DC and Philly to NYC.)
> >  
> >   Put fiber on both sides of a bidge that falls down in a hurricane.
> >   (Raritan River bridge, NJ).
> >  
> >   Have multiple fiber cuts due to fiber crossing an earthquake fault
> >   that gives way.  (This happend in San Diego serving 3 of 4 major
> >   fiber paths to the rest of the world.  International traffic that
> >   went by way of Hawaii to San Diego was affected in a addition to
> >   local traffic.)
> >  
> >   Have some goon dig up the wrong gas line and start breaking a
bunch
> >   of annoying heavy cables that weren't supposed to be their.  (This
> >   happenned somewhere in the midwest along a gasline right of way in
> >   unused natural gas pipes).
> >  
> >   Have a rat explode in the power transfer switch at the POP.
> >   (Happenned in San Jose).
> >  
> >   Have multiple blocks which include POPs of multiple providers shut
> >   down power including backup power due to a major gas leak.  (This
> >   happenned in LA).
> >  
> >   Have a flood take out power in the POP and eventually flood the
POP
> >   with 4 feet of water.  (The Mississippi river flooded St. Louis
and
> >   many other cities in the midwest in 1993).
> >  
> >   Power outage while one generator is being repaired and the other
one
> >   fails to start.  (NYC, affected air traffic control on entire East
> >   Coast for many hours due to outage in FAA private network).
> >  
> >   Etc, etc, etc.
> >  
> > You can read the NANOG for many more example.  Some perhaps even 
> > more ammusing.  This sort of things happens on a fairly regular 
> > basis and increased hurricane activity may make it even more common.
> >  
> > These were all "unexpected" from the standpoint of SRLG 
> > configuration or entirely unexpected.  They were also very well
correlated.
> >  
> > No amount of FRR is going to handle these types of outages.  They 
> > require fallback to resignaling the tunnels for MPLS or fallback to
IP.
> > Providers with "BGP free core" do not have fallback to IP as an
option.
> > BGP free core may become quite popular soon making MPLS resignal 
> > time even more important.
> >  
> > Curtis

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: Curtis Villamizar <[email-masked]>
Date: Wed, 14 Jun 2006 16:40:58 -0400

In message <[email-masked]>
> does not propose methodology for correlated failures. 

Jay,

How about this text.

    Two or more failures are considered correlated if those failures
    occur more or less simultaneously.  Correlated failures are often
    expected where two or more logical resources, such as layer-2
    links, rely on a common physical resource, such as common
    transport.  TDM and WDM provide multiplexing at layer-2 and
    layer-1 that are often the cause of correlated failures.

    Where such correlations are known, such as knowing that two
    logical links share a common fiber segment, the expectation of a
    common failure can be compensated for by specifying Shared Risk
    Link Groupgs [cite something like FRR RFC].

    Not all correlated failures are anticipated in advance of their
    occurance.  Failures due to natural disasters or due to certain
    man-made disasters or mistakes are the most notable causes.
    Failures of this type occur many times a year and generally a
    quite spectacular failure occurs every few years.

    There are two factors impacting service availablity.  One is the
    frequency of failure.  The other is the duration of failure.  FRR
    improves availability by minimizing the duration of the most
    common failures.  Unexpected correlated failures are less common.
    Some routers recover much more quickly than others and therefore
    benchmarking this type of failure may also be useful.

    Benchmarking of unexpected correlated failures should include
    measurement of restoration with and without the availability of IP
    fallback.  The use BGP free core may be growing, making the latter
    case an important test case.

    This document focuses on FRR failover benchmarking with MPLS TE.
    Benchmarking of unexpected correlated failures is out of scope but
    may be covered by a later document.

Trim, snip, cut or dispose of as you pleae.  Consider it a suggestion.

Curtis