Subject: Re: [Idr] Re: BGP path hunting, MRAI timer and Path Length Damping

From: Curtis Villamizar <[email-masked]>
Date: Tue, 26 Jun 2007 21:27:23 -0400

In message <[email-masked]>
Noel Chiappa writes:
>  
>     > From: Curtis Villamizar <[email-masked]>
>  
>     > We've been hearing about the eminent collapse of BGP routing,
>     > exhaustion of IP routes, and other predictions of doom and calls for
>     > radical changes to the architecture for well over a decade, most of it
>     > from people with a limited knowledge of Internet operations if any.
>  
> Curtis, nobody has a perfect crystal ball. I've been on both sides of this
> general situation, making some predictions of problems that didn't come true
> (on the one hand), and opposing alarums about what I thought were
> non-problematic scenarios - later proven to be non-problematic (on the other).
>  
> Do I wish I hadn't called some wrong? Yeah, because obviously it damages ones
> credibility when you blow it. At the same time, I think we're a bit in the
> position of lookouts on a military post; if you don't make some close calls
> that are wrong, *because you're erring on the side of caution*, some day you
> will Lose.Big because you'll miss a real one.

Caution is always good.  Panic isn't.

> In the latest go-round, there are two pieces of real-world data that are
> concerning me. The first is what I can glean from the data gatherered after
> the Pacific earthquake, which I have mentioned before:
>  
>   The Renesys NANOG presentaion Taiwan quake seemed to indicate that "a
>   while" is on the order of an hour or so. (However, it's not clear if that
>   was the time it took BGP to recover, or the time needed to reroute some
>   circuits. The Rensys presentation also covers much longer-term outages, but
>   those were caused by the need to set up new business relationships - not
>   really a routing problem!)
>  
> The other is something else I've mentioned before:
>  
>   I am told that it takes a core router .. [from one major vendor] a
>   significant fraction of an hour to load the current complete routing table
>   (via BGP) when it restarts

I know that taking full routing can be done in a few minutes including
at startup.  Anything taking an hour has an implementation problem.
That would not be a protocol problem.  A widely deployed bad
implementation can cause quite a disruption.  We saw that as far back
as the NSFNET days but the solution is to fix that implementation.

> When Destination-Vector routing systems collapse from overload, they don't
> usually degrade gracefully, there's usually a sharp knee, and then things get
> quite bad pretty quickly. Are we anywhere near that? I have no idea, because
> I don't think we have a really first-class understanding of the overall
> operation of the Internet routing.

It is possible to degrade gracefully.  If not all implementations do
degrade gradefully, that is an implementation problem.

Consider the following points:

  There are a finite number of legitimite prefixes.

  There are mechanisms that can be put into place to distinguish
  between legitimite prefixes and those that are questionable.
  Whether these mechanisms are used enough is a separate question but
  some people do use the IRR.  RPSEC has there own idea of how to do
  this but whether RPSEC has a good solution is a separate topic.

  The set of legitimite prefixes and therefore most important prefixes
  can be bounded.

  Routes change can be magnified but MRAI and other mechanisms and a
  finite number of routers limit that.  Regardless of it being bounded
  the amount of route change that is possible is quite high.

  Proper processing of the Adj-In RIB will read in the input stream
  and the set of changes in very large chuncks approaching the limit
  set by the TCP window and the delay between routers.  This can be a
  very high data.  Multiple changes to the same prefix in the AdjIn
  should simply overwrite each other without incurring any significant
  overhead.

  Periodically the router must pause from reading the Adj-In stream
  and run the decision process, route install, and Adj-Out processing.

  Any relatively stable prefix that makes only a limited number of
  changes (even amplified a bit by multiple paths) will get its change
  through the Adj-RIB-In processing fairly quickly even if other
  things are oscillating wildly.

So a good implementation of BGP (and they do exist) on a well
administered network (some form of verification of legitimate prefixes
is done to keep them finite and the provider knows how to avoid like
BGP RR pitfalls, etc), should be capable of enduring almost arbitrarily
high levels of routing nonsense from some corner of the Internet.

That corner of the Internet might be completely hosed.

This is nothing new.  A decade ago InternetMCI deployed a full mesh of
ATM links and their IGP melted down every few days and didn't
stabilize for days at a time.  This went on for many months until MCI
gave up on that design.  The route churn affected some other providers
but not all.  The NSFNET was fine (actually ANSNET at that point).
I'll pick on them becausse they don't exist anymore.  PSI had their
share of problems too.  Many others.

The point is that BGP can survive this.  A decade ago ANS sold more
expensive network service with the market distinction of using the IRR
to keep routing sane and being the most stable network.  Some others
sold cheaper service with in some cases shoddy administration that was
deemed to be a "low opex" feature.  That still exists today.

Whether the routers can handle the load is an issue for the providers
and for IETF is an issue for BMWG if they choose to take it on.  There
is work in BMWG in that area and providers and vendors have done their
own testing since the mid-1950s, some doing more rigorous testing than
others.

Large commercial customers may want to wise up and start looking to
see how their providers run their networks and ask what is being done
to remain stable if parts or even the rest of the Internet went
unstable.  That might be all that is needed to fix the problem because
the problems at this point may not be primarily technical.

> All I know is that there are a couple of data items out there (particularly
> the Taiwan one) which are somewhat troubling. Yes, the Internet is so big now
> that it's unlikely to 'crash', in the naive sense, but we have historically
> seen several events that screwed up reachability to large parts of the
> Internet for many hours, so something slightly bigger is not that implausible.
>  
> Just because it hasn't up till now is absolutely no guarantee that it never
> will; the 'Challenger' disaster showed NASA how flawed that line of reasoning
> is, and I hope we don't have to learn the same lesson the hard way.
>  
> 	Noel

Right.  Good thing NASA didn't run the NSFNET.

Curtis

btw- this thread is a huge distraction from the topic of the i-d that
Tony put before us.

Regarding the draft:

   At this time, we have no recommendations for possible alternative
   parameter settings.

RIPE made recommendations many years ago.  Almost a decade ago.
You've cited their recommendations so why is this comment there?

The draft continues to ignore the suggestion that we implement BGP
route flap damping as originally specified which is per AS path per
prefix and not just per prefix.  The logs I've seen indicate that
there is not all that high a multiplier of prefixes if AS Path is
included and the data structure for damping is tiny.  It should be
noted in Tony's draft that BGP Route Flap Damping was not implemented
as specified in RFC2439 where the problem of multiple AS Paths for the
same prefix was anticipated and handled.

RFC2439:

   The tuple representing a route for the purpose of route flap damping
   is:

      tuple entry            default      options
      -------------------------------------------
      NLRI
        prefix               required
        length               required
      AS path                included     option to exclude
      last AS set in path    excluded     option to include
      next hop               excluded     option to include
      MED                    excluded     option to include
                                          in comparisons only

   The AS path is generally included in order to identify downstream
   instability which is not being damped or not being sufficiently
   damped and is alternating between a stable and an unstable path.
   Under rare circumstances it may be desirable to exclude AS path for
   all or a subset of prefixes.  If an AS path ends in an AS set, in
   practice the path is always for an aggregate.  Changes to the
   trailing AS set should be ignored.  Ideally the AS path comparison
   should insure that at least one AS has remained constant in the old
   and new AS set, but completely ignoring the contents of a trailing AS
   set is also acceptable.

The RFC recommended implementing AS Path as part of the tuple with the
option to disable its use and tried to discourage disabling it in this
text and elsewhere.

_______________________________________________
Idr mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/idr

---

From: Danny McPherson <[email-masked]>
Date: Tue, 26 Jun 2007 22:02:48 -0600

On Jun 26, 2007, at 7:27 PM, Curtis Villamizar wrote:

> but not all.

I don't recall it being quite that dramatic or fragile and we did
stick with the architecture for more than a couple months, as
we were a bit committed, you might say :-)

That said, with "poor mans" mesh groups for constraining LSP
flooding to carefully engineered overlay topologies, coupled
with a slew of timers to throttle every function, from SPF [re]runs
to LSP lifetimes, hellos and SNPs, guess you could say we had
things tweaked.

The interesting thing is that most of the timers that were put
there at the time to slow things down are the very ones folks
use to speed up convergence today.

Of course, it wasn't a completely ideal situation, as you indicate.

But yeah, this is probably a bit off topic at this point.

-danny