Subject: Re: OSPF WG Charter Proposal

From: "Ash, Gerald R (Jerry), ALASO" <[email-masked]>
Date: Thu, 7 Nov 2002 08:27:38 -0500

Dave,

> > Such failures are not the fault of the service provider 
> > operation or the vendor/equipment implementation.  They are 
> > due to shortcomings in the link-state protocols themselves -- 
> > thus the need for the enhancements proposed in the draft.

> I strongly disagree with this statement.  While the design of the
> protocols can make it challenging, there is ample room in
> implementation to provide stable and scalable networks.
> 
> When a network collapses, the fault lies at the feet of the
> implementers.  In every case I've seen (too many), the collapse was
> inevitable sooner or later, due to naive design choices in software,
> but at the same time was quite nonlinear in its onset (making any
> predictive or self-monitoring approach pretty hopeless.)
> 
> There are some things that would make the job easier, at the cost
> of additional complexity, but pointing at network collapses 
> and blaming the protocols is disingenuous.

I think you should review the ample evidence presented in http://www.ietf.org/internet-drafts/draft-ash-manral-ospf-congestion-control-00.txt that the protocols need to be enhanced to better respond to congestion collapse:

- Section 2: documented failures and their root-cause analysis, across multiple service provider networks (also review the cited references)
- Appendix B: vendor analysis of a realistic failure scenario similar to one experienced as discussed in Section 2 (perhaps you would like to provide your own analysis of this scenario based on your OSPF implementation)
- Appendix C: simulation analysis of protocol performance (other I-D's being discussed provide analysis of proposed protocol extensions)

To say that network collapse in *every* case is due to *naive design choices* ignores the evidence/analysis presented.  Based on the evidence/analysis, there is clearly room for the protocols to be improved to the point where networks *never* go down for hours or days at a time (drawing unwanted headlines & business impact).

Jerry

---

From: Rohit Dube <[email-masked]>
Date: Thu, 7 Nov 2002 10:41:03 -0500

Jerry,

I looked at one of the sections you pointed out.

On Thu, 7 Nov 2002 08:27:38 -0500 "Ash, Gerald R (Jerry), ALASO" writes:
[snip]
=>I think you should review the ample evidence presented in http://www.ietf.org
>/internet-drafts/draft-ash-manral-ospf-congestion-control-00.txt that the prot
>ocols need to be enhanced to better respond to congestion collapse:
=>
=>- Section 2: documented failures and their root-cause analysis, across multip
>le service provider networks (also review the cited references)

The cited references [att, cholewka, jander, pappalardo*] are from trade
rags. Unless there is some other peer reviewed paper or standards document,
these incidents can hardly be used to point to the root cause. No, I am
not saying that something wasn't wrong - just that these references don't
carr much weight.

In one of these reported incidents, I was physically present and happen
to know the root cause. It was an "implementation mistake" which was
triggered by two different versions of the software being present in
the network simultaneously during a network wide s/e upgrade - a flooding
storm resulted. The problem here is _not_ in the flooding. While one
can certainly provide more knobs to limit flooding, one can also solve
the problem equally well or better by fixing the base implementation or
doing better tests before upgrading a network.

=>- Appendix B: vendor analysis of a realistic failure scenario similar to one
>experienced as discussed in Section 2 (perhaps you would like to provide your
>own analysis of this scenario based on your OSPF implementation)
=>- Appendix C: simulation analysis of protocol performance (other I-D's being
>discussed provide analysis of proposed protocol extensions)
=>
=>To say that network collapse in *every* case is due to *naive design choices*
> ignores the evidence/analysis presented.  Based on the evidence/analysis, the
>re is clearly room for the protocols to be improved to the point where network
>s *never* go down for hours or days at a time (drawing unwanted headlines & bu
>siness impact).

I don't think this is what Dave is saying - he is simply referring to all
the cases that _he_ has seen. This matches my experience with the caveat that
in some cases (a) there was faulty hardware (b) network/ospf parameters were
inconsistely or incorrectly applied.

--rohit.

---

From: Tony Przygienda <[email-masked]>
Date: Thu, 7 Nov 2002 17:17:03 +0100

>To say that network collapse in *every* case is due to *naive design choices* ignores the evidence/analysis presented.  Based on the evidence/analysis, there is clearly room for the protocols to be…
Jerry, most of the things you say in your document (which is actually
pretty good) has been
known to people like Dave and other old-time implementors since years
and avoiding exactly
those things by smart implementation techniques was what was
differentiating the have from
the have-nots. I remember myself learning some of those things by hard
experience and some
by looking at old-hands code ;-) [Albeit I remember also picking up a
lot of smart control protocol
ideas from your RTNR work]. I do not think that Dave is putting down
what you say, rather
(and I commit the stupidity to interpret his words by my own beliefs)
that what your document
says are mostly _implementation_ issues, not _standardization_ and
therefore it is not a very wise
idea to add them to the charter of a _standards_ group.  Good protocol
specs are _not_
implementation cookbooks, they are documents governing bits on the wires
in such a way that
two people implementing things in vastly different ways can still talk
to each other. Recommendations
of implementation techniques prove long-term inherently dangerous (like
Joel pointed out, at a
certain point in time adding more code to an implementation introduces
more bugs than the
performance gain is worth) or utterly ridiculous (look at ISIS 0-63
metric to make SPF real fast,
it lead to quite bad contortions).

    thanks

    -- tony

---

From: Dave Katz <[email-masked]>
Date: Thu, 7 Nov 2002 12:19:31 -0800

Other folks have pretty well summarized my feelings on this stuff, and
I've said all this in the past, but at the risk of redundancy I'll
restate.

My problems with the document are as follows:

Firstly, the claim that LS protocol collapses are primarily due to
deficiencies in the protocol design is just not accurate, and lets the
implementors off the hook.  While a different protocol design might
make such collapses more difficult (or at least different) it can be
shown on first principles that the existing protocols as specified can
be implemented in such a way that collapse (defined as adjacency loss,
which is the symptom at the heart of such collapses) will happen only
if the volume of Hello traffic exceeds the capacity of the receiver to
sink it and get its own Hellos out the door.  If properly implemented,
the failure mode under extreme flooding conditions should be to have
convergence time suffer (there's no free lunch, of course) and *not*
to lose adjacencies.  Implementations that suffer from this problem
are essentially doomed, since the failure mode is not predictable.  If
everyone stays lucky, the various workarounds to try to avoid the
problem will keep the implementation from falling over the edge, but
of course you can't control what your neighbor is doing to you.

Secondly, most of what the document describes is not subject to
standardization.  Discussion of implementation techniques is a fine
subject for an informational RFC, and in fact a number of of the
mechanisms described are already in use in some implementations and
are helpful (as are a bunch of other tricks that provide us with
product differentiation.)  All of the MUSTs and so forth give a false
impression that these techniques are required for protocol correctness
and interoperability, which they are not.  The fact that these are
implementation choices underscores my claim that this is not a
protocol design issue, which I define to be the bits on the wire and
the elements of procedure that determine interoperability.  As far as
I can see, there is only one suggestion in this paper that involves a
protocol change (the signalling of congestion.)

The appendices suffer from an excess of academia.  The reality of
routing protocol implementations is that analysis and simulation
seldom bears any resemblance to reality (not to mention that any
vendor-supplied data is likely to be self-serving.)  If we could
actually predict the behavior of networks of the size proposed, we
wouldn't be having collapses--they happen because of what the
implementors don't know, the subtle interactions and plain old
screwups, which of course won't figure into the analysis and
simulation.  It's certainly true that backing away from the cliff will
help avoid collapse, but the problem with this approach is that you
never know where the cliff is.

As Joel and others have pointed out, the addition of even more
complexity to try to fix problems is unlikely to be satisfying.  The
industry's history shows that these protocols are on the hairy edge of
what people are able to implement in a stable, robust way, and making
it even harder is not the path to nirvana.

There are also operational complexities involved--if you try to avoid
disaster by using parametric mechanisms like limiters and such, you
end up in a situation where either values are fixed (based on a wet
finger in the air or some empirical testing) or else a bunch of
mysterious knobs are provided that nobody has any idea how to set.  My
whipping boy for this approach is the multivariate SPF rate control
knob that was recently added by a major vendor.  If you put the
product together properly, the SPF rate shouldn't matter, and nobody
knows how to set the knob anyhow.

I guess my primary point is this--if you build your implementation in
such a way that it is close to impossible to lose adjacencies under
load conditions, the rest of this stuff is gravy (albeit tasty gravy)
but if your implementation can lose adjacencies when it is constipated,
none of this will address the real robustness problems.  The document
operates under the assumption that flooding traffic causes adjacency
loss (which is certainly true in many implementations) but it is this
issue that is the key to stability--the techniques themselves are
band-aids.

I don't have any particular problem with the techniques discussed in the
document (though in-band congestion notification has been shown to be
flawed many years ago--see any of the literature on ICMP Source Quench.)
Matter of fact, I use a lot of this stuff.  It's just that none of this
is really a fix, just a workaround, and the only part of it that I can
see that should be in the standards track are the congestion notification
extensions.

Where these and other techniques are quite valuable is when looking at
convergence/recovery times after major network events (other than load-
related adjacency collapses which, of course, should never happen.)
Expeditious flooding for its own sake is a fine goal indeed.

--Dave