Subject: Re: [GROW] [Idr] I-D Action: draft-ietf-grow-ops-reqs-for-bgp-error-handling-06.txt

From: Jeff Wheeler <[email-masked]>
Date: Mon, 31 Dec 2012 11:59:15 -0500

On Mon, Dec 31, 2012 at 5:52 AM, Rob Shakir <[email-masked]> wrote:
> If we tear down the session based on a single bad UPDATE, then Bad Things™ happen.

This is why I'm confused why you aren't more interested in giving the
vendor (ultimately, the operator) the choice to simply ignore bad
updates.

You know that error-handling already leaves the RIB (network) in a bad
state.  You might as well give the option to reset the session under
even fewer circumstances.

-- 
Jeff S Wheeler <[email-masked]>
Sr Network Operator  /  Innovative Network Concepts

---

From: Jeff Wheeler <[email-masked]>
Date: Mon, 31 Dec 2012 13:13:51 -0500

On Mon, Dec 31, 2012 at 12:54 PM, Jakob Heitz <[email-masked]> wrote:
> I don't think treat-as-withdraw is trying to fix a single session reset. Graceful restart can fix that. It's the rolling resets that need a human to remove a buggy router or a config that triggered the bug. That takes several hours. Treat-as-withdraw limits the damage during those hours.
>
> Could we please settle on that without trying to solve the impossible?

If you read my posts to IDR on this topic, you'll see where I explain
how it is possible to solve the "impossible."

Specifically, you can ignore just about any bad update, or bad message
of any kind, as long as you can figure out where the next message
starts.  This fixes the rolling resets.

You may know that a lot of businesses suffered multi-hour outages in
October simply because of 5 DFZ routes announced by LANL that had
illegal attributes.  This is very hard for operators to troubleshoot
on most routers.  The routers that experienced rolling resets were
buggy but if the operators simply had a panic button, "ignore bad
messages," their networks would have been up and they would not have
been losing money by the minute.

This is not the only time bad updates have propagated through the DFZ
and caused big problems.  It has happened repeatedly.

I believe it will begin to happen more often inside datacenter
networks, because BGP is being used for more and more things, like
EVPN.  Operators are going to need BGP to become more robust.

You can make it a lot more robust just by deciding to ignore
everything in a bad message.  This is not good, but it is a lot better
than session-reset, in most cases.

Please, read my posts on this topic, and do not treat this problem as
an unsolvable one.  It can be largely solved in a way that gives a
very useful fallback option to operators.

This whole draft is about fallback options, and it is pretty stupid to
have a large amount of complexity to solve a small set of potential
bugs, when you could ALTERNATIVELY or IN ADDITION to that, have a very
low-complexity option that solves more problems.

---

From: John Leslie <[email-masked]>
Date: Mon, 31 Dec 2012 13:41:38 -0500

Jakob Heitz <[email-masked]> wrote:
> On Dec 31, 2012, at 1:21 AM, "Robert Raszuk" <[email-masked]> wrote:
>> On Mon, Dec 31, 2012 at 4:09 AM, John Leslie <[email-masked]> wrote:
>>> Brian Dickson <[email-masked]> wrote:
>>>> 
>>>> But, the basic problem is this: missing an UPDATE won't trigger either
>>>> condition, it will at worst cause sub-optimal routing.

   I apologize for careless reading here. I took Brian to mean failing
to process an UPDATE which changed the NLRI from a peer, but didn't
change the _reachability_ of that block from that peer, breaks things
but mildly.

>>>> Missing a WITHDRAW _CAN_ cause Bad Things (TM) to happen.
>>> 
>>>   +1

   I meant to +1 the statement that missing the notice that an NLRI
was being withdrawn and not replaced (thereby changing reachability
from positive to negative) is a much more serious breakage, since
you may keep sending packets to that peer which no longer has the
reachability you think it does.

>> How about if we would mandate Enhance Route Refresh request to be send
>> to such peer who's bad updates triggered treat-as-withdraw action ?

   A Route-Refresh cycle would ensure that we have the actual
reachability status of that peer (if the Route-Refresh completes)
instead of some outdated NLRI the peer intended to withdraw without
replacement.

>> Yes that means that "treat-as-withdraw" should be applied only to
>> those peers which support Enhanced Route Refresh.

   We're discussing an area where "treat-as-withdraw" may no longer
be a useful term, since we're considering an UPDATE so badly formed
that we can't even extract which CIDR blocks are the subject of it.

   Personally, I'd prefer a Route-Refresh approach, where we make it
perfectly clear to the peer in question that we no longer have a
trustworthy state of its current advertised routing.

> I don't think treat-as-withdraw is trying to fix a single session reset.
> Graceful restart can fix that. It's the rolling resets that need a
> human to remove a buggy router or a config that triggered the bug.
> That takes several hours. Treat-as-withdraw limits the damage during
> those hours.

   I agree with Jakob that we're _trying_ to discuss that -- but I
don't agree we're succeeding very well.

> Could we please settle on that without trying to solve the impossible?

   I'm not sure we can...

   First of all, most of us _aren't_ trying to solve the impossible:
we're trying to design an error-management paradigm which passes _enough_
information on which to base intelligent routing decisions.

   And, I'm not sure what it means to "settle on" "treat-as-withdraw"
that some of us consider "good enough" for a "limited" period while
humans are working to resolve the actual problem. We know that the
"limits" on such a period correlate all to well with the amount of
pain being inflicted on paying customers.

--
John Leslie <[email-masked]>

---

From: Jeff Wheeler <[email-masked]>
Date: Mon, 31 Dec 2012 14:59:20 -0500

> pain being inflicted on paying customers.

It means just that, "settle on."  Treat-As-Withdraw leaves plenty of
circumstances where BGP will still be reset, perhaps endlessly until
clue arrives.

Let me just go through the explanation of what happened to many
networks in October 2012 again.  LANL announced some malformed routes
to the DFZ.  Most routers propagated them and ignored the malformed
bits.  Some routers did not, and instead, they reset the BGP session.
That was a bug, but the customers had NO OPTIONS TO WORK AROUND THIS.
Their ONLY CHOICE was to wait for a patch from their vendor (for folks
whose vendor routers did this) or install a patch (for the lucky users
who were running OpenBSD) or to ask their transit provider to install
a prefix-list to block the 5 malformed paths from going to them.

If customers had treat-as-withdraw in the form being discussed today,
it would not have helped.  Why?  Malformed Attribute Flags are not one
of the criteria it addresses.

What networks need is an IGNORE BAD MESSAGES option so BGP can avoid
session-reset, even if the cost is rather high (say a lot of RIB
inconsistency.)  Let the network operator choose when the cost of bad
RIB information becomes so high that he would rather have BGP reset.
After all, this will vary from one network to the next based on
business needs, so there should be flexibility.

That is why treat-as-withdraw is "settling."  Ignore bad messages is a
very simple option that "fixes" (mitigates) virtually all problems so
the operator can buy time to diagnose it in detail, or wait for his
vendor to provide a software update, or whatever.

I'm not arguing this should be the only option or even the default
option.  I'm arguing that it should be ONE OPTION.  As far as I'm
concerned, you can have treat-as-withdraw, and you can build as much
complicated error handling logic into it as you want.  That effort
might help me someday.  But if it doesn't, I know for sure that IGNORE
BAD MESSAGES is something I can fall back on, which is very likely to
buy me the time I need.

---

From: Jeff Wheeler <[email-masked]>
Date: Mon, 31 Dec 2012 15:36:05 -0500

On Mon, Dec 31, 2012 at 3:24 PM, Jakob Heitz <[email-masked]> wrote:
> That covers malformed attribute flags.

Except in the MP_REACH_NLRI or MP_UNREACH_NLRI, which is not a foolish
exception if the goal is to recover all the NLRI; but if the goal is
to avoid session-reset, it is.

Like I keep saying, the goal of error-handling should be to make
session-reset avoidable.  Today it is not.  It is creating complex
rules which must be supported by both sides of the BGP session to try
very hard to keep the network in a good state.

This is a good goal in itself, but not to the exclusion of avoiding
session-reset.

Again, the existing draft also ignores conflicts with RFC4271.  So do
the updates to RFC4760.  You are not allowed to put attributes into
the attribute list in any order you want.  RFC4271 explicitly requires
that attributes be sorted by Attribute Type Code.  A Capability
negotiation is necessary to change the ordering of attributes.

Finally, because the current error-handling concept can't be
implemented without such a capability, it's not useful if the other
BGP speaker lacks support.  That is why IGNORE BAD MESSAGES is needed.
 It doesn't require support on both sides.

---

From: "Smith, Donald" <[email-masked]>
Date: Thu, 3 Jan 2013 17:30:58 +0000

"Pampers use multiple layers of protection to prevent leakage. Rommel used defense in depth to defend European fortresses." (A.White) [email-masked]

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]] On Behalf Of
>Jeff Wheeler
>Sent: Monday, December 31, 2012 11:14 AM
>To: Jakob Heitz
>Cc: [email-masked]; [email-masked]
>Subject: Re: [Idr] [GROW] I-D Action: draft-ietf-grow-ops-reqs-for-bgp-
>error-handling-06.txt
>
>> I don't think treat-as-withdraw is trying to fix a single session
>reset. Graceful restart can fix that. It's the rolling resets that need
>a human to remove a buggy router or a config that triggered the bug.
Exactly. Ignore (not treat as withdraw, not reset session, ...) seems the sanest approach to me also!

>_______________________________________________
>Idr mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/idr

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 3 Jan 2013 19:50:08 +0100

> On Thu, Jan 3, 2013 at 6:30 PM, Smith, Donald <[email-masked]> wrote:

That's equally good as starting to drill little holes in the ship in
the middle of ocean. Sure it will still swim for a while, but when it
starts to drown it will be too late.

You may as well run BGP over UDP and have app level acks for received
datagrams. Problem solved - no sessions resets to worry about.

Cheers,
R.