Subject: Re: [GROW] [Idr] I-D Action: draft-ietf-grow-ops-reqs-for-bgp-error-handling-06.txt

From: Rob Shakir <[email-masked]>
Date: Wed, 2 Jan 2013 11:39:07 +0000

On 1 Jan 2013, at 17:27, Chris Hall <[email-masked]> wrote:

> […snip…]

Hi All.

I think this is a good summary of the different approaches. In ops-reqs-for-bgp-error-handling-06, there is no category of "fatal" essentially because (as you highlight) the line between the fatal and the critical cases is somewhat blurry. I would propose that we do not add another category of error for "fatal".

If I go back to the proposal I made on 31/12:

> Would the GROW working group be happy if we address Chris' concern related to "lost NLRI" (which AFAICS is really the case where we have >1 type of NLRI attribute within a single message) by adding a note that an error remains Non-Critical if _at least one_ NLRI attributes can be successfully parsed? I'm unclear here as to whether we're addressing a common situation where multiple sets of NLRI are being contained within a single message [1]. If so, then adding "at least one" and then a further point that an implementation SHOULD use a single NLRI attribute per UPDATE message, and put this at the start of the attributes would seem to be a fair way forward.

I would suggest that adding the following wording to § 3 of the draft addresses this, and clarifies the issue of "lost" NLRI:

"An error SHOULD be defined as Non-Critical if at least one NLRI attribute within an erroneous message can be successfully parsed. In cases where more than one attribute containing NLRI is included within a single UPDATE message, this may result in cases where some NLRI contained within subsequent attributes are missed, particularly where length errors exist in the message. In order to minimise the risk of such occurrences, it is recommended that an implementation SHOULD include only one attribute containing NLRI per message." 

Additionally -- from the discussions that Jeff Wheeler raised, around repeated errors. In § 5, it seems that there is a further requirement, which I would suggest that we state as:

"In order to address repeated instances of critical errors, an implementation SHOULD provide a means by which an operator can enable such errors to be ignored. Where a mechanism of this nature is implemented, it provides a means by which an operator may avoid prolonged session failure which results in isolation from one, or more, routing domains. An operator deploying such a mechanism MUST be aware that holding such sessions up may result in inconsistency within the RIB, which may cause incorrect forwarding of traffic (e.g., loops, or blackholing)."

Some feedback on whether this addresses the points discussed over the last few days would be appreciated.

Happy New Year,
r.

---

From: Jared Mauch <[email-masked]>
Date: Wed, 2 Jan 2013 10:26:33 -0500

On Jan 1, 2013, at 12:27 PM, Chris Hall wrote:

> It is a truth universally acknowledged (AFAICS), that if NLRI in a
> broken UPDATE are treated-as-withdraw, that is no worse than
> session-reset and much to be preferred.
> 
> So, treat-as-withdraw is a reasonable thing for any implementation to
> do, by default, where it can.
> 
> The problem is that when things are broken, it may not be possible to
> identify all the NLRI -- some may be "lost".  [At this point I
> recommend: "The Engineer", AA Milne.]

I've been following this draft and wanted to chime in with some concerns.

1) in the (big) "Internet" space, lost NLRI are a huge deal for customers.

2) When we see a software defect that results in a session being closed, we regularly need help from the vendor to identify the offending NLRI/message.

3) The biggest problems we've seen are where vendor A and vendor B (or their sub-variants that run another OS) behave differently with the same NLRI.

I've seen a small number of theses cases over the past ~12 years and am very concerned with the amount of effort trying to error correct the error handling system when one side has a software defect.  I want to make it clear that all these cases attempting to resync at 0xffff etc are correcting for a defect.  Some of these defects resulted in an improperly formatted UPDATE message from a sender, and others were the result of problems on the receiver.

The instability this possibly introduces into an enterprise or large scale network is of significant concern.

To solve the "treat as withdraw" (aka: possibly create a per-node routing loop) problem, I would expect BGP users to demand a "periodic resend all NLRIs" feature to flush the state, creating further entropy in the system unnecessarily.

At some point, the equipment that is sending or receiving the BGP message will need to be maintained by the owner.  The dropping of the session is meant to draw attention to the problem in the same way as others that use ABORT or ASSERT in their code to handle an unexpected condition.

We can not correct for every error, nor should we make that a goal as the results in writing the error handling code quickly get complex.  (Speaking as someone who attempted to write a BGP implementation once.. ugh).

Asbestos suit on,

- Jared

---

From: "Chris Hall" <[email-masked]>
Date: Tue, 1 Jan 2013 17:27:48 -0000

Jeff Wheeler wrote (on Mon 31-Dec-2012 at 20:36 +0000):
....
> Like I keep saying, the goal of error-handling should be to make
> session-reset avoidable.  Today it is not.  It is creating complex
> rules which must be supported by both sides of the BGP session to
> try very hard to keep the network in a good state.

The effects of "lost NLRI" range from "ho hum" to "arghh", depending.
The possibility of "lost NLRI" ranges from "small, I think" through
"dunno" to "don't care".

The incidence of "lost NLRI" can be reduced by some simple rule
changes on the ordering of attributes.

But, there is a desire to improve things without changes at the sender
end.  Also, there is a desire to avoid session-reset at (almost) all
costs -- right up to the point where sessions are never reset
(re-syncing on 16 x 0xFF as required).  So, since we cannot eliminate
"lost NLRI" let us learn to love them (or at least learn some
tolerance).

The risks posed by "lost NLRI" are unknown, and the effects context
dependent.  Given the uncertainty, I do not think it is reasonable to
accept anything more than a (vanishingly) small risk of "lost NLRI",
by default.  So, AFAICS, we are in the land of the knob.

Suppose three categories of error in an UPDATE message:

  * Non-Critical -- ie treat-as-withdraw or otherwise

    in general terms: all NLRI accounted for.

    More accurately: the risk of "lost NLRI" is deemed
    negligible.

  * Critical -- ie bad, but not session-reset

    in general terms: "lost NLRI", ie:

      either: there is reason to believe that there
        are "lost NLRI" (eg. a broken MP_UNREACH_NLRI
        attribute)

      or: it is not sufficiently clear that all NLRI
        are accounted for.

    NLRI that can be accounted for may be treated in
    a Non-Critical sort of a way, but some other
    response(s) may be triggered to mitigate the
    effect of "lost NLRI".

   * Fatal -- ie session-reset

     things are FUBAR -- by some definition.

     (Graceful Restart and enhancements thereof may blur
      the line between Fatal and Critical errors.  But
      that's another story.) 

The message is broken... things are uncertain, so: a key knob is the
one which allows the operator to select criteria for Criticality --
anything less is Non-Critical, anything more is Fatal.  Other knobs
may select for specific responses to different forms of Critical and
Non-Critical errors.

This is all starting to look complicated :-(  But at least it avoids
trying to square the circle.  [When you have eliminated the
impossible, what remains, ....]

At a practical level, I observe:

  1) if (or as) most implementations only send one
     collection of NLRI per message, then extracting
     one collection is enough to be reasonably sure
     there are no "lost NLRI".

  2) the worst case of "lost NLRI" is lost Withdrawn
     NLRI... which, per (1), are going to be the
     only NLRI in the message, and hence unlikely
     to be lost.

So, a knob or two can allow the operator to settle on what works best
for them, and allow the implementer to provide stuff which an adult
operator may use in the privacy of their own network.  And, one can
envisage Emergency Knobs -- for when there is limited time to Save the
World, and being fastidious about the specification just gets in the
way.

Stepping back from the minutiae of unpacking UPDATE messages, in
routeing terms the above categories are (I think):

  * Non-Critical:

      - some routes which the sender has offered have been
        filtered out, because the attributes were garbled,
        (ie treat-as-withdraw),

      - some routes which the sender offered had partly
        invalid attributes, but they have been accepted,
        in some form (ie other mechanisms),

      - BUT the routes which remain are *valid*,

      - AND the receiver knows which prefixes have been
        affected.

  * Critical:

      - some routes which the sender has offered are not
        available to the receiver,

      - some routes which should have been withdrawn
        may still be in use,

      - some routes whose attributes should have been
        changed may still be in use,

      - AND the receiver does not know which of the
        routes which remain are in doubt.

   * Fatal -- FUBAR

Incidentally, in the absence of any better, novel mechanism: having
reset and restarted a session, there is (much) less reason to worry
about "lost NLRI" -- at least until End-of-RIB rolls up (or some
time-out).  But in any case, some things which are Fatal in normal
running could be downgraded, for some period, after a session-reset ?

Happy New Year,

Chris

---

From: Tony Li <[email-masked]>
Date: Wed, 2 Jan 2013 09:56:30 -0800

On Jan 2, 2013, at 7:26 AM, Jared Mauch <[email-masked]> wrote:

> We can not correct for every error, nor should we make that a goal as the results in writing the error handling code quickly get complex. 

+1

I'd also like the folks in the operational community who feel that treat-as-withdraw is a too liberal policy to please speak up.  I know folks still call their vendors and raise the roof when a single prefix is improperly handled.  Your voices need to be heard here too.

While we can do SOME things to decrease session resets, we cannot fix all cases and simply treating things as a withdraw and walking away is wholly unacceptable, as some of you will hopefully agree.  Creating arbitrary hair here is NOT going to help as the error handling code itself will become fraught with errors.

Ergo: KISS

Tony

---

From: Jakob Heitz <[email-masked]>
Date: Wed, 2 Jan 2013 20:05:11 +0000

On , Tony Li <> wrote:

> simply treating things as a withdraw
> and walking away is wholly unacceptable,

It's worth reiterating that Robert and I (at least) have advocated
for using
http://tools.ietf.org/html/draft-ietf-idr-operational-message-00
when an error is detected but no NOTIFICATION is sent.

BTW, I notice that this draft has expired.

-- 
Jakob Heitz.

---

From: "Chris Hall" <[email-masked]>
Date: Wed, 2 Jan 2013 18:15:31 -0000

Rob Shakir wrote (on Wed 02-Jan-2013 at 11:39 +0000):
> On 1 Jan 2013, at 17:27, Chris Hall wrote:
> > […snip…]
 
> I think this is a good summary of the different approaches. In ops-
> reqs-for-bgp-error-handling-06, there is no category of "fatal"
> essentially because (as you highlight) the line between the fatal
> and the critical cases is somewhat blurry. I would propose that we
> do not add another category of error for "fatal".

OK.  Old-fashioned session-reset (unadorned by Graceful Restart or
other mitigation, past present or future) is Bad.  The Fatal category
would capture cases where an old-fashioned session-reset is the only
available response.  As old-fashioned session-reset fades into distant
memory, that distinction will be less useful.

I think that what this boils down to is:

  Non-Critical => per-Message response is sufficient

  Critical     => per-Session (or possibly per-AFI/SAFI)
                  response is required.

The anatomy of per-Message response may include:

  1) treat-as-withdraw all NLRI that can be identified

  2) scrubbing round some attribute errors and proceeding
     with some or all announced NLRI.

  3) other (novel) means to recover the state of particular
     NLRI.

  4) (implicitly) ignoring any known or unknown "lost-NLRI".

The anatomy of per-Session response may include:

  a) old-fashioned session-reset, where nothing else is
     available.

  b) complete session-drop, where patience is exhausted.

  c) other means, old and new, to restore the session,
     to some level of health.

  d) some means to control cycles of repeated errors
     generating the same (inadequate) response.

     [Definition of madness: doing the same thing over
      and over again in the expectation of a different
      outcome.] 

It seems to me that there are two contexts for error-handling: (i) in
normal running, and (ii) during error-recovery.

Things which are deemed Critical in normal running may be deemed
Non-Critical during error-recovery.  That may be part of the automatic
per-Session response to an error, or may be some override settable by
the Operator.

What an operator deems Non-Critical in normal running will depend on
their assessment of the risk/impact of "lost NLRI" and the impact/cost
of the available per-Session response(s).  The risk of "lost NLRI"
depends on sender behaviour and on the acceptable
"degree-of-broken-ness".  The impact of "lost NLRI" is context
dependent.  The impact/cost of any per-Session response depends on
what is supported by both ends of the session.

My conclusion is that there is no "one size fits all" allocation of
errors to class of error.  Hence, the requirement is for operator
control over error classification and over error response/recovery --
on a per Session basis -- both for normal running and error-recovery.

Having got this far... I'm tempted to back away from talking about
Critical/Non-Critical *Errors*, and talk, instead, about
Message-Level/Session-Level *Recovery*... not much of the existing
draft would be affected if the focus shifted from Errors to Recovery.

.....
> I would suggest that adding the following wording to § 3 of the
> draft addresses this, and clarifies the issue of "lost" NLRI:
> 
> "An error SHOULD be defined as Non-Critical if at least one NLRI
> attribute within an erroneous message can be successfully parsed. In
> cases where more than one attribute containing NLRI is included
> within a single UPDATE message, this may result in cases where some
> NLRI contained within subsequent attributes are missed, particularly
> where length errors exist in the message. In order to minimise the
> risk of such occurrences, it is recommended that an implementation
> SHOULD include only one attribute containing NLRI per message."

The way in which attributes and NLRI are packed in a (current) UPDATE
message is unhelpful.  But, I don't think this is the place to solve
that problem.

Further, changes to the specification of UPDATE messages cannot ensure
that software will follow that (or any other) specification.  After
all, we are only here because software is less than perfect !  And,
there is some desire to cope with errors which cannot be addressed by
any amount of specification-tweaking.

And, there is the need to do better without changes at the sender end.

Hence, IMHO the question of what should be handled at the
Message-Level, and what should be handled at the Session-Level, is
best decided by the operator... as above.

Further, this approach would allow the requirements to avoid the
quicksand which is UPDATE message parsing.  [Huzzah !]

-----

The requirements could recommend new defaults for the classification
of errors.  There's no absolute need for this: given the ability to
choose something which suits them better, I'm sure operators will
happily enable what they want.

I guess any *default* would err on the side of caution, ie:
Message-Level Recovery is appropriate only where there is a
(vanishingly) small risk of "lost NLRI".

The appropriate default for a given session may depend on the
behaviour of the peer, which may be the subject of negotiation,
configuration or sweeping generalisation.

The requirements could recommend that UPDATE messages should be
altered to reduce the risk of "lost NLRI" or be generally more robust.
If that is constrained by the need for new-form UPDATE messages to be
downwards compatible with existing ones, the requirements should
mention that.  If the default classification depends on new sender
behaviour, I think that implies the requirement for a C.... but I
won't repeat that heresy :-)

I'm skirting around the quicksand here, I don't want to get sucked
down again...

Chris

---

From: Randy Bush <[email-masked]>
Date: Thu, 03 Jan 2013 09:03:53 +0900

> 1) in the (big) "Internet" space, lost NLRI are a huge deal for

indeed.

> At some point, the equipment that is sending or receiving the BGP
> message will need to be maintained by the owner.

early.  i want to hear it from my network management system before i get
a call from the customer.

> We can not correct for every error, nor should we make that a goal as
> the results in writing the error handling code quickly get complex.

as a naggumite, i call this 'do-gooder' software.  when it does the
right thing, no one notices and says thanks.  when it does the wrong
thing, the flamage gets pretty hot, and justifiably so.

randy

---

From: Jared Mauch <[email-masked]>
Date: Thu, 3 Jan 2013 12:40:14 -0500

Jeff,

On Jan 3, 2013, at 11:51 AM, Jeff Wheeler wrote:

> On Wed, Jan 2, 2013 at 12:56 PM, Tony Li <[email-masked]> wrote:
> Tony,
> Every operator I've asked thinks "ignore bad BGP messages," which is
> even more extreme than treat-as-withdraw, is a good idea.

Not sure who you're asking, but I think this whole draft is an idealist attempt to workaround software defects that may be uncorrectable as a whole.  (I say this as a $large_operator as measured here: http://as-rank.caida.org/?mode0=as-ranking&n=10&ranksort=1 )

The missing prefix because it was ignored, or routing loop because a withdraw was ignored will quickly change these folks minds.

Take one of the most recent defects:

http://www.cisco.com/en/US/products/csa/cisco-sa-20100827-bgp.html

The device takes a valid route on the receive side and corrupts it as it forwards it.  While ignoring may be one solutions, there is no way to actually know or get remediation of this prefix and software defect.

There are 3 classes of BGP operators:

1) Core networks
2) Edge/Mid-tier networks
3) People using it as their IGP/datacenter/vpn/private networks (these may eventually connect to the internet, but those UPDATE messages won't necessarily reach)

> I'm not saying anyone thinks this would be a good default.  These
> things can just be knobs used when they are appropriate or necessary.
> 

Methinks you underestimate the complexity that would be added to the error handling code.  While finding a marker/0xff may be easier, understanding the large block of updates in the flood of activity and low latency of large tcp windows make this much harder and more prone to error.

> Respectfully, you are about as wrong as one could get on this issue.
> Of course customers don't want one prefix to be broken.  Fifteen years
> of "CEF problem" have taught us all that these conditions are hard to
> troubleshoot.  However, it is often preferable to have one or many
> prefixes broken, than have a BGP session flap endlessly due to some
> bug.

For some operators, the only chance to workaround defects is to have something catastrophic happen to provide the justification to management to actually pick up the $new_software that corrects the problems you've been applying band-aids to. 

There are many people who now are having trouble diagnosing network problems due to a workaround from early 2009:

http://puck.nether.net/pipermail/cisco-nsp/2009-February/058512.html

> The vendor should have some standards body coverage for giving the
> operator this knob, and customers are right to ask for it.  Sure,
> you're giving us more rope.  Sometimes that is what we need.

I'm generally in the more-rope camp, but this is effectively throwing out years of well worn code path and introducing new code (and likely defects) in the handling of error cases which can only make things worse.  There wasn't a broad-reaching bgp attribute problem in 2012 that i'm aware of, putting us in the "there is stability in the core" camp.  I'm seeing a well intentioned but unneeded element of meddling here.

People on the edge also need to learn to maintain their devices.  This isn't a standards body issue, and IMHO off-topic for here, but an important datapoint.

- Jared

---

From: Tony Li <[email-masked]>
Date: Thu, 3 Jan 2013 10:00:00 -0800

Jeff,

> even more extreme than treat-as-withdraw, is a good idea.

I'm sure if you asked folks if they wanted anything that helped them and completely ignored the costs, they would say yes.  However, if you want to make a reasonable, rational, and justified decision, you must consider the implications of the request.  That's what I'm asking.  Consider implementor input as well, because there are some practical considerations here.

> Respectfully, you are about as wrong as one could get on this issue.

Respectfully, I think you're misunderstanding my position completely.  My point is that a reasonable implementation cannot possibly live up to the expectations that you're setting up here.  To be specific, once an implementation loses the syntactic parsing of the data stream, realistically, the session is corrupt and an eventual reset is inevitable.  Or, in other words, BGP cannot possible ignore bad messages.  That's not the way it works.

> prefixes broken, than have a BGP session flap endlessly due to some

All of the marketing that you're doing here is positioning this as a 'solution'.  It's not.  Yes, it will stop the flap, but it does NOTHING to fix or deal with the underlying bug.  All it does is gloss it over, and as such, it will have implications in the field whereby this papers over real bugs and we have now promoted BGP errors into RIB errors.  That's NOT making things easier to debug, that's just applying a band-aid.

A more constructive way to address the real problem here would be to talk about whether we should even re-establish the session after an error.  Long ago, we made an implementation decision to simply retry.  That would seem to be the real issue at hand.

> you're giving us more rope.  Sometimes that is what we need.

Sorry, but the point of the standards body is to standardize PROTOCOL changes.  Everything that has been discussed here are IMPLEMENTATION ISSUES.  We don't standardize those, for very good reasons.  And the vendors need zero help from the IETF in making implementation issues.  If real customers want a particular behavior, they can always just ask for it, as always.

> Related to this, what is your plan for dealing with BGP Attribute
> re-ordering in the rewrite of RFC4760?  

My plan?  My personal plan is to ban the use of all MP extensions, as all of that is simply evil and should be scrubbed off the face of the earth.  

I'll be putting this in place as soon as I'm elected Emperor of the Universe.  ;-)

> Currently there is no
> mechanism for telling a BGP neighbor that you intend to send the
> MP-NLRIs before other Attributes.  Doing that goes against RFC4271's
> recommendations.  Relying on that behavior (e.g. for error-handling)
> is not possible unless the neighbor promises to do it.  Thus far,
> there seems to be no intent to allocate another Capability Code for
> this.  The MP-BGP Capabilities Optional Parameter was not designed to
> be extended to do other things besides announce to the neighbor what
> AFI/SAFIs it supports.

RFC 4271 doesn't require a specific ordering because it would be bad protocol design.  As soon as you require ordering, some implementation is going to check that ordering.  There will be more bugs that occur because the mandated ordering was not followed, and more sessions will be dropped.   As always, the right thing is to follow Postel's law: be liberal in what you accept.  

Requiring a specific ordering in order to improve error handling is a bad tradeoff: you're creating a host of additional bugs so that you can try to simplify error handling on another set of bugs.

Tony

---

From: Jeff Wheeler <[email-masked]>
Date: Thu, 3 Jan 2013 11:51:59 -0500

Tony,

re-ordering in the rewrite of RFC4760?  Currently there is no

-- 
Jeff S Wheeler <[email-masked]>
Sr Network Operator  /  Innovative Network Concepts

---

From: Michael Long <[email-masked]>
Date: Thu, 3 Jan 2013 11:35:54 -0800

> All of the marketing that you're doing here is positioning this as a 'solution'.  It's not.  Yes, it will stop the flap, but it does NOTHING to fix or deal with the underlying bug.  All it does is…

I understand what you are saying and I agree 100%, however, from an my operations perspective the "fix" is the same. Either upgrade to fixed code or policy out the offending announcement. I would rather deal with a customer routing issue vs a frantic call from our noc saying 15+ att peers globally are bouncing. The latter being a much bigger impact on our network. 

I can live with a couple of /24's not working for a few customers. I can't have 15+ peers bouncing because of bad updates and even more peers bouncing because of missed keepalives due to cpu pegged trying to deal with 15 peers bouncing globally. 

> A more constructive way to address the real problem here would be to talk about whether we should even re-establish the session after an error.  Long ago, we made an implementation decision to simp…

I would back this provided adequate logging as to why the session is down. It would be much like tripping max-prefixes where we could hard clear a single single session for debug. I could live with this. 

Mike
-- 
Michael Long
NTT Communications Global IP Network
ph. 214.915.1352
jabber: [email-masked]

---

From: Enke Chen <[email-masked]>
Date: Thu, 03 Jan 2013 11:36:03 -0800

hi, Jeff:

On 1/3/13 11:22 AM, Jeff Wheeler wrote:

[snip]

> Of course BGP can ignore bad messages.  To say otherwise is simply
> telling a lie because you haven't made a good argument.
>
>

A good argument (IMO) has been made in the error handling draft:

---------
http://datatracker.ietf.org/doc/draft-ietf-idr-error-handling/

4. Operational Considerations

    Note that "treat-as-withdraw" is different from discarding an UPDATE
    message.  The latter violates the basic BGP principle of incremental
    update, and could cause invalid routes to be kept.  (See also
    Appendix A.)

---------

-- Enke

---

From: Jared Mauch <[email-masked]>
Date: Thu, 3 Jan 2013 14:58:02 -0500

> I understand what you are saying and I agree 100%, however, from an my operations perspective the "fix" is the same. Either upgrade to fixed code or policy out the offending announcement. I would r…

I'm very concerned with the case of ignoring a route update and having a month-long discussion about why some route is missing from the $carrier_a network when it's being sent from $carrier_b and they show it going out just fine.

You don't know there's an issue until someone reports it and your long-tail to problem resolution takes forever.

> I can live with a couple of /24's not working for a few customers. I can't have 15+ peers bouncing because of bad updates and even more peers bouncing because of missed keepalives due to cpu pegged…

While related, this is an implementation defect on the part of vendors and their poorly optimized TCP and BGP implementations being unable to get their basic job done.  I recall vendors blaming our "slow" system CPU then finally fixing their logic defect that always returned 1 or 0 when it thought it was idle.  (sometimes those if statements look really complex).

> I would back this provided adequate logging as to why the session is down. It would be much like tripping max-prefixes where we could hard clear a single single session for debug. I could live with…

I certainly agree there needs to be better logging from the vendors.

I remain convinced that attempts to address this problem will create more complex situations vs provide the desired result of a stable BGP core.

- Jared

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 3 Jan 2013 21:18:09 +0100

Hi Jeff,

> Ignore-bad-message is very simple and provides a band-aid for more
> problems, including unforeseen problems.

How are you going to clean the NLRIs in your network (both transit or
stub) which were withdrawn in the messages your BGP implementation
declared "bad" and decided to ignore ?

r.

---

From: Jared Mauch <[email-masked]>
Date: Thu, 3 Jan 2013 16:28:01 -0500

On Jan 3, 2013, at 4:19 PM, Jeff Wheeler wrote:

> On Thu, Jan 3, 2013 at 3:18 PM, Robert Raszuk <[email-masked]> wrote:
> I can fix them later, maybe even after I've had time to fully analyze the problem and get a software update from my vendor.  Maybe I'll try a refresh or a session-reset, but I won't be at the mercy of repeatedly flapping session and phone ringing off the hook with angry customers!

They're going to complain no matter what.

Also, if you're not a fully-peered DFZ ISP, perhaps you don't need full routes to make your decisions anyways.

I am sympathetic to a point, but either way someone has to look at the problem and correct it (likely with a code spin) assuming you don't take the offending device offline.

This is the shared fate everyone has in the BGP world for the distributed online database we operate.

Also, if you're seeing some problem, I'm sure that some other set of people out there will be seeing it as well.  The shared pain/cost will exist.  If you can't take the risk of speaking BGP, have your ISP send you default (or nothing) and advert your prefixes and call it a day.

- jared

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 3 Jan 2013 22:28:42 +0100

Jeff,

> I can fix them later, maybe even after I've had time to fully analyze the
> problem and get a software update from my vendor.

Well that assumes you have even noticed the problem in the first place.

On the point of flapping - completely agree. But the knob - already
available in some implementations - not to flap, but to keep the
session down till manual intervention - is completely different thing
and this is completely safe solution from protocol correctness pov.

--

Yes I understand your motivations, but the problem with BGP doing
things like treat-as-withdraw by default are really not what you are
describing.

Cheers,
R.

On Thu, Jan 3, 2013 at 10:19 PM, Jeff Wheeler <[email-masked]> wrote:
> I can fix them later, maybe even after I've had time to fully analyze the
> problem and get a software update from my vendor.  Maybe I'll try a refresh
> or a session-reset, but I won't be at the mercy of repeatedly flapping
> session and phone ringing off the hook with angry customers!
>
> A lot of folks are thinking about this problem in the context of the big
> carrier who doesn't want a hard-to-diagnose problem of 1 RIB entry being
> wrong.  That's okay, it is one way to think about it.
>
> A second way to think of it is as a small/regional ISP.  If one or more of
> his transits are flapping because of a bad path on the DFZ, that is going to
> cost him money and customers.  If he has no way to mitigate it, he is at the
> mercy of external parties.  He could just use "ignore bad messages" and at
> least stop bleeding money.  He does not care if he can't reach  5 /24s at
> LANL, they are unimportant to him.  What is important is if he has any
> customers left next week.
>
> A third way is the small- or medium-datacenter network.  Imagine you are a
> typical small/medium shop and you have some Cisco/Juniper/Brocade stuff for
> your ASBRs and your core, but you bought a bunch of RainbowPoop Router Co
> switches for your racks, because they are inexpensive and they support EVPN,
> L3VPN, VPLS, or some other feature you want but Cisco/Juniper/Brocade don't
> put into their inexpensive product.
>
> So your network looks like this:
>
> ISP1    ISP2
>
>   CISCO  JUNIPER
>   |    \/
>   |    /\                \   |
>   |   /  \                \  |
>   TOR1    TOR2    ....    TOR99
>
> Now imagine your JUNIPER supports NewVpnThing and that's a feature you
> decided to use on the RainbowPoop TOR devices.  But TOR1 sends a bad BGP
> update.  JUNIPER knows about NewVpnThing and sees a bad BGP attribute (that
> it recognizes) so it does whatever the NewVpnThing spec says, and tears down
> the session to TOR1.
>
> CISCO on the other hand, does not know about NewVpnThing so this router
> doesn't even understand the update is bad.  It just passes it along to TOR2
> .. TOR99.  Now those boxes all tear down their session to the CISCO.  Then
> they re-establish.  Then they go down again.  They keep on doing this and
> the network is freaking out.
>
> By the time your in-house clue notices, your symptom is that 99 identical
> TORs are flapping their BGP to your CISCO.  You probably don't even notice
> the 1 TOR that is flapping to JUNIPER.  Maybe JUNIPER even logs something
> helpful but you may not investigate it for a while.
>
> So your CISCO which is following the base spec is carrying a buggy update to
> your 99 other RainbowPoop TORs and they are all failing.  Your JUNIPER which
> knows about the NewVpnThing is following its spec and protecting the other
> TORs from this problem, but it is probably not helpful since your network is
> in chaos from all the flapping.
>
> What do you do?  Call vendor support.  Probably for CISCO and RainbowPoop.
> Well, now you are expecting the TAC of Cisco and the TAC of RainbowPoop to
> cooperate, which they'll have trouble doing; and it may take ages before
> anyone identifies the root cause of the problem is really TOR1.
>
> There are going to be a lot of RainbowPoop routers in the future, and many
> of them may use BGP.  We should make BGP more robust.
>
>
> --
> _______________________________________________
> Idr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idr
>

---

From: Jared Mauch <[email-masked]>
Date: Thu, 3 Jan 2013 17:05:28 -0500

On Jan 3, 2013, at 5:01 PM, Jeff Wheeler wrote:

> On Thu, Jan 3, 2013 at 4:28 PM, Jared Mauch <[email-masked]> wrote:
>> Also, if you're seeing some problem, I'm sure that some other set of
>> people out there will be seeing it as well.  The shared pain/cost will
>> exist.  If you can't take the risk of speaking BGP, have your ISP send you
>> default (or nothing) and advert your prefixes and call it a day.
> 
> There may be no shared pain if you are using BGP only within your
> datacenter network.  This is precisely why I provided the RainbowPoop
> scenario.  This is not your 1990s BGP anymore.  Anyone who has MPLS
> VPNs signaled using BGP already knows this but maybe isn't cognizant
> of it in this context.
> 
> You continue to make this argument that breaking your network so you
> notice a problem is good.  I would like to have the option to notice
> the problem by actually parsing my syslogs and NOT having my network
> break and cost me and customers a bunch of money.  That's not what I
> would really do, though.  I would just re-actively configure
> ignore-bad-messages if I needed to.  Because I agree with you, I would
> like BGP to flap if something unexpected happens.  However, I might
> need to stop it from flapping to remain in business.

Oh, I understand all these use-cases, but there is a case for a well-designed network not always sharing/mixing the NLRI.  eg: We don't transport v4 NLRI in v6 transport, nor v6 NLRI in v4 transport.  If you have a single session with massive shared fate, perhaps it's not a protocol design error but a network design error.

- Jared

---

From: Jeff Wheeler <[email-masked]>
Date: Thu, 3 Jan 2013 17:15:40 -0500

> Oh, I understand all these use-cases, but there is a case for a well-designed network not always sharing/mixing the NLRI.  eg: We don't transport v4 NLRI in v6 transport, nor v6 NLRI in v4 transpor…

it is not just a shared-fate problem.  Even if you have distinct BGP
sessions per AFI, the RainbowPoop scenario still breaks the datacenter
network if they actually depend on NewVpnThing.  Most networks will
still have a very hard time troubleshooting it, and they'll be
bleeding customers until they do.

I noticed you've omitted MPLS VPNs.  We also do not use the same BGP
sessions for IPv4 and IPv6, but we do use IPv4 sessions for MPLS VPNs.
 It is avoidable but I would bet that most networks do not avoid it.
Do yours?

On Thu, Jan 3, 2013 at 5:05 PM, Tony Li <[email-masked]> wrote:
> On Jan 3, 2013, at 1:48 PM, "Chris Hall" <[email-masked]> wrote:
>> However bad or difficult to clean up, what if that's not as bad as the
>> alternative ?
>
> Clearly, the treat-as-withdraw alternative is better.  You're not leaving bogus state in the rest of the network.

Does this mean you support treat-as-withdraw, even though you dislike
its high complexity?  My read of your postings today is that you
dislike the whole error-handling idea.

---

From: Tony Li <[email-masked]>
Date: Thu, 3 Jan 2013 14:16:19 -0800

On Jan 3, 2013, at 1:19 PM, Jeff Wheeler <[email-masked]> wrote:

> A lot of folks are thinking about this problem in the context of the big carrier who doesn't want a hard-to-diagnose problem of 1 RIB entry being wrong.  That's okay, it is one way to think about it.

Unfortunately, the code is NOT going to be that sensitive to the variations of the use case.  The same code that the Tier 1 carrier is using is the same code on much smaller systems.  We need to find a behavior that is amenable to all.

> We should make BGP more robust.

I'm glad we agree on one thing.

Tony

---

From: Jeff Wheeler <[email-masked]>
Date: Thu, 3 Jan 2013 17:23:56 -0500

> Unfortunately, the code is NOT going to be that sensitive to the variations of the use case.  The same code that the Tier 1 carrier is using is the same code on much smaller systems.  We need to fi…

One behavior may not suit all cases.  That's why we have options.

If we only had one router I guess it would be the biggest, most
expensive router possible; because big networks need these.
Fortunately vendors understand that they need to produce options to
suit different kinds of customers.  They also need to produce
configuration knobs to suit different kinds of customers.  This
standards body creates options every time it uses a capitalized word
except MUST or MUST NOT.  If options weren't needed, RFC2119 would be
a lot shorter.

---

From: Tony Li <[email-masked]>
Date: Thu, 3 Jan 2013 14:31:16 -0800

Jeff,

I support the general concept of improved error handling and softer failures when possible.

As I've said before, errors can be reasonably classified into two groups: syntactic and semantic.  

A syntactic error would be any inconsistency of length fields, incompatibility between a type and a length, or any other error that would introduce any doubt about the parsing of the message.  Once this type of error has occurred, then the remainder of the data stream is wholly in doubt.  A session reset seems like the only (conservative) way of handling this, as it's unclear that further data would be accurate.

I'm very open to a discussion of alternatives to session flap for these cases.  Should we require manual intervention before session restart?  This would seem reasonable.

Semantic errors would be consistency violations within the contents of the message.  In these cases, treat-as-withdraw seems reasonable.

Tony

---

From: Tony Li <[email-masked]>
Date: Thu, 3 Jan 2013 14:36:26 -0800

On Jan 3, 2013, at 2:23 PM, Jeff Wheeler <[email-masked]> wrote:

> suit different kinds of customers.  

Fortunately, this vendor has been around long enough to understand that adding options to suit every possible behavior results in complexity in the underlying code.  We, as a collective IETF group, learned long ago that adding 'nerd knobs' is generally a bad idea.

Tony

---

From: "Smith, Donald" <[email-masked]>
Date: Thu, 3 Jan 2013 22:54:50 +0000

"Pampers use multiple layers of protection to prevent leakage. Rommel used defense in depth to defend European fortresses." (A.White) [email-masked]

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]] On Behalf Of
>Tony Li
>Sent: Thursday, January 03, 2013 3:31 PM
>To: Jeff Wheeler
>Cc: [email-masked]; [email-masked]
>Subject: Re: [GROW] [Idr] I-D Action: draft-ietf-grow-ops-reqs-for-bgp-
>error-handling-06.txt
>
>
>>> Clearly, the treat-as-withdraw alternative is better.  You're not
>leaving bogus state in the rest of the network.
>>
>A syntactic error would be any inconsistency of length fields,
>incompatibility between a type and a length, or any other error that
>would introduce any doubt about the parsing of the message.  Once this
>type of error has occurred, then the remainder of the data stream is
>wholly in doubt.  A session reset seems like the only (conservative) way
>of handling this, as it's unclear that further data would be accurate.
I disagree. Does a session reset fix the issue? It hasn't the last few times I have seen this.
It exasperated it by causing peers to drop and having to rebuild ribs/fibs etc... continuously.

Other than getting it noticed I doubt a reset will ever make a misbehaving router stop misbehaving.

>
>I'm very open to a discussion of alternatives to session flap for these
>cases.  Should we require manual intervention before session restart?
Manual intervention will be required in most cases but if we "require" manual intervention haven't we left the programming side (state machine) and gone into the operational control side of things?

>This would seem reasonable.
>
>Semantic errors would be consistency violations within the contents of
>the message.  In these cases, treat-as-withdraw seems reasonable.
I would also like the ignore option here. You know your neighbor is saying something you don't understand, possibly due to a lack of in your vocabulary but you understand other elements so you nod and continue listening to the rest:)

This is what happened recently. ALU announced attributes with 1 bit set that others didn't understand. If the routers that didn't understand that bit could have ignored just that element everything would have been fine.

>GROW mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/grow

---

From: Tony Li <[email-masked]>
Date: Thu, 3 Jan 2013 15:14:45 -0800

Hi Donald,

>> A session reset seems like the only (conservative) way

Which is why I suggested that re-opening the connection is perhaps not optimal.

> Other than getting it noticed I doubt a reset will ever make a misbehaving router stop misbehaving.

The point is that once you've lost context in the data stream, subsequent messages cannot be accepted with a reasonable certainty of correct parsing.  

> Manual intervention will be required in most cases but if we "require" manual intervention haven't we left the programming side (state machine) and gone into the operational control side of things?

Correct.  The programming side has failed.  As you note, trying again is not likely to help.  Adding Artificial Intelligence to BGP to recover is unlikely to be effective or an efficient means of making progress.

> I would also like the ignore option here. You know your neighbor is saying something you don't understand, possibly due to a lack of in your vocabulary but you understand other elements so you nod…

The entire problem with the ignore option is that it leaves bogus information in the network.  Suppose that the update contains an AS path change for a prefix.  If you ignore the update, then you ignore that path change, AND, you fail to propagate that change to your upstream neighbors.  Now, BGP's loop prevention mechanism is out the window and, in the worst case, you've created an inter-domain forwarding loop.

If, on the other hand, you withdraw that prefix, then any alternate connectivity for that prefix can come into play.

In short, treat-as-withdraw is a fail safe approach.  Ignoring updates is not.

Tony

---

From: Brian Dickson <[email-masked]>
Date: Thu, 3 Jan 2013 18:34:29 -0500

> Other than getting it noticed I doubt a reset will ever make a misbehaving

The "continuously" points to the problem. If a session drops because of
syntactic issues, it should stay down.
Does that make sense? It is analogous to "treat-as-withdraw", just using a
much larger hammer. :-)

> I would also like the ignore option here. You know your neighbor is saying
> something you don't understand, possibly due to a lack of in your
> vocabulary but you understand other elements so you nod and continue
> listening to the rest:)
>
> This is what happened recently. ALU announced attributes with 1 bit set
> that others didn't understand. If the routers that didn't understand that
> bit could have ignored just that element everything would have been fine.

Here's a couple of problems with what you're suggesting here.

You don't know whether that 1 bit changes the logic that should be used
(e.g. "This is a back-up route only", or "Black-hole this prefix, please".)
Presuming that you should use the announcement is dangerous.

And then what should you do with the announcement - forward to appropriate
peers, or not?
If you forward it, should you keep the bit set, or not?

IMHO, the safe thing to do with "unknown bit set", is treat-as-withdraw.

Maybe, MAYBE, have a knob to "listen to but ignore unknown bit".

DEFINITELY, do NOT announce bits you don't understand. (liberal accept,
conservative send).

And PROBABLY do NOT announce things if you twiddled unknown bit(s).
You are potentially messing with unknown semantics, to the probable
detriment of your peers.
(Here there be dragons.)

The safest thing is to not announce. The conservative approach is to "treat
as withdraw".
And put a suitable error message wherever error messages normally go.

Again, IMHO.

Brian

---

From: Jared Mauch <[email-masked]>
Date: Thu, 3 Jan 2013 18:42:07 -0500

On Jan 3, 2013, at 6:34 PM, Brian Dickson <[email-masked]> wrote:

> 
> 
> On Thu, Jan 3, 2013 at 5:54 PM, Smith, Donald <[email-masked]> wrote:
> The "continuously" points to the problem. If a session drops because of syntactic issues, it should stay down.
> Does that make sense? It is analogous to "treat-as-withdraw", just using a much larger hammer. :-)
> 

I think that Mike Long had it right earlier today in saying that the vendor implementation should shut the session just like max-prefix. (perhaps with a default restart-timer measured in tens of minutes)?  This doesn't seem to need a protocol change though.  I am also a bit wary of as Tony put it .. "nerd knobs".  I've seen how some of these legacy knobs break in some new/future release as they migrate through our templates/platforms over the years.  Nobody even knows what it does anymore, and the original purpose is gone but the parser still takes it ...

- Jared

---

From: Michael Hallgren <[email-masked]>
Date: Fri, 04 Jan 2013 00:51:21 +0100

+1

mh

---

From: Jeff Wheeler <[email-masked]>
Date: Thu, 3 Jan 2013 19:26:40 -0500

> I'm very open to a discussion of alternatives to session flap for these cases.  Should we require manual intervention before session restart?  This would seem reasonable.

I've already posted an alternative.  The BGP MARKER makes it possible
to recover Message framing.  I'm not sure I think this is a good idea.
 I am sure I would wish for it at 2am on a Saturday morning, if it
fixed my problem.  So if my vendor supplied it as an option, I might
be glad for that someday.

> Semantic errors would be consistency violations within the contents of the message.  In these cases, treat-as-withdraw seems reasonable.

The current draft for "treat-as-withdraw" has a great deal of
complexity.  You seem to both favor this solution, and hate the
complexity.  I also hate the complexity, and "ignore-bad-message" is
my alternative.

Would you rather the draft simply eliminate a lot of the complexity?
I would, because the complexity is what creates more opportunities for
session-reset.  It also may catalyze bugs.

None of these are "nerd knobs" when they fix your operational problem
and allow you to stop bleeding money.  TCP Path MTU Detection for BGP
is a "nerd knob."  Yet, there it is, along with plenty of other things
that might tweak your router a bit; but they're not there to save your
butt in a crisis.  That's what error-handling should do -- save your
butt.

> Other than getting it noticed I doubt a reset will ever make a misbehaving router stop misbehaving.

Maybe not, but I've certainly seen it solve problems.  For example,
Cisco 6500/7600s used to send > MTU sized Ethernet frames to eBGP
neighbors if TCP MD5 for BGP was in use.  I had a eBGP session to a
transit network which would fail every few days because the Cisco box
would repeatedly try to send an illegally-large frame, my router would
discard the frames, and the Cisco would keep trying to retransmit it.
Eventually my HoldTime would expire.  As you might imagine, this took
quite some time for us to diagnose; but at least the auto-reset caused
the network to return to a good state pretty quickly.

> This is what happened recently. ALU announced attributes with 1 bit set that others didn't understand. If the routers that didn't understand that bit could have ignored just that element everything…

Actually the problematic routers that reset their sessions "MUST have
ignored [this bit] when received," according to RFC4271, but they
didn't do that.  So the reason why those routers broke is that THEY
were buggy AND the originating router was buggy.  It is also arguable
that routers in the DFZ should not have propagated this 1 bit.  Some
routers didn't while others did.

So if the receiving routers weren't buggy in the first place, they
would not have experienced any problem.  However, ignore-bad-message
or treat-as-withdraw would be a good catch-all that an operator could
use, even if his router was buggy, to keep his network functioning.

> In short, treat-as-withdraw is a fail safe approach.  Ignoring updates is not.

Treat-as-withdraw is NOT fail-safe.  It can create loops or blackholes
in your network.  Claiming over and over that it is safe will not make
it true.

Ignore-bad-message has bad qualities, which are just as bad as
treat-as-withdraw, except it is less complicated to implement and
there is a greater chance it can avoid a potentially crippling
session-reset loop.

---

From: Tony Li <[email-masked]>
Date: Thu, 3 Jan 2013 17:01:50 -0800

Hi Jeff,

> to recover Message framing.  I'm not sure I think this is a good idea.

I'm pretty convinced that it's NOT a good idea.   What happens if the marker is preceded by 0xFF?   Ok, you can make some educated guesses based on what looks to be a length field (probably NOT starting with 0xFF), but you are just guessing.    

And what if the error was egregious and your parsing is off by more than a byte or two?  That marker might be a path attribute.  Now what?

Ok, yes, you could take a risk and see if you can keep parsing, but then what happens if you hit another error?  Just keep guessing along?  Your odds of guessing correctly are falling.  Or do you try to be really, really smart?

While you may be willing to gamble with your domain, please remember that you are also gambling with every network that accepts updates from you, transitively.  For lots of folks, that's the bulk of the Internet.  Is this a good gamble?  I'm not convinced.

> complexity.  I also hate the complexity, and "ignore-bad-message" is

To be specific, treat-as-withdraw seems to be less dangerous.  As implemented (and perhaps NOT as envisioned), treat-as-withdraw can be done with an acceptable amount of complexity, IMHO.  It needs to be done VERY simply.

> session-reset.  It also may catalyze bugs.

I would welcome that, but we should recognize that we're making a tradeoff.  4271 opts for maximum simplicity: if there's an error, shut it down.  Anything more than that will be added complexity.  What we're trying to do is to add some small amounts of complexity for a hopefully significant amount of robustness.  If done right, that would be a reasonable tradeoff.  If done wrong, this could be a major disaster.

> in your network.  Claiming over and over that it is safe will not make

Example please?  Assuming that the prefixes are parsed out of the message correctly, then treating them as withdrawn should be hazard free.

> there is a greater chance it can avoid a potentially crippling

Non-sequitur.  Cyclic resets are orthogonal to treat-as-withdraw.  As enumerated above, ignore-bad-message is harder to implement and is more dangerous.
  
Tony

---

From: Jeff Wheeler <[email-masked]>
Date: Thu, 3 Jan 2013 21:33:48 -0500

> Example please?  Assuming that the prefixes are parsed out of the message correctly, then treating them as withdrawn should be hazard free.

If a transit router in your AS treats-as-withdraw a prefix, other
routers may try to send traffic across it, but the router which has
treated-as-withdraw will either have no information for that route, or
it will have made a different bestpath choice.  This could cause a
loop or a blackhole.

> Non-sequitur.  Cyclic resets are orthogonal to treat-as-withdraw.  As enumerated above, ignore-bad-message is harder to implement and is more dangerous.

No, ignore-bad-message is not harder to implement.  To say so is simply a lie.

To treat-as-withdraw you have to follow some rules, parse a damaged
message to look for NLRI, and then withdraw them.

Treat-as-withdraw is unquestionably harder than simply ignoring the
whole contents of a message.

Your argument is some hand-waving about how hard it is to ignore a
message, and your incorrect belief that treat-as-withdraw can't leave
the RIB in state that can cause forwarding problems.

I'm not picking at your position just for the sake of argument.  I
agree with you that error-handling (current draft) is too complicated.
 I'm suggesting alternatives, as well as modifications that should be
made to either RFC4760bis or to error-handling to make it actually
work.

Just to move beyond ignore-bad-message vs. treat-as-withdraw arguments
for now, I believe some basic questions need to be answered for
treat-as-withdraw to be a useful option.

1) should a BGP speaker be able to promise that MP attributes will be
sent first?
   if so, a capability code is needed
   this could be done by error-handling or RFC4760bis

2) should non-MP NLRIs in the message be moved in the way
error-handling suggests?
   if so, a capability is absolutely required
   the basic encoding of BGP Messages changes if this is to be done

3) there are numerous attribute-specific rules in error-handling now.
They are not crazy but they are complicated.  Is this complexity
enough to deter vendors from implementing error-handling at all?  Are
they likely to catalyze bugs?  Do the attribute-specific rules do more
harm than good?

If you are in favor of treat-as-withdraw then the above are important
questions to answer.

---

From: Jeff Wheeler <[email-masked]>
Date: Thu, 3 Jan 2013 14:22:47 -0500

> The missing prefix because it was ignored, or routing loop because a withdraw was ignored will quickly change these folks minds.

Yes, the existing draft contains a great deal of complexity to work
around specific problems that have been imagined.

It would be very simple to just ignore bad updates.  That is not
complicated.  It's not "good" but it's less bad than your network
being down because you received one bad update from your DFZ neighbor,
which in the case of a small network, might be one or all of their
transit providers.

Having had to support networks who were down because of this condition
recently, I understand their pain, because they were down with no hope
of being back up until external parties helped them.

> Methinks you underestimate the complexity that would be added to the error handling code.  While finding a marker/0xff may be easier, understanding the large block of updates in the flood of activi…

I don't think I underestimate it at all.  I think the complexity
introduced by the error-handling draft is a bad idea.  I know that
"ignore bad updates" is extremely simple and a very broad catch-all
tool which can be utilized if there are no better options.  Having
your network be down because you can't keep BGP sessions to your
transit providers established is not an option.

> For some operators, the only chance to workaround defects is to have something catastrophic happen to provide the justification to management to actually pick up the $new_software that corrects the…

Your position is that you don't want a feature that can mitigate
problems, because it would provide you with an opportunity to mitigate
problems instead of fix them?  Do you have to sabotage your network to
get problems fixed or capacity upgraded, too?

> I'm generally in the more-rope camp, but this is effectively throwing out years of well worn code path and introducing new code (and likely defects) in the handling of error cases which can only ma…

LANL's announcement of invalid paths caused about 3000 prefixes to
instantly disappear from the DFZ.  This lasted for several hours.
Those missing prefixes are just networks who didn't have any working
transit at all.  Some small networks that I worked with were affected
on only some transit sessions and not others.  The affect of this is
difficult to measure, but almost 1% of the DFZ disappearing because of
one bad update demonstrates just why BGP is a very vulnerable single
point-of-failure.

> People on the edge also need to learn to maintain their devices.  This isn't a standards body issue, and IMHO off-topic for here, but an important datapoint.

While true, if their vendor hasn't identified and corrected a bug yet,
they don't have any software upgrade option.  I believe this was
recently the case for Alcatel during same event caused by the LANL
announcements.

> I'm sure if you asked folks if they wanted anything that helped them and completely ignored the costs, they would say yes.  However, if you want to make a reasonable, rational, and justified decisi…

Do you think the error-handling draft is too complex to be worth
implementing?  I do.  That is exactly why I suggest "ignore bad
messages" as an alternative.  It is not complicated.

There are three types of situations worth considering.

1) message is bad and we don't know if Message Length is, because
another MARKER hasn't yet arrived
This is very easy to deal with.  Simply ignore the message.  If a
MARKER doesn't arrive next, reset the session.  Avoids complexity.
Your code should already be able to deal with a message that contains
nothing it understands -- for example, a message with no NLRI and
nothing but an optional, non-transitive attribute that it doesn't
recognize.

1) message is bad but Message Length is fine, and the next MARKER
appears as expected
This is also very easy to deal with.  Just ignore the message.  It is
exactly the same as the above case except perhaps you waited for the
next message to start arriving before you decided what to do about the
previous, corrupt message.

2) message is bad and Message Length is bad, so the next MARKER is not
where it should be
This is a little harder to deal with.  It may be a lot harder for a
BGP implementation that pathologically integrates TCP and BGP Message
parsing.  Perhaps some implementations would choose to try to recover
without session-reset by hoping a new MARKER will arrive soon to
"re-sync" the session.  Perhaps not.  I think the value of this is
quite questionable and highly dependent on how much work it is for the
implementor -- and what he feels are the chances of introducing new
bugs.

> All of the marketing that you're doing here is positioning this as a 'solution'.  It's not.  Yes, it will stop the flap, but it does NOTHING to fix or deal with the underlying bug.  All it does is…

It does do something to keep the network functioning.  Is it
functioning well?  No, of course not.  But there may be only 1 or a
few routes that are bad.

In the case of the LANL incident, 5 /24s were bad.  I can live with
not being able to reach 5 /24s that were announced with bad attribute
flags.  I can even live with a loop in my network for those /24s.  I
can't live with my network down because BGP is flapping.

You are right, this is a band-aid.  It is a very good one when you are
bleeding money.

> A more constructive way to address the real problem here would be to talk about whether we should even re-establish the session after an error.  Long ago, we made an implementation decision to simp…

If you have received bad messages from all your transit providers, it
will not matter if you re-establish the sessions or not, if the bad
messages keep coming.  All your transit will be down and you'll be
bleeding money.

> Sorry, but the point of the standards body is to standardize PROTOCOL changes.  Everything that has been discussed here are IMPLEMENTATION ISSUES.  We don't standardize those, for very good reasons…

Would you like a list of some standards-track documents that do not
modify protocols?

> I'll be putting this in place as soon as I'm elected Emperor of the Universe.  ;-)

I agree that MP-BGP is less than ideal.  That's true of many
extensions to BGP.  Sadly, there isn't BGP-5, and BGP-5 would be
needed to clean all of this up.

The reason I ask is because the work on MP-BGP makes a specific
recommendation to re-order the attributes in such a way that is the
opposite of a specific recommendation in the base BGP spec.  If other
work, such as error-handling, wants to depend on the MP-BGP
recommendation being followed, then MP-BGP should actually provide a
way for the neighbor to signal its intent to follow that
recommendation.

That can and perhaps should be done with a new Capability Code.

If that is not done, then a lot of the rules in the error-handling
draft are not useful unless error-handling itself allocates such
Capability Code.

I don't understand the reason for adding this recommendation to MP-BGP
if you don't also think there should be a mechanism by which you and
your neighbor can agree to depend on it.

Also, I'd like to repeat that a lot of the rules in error-handling are
not useful unless the neighbor supports it.  This means it might be
useful within your datacenter network but is probably not useful to
most transit or peers for a long time.

> Requiring a specific ordering in order to improve error handling is a bad tradeoff: you're creating a host of additional bugs so that you can try to simplify error handling on another set of bugs.

I gather that your general opinion is error-handling is bad.

In addition, you think that specifically promising to order attributes
in the way needed by error-handling is further bad.  Do you think the
recommendation for ordering attributes should be removed from
RFC4760bis?

IMO this is important.

---

From: Jeff Wheeler <[email-masked]>
Date: Thu, 3 Jan 2013 15:05:44 -0500

On Thu, Jan 3, 2013 at 2:35 PM, Michael Long <[email-masked]> wrote:
> Either upgrade to fixed code or policy out the offending announcement.

You actually can't policy out the announcement on the receiving side.

This is why I continue to state that you need help from external
parties to solve these problems.  Vendor, BGP neighbor, etc.

When I got called because some of my transit customers were having
their sessions flap endlessly and their routers were logging errors
they did not understand, it took be about an hour to figure out what
prefixes were causing it.  That's because my router had equally poor
logging.  Then I asked the customers if they wanted me to filter out
the 5 bad prefixes from going to them.  That is what they needed until
they could upgrade their software.

>    update, and could cause invalid routes to be kept.  (See also

I hope we all understand that treat-as-withdraw violates exactly the
same principle.  It just throws away routes which may be valid,
instead of keeping them.

I don't understand the perspective from which you are looking at this
problem.  From mine, there could be a buggy router in my AS, and it
could be withdrawing routes from its RIB.  Then other routers could be
trying to transit traffic across that router, which is then mis-routed
(could be looped) or discarded.

The error-handling draft does not keep the RIB in a good state.
Anyone who thinks otherwise is simply incorrect.

So treat-as-withdraw puts the RIB in a bad state.  Ignore-bad-message
puts the RIB in a bad state.  Exactly how it is bad differs, but
treat-as-withdraw in the current draft is very complicated and does
not provide a band-aid for "all" problems.  Ignore-bad-message is very
simple and provides a band-aid for more problems, including unforeseen
problems.

---

From: Jeff Wheeler <[email-masked]>
Date: Thu, 3 Jan 2013 16:19:57 -0500

A second way to think of it is as a small/regional ISP.  If one or more of
his transits are flapping because of a bad path on the DFZ, that is going
to cost him money and customers.  If he has no way to mitigate it, he is at
the mercy of external parties.  He could just use "ignore bad messages" and
at least stop bleeding money.  He does not care if he can't reach  5 /24s
at LANL, they are unimportant to him.  What is important is if he has any
customers left next week.

A third way is the small- or medium-datacenter network.  Imagine you are a
typical small/medium shop and you have some Cisco/Juniper/Brocade stuff for
your ASBRs and your core, but you bought a bunch of RainbowPoop Router Co
switches for your racks, because they are inexpensive and they support
EVPN, L3VPN, VPLS, or some other feature you want but Cisco/Juniper/Brocade
don't put into their inexpensive product.

ISP1    ISP2

  CISCO  JUNIPER
  |    \/
  |    /\                \   |
  |   /  \                \  |
  TOR1    TOR2    ....    TOR99

Now imagine your JUNIPER supports NewVpnThing and that's a feature you
decided to use on the RainbowPoop TOR devices.  But TOR1 sends a bad BGP
update.  JUNIPER knows about NewVpnThing and sees a bad BGP attribute (that
it recognizes) so it does whatever the NewVpnThing spec says, and tears
down the session to TOR1.

So your CISCO which is following the base spec is carrying a buggy update
to your 99 other RainbowPoop TORs and they are all failing.  Your JUNIPER
which knows about the NewVpnThing is following its spec and protecting the
other TORs from this problem, but it is probably not helpful since your
network is in chaos from all the flapping.

---

From: Jeff Wheeler <[email-masked]>
Date: Thu, 3 Jan 2013 17:01:08 -0500

---

From: Randy Bush <[email-masked]>
Date: Fri, 04 Jan 2013 08:48:26 +0900

> wholly in doubt.  A session reset seems like the only (conservative)
> way of handling this, as it's unclear that further data would be
> accurate.

both ears

> I'm very open to a discussion of alternatives to session flap for
> these cases.  Should we require manual intervention before session
> restart?  This would seem reasonable.

you may have more faith in the alacrity of manual response than i

> the message.  In these cases, treat-as-withdraw seems reasonable.

and the tail

randy

---

From: "Saikat Ray (sairay)" <[email-masked]>
Date: Fri, 4 Jan 2013 02:37:16 +0000

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Jeff Wheeler
Sent: Thursday, January 03, 2013 6:34 PM
To: Tony Li
Cc: [email-masked]; [email-masked]
Subject: Re: [Idr] [GROW] I-D Action: draft-ietf-grow-ops-reqs-for-bgp-error-handling-06.txt

> Example please?  Assuming that the prefixes are parsed out of the message correctly, then treating them as withdrawn should be hazard free.

If a transit router in your AS treats-as-withdraw a prefix, other routers may try to send traffic across it, but the router which has treated-as-withdraw will either have no information for that route, or it will have made a different bestpath choice.  This could cause a loop or a blackhole.

[SR] If the transit router treated the prefix as "withdrawn", then it will remove the corresponding path. If there are no other paths for that net, then the router would send withdraws for that prefix to its own peers. So no other router would send traffic to it in the steady state.

---

From: Jeff Wheeler <[email-masked]>
Date: Thu, 3 Jan 2013 21:48:35 -0500

> [SR] If the transit router treated the prefix as "withdrawn", then it will remove the corresponding path. If there are no other paths for that net, then the router would send withdraws for that pre…

Perhaps you read "ASBR" where in fact I wrote "transit router?"  The
behavior you describe is wrong.

---

From: "Saikat Ray (sairay)" <[email-masked]>
Date: Fri, 4 Jan 2013 02:53:05 +0000

-----Original Message-----
From: Jeff Wheeler [mailto:[email-masked]] 
Sent: Thursday, January 03, 2013 6:49 PM
To: Saikat Ray (sairay)
Cc: [email-masked]; [email-masked]
Subject: Re: [Idr] [GROW] I-D Action: draft-ietf-grow-ops-reqs-for-bgp-error-handling-06.txt

> [SR] If the transit router treated the prefix as "withdrawn", then it will remove the corresponding path. If there are no other paths for that net, then the router would send withdraws for that pre…

Perhaps you read "ASBR" where in fact I wrote "transit router?"  The behavior you describe is wrong.

[SR] Wrong in what sense? BGP code does not care whether it is a transit or an ASBR. What I described is exactly what the code will do.
                (i) a path for a prefix (a "net") is withdrawn.
              (ii) BGP deletes the path from that net.
            (iii) BGP reruns bestpath computation.
            (iv) If the bestpath changes, 
                    (a) if there is a new bestpath, then BGP readvertises the new bestpath to all its peers
                    (b) if there is no bestpath (which would be the case when there is no path for the net), BGP will send withdraws to all its peers.

---

From: Tony Li <[email-masked]>
Date: Thu, 3 Jan 2013 18:54:14 -0800

> No, ignore-bad-message is not harder to implement.  To say so is simply a lie.

I'm sorry, but this conversation has taken a turn for the worse.   I may be wrong (I really don't think so), but I am not lying.

You owe me an apology and I'm not going to continue this conversation until you do.

Tony

---

From: Jeff Wheeler <[email-masked]>
Date: Fri, 4 Jan 2013 01:41:42 -0500

>                     (b) if there is no bestpath (which would be the case when there is no path for the net), BGP will send withdraws to all its peers.

Bugs can be present at other places than an ASBR.  Imagine the
following very ordinary network:

AS100 --- [ ASBR2 --- AR3 --- ASBR4 --- ] AS500

ASBR2 will have iBGP routing exchange with ASBR4 by some means.  This
may be an ordinary iBGP session, via a route-reflector, or whatever.
If AR3 is buggy but it is in the forwarding path between ASBR2 and
ASBR4, then ASBR4 potentially will send packets to AR3, yet AR3 has
treat-as-withdraw a route.  ASBR4 is not subject to the bug, receives
and installs the iBGP-learnt route from ASBR2, and happily announces
it to AS500.

Now you see that ASBR4 and AS500 can both send traffic into the
network which is either blackholed or looped because of a bug in AR3.

This is exactly why the base BGP specification has a lot of rules
about what kind of routing policy you shouldn't apply to iBGP-learnt
routes.

You can create loops or blackholes with no malfunction at all, if your
vendor has given you a little bit of rope -- the ability to apply a
route-map to an iBGP neighbor and alter local-preference or various
other path attributes which influence the bestpath decision.

You certainly will encounter blackholes or loops if a router in the
middle of your network is buggy and does treat-as-withdraw.

If every AS contained only one router (or two), you would be correct.
However, you have overlooked the possibility of a bug affecting just a
path learnt by a transit router in the middle of an AS.

> You owe me an apology and I'm not going to continue this conversation until you do.

If calling your statement a lie has offended you, I apologize.  It is,
however, absolutely wrong; and you've framed it as absolutely right.

To review, you've said that ignoring a BGP message is not even
possible.  Of course we know that isn't true.  You've also said it is
harder to ignore a message than to treat-as-withdraw.  This is not
true either.  To do treat-as-withdraw you have to, at minimum, extract
NLRI from a known to be damaged message.  The current error-handling
draft then requires you to apply numerous rules.  Only then may you
decide if you treat-as-withdraw or terminate the BGP session.  Your
assertions cannot possibly be correct in any implementation, period.

The fact is you don't want to ignore messages.  I am much more
persuaded by your argument that ignoring messages may result in the
persistence of bad RIB entries which may be re-forwarded within or
outside the AS.  That is a good argument.  It's not constructive,
however, to cloud the issue by making absurd statements like "ignoring
a message is impossible."  It is possible, you just feel strongly that
it is the wrong approach.

Operators continuously feel that the IETF does not understand their
concerns.  The IETF feels that operators don't participate.  Both are
right.

I'm giving you the operator perspective and I am perfectly willing to
admit that my suggestion is both extreme and potentially stupid; but I
think it is still be smarter than the one on the table right now.  It
would be more productive for you to state the reason why you think
ignoring messages is stupid, than to simply say it's not possible or
it's too hard.

If you want operators to participate in the process, you have got to
be forthright with people who may have a very different view than you.
 I can say, I want to crush my thumb with a hammer, and you can tell
me that's stupid and be right.  If you say it's not possible because
hammers don't work that way, of course I'm going to call you out
because I know hammers can be swung in all manner of dangerous ways.
If you tell me my position is stupid, at least that is a matter of
opinion and open to argument.

---

From: "Saikat Ray (sairay)" <[email-masked]>
Date: Fri, 4 Jan 2013 06:54:10 +0000

Bugs can be present at other places than an ASBR.  Imagine the following very ordinary network:

ASBR2 will have iBGP routing exchange with ASBR4 by some means.  This may be an ordinary iBGP session, via a route-reflector, or whatever.
If AR3 is buggy but it is in the forwarding path between ASBR2 and ASBR4, then ASBR4 potentially will send packets to AR3, yet AR3 has treat-as-withdraw a route.  ASBR4 is not subject to the bug, receives and installs the iBGP-learnt route from ASBR2, and happily announces it to AS500.

[SR] Ok. This however does not happen in bgp-free core (ASBR2 will tunnel 
          the packet to ASBR4 even if AR3 is in the forwarding path).

---

From: Tony Li <[email-masked]>
Date: Thu, 3 Jan 2013 23:01:53 -0800

>          the packet to ASBR4 even if AR3 is in the forwarding path).

And we should also note that this can happen with ANY error recovery mechanism.  Anytime AR3 and ASBR4 receive different information or process it differently, you'll have a problem.  Also true with simple session reset: if AR3 detects an error and shuts down its iBGP connection, then ASBR4 won't know it and you've got a blackhole.

Bottom line: this situation already occurs with session reset, will occur with treat-as-withdraw, and will occur with ignore-bad-messages.

So, what was the point of this argument again?

Tony

---

From: Tony Li <[email-masked]>
Date: Thu, 3 Jan 2013 23:15:45 -0800

Jeff,

> If calling your statement a lie has offended you, I apologize.  

Thank you.

> It is,

You're welcome to disagree.  It remains my opinion about the coding complexity.  I think my experience with BGP coding speaks for itself.  What's yours again?

> possible.  Of course we know that isn't true.  

Do we?  Given an arbitrary set of errors on a byte stream where there are no guaranteed delimiters, how do you propose to resynchronize with absolute certainty.  My position is that it is risky as it requires certain inferences.

> You've also said it is
> true either.

So you can tell us what's true and what's not?

I'm dying to know about O.J., L.H. Oswald, and Marilyn.  Do tell.  ;-)

Yes, in my professional opinion, trying to recover from the semantic errors is pretty straightforward.  Recovering from the syntactic errors is not going to work and will end up as a reset.  Guessing about the message delimiters is going to be messy and hard.

>  To do treat-as-withdraw you have to, at minimum, extract

Again, I was speaking to a practical implementation, not necessarily what the draft is currently specifying.  The reality is simple: extract the NLRI.  If you can do so cleanly, then treat them all as withdrawn and syslog loudly.  If you can't extract the NLRI cleanly, then it's reset time.

> it is the wrong approach.

It is impossible to do so in the general case with 100% certainty.  You have lost context on the data stream.  You can no longer distinguish between data and TLV overhead.  There is (unfortunately) no redundancy that you can use to determine that any hypothetical recovery is correct.  That makes it a poor error recovery approach.

Now, if we had thought to have put a CRC-16 at the end of each message, we'd be in much better shape.  Sigh.

Tony

---

From: heasley <[email-masked]>
Date: Fri, 4 Jan 2013 07:59:44 +0000

> A syntactic error would be any inconsistency of length fields, incompatibility between a type and a length, or any other error that would introduce any doubt about the parsing of the message.  Once…

Amen.  if you've come upon a syntactic error and therefore have zero
confidence that any bit in the message is correct, I dont understand how you
could have any confidence that treat-as-withdraw is treating the right
prefix(es) as withdrawn, or even logging the right prefix(es).  you cant
even have confidence about the length of the message.

The aforementioned sesson re-establishment suppression seems a far better
and consistent approach; it also doesnt seem to be the subject for an rfc.

---

From: Jeff Wheeler <[email-masked]>
Date: Fri, 4 Jan 2013 03:29:54 -0500

>           the packet to ASBR4 even if AR3 is in the forwarding path).

Yes, absolutely true.

> So, what was the point of this argument again?

The point is that treat-as-withdraw is relatively complicated and does
not solve all the problems that many list participants might expect it
to.

Ignore-bad-message is less complicated.  It has similar problems to
treat-as-withdraw but the implementation cost, and bug risk, is lower.

Because several posters have expressed concern about the complexity of
additional code required to implement treat-as-withdraw, and because
some aspects of it truly haven't been thought through yet, it is
useful to compare treat-as-withdraw to something simple, even if it
may seem stupid at first.

> You're welcome to disagree.  It remains my opinion about the coding complexity.  I think my experience with BGP coding speaks for itself.  What's yours again?

For 13 years I've maintained a reference implementation of BGP for the
purpose of troubleshooting vendor implementations.

I'm posting, however, to speak from the operator perspective.  There
are plenty of programmers participating in the IETF process, and not
enough operators.

> Do we?  Given an arbitrary set of errors on a byte stream where there are no guaranteed delimiters, how do you propose to resynchronize with absolute certainty.  My position is that it is risky as…

If you read back a few posts, you'll note that I suggest it is
possible (without certainty, though) to re-synchronize using the
MARKER.

I'm not saying you must do that.  To ignore a Message you can just
ignore all of its contents.  If the Message Length is correct, then
you do not need to re-synchronize, because the Message framing won't
have been lost.  I also suggested this in the same post.

You've tied a need to re-sync the Message stream, to the simple
ability to ignore Messages.  You don't have to do both.  If your
argument is that re-sync is too hard to be worthwhile, I'll buy into
your opinion; but you should still consider the simple case if
ignoring Messages with no re-sync feature.

All you need to do to just ignore Messages is ignore the content of a
malformed message and move on.  Either the Message Length will be
correct and the following Message will begin with a MARKER, or it
won't.  What this buys you is much of what treat-as-withdraw intends
to fix with almost no implementation cost.

> I'm dying to know about O.J., L.H. Oswald, and Marilyn.  Do tell.  ;-)

I've written several times how to ignore messages easily.  You already
know how though, you just think it's a bad idea.  This is a reasonable
place to disagree.

I'll save my opinion about JFK for another time.

> Now, if we had thought to have put a CRC-16 at the end of each message, we'd be in much better shape.  Sigh.

Indeed.

---

From: Rob Shakir <[email-masked]>
Date: Fri, 4 Jan 2013 08:52:20 +0000

> Oh, I understand all these use-cases, but there is a case for a well-designed network not always sharing/mixing the NLRI.  eg: We don't transport v4 NLRI in v6 transport, nor v6 NLRI in v4 transpor…

I would highlight that public Internet networks are NOT the only implementation of BGP. Yes, in the Internet you can divide the two topologies that you have (the v4 DFZ and the v6 DFZ) into separate sessions to separate fate. However, a (I will assert, significant) number of operators run networks which run one address family that has numerous entirely separate topologies within it (e.g., L3VPN). There are no mechanisms that provide separation between such routing topologies within that AFI, SAFI, and it is not practical or scalable to divide these into individual sessions per-topology.

One of the taxes that we have with MP-BGP is that there are multiple applications which may not all have exactly the same deployment characteristics, or sensitivities. IMHO, we need to make sure we are considering all applications of BGP when considering this subject (this is one of the points that the draft that this thread relates to tries to cover). Having said that, where there are means where failure domains can be limited e.g. multi-session and/or separating AFI,SAFIs onto different transport sessions, of course, these form part of the solution.

Cheers,
r.

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 4 Jan 2013 11:45:25 +0100

Saikat,

>           the packet to ASBR4 even if AR3 is in the forwarding path).

True but http://tools.ietf.org/html/draft-ietf-idr-error-handling-03
does not mandate nor recommend to use "treat-as-withdraw" in IBGP in
tunneled only networks. Once you upgrade your router to new release it
careless if there is MPLS or GRE running in your domain.

In fact it says quite clearly:

   Although the "treat-as-withdraw" error-handling behavior defined in
   Section 2 makes every effort to preserve BGP's correctness, we note
   that if an UPDATE received on an IBGP session is subjected to this
   treatment, inconsistent routing within the affected Autonomous System
   may result.  The consequences of inconsistent routing can include
   long-lived forwarding loops and black holes.

Rgs,
R.

---

From: Jared Mauch <[email-masked]>
Date: Fri, 4 Jan 2013 08:51:36 -0500

Jeff,

The challenge I see here is that the NLRI is always at the END of the message.  The parsing problem is going to happen before the NLRI.  I certainly don't want my BGP implementation doing 'fuzzy' things with the message and producing further poor results.  It's no way to run a network, and can be difficult to code.

Once again I must assert that this is a problem best left to be fixed in the individual implementations.  We can certainly expect a high level of standard for software, but expecting no defects is unrealistic.

- Jared

---

From: Tony Li <[email-masked]>
Date: Fri, 4 Jan 2013 09:02:06 -0800

Jeff,

> have been lost.  I also suggested this in the same post.

True, but again, we're trying to deal with error handling.  When the interesting syntactic errors occur, then the Message Length is very much in doubt.  A typical syntactic error is an off-by-one inconsistency between the Message Length and the sum of all of the lengths of the TLVs (and overhead).  There is no good reason to believe the Message Length or the lengths of the TLVs is correct, so our code should probably check both.  Then check to see if the Marker (no need to shout ;-) is in the right place.

That's reasonable for off-by-one, but that's not the only possible mistake.  It's easy to have an inconsistency which is in the tens or hundreds of bytes.  Now what?  We could again check both alternatives, but again, the presence of a Marker, even in one of the possible locations, is no guarantee.  You could easily skip over an entire message, for example, and end up ignoring two updates, not just one.

> ignoring Messages with no re-sync feature.

As I've argued above, yes, I'd say that the re-sync risk vs. effort vs. return tradeoff isn't worthwhile.

I'm not sure why we would discuss a different error recovery mechanism that doesn't deal with the syntactic errors.  Again, if there are no syntactic errors, then recovery by ignoring the message leaves the network in an inconsistent state, as you've stipulated.  Clearly, treat-as-withdraw is preferable for this case.  The remaining question is how to deal with the syntactic errors.  I've argued that we shouldn't mess around and should just accept the session reset.

> to fix with almost no implementation cost.

I agree that the Message Length will either be correct or it won't.  Now, all us need is an oracle (the computer science kind ;-), to tell us which is which.  Got any handy?  ;-)

Regards,
Tony

---

From: "Smith, Donald" <[email-masked]>
Date: Fri, 4 Jan 2013 17:54:20 +0000

>From: Tony Li [mailto:[email-masked]]
>Sent: Thursday, January 03, 2013 4:15 PM
>To: Smith, Donald
>Cc: 'Jeff Wheeler'; [email-masked]'; [email-masked]'
Agreed.

>The point is that once you've lost context in the data stream,
>subsequent messages cannot be accepted with a reasonable certainty of
>correct parsing.
>
>
Understood on over runs or under runs (length errors). I am not sure that is true for ALL other issues.
Such as the recent issue where the attribute had non-zero bits in the lower nibble.

>these
>> Manual intervention will be required in most cases but if we "require"
>manual intervention haven't we left the programming side (state machine)
>and gone into the operational control side of things?
>
>
>Correct.  The programming side has failed.  As you note, trying again is
>not likely to help.  Adding Artificial Intelligence to BGP to recover is
>unlikely to be effective or an efficient means of making progress.
>
>
>> I would also like the ignore option here. You know your neighbor is
>saying something you don't understand, possibly due to a lack of in your
>The entire problem with the ignore option is that it leaves bogus
>information in the network.  Suppose that the update contains an AS path
>change for a prefix.  If you ignore the update, then you ignore that
>path change, AND, you fail to propagate that change to your upstream
>neighbors.  Now, BGP's loop prevention mechanism is out the window and,
>in the worst case, you've created an inter-domain forwarding loop.
Agreed I now have bad data (incorrect or incomplete) from one peer but everything else is still working.
I haven't flapped the session. Now I have to talk to that peer (assuming this is noticed via syslog or snmp or something). Or that peer calls us and says "why aren't you propagating this announcement" and we begin joint AS troubleshooting. That is probably the best solution:)

>
>If, on the other hand, you withdraw that prefix, then any alternate
>connectivity for that prefix can come into play.
That makes sense and in big ISPs there will nearly always be another path.

>In short, treat-as-withdraw is a fail safe approach.  Ignoring updates
But in a peering world you have potentially just withdrawn all of your routes to a peer from that peer.

You will now prefer some other peer for all of that peers traffic (or another connection to that peer in another location).

If you can ignore a single update error from a peer while you troubleshoot the issue with that peer it would allow a better path for the rest of that peers announcements. Again anything with a overrun or under run issue sort of implies the rest of the updates from that point forward are corrupted/untrustworthy.
Can you detect overrun and under runs without FPs or FNs? Probably not in all cases.

---

From: Tony Li <[email-masked]>
Date: Fri, 4 Jan 2013 10:02:43 -0800

> Can you detect overrun and under runs without FPs or FNs? Probably not in all cases.

Donald,

As I would implement treat-as-withdraw, the scope of the withdrawal would ONLY be for the message in question, and again it would only apply to semantic errors where the NLRI could be cleanly extracted.  So only the prefixes in error would be affected, not necessarily everything from that peer.  

Detecting overrun and under run is not too hard.  Again, the most common cases are when the Message Length and TLV lengths are inconsistent.  The presence of the marker for the next message and a sane length field for the next message are also helpful checks for detection.

Again, some kind of redundancy for framing would have been a better long-term design.  I'll note that if folks are sufficiently motivated, this COULD be retro-fit.  Add an attribute that is a CRC-16 of the entire message.

Tony

---

From: Jakob Heitz <[email-masked]>
Date: Fri, 4 Jan 2013 18:27:49 +0000

On , Tony Li <> wrote:

> possible mistake.  It's easy to have an inconsistency which
> is in the tens or hundreds of bytes.  Now what?  We could
> again check both alternatives, but again, the presence of a
> Marker, even in one of the possible locations, is no
> guarantee.  You could easily skip over an entire message, for
> example, and end up ignoring two updates, not just one.

Or waiting for a very long time for that marker to arrive.
Suppose the length is 40. You think it's 4000. You will sit
there patiently for an eternity waiting for more bytes to
arrive so you can read your next marker.

I support
http://tools.ietf.org/html/draft-ietf-idr-error-handling-03

with the understanding that it's the best we can do
in a tricky situation to limit the damage. Damage will
occur, but this limits it as best we can.

-- 
Jakob Heitz.

---

From: Jeff Wheeler <[email-masked]>
Date: Fri, 4 Jan 2013 14:50:50 -0500

Almost no one has commented on whether or not a capability code should
be used to re-order the contents of BGP Messages, and Attributes, as
suggested by error-handling.

This is really important if error-handling is to rely on such
re-ordering.  Many folks are saying error-handling needs it, the draft
says it, yet no mechanism is proposed to do it?  The closest thing is
a recommendation in RFC4760bis but it is the opposite of what RFC4271
recommends.  You can't actually know if your neighbor plans to follow
this new recommendation because there's no way to signal that intent.

In short, I suggest RFC4760bis should add a second capability that
indicates the BGP speaker promises to send the MP attributes before
all other attributes, as it recommends.

I further suggest any re-ordering of native NLRI information in the
Message needs to be addressed .. somewhere.

> That's reasonable for off-by-one, but that's not the only possible mistake.  It's easy to have an inconsistency which is in the tens or hundreds of bytes.  Now what?  We could again check both alte…

I'm glad you raise this possibility.  Indeed, if you tried to re-sync
the message stream by searching for a MARKER, you might skip many
messages without realizing it.  The potential damage to the RIB is
high, so it is a high price to pay to avoid session-reset.

> I agree that the Message Length will either be correct or it won't.  Now, all us need is an oracle (the computer science kind ;-), to tell us which is which.  Got any handy?  ;-)

As you know, when more data arrives from the neighbor, you can make a
good guess based on the marker.  I don't think divine intervention is
necessary.

> arrive so you can read your next marker.

Sure, but this is not different than many other cases where the
neighbor fails and you sit there waiting for HoldTime to expire.

---

From: Chandra Appanna <[email-masked]>
Date: Mon, 14 Jan 2013 18:37:27 -0800 (PST)

Rather late reply on this topic.. But I happen to agree with Jared.
I did start out thinking that the bgp error handling idea was a new
twist of a solution that was very interesting but after all this debate
and my own introspection, I am actually back in the camp of not believing
in the idea of soft corrections in the face of malformed updates and such
situations. I for one have not seen a lot of real data from the SPs and
other newerusers of BGP in my dealings of many years that reflect the need for
such solutions in the real world. I do remain convinced that this is
still a very interesting and challenging and 's..y' software and protocol
development point but from a practical point of view I am not sure.
I have also seen discussions in development teams about how to structure
code to handle such treat-as-withdraw scenarios and it seems to me
that it is very possible that it might only make code more fragile than robust.
If only I had a penny for every time I thought I had written bug free
automatic bug handling code... :)

0.02,
Chandra.

________________________________
 From: Jared Mauch <[email-masked]>
To: Michael Long <[email-masked]> 
Cc: [email-masked]; [email-masked]; Tony Li <[email-masked]> 
Sent: Thursday, January 3, 2013 11:58 AM
Subject: Re: [Idr] [GROW] I-D Action: draft-ietf-grow-ops-reqs-for-bgp-error-handling-06.txt

- Jared

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 15 Jan 2013 15:57:10 +0000

On 15/01/2013 02:37, Chandra Appanna wrote:
> I for one have not seen a lot of real data from the SPs and
> other newer users of BGP in my dealings of many years that reflect the need for
> such solutions in the real world.

Chandra,

This particular problem tends only to crop up every so often, but when it
happens, the results can be devastating on a global scale, causing
wide-scale outages across the entire Internet.  The last couple of times,
the problem has been caused by malformed attributes and has affect entire
products lines, e.g. all Junipers routers in the propagation path, or all
Cisco IOS boxes, or whatever.  Or better still, when you have one vendor
for your routers and another vendor for your RRs, you can be guaranteed of
failure until all your upstreams figure out some way of filtering out the
offending UPDATEs.  It is a very real and very serious problem.  There is
lots of noise about this on nanog-l.

Nick