Subject: Re: [Idr] Fwd: [GROW] I-D Action: draft-ietf-grow-ops-reqs-for-bgp-error-handling-06.txt

From: Rob Shakir <[email-masked]>
Date: Fri, 28 Dec 2012 13:10:04 +0000

Hi Chris,

(re: CCing IDR & GROW)

On 28 Dec 2012, at 12:28, Chris Hall wrote:

> Rob Shakir wrote (on Thu 27-Dec-2012 at 18:44):
>> 
>> Any comments very welcome (to me or grow@).
> 
> I'm afraid I still don't get it :-(  What am I missing ?
> 
> UPDATE Message Length errors are Critical because they (1) "result in
> cases whereby the NLRI attribute cannot be correctly extracted".
> 
> The implication is that a failure to extract all NLRI is Critical.  Is
> that a requirement ?

If the NLRI cannot be determined, then this is a Critical error, yes. I left the wording relatively open on whether this is *all* NLRI, as I am not sure that in the requirements draft we should specify direct solutions to specific issues, to e.g., say how to handle cases where MP_REACH_NLRI and MP_UNREACH_NLRI are in the same message [this is a case that I do not believe is forbidden by rfc2858 - if the working group could clarify whether this is something that we feel the draft needs to handle or can explicitly be omitted, then that would be appreciated].

> 
> Later: 
> 
>  (2) "All errors whereby the contained NLRI can be
>       extracted are referred to as Non-Critical". 
> 
> And that includes:
> 
>  (3) "where the length of all path attributes contained
>       within the UPDATE does not correspond to the
>       total path attribute length."
> 
> That is, at least, more explicit than
> draft-ietf-idr-error-handling-03, which glosses over (3).
> 
> But if (3) is non-critical then there is some chance that some NLRI
> will not be extracted, which appears to violate (1) and (2).

Disclaimer: As I am sure that my comments previously have made clear, I do not maintain a code base for a BGP daemon/implementation - so please feel free to correct my logic below.

I do not believe that (3) implies that the NLRI cannot be correctly found. If the sum of total length is incorrect, then we can still extract the individual attributes - we just find that there is not enough data to fill the overall length we were told and/or we have too much attribute data compared to the total attribute length. In the case where the NLRI attribute itself has a length error, then this is a critical error (based on the "Errors parsing the NLRI attribute of an UPDATE message" definition of Critical error), and a similar Critical error occurs in the latter case, where the Total Path Attributes + Withdrawn Routes are not equal to total UPDATE message length.

Either way -- again, I would say that this is something that we need to put text together for draft-ietf-idr-error-handling rather than the requirements document (this sounds like a solution, and the requirement does not have a SHOULD or MUST here, it is an "it is expected that…" comment).

> Then (4) "In order to maximise the number of cases whereby the NLRI
> attributes [plural, now, BTW] can be reliably extracted from a
> received message...".  Ah.  So it is not a Critical Error if "the NLRI
> attribute cannot be correctly extracted".

No - it is a Critical error if we cannot extract the NLRI. This recommendation is to give an increased chance that the NLRI can be extracted as per the IDR error handling draft. This then (by virtue of resulting in the NLRI being extracted) minimises the number of cases that result in a Critical error. The plural here is to reflect that the existence of >1 type of NLRI attribute.

> For me the requirement remains "conflicted".  On the one hand it seems
> to say that it is a Critical Error if the NLRI cannot be extracted and
> parsed.  On the other it seems to say it's OK if you cannot extract
> some NLRI.

If you'll forgive me for removing a significant proportion of your message, I think that we need to take another step back here. It seems to me that the key question that you are highlighting is "What level of confidence do we need to have before we declare that the NLRI cannot be extracted?" -- do you agree?

From an operator perspective, I would like to compromise *certainty* for *robustness*. You are right, we are compromising correctness here, we might end up withdrawing an incorrect NLRI and impacting service operation for that prefix - however, it is somewhat preferable to me to withdraw a a subset of the NLRI incorrectly, rather than impact all NLRI in one single action. We clearly need to provide some bounds on how much we compromise the certainty (and live within the realms of possibility, such that we are not just taking a shot in the dark). This is what the definitions of Critical and Non-Critical within the document are intended to provide. Once again I will refer to the requirement that there is a balance between correctness and robustness - rather than a locally risk averse approach that results in harmful wider behaviour.

Is it acceptable that we leave this as guidance within the requirements? If not, please could you suggest how the definitions of Critical/Non-Critical could be altered to address your concerns? I would also appreciate further input from IDR as to whether this is sufficient requirement from GROW to allow a solution document to be written?

Many thanks,
r.

---

From: "Chris Hall" <[email-masked]>
Date: Fri, 28 Dec 2012 18:04:09 -0000

> > UPDATE Message Length errors are Critical because they:
> > 
> > (1) "result in cases whereby the NLRI attribute cannot
> >      be correctly extracted".
> >
> > The implication is that a failure to extract all NLRI is Critical.
> > Is that a requirement ?
 
> If the NLRI cannot be determined, then this is a Critical error,
> yes. I left the wording relatively open on whether this is *all*
> NLRI, ...

OK.  So, if the message is broken (in any way):

  * if no NLRI can be found, that's a Critical Error.

    That would include the case where some broken
    attribute upstream of any MP_XXX obscures that
    MP_XXX. 

  * if some NLRI can be found, that's a Non-Critical error.

    Such NLRI as can be found are treated-as-withdraw.

    AND any NLRI that may or may not have been lost, are
    ignored.

    AND whatever dangers lost NLRI may pose are covered
    by the caveats in 4.1.

Yes ?

> ... as I am not sure that in the requirements draft we should
> specify direct solutions to specific issues, to e.g., say how to
> handle cases where MP_REACH_NLRI and MP_UNREACH_NLRI are in the same
> message [this is a case that I do not believe is forbidden by
> rfc2858 - if the working group could clarify whether this is
> something that we feel the draft needs to handle or can explicitly
> be omitted, then that would be appreciated].

The RFCs also allow any mix of IPv4 Unicast Reachable/Unreachable in
the body of the message, with or without MP_REACH_NLRI and/or
MP_UNREACH_NLRI.  If we call each of these a "collection of NLRI",
then there may be between 1 and 4 collections of NLRI in a message
(ignoring the End-of-RIB pseudo-UPDATE).  It is an error to have more
than one MP_REACH_NLRI attribute or more than one MP_UNREACH_NLRI
attribute.  It is not clear to me whether those should be Critical
Errors -- but while we are relaxing all error checking, I don't see
why they should be.

It is possible that most implementations actually only send one
collection per message... so, as you suggest, on a you-know-and-I-know
basis, we'all could ignore the "theoretical" issues of more than one
collection in a message.

But, this does not matter if the requirements allow lost NLRI to
simply be ignored.  If the sender sends any MP_XXX first, then there
is little chance that NLRI will be lost, and things work better.  If
the sender follows the old rules, and particularly if it sends more
than one collection at a time, then there is more chance that NLRI
will be lost.

> > But if (3) is non-critical then there is some chance that
> > some NLRI will not be extracted, which appears to violate
> > (1) and (2).

> Disclaimer: As I am sure that my comments previously have made
> clear, I do not maintain a code base for a BGP daemon/implementation
> - so please feel free to correct my logic below.
> 
> I do not believe that (3) implies that the NLRI cannot be correctly
> found.

What (3) implies to me is that somewhere amongst the attributes
something is broken.  As Jakob Heitz succinctly puts it: "Once you
have a malformed update, NOTHING is certain."  In particular, you
cannot be certain that all NLRI referred to by the sender can be found
by the receiver.

> If the sum of total length is incorrect, then we can still
> extract the individual attributes - we just find that there is not
> enough data to fill the overall length we were told and/or we have
> too much attribute data compared to the total attribute length.

This is precisely where I think the requirements come unstuck.  There
is almost no redundancy in the attribute encoding.  And almost all the
redundancy there is is discarded by the new-form-error-handling.  So,
if the sum of the attribute lengths is incorrect, then (inter alia)
the receiver CANNOT know whether the attributes it has extracted are
the attributes the sender intended to send.  In particular, the
receiver cannot know it has extracted all NLRI (except in the,
probably obscure, case of having extracted both MP_REACH_NLRI and
MP_UNREACH_NLRI).  "Once you have a malformed update, NOTHING is
certain."

Conversely, if the sum of attribute lengths is correct, then there is
a fighting chance that the attributes received are the attributes
sent.  But, if one of those attributes is (say) nominally an
ATOMIC_AGGREGATE which is apparently 200 octets long, that might be a
worry.

...
> > Then (4) "In order to maximise the number of cases whereby the
> > NLRI attributes [plural, now, BTW] can be reliably extracted
> > from a received message...".  Ah.  So it is not a Critical
> > Error if "the NLRI attribute cannot be correctly extracted".

> No - it is a Critical error if we cannot extract the NLRI. This
> recommendation is to give an increased chance that the NLRI can be
> extracted as per the IDR error handling draft. This then (by virtue
> of resulting in the NLRI being extracted) minimises the number of
> cases that result in a Critical error.

By this do you mean "all" NLRI or only "some" NLRI ?  As above.

> The plural here is to reflect
> that the existence of >1 type of NLRI attribute.

That would imply that it should be "NLRI attributes" throughout ?
Though in most places where the draft speaks of "NLRI attribute" I
think it means one, some, any or all of the "collections of NLRI" (as
defined above).
 
> > For me the requirement remains "conflicted".  On the one hand it
> > seems to say that it is a Critical Error if the NLRI cannot be
> > extracted and parsed.  On the other it seems to say it's OK if
> > you cannot extract some NLRI.

> If you'll forgive me for removing a significant proportion of your
> message, I think that we need to take another step back here. It
> seems to me that the key question that you are highlighting is "What
> level of confidence do we need to have before we declare that the
> NLRI cannot be extracted?" -- do you agree?

Absolutely.

> From an operator perspective, I would like to compromise *certainty*
> for *robustness*. You are right, we are compromising correctness
> here, we might end up withdrawing an incorrect NLRI and impacting
> service operation for that prefix - however, it is somewhat
> preferable to me to withdraw a a subset of the NLRI incorrectly,
> rather than impact all NLRI in one single action. We clearly need to
> provide some bounds on how much we compromise the certainty (and
> live within the realms of possibility, such that we are not just
> taking a shot in the dark). This is what the definitions of Critical
> and Non-Critical within the document are intended to provide. Once
> again I will refer to the requirement that there is a balance
> between correctness and robustness - rather than a locally risk
> averse approach that results in harmful wider behaviour.

Where one can extract the NLRI from a broken message, then
treat-as-withdraw is AFAICS no worse for the NLRI in question than
session-reset, and hugely better for all the other NLRI received from
the peer.

Where not *all* the NLRI in a message have been extracted, then we
take a step beyond withdrawing something which should not have been
withdrawn.  For each of those NLRI the receiver will (unknowingly) do
one of:

  (a) continue using a route which the sender has
      withdrawn;

  (b) continue to use an out of date version of a route
      which the sender has changed in some way, possibly
      materially;

  (c) to fail to use a new route the sender has now made
      available.

Of these (a) looks serious and (b) could be, but certainly the effect
is different from session-reset; while (c) is, essentially,
treat-as-withdraw.  If some risk "lost routes" is taken, then the
impact of that clearly must be weighed against the impact of
session-reset.  I have not found a discussion of the impact of "lost
routes" in the draft, so I have no idea what the trade-off is, here.

For completeness, if all forms and degrees of attribute broken-ness
are required to be acceptable, then there are cases where the receiver
cannot know whether *all* NLRI have been extracted, or not.

> Is it acceptable that we leave this as guidance within the
> requirements? If not, please could you suggest how the definitions
> of Critical/Non-Critical could be altered to address your concerns?

Without a change at the sender end, the problem is that: "Once you
have a malformed update, NOTHING is certain."

So, as I said earlier, the fundamental question is:

  (F) is it OK to continue with a session after processing
      a message which may have contained some NLRI which
      could not be extracted ?

As you say, the question hinges on the "which may have contained".  As
above, it seems to me that the draft skates over the issue of "lost
routes".

It also hinges on the operational impact of such "lost routes", on
which I am unqualified to pronounce.

It seems to me there is a range of possibilities:

  1) at one end of the spectrum, if "lost routes" are to
     be avoided at all costs, then the rules for parsing
     attributes need to be tight and without a change at
     the sender end, what is achievable is constrained.

  2) if "lost routes" are acceptable in the "theoretical"
     cases, but you-know-and-I-know that most of the
     time the case will not arise...

     ...or "lost routes" are acceptable provided some
     defined steps are taken to identify as much NLRI
     as is reasonably possible...

     ...then the requirements need to (a) make the case
     and (b) describe the trade-off (so that designs
     made to follow the requirements can be judged in
     those terms).

  3) at the other end of the spectrum, if "lost routes"
     are always preferable to session-reset, then this
     opens up an even looser approach to UPDATE message
     handling, in which practically nothing need cause
     a session-reset -- ie, practically nothing is a
     Critical Error.

And, as I have suggested in previous discussions, it is possible to
consider case (2) in two parts:

  a) where the sender is unchanged.

     In which case the issue is the degree of attribute
     broken-ness vs the likelihood of "lost routes".

     If this is viewed as an intermediate step, then
     perhaps the requirements could err on the side of
     safety ?

     Perhaps this can be addressed by a "knob" allowing
     more or less strictness in the parsing of
     attributes ?

  b) where the sender is changed.

     In which case the issue ought to be moot, because
     the receiver will then be able to know that there
     are no "lost routes".

> I would also appreciate further input from IDR as to whether this is
> sufficient requirement from GROW to allow a solution document to be
> written?

The new requirements specify only Critical (session-reset) and
Non-Critical (treat-as-withdraw) Errors.  This appears to rule out the
"attribute discard" option in draft-ietf-idr-error-handling-03.  Is
that intended ?

Chris

---

From: Brian Dickson <[email-masked]>
Date: Fri, 28 Dec 2012 14:02:23 -0500

Here's a quick question, about possible mechanisms for determining whether
or not we need to tear down a session.

If there is a chance that some NLRI weren't properly decoded:

- What about requesting (presuming the option was negotiated) a
route-refresh, or a "confirm per-AFI-SAFI prefix list"?

It's an expensive "parity check" but is one way of ensuring that we haven't
missed a withdrawn prefix.
If it isn't in the subsequently received list of prefixes, we missed it and
should withdraw it.
Everything else is a no-op - maybe we missed a new prefix (with new
attribute?), but that is what "treat as withdraw" gets you.

Just trying to make sure the cure isn't worse than the disease, in all
situations.
(Where "cure" is fail to withdraw because we "lost" the withdrawal because
of malformed update, or "cure" is tear down the session.)

Brian

On Fri, Dec 28, 2012 at 1:04 PM, Chris Hall <[email-masked]>wrote:

> _______________________________________________
> Idr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idr
>

---

From: Jakob Heitz <[email-masked]>
Date: Fri, 28 Dec 2012 21:35:03 +0000

Treat-as-withdraw is not a cure.
Tear down the session is not a cure.

All we can hope for after a malformed update is a
temporary mitigation until human intervention can
fix the problem.

IMO, the goal of error handling is to limit the
damage, not to cure the problem.

--
Jakob Heitz.

________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Brian Dickson
Sent: Friday, December 28, 2012 11:02 AM
To: Chris Hall
Cc: [email-masked]; [email-masked]
Subject: Re: [GROW] [Idr] Fwd: I-D Action: draft-ietf-grow-ops-reqs-for-bgp-error-handling-06.txt

Here's a quick question, about possible mechanisms for determining whether or not we need to tear down a session.

- What about requesting (presuming the option was negotiated) a route-refresh, or a "confirm per-AFI-SAFI prefix list"?

It's an expensive "parity check" but is one way of ensuring that we haven't missed a withdrawn prefix.
If it isn't in the subsequently received list of prefixes, we missed it and should withdraw it.
Everything else is a no-op - maybe we missed a new prefix (with new attribute?), but that is what "treat as withdraw" gets you.

Just trying to make sure the cure isn't worse than the disease, in all situations.
(Where "cure" is fail to withdraw because we "lost" the withdrawal because of malformed update, or "cure" is tear down the session.)

Brian

On Fri, Dec 28, 2012 at 1:04 PM, Chris Hall <[email-masked]<mailto:[email-masked]>> wrote:

Yes ?

...

Absolutely.

Chris

_______________________________________________
Idr mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/idr

---

From: "Chris Hall" <[email-masked]>
Date: Sat, 29 Dec 2012 14:21:01 -0000

Brian Dickson wrote (on Fri 28-Dec-2012 at 19:02 +0000):
>  - What about requesting (presuming the option was negotiated)
>    a route-refresh, or a "confirm per-AFI-SAFI prefix list"?

If the sender is sending broken updates, then a route-refresh might
repeat the error.  But that doesn't make things any worse.  In fact,
since one would only keep those routes which the route-refresh sent
correctly, that would improve things :-)  The draft (section 5) talks
about Graceful Restart and extensions thereof, but if those are not
available, Route-Refresh might be.

Is Route-Refresh a sufficiently common option that one can stop trying
to square the "Once you have a malformed update, NOTHING is certain"
circle ?

It might be nice to only refresh/confirm the affected AFI/SAFI... but
if MP_XXX attributes have gone walkabout, it may not be clear which
AFI/SAFI those are :-(

Chris

---

From: Rob Shakir <[email-masked]>
Date: Sun, 30 Dec 2012 00:05:08 +0000

Hi Brian,

We discussed this at some length previously (the -00 of the requirements draft discussed automatically triggering some form of means to recover from the error). I think that there is some merit in doing so, but there are clearly also scaling considerations around this (in terms of requiring both BGP speakers to re-generate/re-process UPDATEs). 

The consensus that was reached in the previous discussion was that this is not necessarily going to be of advantage in terms of causing a different UPDATE message to be generated. Particularly, since a ROUTE REFRESH would result in the same UPDATE packing (generally). To that end, what the requirements draft currently says is that:

- ROUTE-REFRESH is a reasonable way to re-run a consistency check between two peers (especially with start-of-refresh and end-of-refresh markers, since this means that we can purge any NLRI that we missed being withdrawn).
- If there is anything automatically triggered, it would be prudent to try and ensure that we can make as specific a request to the other speaker as possible - this is where mechanisms such as One-Time ORF were suggested, and is a similar space to that which would be solved through having the UPDATE-VERSION message described in draft-ietf-idr-enhanced-gr if one were to be able to have a REFRESH following a "last known good" position. Making a more specific request may result in a different UPDATE being generated, which may mean the error is fixed, but this is not guaranteed.
- If the selected best path changes on the remote speaker, then it will be re-advertised anyway - resulting in another chance to re-parse these NLRI.

So, certainly, if we relax the "correctness" requirement, then it would seem a good idea to have some means to be able to consistency check -- it's just a matter of ensuring that this is done in an effective and scalable manner, that does not affect normal BGP operation.

Cheers,
r.

On 28 Dec 2012, at 19:02, Brian Dickson <[email-masked]> wrote:

> On Fri, Dec 28, 2012 at 1:04 PM, Chris Hall <[email-masked]> wrote:

---

From: Brian Dickson <[email-masked]>
Date: Sun, 30 Dec 2012 16:34:37 -0500

Here's the basic problem in a nutshell:

If a WITHDRAW is missed (either standard or MP), there are two
possibilities concerning withdrawn NLRIs:
- the WITHDRAW was sent because the sender no longer HAS a route to the
destination, or
- the WITHDRAW was sent because the sender now prefers the route via the
recipient

In the former case, losing the WITHDRAW means that the recipient, if it
selects that route as BEST, will now BLACKHOLE the traffic.
In the latter case, it might be possible for a routing loop to be caused
(although, I think most of the time this won't be possible unless there was
already a missed WITHDRAW in the other direction).

But, the basic problem is this: missing an UPDATE won't trigger either
condition, it will at worst cause sub-optimal routing. Missing a WITHDRAW
_CAN_ cause Bad Things (TM) to happen.

So, it is important to accommodate this in the requirements, possibly
requiring a very slight change in required message packing, in order to
guarantee that these Bad Things do not happen.

Today, those Bad Things can't happen when a session tear-down occurs.

There are a couple of easy ways to ensure the WITHDRAW is not missed:
- Pack all the WITHDRAW stuff at the start
- Require that no UPDATE be sent in the same MESSAGE as a WITHDRAW, and
preferably limit WITHDRAW to one AFI/SAFI per MESSAGE

It's clean, has relatively modest requirements, and ensures that a broken
WITHDRAW can trigger the correct result (tear-down), and any other broken
UPDATE can be handled with Treat-As-Withdraw.

IMNSHO, routing loops and/or black holes (for even a single prefix) are not
an acceptable situation.

There are other approaches that can be take to avoid this as well, but I
think KISS is the best approach.

Brian

On Sat, Dec 29, 2012 at 7:05 PM, Rob Shakir <[email-masked]> wrote:

> Hi Brian,
>
> We discussed this at some length previously (the -00 of the requirements
> draft discussed automatically triggering some form of means to recover from
> the error). I think that there is some merit in doing so, but there are
> clearly also scaling considerations around this (in terms of requiring both
> BGP speakers to re-generate/re-process UPDATEs).
>
> The consensus that was reached in the previous discussion was that this is
> not necessarily going to be of advantage in terms of causing a different
> UPDATE message to be generated. Particularly, since a ROUTE REFRESH would
> result in the same UPDATE packing (generally). To that end, what the
> requirements draft currently says is that:
>
> - ROUTE-REFRESH is a reasonable way to re-run a consistency check between
> two peers (especially with start-of-refresh and end-of-refresh markers,
> since this means that we can purge any NLRI that we missed being withdrawn).
> - If there is anything automatically triggered, it would be prudent to try
> and ensure that we can make as specific a request to the other speaker as
> possible - this is where mechanisms such as One-Time ORF were suggested,
> and is a similar space to that which would be solved through having the
> UPDATE-VERSION message described in draft-ietf-idr-enhanced-gr if one were
> to be able to have a REFRESH following a "last known good" position. Making
> a more specific request may result in a different UPDATE being generated,
> which may mean the error is fixed, but this is not guaranteed.
> - If the selected best path changes on the remote speaker, then it will be
> re-advertised anyway - resulting in another chance to re-parse these NLRI.
>
> So, certainly, if we relax the "correctness" requirement, then it would
> seem a good idea to have some means to be able to consistency check -- it's
> just a matter of ensuring that this is done in an effective and scalable
> manner, that does not affect normal BGP operation.
>
> Cheers,
> r.
>
>

---

From: John Leslie <[email-masked]>
Date: Sun, 30 Dec 2012 22:09:25 -0500

> _CAN_ cause Bad Things (TM) to happen.

   +1

--
John Leslie <[email-masked]>

---

From: Jeff Wheeler <[email-masked]>
Date: Sun, 30 Dec 2012 23:02:41 -0500

> preferably limit WITHDRAW to one AFI/SAFI per MESSAGE

The change you are proposing is sensible and has minimal cost to the
control-plane CPU.

BGP already can only have one, two, or three AFI/SAFI per Message.
The reason it's "one, two, or three" is there can be native NLRI and
MP-NLRI in one Message.

You are not allowed to send more than one Attribute of the same Type
Code in a BGP Message.  There can't be two MP_REACH_NLRI or two
MP_UNREACH_NLRI.

The rule about one attribute per type code is kind of buried in
RFC4271 section 6.3 paragraph 14.  I imagine not everyone is aware of
this because it really is not written into the spec in a way that it
jumps out at you!

Here is how this plus MP-BGP combine to allow up to three NLRI in the
same Message:
1) native NLRI which are either reachable or withdrawn
2) NLRI inside an MP_REACH_NLRI Attribute
3) NLRI inside an MP_UNREACH_NLRI Attribute

All three of those could possibly have distinct AFI/SAFI today.  But
it's not like it is possible to have NLRIs for 10 different AFIs in
one Message today.

I'm sure this discussion is related to draft-ietf-idr-error-handling.
This draft does not call for a BGP Capability Code.  It needs one if
it expects the neighbor to behave in a certain way which differs from
existing BGP requirements.  I have posted some discussion about this
on the IDR list recently but I do not read GROW.

-- 
Jeff S Wheeler <[email-masked]>
Sr Network Operator  /  Innovative Network Concepts

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 31 Dec 2012 10:21:00 +0100

How about if we would mandate Enhance Route Refresh request to be send
to such peer who's bad updates triggered treat-as-withdraw action ?

If we resync databases OK the likehood of "bad things" should be
minimal. If we fail to sync (get another malformed update(s)) then
IMHO resetting the session should be granted.

Yes that means that "treat-as-withdraw" should be applied only to
those peers which support Enhanced Route Refresh.

Happy New Year,
R.

> On Mon, Dec 31, 2012 at 4:09 AM, John Leslie <[email-masked]> wrote:
>

---

From: Rob Shakir <[email-masked]>
Date: Mon, 31 Dec 2012 10:52:08 +0000

On 30 Dec 2012, at 21:34, Brian Dickson wrote:

> But, the basic problem is this: missing an UPDATE won't trigger either condition, it will at worst cause sub-optimal routing. Missing a WITHDRAW _CAN_ cause Bad Things (TM) to happen.

If we tear down the session based on a single bad UPDATE, then Bad Things™ happen. Worse still, these bad things happen to  NLRI that are not associated with the UPDATE that was erroneous and were supporting live customer services. With modern deployments of BGP, the sessions that are torn down can be relatively fundamental to network operation, carry multiple (unrelated) customer topologies, and have significant recovery times. Based on this, the tear-down behaviour has been shown to have harmful effects to the operation of real networks based on real world errors [0], and as per the draft that this thread is discussing. There are a number of deployments in which this is not acceptable.

We're spinning round the same loop here that was discussed a number of times. Let me try and summarise this as accurately as I  can:
 
As soon as we accept any kind of mechanism that targets error handling to particular NLRI, we KNOW that we will compromise correctness. (As per Section 4.1 of the draft) We also KNOW that this has a cost - which is that we cannot be sure that the routing system is entirely consistent any more, and we might end up with stale NLRI, routing loops, or blackholing. The operational requirement described in this draft highlight that this _IS_ the cost of robustness, and highlight the need for operational monitoring and recovery mechanisms to support this relaxation. Further to this, they highlight ways that the current session-reset behaviour can be improved, particularly relevant in cases where we CANNOT apply more targeted error handling.

We are discussing whether we can wholly address the "when there has been an error in an UPDATE, NOTHING is certain" - I would assert that we do not need to. The fundamental premise here is that we are compromising protocol correctness, which has a cost in terms consistency. Operationally, paying this cost in order to keep numerous other services up and running is acceptable. If this cost is not acceptable to an operator, then such mechanisms SHOULD NOT be deployed.

There is an argument that says that we could use a modified session-reset based on NOTIFICATION after erroneous UPDATE approach for all errors, however, since we also have a concern around longer-lived errors (that are sourced by a remote speaker, and hence just repeated on session reset), and the scalability of BGP routers, optimising such that where it is possible to do so, we make minimal compromises to correctness (i.e.., we compromise the correctness of a limited subset of NLRI, where we can identify them) and maintain sessions is a key requirement.

If there are ways that we can improve the message packing in BGP such that we minimise the risk of compromising correctness with NLRI-targeted error handling, absolutely we should try and pursue these in draft-ietf-idr-error-handling - however, I think that there is consensus that error handling as a problem space is something we need to address, since there are work items for this in both IDR and GROW.

Would the GROW working group be happy if we address Chris' concern related to "lost NLRI" (which AFAICS is really the case where we have >1 type of NLRI attribute within a single message) by adding a note that an error remains Non-Critical if _at least one_ NLRI attributes can be successfully parsed? I'm unclear here as to whether we're addressing a common situation where multiple sets of NLRI are being contained within a single message [1]. If so, then adding "at least one" and then a further point that an implementation SHOULD use a single NLRI attribute per UPDATE message, and put this at the start of the attributes would seem to be a fair way forward.

Does this sound a reasonable way forward? If so, I will update this in an -07, as well as addressing the editorial issues that Chris highlighted earlier.

Many thanks for your feedback.

HNY!
r.

[0]: I enumerated a small number of real-world incidents back in 2011 when I presented this work at NANOG (slides: http://rob.sh/files/nanog-slides.pdf). There have been numerous incidents since, alongside many incidents that the operators involved were not willing to share more public details of. It would be a massive shame to me that when the inevitable next incident like this turns up in the DFZ, or internally to a SP network, we are still questioning the validity of the problem space.

[1]: Trawling a number PCAPs that I have, I can't find an UPDATE that contained MP_REACH_NLRI and MP_UNREACH_NLRI simultaneously from a multi-vendor L3VPN network. Another operator informs me that they have a certain implementation that appears to include both in a single message.

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 31 Dec 2012 13:49:41 +0100

Hi Rob,

> we are still questioning the validity of the problem space.

I am not sure where do you draw such impression from ? I think we are
all clear that there is problem .. from time to time a bleeding wound
appears.

However it seems that there is serious concern if the cure proposed is
the right one. It appears that the treat-as-withdraw is a piece of
duck tape so the rest of the body can sort of function. One could
argue that while for some this may even work if the wound is small,
imagine such duck tape being applied on 1 meter wound ? Moreover such
cut will not recover under duck tape.

So to me it seems this is a valid debate if it is better to reset the
guy and send him to hospital for proper bandage or stitching while
counting on the co-pilot(s) (other BGP sessions) to seamlessly carry
the services for customers in the mean time.

I think your draft and even the proposed solution is the best for the
current BGP4 transport of messages for the cases where some bug is
remote and you would likely end-up resetting perhaps all of your
upstream sessions .. no doubt. But we have no way of knowing that
network wide and I think this one of the problems. Clearly in this
case it is better to put even duck tape on the pilots so they can try
to land rather then sit and watch the plane crash.

Actually real life example:

I have discovered last week in one of BGP implementation that BGP
conditional advertisement will not kick in till the trigger  (check
condition) disappears completely from BGP table even if it's next hop
is no longer present in global RIB. One could call this a bug, but it
is one example where while today I can automatically prefer other exit
when session goes down with treat-as-withdraw and keep-the-session-up
this will no longer work as trigger route will happily continue to sit
in BGP table. And with treat-as-withdraw being default in some cases
after new software upgrade your expected BGP behavior will change. I
doubt you will find this example described in the release notes too ;)

Best,
R.

---

From: Rob Shakir <[email-masked]>
Date: Mon, 31 Dec 2012 15:46:42 +0000

Hi Robert,

> to land rather then sit and watch the plane crash.

The previous discussions on this subject would seem to indicate that there are a significant number of cases where treat-as-withdraw is seen as the best approach for both non-Internet and Internet deployments. In my view, the presence of other paths for particular NLRI does not necessarily imply that they are equally viable (for both capacity and cost reasons) - so maintaining as much of the network in the normal operating state seems the most beneficial approach. To do this we need some form of NLRI-level error handling.

There are undoubtedly some deployment scenarios where it is not what the operator wants for their network - and this may be the case for your example(s). The requirements draft is not asserting that this is a MUST and should be the default -- but rather outlining the motivation for those operators who want error handling that applies to particular NLRI, as well as the tools that are needed to make such error handling deployable. If your preference is to err more on the side of caution (as an operator, or an implementor) then there is no reason that one could not always just have session-level error handling, with Enhanced GR deployed within your network.

I think we need to have both approaches - and note the caveats around each. At the moment, the approach of having no choice for operators is concerning to me.

Kind regards,
r.

---

From: Jeff Wheeler <[email-masked]>
Date: Mon, 31 Dec 2012 11:59:15 -0500

On Mon, Dec 31, 2012 at 5:52 AM, Rob Shakir <[email-masked]> wrote:
> If we tear down the session based on a single bad UPDATE, then Bad Things™ happen.

This is why I'm confused why you aren't more interested in giving the
vendor (ultimately, the operator) the choice to simply ignore bad
updates.

You know that error-handling already leaves the RIB (network) in a bad
state.  You might as well give the option to reset the session under
even fewer circumstances.

---

From: Rob Shakir <[email-masked]>
Date: Mon, 31 Dec 2012 17:33:30 +0000

> even fewer circumstances.

This is, of course, another route that we could go down, and has been discussed previously. If we receive an UPDATE we know that *something* to do with this path has changed, and if this UPDATE is erroneous know that *something* about the signalling path is not correct. If we do have other paths to that particular prefix, then treat-as-withdraw lets us use these. If we simply ignore the UPDATE, then we trust this path that has something suspect about it just as much as those that we have not seen any error in the messaging for.

Cheers,
r.

---

From: Jakob Heitz <[email-masked]>
Date: Mon, 31 Dec 2012 17:54:59 +0000

I don't think treat-as-withdraw is trying to fix a single session reset. Graceful restart can fix that. It's the rolling resets that need a human to remove a buggy router or a config that triggered the bug. That takes several hours. Treat-as-withdraw limits the damage during those hours.

Could we please settle on that without trying to solve the impossible?

Cheers,
Jakob

Sent from my iPoo.

On Dec 31, 2012, at 1:21 AM, "Robert Raszuk" <[email-masked]> wrote:

> GROW mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/grow

---

From: Jeff Wheeler <[email-masked]>
Date: Mon, 31 Dec 2012 13:13:51 -0500

> Could we please settle on that without trying to solve the impossible?

If you read my posts to IDR on this topic, you'll see where I explain
how it is possible to solve the "impossible."

Specifically, you can ignore just about any bad update, or bad message
of any kind, as long as you can figure out where the next message
starts.  This fixes the rolling resets.

You may know that a lot of businesses suffered multi-hour outages in
October simply because of 5 DFZ routes announced by LANL that had
illegal attributes.  This is very hard for operators to troubleshoot
on most routers.  The routers that experienced rolling resets were
buggy but if the operators simply had a panic button, "ignore bad
messages," their networks would have been up and they would not have
been losing money by the minute.

This is not the only time bad updates have propagated through the DFZ
and caused big problems.  It has happened repeatedly.

I believe it will begin to happen more often inside datacenter
networks, because BGP is being used for more and more things, like
EVPN.  Operators are going to need BGP to become more robust.

You can make it a lot more robust just by deciding to ignore
everything in a bad message.  This is not good, but it is a lot better
than session-reset, in most cases.

Please, read my posts on this topic, and do not treat this problem as
an unsolvable one.  It can be largely solved in a way that gives a
very useful fallback option to operators.

This whole draft is about fallback options, and it is pretty stupid to
have a large amount of complexity to solve a small set of potential
bugs, when you could ALTERNATIVELY or IN ADDITION to that, have a very
low-complexity option that solves more problems.

---

From: Jakob Heitz <[email-masked]>
Date: Mon, 31 Dec 2012 18:15:32 +0000

On Dec 31, 2012, at 4:49 AM, "Robert Raszuk" <[email-masked]> wrote:
> is no longer present in global RIB. One could call this a bug,

Before we even consider this case, perhaps someone could update (and explain) rfc4271 to remove the following sentence from page 76:

The function that calculates the degree of preference for a given route SHALL NOT use any of the following as its inputs: the existence of other routes, the non-existence of other routes, or the path attributes of other routes.

Cheers,
Jakob.

---

From: Jakob Heitz <[email-masked]>
Date: Mon, 31 Dec 2012 18:35:42 +0000

The difference between ignore and treat-as-withdraw is discussed in the draft.
Anyway, the feature will be knobbed. So far, I see 2 knob options:
1. Treat-as-withdraw / ignore
2. If stream sync is lost, search ahead for the next 16*0xFF / reset the session.

Cheers,
Jakob

---

From: John Leslie <[email-masked]>
Date: Mon, 31 Dec 2012 13:41:38 -0500

Jakob Heitz <[email-masked]> wrote:
>>>> condition, it will at worst cause sub-optimal routing.

   I apologize for careless reading here. I took Brian to mean failing
to process an UPDATE which changed the NLRI from a peer, but didn't
change the _reachability_ of that block from that peer, breaks things
but mildly.

>>>> Missing a WITHDRAW _CAN_ cause Bad Things (TM) to happen.
>>> 
>>>   +1

   I meant to +1 the statement that missing the notice that an NLRI
was being withdrawn and not replaced (thereby changing reachability
from positive to negative) is a much more serious breakage, since
you may keep sending packets to that peer which no longer has the
reachability you think it does.

>> to such peer who's bad updates triggered treat-as-withdraw action ?

   A Route-Refresh cycle would ensure that we have the actual
reachability status of that peer (if the Route-Refresh completes)
instead of some outdated NLRI the peer intended to withdraw without
replacement.

>> those peers which support Enhanced Route Refresh.

   We're discussing an area where "treat-as-withdraw" may no longer
be a useful term, since we're considering an UPDATE so badly formed
that we can't even extract which CIDR blocks are the subject of it.

   Personally, I'd prefer a Route-Refresh approach, where we make it
perfectly clear to the peer in question that we no longer have a
trustworthy state of its current advertised routing.

> I don't think treat-as-withdraw is trying to fix a single session reset.
> Graceful restart can fix that. It's the rolling resets that need a
> human to remove a buggy router or a config that triggered the bug.
> That takes several hours. Treat-as-withdraw limits the damage during
> those hours.

   I agree with Jakob that we're _trying_ to discuss that -- but I
don't agree we're succeeding very well.

> Could we please settle on that without trying to solve the impossible?

   I'm not sure we can...

   First of all, most of us _aren't_ trying to solve the impossible:
we're trying to design an error-management paradigm which passes _enough_
information on which to base intelligent routing decisions.

   And, I'm not sure what it means to "settle on" "treat-as-withdraw"
that some of us consider "good enough" for a "limited" period while
humans are working to resolve the actual problem. We know that the
"limits" on such a period correlate all to well with the amount of
pain being inflicted on paying customers.

---

From: Jeff Wheeler <[email-masked]>
Date: Mon, 31 Dec 2012 14:59:20 -0500

> pain being inflicted on paying customers.

It means just that, "settle on."  Treat-As-Withdraw leaves plenty of
circumstances where BGP will still be reset, perhaps endlessly until
clue arrives.

Let me just go through the explanation of what happened to many
networks in October 2012 again.  LANL announced some malformed routes
to the DFZ.  Most routers propagated them and ignored the malformed
bits.  Some routers did not, and instead, they reset the BGP session.
That was a bug, but the customers had NO OPTIONS TO WORK AROUND THIS.
Their ONLY CHOICE was to wait for a patch from their vendor (for folks
whose vendor routers did this) or install a patch (for the lucky users
who were running OpenBSD) or to ask their transit provider to install
a prefix-list to block the 5 malformed paths from going to them.

If customers had treat-as-withdraw in the form being discussed today,
it would not have helped.  Why?  Malformed Attribute Flags are not one
of the criteria it addresses.

What networks need is an IGNORE BAD MESSAGES option so BGP can avoid
session-reset, even if the cost is rather high (say a lot of RIB
inconsistency.)  Let the network operator choose when the cost of bad
RIB information becomes so high that he would rather have BGP reset.
After all, this will vary from one network to the next based on
business needs, so there should be flexibility.

That is why treat-as-withdraw is "settling."  Ignore bad messages is a
very simple option that "fixes" (mitigates) virtually all problems so
the operator can buy time to diagnose it in detail, or wait for his
vendor to provide a software update, or whatever.

I'm not arguing this should be the only option or even the default
option.  I'm arguing that it should be ONE OPTION.  As far as I'm
concerned, you can have treat-as-withdraw, and you can build as much
complicated error handling logic into it as you want.  That effort
might help me someday.  But if it doesn't, I know for sure that IGNORE
BAD MESSAGES is something I can fall back on, which is very likely to
buy me the time I need.

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 31 Dec 2012 21:15:02 +0100

Hi Jeff,

> whose vendor routers did this)

+

> vendor to provide a software update, or whatever.

I think you have just hit the nail on it's head above.

I think waiting for vendor's fix is just not acceptable for production
environments .. it just takes too long [read days if not weeks] (not
to mention that a lot of deployed hardware requires reload when you
upgrade).

When we originally discussed error handling solutions in junisco one
option was to allow operator to apply filter to BGP messages between
TCP and BGP. If the error is known it is pretty trivial to match on
arbitrary string in such BGP pre-parser and filter it out or clear or
reset the bit/bits if NOC decides it is operationally safe.

The point is that for critical protocol like BGP4 (with current choice
of transport) I am of the opinion that we should build a much more
universal mechanism rather then add fixed twicks which may or may not
always help and when operator has zero control on what such "session
saver" twick actually does.

Cheers,
R.

---

From: Jakob Heitz <[email-masked]>
Date: Mon, 31 Dec 2012 20:24:30 +0000

On , Jeff Wheeler <> wrote:

> Malformed Attribute Flags are not
> one of the criteria it addresses. 

http://tools.ietf.org/html/draft-ietf-idr-error-handling-03
says

   When a path attribute (other than the MP_REACH_NLRI attribute
   [RFC4760] or the MP_UNREACH_NLRI attribute [RFC4760]) in an UPDATE
   message is determined to be malformed, the UPDATE message containing
   that attribute MUST be treated as though all contained routes had
   been withdrawn...
   ...In the case of an
   attribute which has no effect on route selection or installation, the
   malformed attribute MAY instead be discarded...

That covers malformed attribute flags.

-- 
Jakob Heitz.

---

From: Jeff Wheeler <[email-masked]>
Date: Mon, 31 Dec 2012 15:36:05 -0500

> That covers malformed attribute flags.

Except in the MP_REACH_NLRI or MP_UNREACH_NLRI, which is not a foolish
exception if the goal is to recover all the NLRI; but if the goal is
to avoid session-reset, it is.

Like I keep saying, the goal of error-handling should be to make
session-reset avoidable.  Today it is not.  It is creating complex
rules which must be supported by both sides of the BGP session to try
very hard to keep the network in a good state.

This is a good goal in itself, but not to the exclusion of avoiding
session-reset.

Again, the existing draft also ignores conflicts with RFC4271.  So do
the updates to RFC4760.  You are not allowed to put attributes into
the attribute list in any order you want.  RFC4271 explicitly requires
that attributes be sorted by Attribute Type Code.  A Capability
negotiation is necessary to change the ordering of attributes.

Finally, because the current error-handling concept can't be
implemented without such a capability, it's not useful if the other
BGP speaker lacks support.  That is why IGNORE BAD MESSAGES is needed.
 It doesn't require support on both sides.