Subject: [Idr] draft-ietf-idr-error-handling

From: <[email-masked]>
Date: Thu, 12 Jan 2012 14:56:30 +0000

Hi all,

The draft currently does not discuss nor address how to recover from error i.e. for the BGP receiver to get back all the NLRI / attributes after it has withdrawn/discarded some. IMHO it should as the current mechanism it is replacing (session reset) address both error handling and error recovery.
In order to initiate discussion on this, may I suggest the following text:

When the approach "treat-as-withdraw" or "attribute discard" has been used on a BGP session, the BGP receiver has not the full set of attributes or NLRI that the BGP speaker sent. There MUST be a way to latter recover from this inconsistency.

In some cases, this inconsistency will naturally be resolved. As the error is fixed on the speaker or on an upstream speaker (possibly the originator of the route), the BGP attribute in error will be updated and a new UPDATE message will be propagated on downstream BGP routers. If the error has indeed been fixed, the new BGP UPDATE message will be accepted and the dropped attribute or withdrawn NLRI will be learnt again.

To cover remaining cases, an automatic recovery mechanism SHOULD be implemented. The use of the ROUTE-REFRESH message is suggested but others non traffic disruptive mechanisms may be used. This recovery mechanism SHOULD be initiated after a configurable delay. If the mechanism failed to recover all NLRI or attributes in error, subsequent tentative SHOULD be initiated after a timer. Timer value SHOULD be carefully defined to allow for both a reasonably quick recovery and a reasonable impact on the BGP control plane load. The use of a (exponential) backof algorithm may be considered. The maximum value should be high enough (e.g. days) in order for the mechanism to keep trying the recovery possibly for ever while having a low impact on the BGP control plane. Timer MAY be on a per BGP session basis or MAY be global to the BGP speaker.

If an implementation chooses not the implement an automatic recovery mechanism, when or prior to configure this revised error handling, network operator MUST be made aware that the recovery will require a manual action (typically a session (soft) reset).

Thanks,
Regards,
Bruno

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
France Telecom - Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorization.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, France Telecom - Orange shall not be liable if this message was modified, changed or falsified.
Thank you.

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 12 Jan 2012 19:04:48 +0100

Hi Bruno,

The recovery you are proposing is a valid concern. However what you are 
proposing will be just local across given ebgp session while the error 
as many have pointed out on this topic before may have been originated 
far away.

In such case periodic refresh requests or even soft clear will just 
result in continued reception of bad attributes and their local drops 
and it will not help to recover at all.

The session drop had the nice property of raising red alarms on NOC 
screens. It normally (other then in some implementations periodic 
reestablishment attempts) did not correct the problem.

So in the "treat-as-withdraw" cases perhaps what needs to happen the 
same level of red alarms will be seen by NOCs and either manual or 
automated triggered scripting will attempt to recover from the problem.

Therefor I am not sure if adding any mechanism for recovery to the draft 
is required. Worse it may even further be received by some NOCs as sort 
of let the network fix itself message.

Best regards,
R.

> Hi all,
>
> The draft currently does not discuss nor address how to recover from
> error i.e. for the BGP receiver to get back all the NLRI / attributes
> after it has withdrawn/discarded some. IMHO it should as the current
> mechanism it is replacing (session reset) address both error handling
> and error recovery. In order to initiate discussion on this, may I
> suggest the following text:
>
>
> When the approach "treat-as-withdraw" or "attribute discard" has been
> used on a BGP session, the BGP receiver has not the full set of
> attributes or NLRI that the BGP speaker sent. There MUST be a way to
> latter recover from this inconsistency.
>
> In some cases, this inconsistency will naturally be resolved. As the
> error is fixed on the speaker or on an upstream speaker (possibly the
> originator of the route), the BGP attribute in error will be updated
> and a new UPDATE message will be propagated on downstream BGP
> routers. If the error has indeed been fixed, the new BGP UPDATE
> message will be accepted and the dropped attribute or withdrawn NLRI
> will be learnt again.
>
> To cover remaining cases, an automatic recovery mechanism SHOULD be
> implemented. The use of the ROUTE-REFRESH message is suggested but
> others non traffic disruptive mechanisms may be used. This recovery
> mechanism SHOULD be initiated after a configurable delay. If the
> mechanism failed to recover all NLRI or attributes in error,
> subsequent tentative SHOULD be initiated after a timer. Timer value
> SHOULD be carefully defined to allow for both a reasonably quick
> recovery and a reasonable impact on the BGP control plane load. The
> use of a (exponential) backof algorithm may be considered. The
> maximum value should be high enough (e.g. days) in order for the
> mechanism to keep trying the recovery possibly for ever while having
> a low impact on the BGP control plane. Timer MAY be on a per BGP
> session basis or MAY be global to the BGP speaker.
>
> If an implementation chooses not the implement an automatic recovery
> mechanism, when or prior to configure this revised error handling,
> network operator MUST be made aware that the recovery will require a
> manual action (typically a session (soft) reset).
>
>
> Thanks, Regards, Bruno
>
>
>  Ce message et ses pieces jointes peuvent contenir des informations
> confidentielles ou privilegiees et ne doivent donc pas etre diffuses,
> exploites ou copies sans autorisation. Si vous avez recu ce message
> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi
> que les pieces jointes. Les messages electroniques etant susceptibles
> d'alteration, France Telecom - Orange decline toute responsabilite si
> ce message a ete altere, deforme ou falsifie. Merci
>
> This message and its attachments may contain confidential or
> privileged information that may be protected by law; they should not
> be distributed, used or copied without authorization. If you have
> received this email in error, please notify the sender and delete
> this message and its attachments. As emails may be altered, France
> Telecom - Orange shall not be liable if this message was modified,
> changed or falsified. Thank you.
>
> _______________________________________________ Idr mailing list
> [email-masked] https://www.ietf.org/mailman/listinfo/idr
>
>

---

From: Jakob Heitz <[email-masked]>
Date: Thu, 12 Jan 2012 14:13:35 -0500

One thing is clear: the originator of the malformed message
needs manual intervention. No automatic fix is possible.

Question is: once the originator is fixed, will the corrected
updates propagate properly throughout the Internet?

--
Jakob Heitz.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Robert Raszuk
Sent: Thursday, January 12, 2012 10:05 AM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [Idr] draft-ietf-idr-error-handling

Hi Bruno,

Best regards,
R.

> manual action (typically a session (soft) reset).

_______________________________________________
Idr mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/idr

---

From: <[email-masked]>
Date: Fri, 13 Jan 2012 14:59:38 +0100

Hi,

I agree that a route-refresh may not correct the issue. Based on our experience, sometimes it works (not really often :) ), sometimes not. In this case, as you mentionned Jakob, only manual action on originator would fix the issue.
If we decide to implement a route-refresh to correct the issue, we need to care on not doing an infinite loop in case of persistent error. I mean : receiving a bad attribute, sending a rr,receiving same bad attribute, sending a rr  , receiving same bad attribute, sending a rr ... If people agree on implementing such correction mechanism, I would propose to only try to fix one time, and not going into a loop.
When a correct update (after fix) is received from originator, it should be propagated correctly by peers through the Internet. I don't see why this could cause an issue ...

Still a  question on informing NOCs about the issue, do we agree that only BGP speaker that detect the issue will inform "locally" the NOC (using syslog, SNMP or other monitoring mechanism ...), so only the service provider that detected the issue is informed about the issue , others are receiving withdraws or path update (if another path is available in local provider where the error occurred) or path with some stripped attributes ? Or do you think it could be interresting to inform peers (in a transitive way) that an error occurred ? There was already some topics on this, but I can't see any clear position on this (but maybe I missed it).

Regards,

--- 
Stephane

-----Message d'origine-----
De : [email-masked] [mailto:[email-masked]] De la part de Jakob Heitz
Envoyé : jeudi 12 janvier 2012 20:14
À : [email-masked]; DECRAENE Bruno RD-CORE
Cc : [email-masked]
Objet : Re: [Idr] draft-ietf-idr-error-handling

One thing is clear: the originator of the malformed message needs manual intervention. No automatic fix is possible.

Question is: once the originator is fixed, will the corrected updates propagate properly throughout the Internet?

--
Jakob Heitz.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Robert Raszuk
Sent: Thursday, January 12, 2012 10:05 AM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [Idr] draft-ietf-idr-error-handling

Hi Bruno,

The recovery you are proposing is a valid concern. However what you are proposing will be just local across given ebgp session while the error as many have pointed out on this topic before may have been originated far away.

In such case periodic refresh requests or even soft clear will just result in continued reception of bad attributes and their local drops and it will not help to recover at all.

The session drop had the nice property of raising red alarms on NOC screens. It normally (other then in some implementations periodic reestablishment attempts) did not correct the problem.

So in the "treat-as-withdraw" cases perhaps what needs to happen the same level of red alarms will be seen by NOCs and either manual or automated triggered scripting will attempt to recover from the problem.

Therefor I am not sure if adding any mechanism for recovery to the draft is required. Worse it may even further be received by some NOCs as sort of let the network fix itself message.

Best regards,
R.

> and a new UPDATE message will be propagated on downstream BGP routers. 
> If the error has indeed been fixed, the new BGP UPDATE message will be 
> accepted and the dropped attribute or withdrawn NLRI will be learnt 
> again.
>
> use of a (exponential) backof algorithm may be considered. The maximum 
> value should be high enough (e.g. days) in order for the mechanism to 
> keep trying the recovery possibly for ever while having a low impact 
> on the BGP control plane. Timer MAY be on a per BGP session basis or 
> MAY be global to the BGP speaker.
>
> ______________________________________________________________________
> ___________________________________________________
>
> received this email in error, please notify the sender and delete this 
> message and its attachments. As emails may be altered, France Telecom 
> - Orange shall not be liable if this message was modified, changed or 
> falsified. Thank you.
>

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 13 Jan 2012 10:07:59 -0500

On Fri, Jan 13, 2012 at 02:59:38PM +0100, [email-masked] wrote:
> If we decide to implement a route-refresh to correct the issue, we need to care on not doing an infinite loop in case of persistent error.

Never mind the fact that route refresh is a very large and expensive hammer.
"I received one Bad route from you.  Mind sending me 349,999 of the good
ones again so I can see if the Bad route is better this time around?"

Mechanisms that request the same prefix again are potentially better, but
not be well received by operators.  (This opinion is under the "don't let
people ask me for stuff" operational umbrella.  Some operators really prefer
a push-only model.)

-- Jeff

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 13 Jan 2012 16:21:43 +0100

Jeff,

> ones again so I can see if the Bad route is better this time around?"

I would rather see draft-zeng-idr-one-time-prefix-orf-01 which is I 
think what you are referring to below.

> people ask me for stuff" operational umbrella.  Some operators really prefer

Such operators are free to disable functionality they do not appreciate. 
However I am not aware about any operator who forcefully disables route 
refresh capability on their BGP sessions.

And those who disable route refresh would be subject to session reset 
anyway as even soft clear would not be possible therefor it seems 
obvious that they are least interested in draft-ietf-idr-error-handling 
which struggles to preserve BGP session.

Cheers,
R.

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 13 Jan 2012 10:35:38 -0500

> think what you are referring to below.

Indeed.  Or at least things in that solution space.

> Such operators are free to disable functionality they do not
> appreciate. However I am not aware about any operator who forcefully
> disables route refresh capability on their BGP sessions.

But I believe you're aware of implementations that rate limit refreshes.
Even though the benefits of route refresh are clear, I think it'll take some
discussion to determine if such prefix polling is similarly useful.

Of course, a big part of my opinion on this is still prompted by the belief
that if you're getting bad UPDATE messages that trusting the NLRI are
correct is a wager at best, albeit one with good odds.

-- Jeff

---

From: John Leslie <[email-masked]>
Date: Fri, 13 Jan 2012 10:37:10 -0500

Jeffrey Haas <[email-masked]> wrote:
>> If we decide to implement a route-refresh to correct the issue, we need
>> to care on not doing an infinite loop in case of persistent error.
> 

   (But one that already exists...)

> ones again so I can see if the Bad route is better this time around?"

   Yes, it sounds silly...

> Mechanisms that request the same prefix again are potentially better,
> but not be well received by operators.

   Clearly, no particular response can be required. Indeed, if there
is a response, it would be wise to be especially "conservative in what
you send".

> (This opinion is under the "don't let people ask me for stuff"
> operational umbrella.

   Which I regard as "silly". I much prefer telling them to go to hell.

> Some operators really prefer a push-only model.)

   Which is perfectly reasonable for some, not all, operators.

   When I advertise customer routes, I want them to reach as much of
the Internet as possible.

   There definitely are cases, though, where something you advertise
is merely an optimization, not _expected_ to reach the whole Internet.

   IMHO, it would be good to have a way to tell a peer you want his
route(s) to a CIDR block (and anything you have will be treated as
withdrawn if there is no response). Simultaneously, it would help to
have a way to recognize the response.

   Our BGP paradigm is kinda simplistic (and fragile IMHO) for the 21st
century.

--
John Leslie <[email-masked]>

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 13 Jan 2012 11:50:55 -0500

> correct is a wager at best, albeit one with good odds.

In thinking about this a bit more over lunch, here's something else I'd like
for the list archives.  I think this was discussed during an IETF session,
but I can't recall who would have made the statement if so:

Part of our problem is that when errors are happening, they're typically
transitive errors that aren't noticed until they've traveled some distance
down stream.  Thus, if an implementation has noted an error and is applying
the error-handling draft's semantics to treat as a withdrawal, there's a
very *low* chance that re-requesting the destination again will result in
a corrected announcement.

For a request to have an impact, I believe one of the following conditions must
apply:
- It might have to be transitive.  This means the request might have to
  travel upstream all the way to the originator.
- The error would have to had occurred at the adjacent peering session.

-- Jeff

---

From: John Scudder <[email-masked]>
Date: Fri, 13 Jan 2012 12:56:04 -0800

Folks,

I see little to no benefit from attempting some kind of try-to-fix-the-problem-automatically mechanism, and on the other hand I do see drawbacks. Consider the different kinds of errors that can be handled by the scheme in this draft:

- Updates that are malformed due to some spec implementation error on the part of the update's originator, or one of the intermediate routers. In this case, one can expect that requesting the update again, will fetch the same data as before. "The definition of insanity is doing the same thing over and over and expecting different results." Let's not.

- Updates that are malformed due to some more general implementation error on the part of some upstream router, e.g. memory corruption. Such an error might be effectively 'random' and amenable to correction through a refresh, but OTOH the originating implementation is very badly broken and might be expected to crash soon anyway. The class of problems where there's just an itty bit of corruption that breaks one update and then clears up forever is probably very small.

If you know of some plausible class of errors that would be expected to be corrected by some self-healing mechanism, please explain in more detail.

When the originator of the error is fixed, I would expect the proper update to be propagated and installed without any special-case "healing" mechanisms. I think this is self-evident, so I'm not going to present arguments for why it is so; rather, if you think it's not so, please explain why.

Remember that we are proposing to replace a simple and reliable (but problematic!) error handling mechanism that has many years of deployment experience. In doing so, I think one worthwhile goal is to keep the replacement also as simple as possible. To the extent that any bells and whistles are to be added, I suggest they be on the error reporting and diagnostics side and not on an attempt to introduce self-repair mechanisms. Remember that additional complexity has cost and such code paths may themselves be subject to bugs (are more likely to be actually, since less frequently exercised) and may complicate debugging in the field.

Finally, I strongly agree that a treat-as-withdraw error should be treated as though it's (almost) equal in severity to a downed session. On the router side, this means raising appropriate alarms.

--John

---

From: Jakob Heitz <[email-masked]>
Date: Fri, 13 Jan 2012 16:23:33 -0500

Strongly agree.

As I said before, we are only trying to define how to limp along
instead of needlessly spreading damage.

--
Jakob Heitz.

---

From: Randy Bush <[email-masked]>
Date: Fri, 13 Jan 2012 15:58:32 -0800

<aol>

there is an approach i call "do-gooder software" which attempts to fix
the broken.  when it works, no one notices and says "thanks."  when it
screws up, and it always does, the resulting poolpah results in more
problems than the one it was trying to whitewash.

and i do not believe in being liberal in what one accepts, that way lies
entropic death.

randy, a naggumite

---

From: Jie Dong <[email-masked]>
Date: Mon, 16 Jan 2012 06:26:24 +0000

Robert, Jeff, Bruno,

> >> If we decide to implement a route-refresh to correct the issue, we need to
> care on not doing an infinite loop in case of persistent error.
> >

Yes. One use case of one-time prefix ORF is to trigger this selective refresh after some routes have been "treat-as-withdraw". 

Though treat-as-withdraw will not turn down the BGP session, reachability for some specific prefixes may already be lost, which may not be accepted by customers. Request those prefixes again may solve this issue (though not in all cases). Definitely an infinite loop should be avoided by limiting the number of requests and with some back-off mechanism.

Note that now the mechanism in draft-ietf-idr-error-handling is also applicable to the well-known attributes, thus it could be more possible that the fault is at the local BGP peer.

Best regards,
Jie

---

From: <[email-masked]>
Date: Mon, 16 Jan 2012 10:06:00 +0100

>> If we decide to implement a route-refresh to correct the issue, we
need to care on not doing an infinite loop in case of persistent error.
>"I received one Bad route from you.  Mind sending me 349,999 of the
good ones again so I can see if the 
>Bad route is better this time around?"
>Mechanisms that request the same prefix again are potentially better

I agree that route-refresh is mostly wasted but it's a "quick win" as
it's already available in most (all) codes and moreover I hope there is
no more bugs in it ! ;) ... But optimization (like
draft-zeng-idr-one-time-prefix-orf-01) are fine but not available yet.

I agree with Jakob and John that we should try to keep the error
processing as simple as possible, and introducing new mechanisms would
introduce complexity. Maybe for a future , it will be interresting.

I agree too that the number of cases that would be solved by the
route-refresh would be small.
But IMHO, introducing a route-refresh in code path is not costly, this
is something stable, available in all codes (no need of new code on peer
too), and route-refresh is not costly for the router (remember that for
VPNs, each time we provision an RT, a route-refresh is sent, and number
of routes are really higher than in Internet !!). Question to answer :
how much it cost vs number of occurences solved. I think cost is about
0, but I'm not a BGP coder ... So maybe I'm wrong.

--- 
Stephane

---

From: Jakob Heitz <[email-masked]>
Date: Mon, 16 Jan 2012 12:04:42 -0500

The solution is to fix the root cause. Once that's done, is there a scenario in which the updates will not propagate naturally?

--
Jakob Heitz.

---

From: <[email-masked]>
Date: Tue, 17 Jan 2012 09:52:23 +0100

>The solution is to fix the root cause. Once that's done, is there a scenario in which the updates will not propagate naturally?

I don't think so ... As soon as issue is cleared on originator (of the failure), the updates will propagate naturally. Why do you think there could not ?

--
Stephane

-----Message d'origine-----
De : Jakob Heitz [mailto:[email-masked]] 
Envoyé : lundi 16 janvier 2012 18:05
À : LITKOWSKI Stephane DTF/DERX
Cc : Jeffrey Haas; [email-masked]; DECRAENE Bruno RD-CORE; [email-masked]
Objet : Re: [Idr] draft-ietf-idr-error-handling

--
Jakob Heitz.

On Jan 16, 2012, at 1:06 AM, "[email-masked]" <[email-masked]> wrote:

> it's already available in most (all) codes and moreover I hope there 
> is no more bugs in it ! ;) ... But optimization (like
> is something stable, available in all codes (no need of new code on 
> peer too), and route-refresh is not costly for the router (remember 
> that for VPNs, each time we provision an RT, a route-refresh is sent, 
> and number of routes are really higher than in Internet !!). Question to answer :
> privileged information that may be protected by law; they should not be distributed, used or copied without authorization.

---

From: Jakob Heitz <[email-masked]>
Date: Tue, 17 Jan 2012 11:40:52 -0500

In that case, a route refresh will fix nothing.

--
Jakob Heitz.

---

From: <[email-masked]>
Date: Wed, 18 Jan 2012 10:31:39 +0100

It depends of kind of errors.
I can reuse what John said about type of errors : 

[John] 
"- Updates that are malformed due to some spec implementation error on the part of the update's originator, or one of the intermediate routers. In this case, one can expect that requesting the update again, will fetch the same data as before. "The definition of insanity is doing the same thing over and over and expecting different results." Let's not.

- Updates that are malformed due to some more general implementation error on the part of some upstream router, e.g. memory corruption. Such an error might be effectively 'random' and amenable to correction through a refresh, but OTOH the originating implementation is very badly broken and might be expected to crash soon anyway. The class of problems where there's just an itty bit of corruption that breaks one update and then clears up forever is probably very small."

In case two, when there is a random memory corruption when update is formatted and sent, a route-refresh may clear the issue. As I mentionned, and as John said too, probability is low but it exists : just a question of cost (to implement and on router ressources), vs nb of occurences.

--
Stephane

-----Message d'origine-----
De : Jakob Heitz [mailto:[email-masked]] 
Envoyé : mardi 17 janvier 2012 17:41
À : LITKOWSKI Stephane DTF/DERX
Cc : Jeffrey Haas; [email-masked]; DECRAENE Bruno RD-CORE; [email-masked]
Objet : Re: [Idr] draft-ietf-idr-error-handling

--
Jakob Heitz.

On Jan 17, 2012, at 12:52 AM, "[email-masked]" <[email-masked]> wrote:

> De : Jakob Heitz [mailto:[email-masked]] Envoyé : lundi 16 
> janvier 2012 18:05 À : LITKOWSKI Stephane DTF/DERX Cc : Jeffrey Haas; 
> [email-masked]; DECRAENE Bruno RD-CORE; [email-masked] Objet : Re: 
> [Idr] draft-ietf-idr-error-handling
> 
>> processing as simple as possible, and introducing new mechanisms 
>> would introduce complexity. Maybe for a future , it will be interresting.
>> 
>> But IMHO, introducing a route-refresh in code path is not costly, 
>> this is something stable, available in all codes (no need of new code 
>> on peer too), and route-refresh is not costly for the router 
>> (remember that for VPNs, each time we provision an RT, a 
>> route-refresh is sent, and number of routes are really higher than in Internet !!). Question to answer :
>> how much it cost vs number of occurences solved. I think cost is 
>> about 0, but I'm not a BGP coder ... So maybe I'm wrong.
>> 
>> ---
>> Stephane
>> 
>> _____________________________________________________________________
>> _ ___________________________________________________
>>