Subject: [Idr] I-D Action: draft-ietf-idr-error-handling-01.txt

From: [email-masked]
Date: Thu, 15 Dec 2011 15:38:45 -0800

A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the Inter-Domain Routing Working Group of the IETF.

	Title           : Revised Error Handling for BGP UPDATE Messages
	Author(s)       : John G. Scudder
                          Enke Chen
                          Pradosh Mohapatra
                          Keyur Patel
	Filename        : draft-ietf-idr-error-handling-01.txt
	Pages           : 10
	Date            : 2011-12-15

   According to the base BGP specification, a BGP speaker that receives
   an UPDATE message containing a malformed attribute is required to
   reset the session over which the offending attribute was received.
   This behavior is undesirable as a session reset would impact not only
   routes with the offending attribute, but also other valid routes
   exchanged over the session.  This document partially revises the
   error handling for UPDATE messages, and provides guidelines for the
   authors of documents defining new attributes.  Finally, it revises
   the error handling procedures for several existing attributes.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-idr-error-handling-01.txt

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

This Internet-Draft can be retrieved at:
ftp://ftp.ietf.org/internet-drafts/draft-ietf-idr-error-handling-01.txt

---

From: Jakob Heitz <[email-masked]>
Date: Fri, 16 Dec 2011 10:52:17 -0500

I agree with the added sentence:

   If an attribute appears more than once in an UPDATE message, then all
   the occurrences of the attribute other than the first one SHALL be
   discarded and the UPDATE message continue to be processed.

--
Jakob Heitz.

On Dec 15, 2011, at 3:39 PM, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>> wrote:

_______________________________________________
Idr mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/idr

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 16 Dec 2011 17:04:51 +0100

Hi Jakob,

Is this paragraph referring to any attribute ? If so I think more proper 
would be to say:

"If an identical attribute appears more than once .."
        ^^^^^^^^^

Example:

You have received Update message with two MP_UNREACH attributes first 
containing 5 NLRIs and the second 50 I don't think it is safe to drop 
the second MP_UNREACH and continue processing the message as usual.

Thx,
R.

---

From: <[email-masked]>
Date: Fri, 16 Dec 2011 16:11:42 +0000

>   If an attribute appears more than once in an UPDATE message, then all

What does "an attribute" refer to?
a) any attribute?
b) any attribute in error?
c) any attribute in error which does not impact the routing ("attribute discard")?

I understand "c". I'm not sure about the others ("a", "b") (in particular "a" would mean to silently discard an AS_PATH, MP_REACH attribute while processing the NLRI)

>On Dec 15, 2011, at 3:39 PM, "[email-masked]<mailto:internet-
>[email-masked]>" <[email-masked]<mailto:[email-masked]>>
>wrote:
>
>
>A New Internet-Draft is available from the on-line Internet-Drafts directories.
>This draft is a work item of the Inter-Domain Routing Working Group of the
>IETF.
>
>[email-masked]

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
France Telecom - Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorization.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, France Telecom - Orange shall not be liable if this message was modified, changed or falsified.
Thank you.

---

From: Jakob Heitz <[email-masked]>
Date: Fri, 16 Dec 2011 11:25:30 -0500

An attribute of a specific type. For example if an update message contains 2 LOCAL_PREF attributes, the second would be discarded. The first one would be processed. If the first was in error, then it would be processed as such, but the second would be discarded nonetheless.

--
Jakob Heitz.

---

From: Jakob Heitz <[email-masked]>
Date: Fri, 16 Dec 2011 11:34:12 -0500

Say I get 2 LOCAL_PREF, one with value 100, the other 99. They are not identical, but I can clearly choose only one.

Robert, please go through the list of bgp attributes in http://www.iana.org/assignments/bgp-parameters/bgp-parameters.xml and enumerate the identical ones.

--
Jakob Heitz.

---

From: <[email-masked]>
Date: Fri, 16 Dec 2011 16:51:35 +0000

>An attribute of a specific type. For example if an update message contains 2
>LOCAL_PREF attributes, the second would be discarded. The first one would be
>processed.

That would be my "a" case.
Why would it be valid and safe to discard ("attribute discard") this second LOCAL_PREF and not the first/only one?

> If the first was in error, then it would be processed as such, but
> the second would be discarded nonetheless.

If the first is treated as "treat-as-withdraw" (My case "b"), the second will never be processed in the first place. No need to add a sentence.

If the first is treated as "attribute discard" (My case "c"), I'm fine with the proposed new addition but yet why would we want to discard the second attribute?

In short, IMHO, the additional sentence is mostly applicable to attribute treated with the approach "attribute discard". I would propose the following formulation:
If an attribute _using the "attribute discard" approach_ appears more than once in an UPDATE message, then all the occurrences of the attribute other than the first _valid_ one SHALL be discarded and the UPDATE message continue to be processed.

>> I understand "c". I'm not sure about the others ("a", "b") (in particular "a"
>would mean to silently discard an AS_PATH, MP_REACH attribute while processing
>the NLRI)
>>
>_______________________________________________________________________________
>__________________________________________
>>
>> Ce message et ses pieces jointes peuvent contenir des informations
>confidentielles ou privilegiees et ne doivent donc
>> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu
>ce message par erreur, veuillez le signaler
>> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
>electroniques etant susceptibles d'alteration,
>> This message and its attachments may contain confidential or privileged
>information that may be protected by law;
>> As emails may be altered, France Telecom - Orange shall not be liable if this
>message was modified, changed or falsified.
>> Thank you.
>>

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 16 Dec 2011 17:59:17 +0100

Jakob,

> Say I get 2 LOCAL_PREF, one with value 100, the other 99. They are
> not identical, but I can clearly choose only one.

If you want to know my preference .. I would prefer to drop the session 
rather then to deal with peer which can not even build the BGP Update 
message properly.

> Robert, please go through the list of bgp attributes in
> http://www.iana.org/assignments/bgp-parameters/bgp-parameters.xml and
> enumerate the identical ones.

Entirely irrelevant comment.

Thx,
R.

> On Dec 16, 2011, at 8:04 AM, "Robert Raszuk"<[email-masked]>
> wrote:
>
>> Hi Jakob,
>>
>> Is this paragraph referring to any attribute ? If so I think more
>> proper would be to say:
>>
>> "If an identical attribute appears more than once .." ^^^^^^^^^
>>
>> Example:
>>
>> You have received Update message with two MP_UNREACH attributes
>> first containing 5 NLRIs and the second 50 I don't think it is safe
>> to drop the second MP_UNREACH and continue processing the message
>> as usual.
>>
>> Thx, R.
>>
>>
>>> If an attribute appears more than once in an UPDATE message, then
>>> all the occurrences of the attribute other than the first one
>>> SHALL be discarded and the UPDATE message continue to be
>>> processed.
>>>
>>> -- Jakob Heitz.
>
>

---

From: Enke Chen <[email-masked]>
Date: Fri, 16 Dec 2011 11:04:12 -0800

It is to address the following case specified in Sect. 6.3, RFC 4271:

    If any attribute appears more than once in the UPDATE message, then
    the Error Subcode MUST be set to Malformed Attribute List.

We missed the case in the earlier version, although we said:

    The error handling of all other cases described in Section 6.3 of
    [RFC4271] that specify a session reset is revised as follows.
    ...

-- Enke

On 12/16/11 7:52 AM, Jakob Heitz wrote:
> On Dec 15, 2011, at 3:39 PM, "[email-masked]<mailto:[email-masked]>"<[email-masked]<mailto:[email-masked]>>  wrote:
>
>

---

From: Claudio Jeker <[email-masked]>
Date: Fri, 16 Dec 2011 23:07:20 +0100

On Fri, Dec 16, 2011 at 05:59:17PM +0100, Robert Raszuk wrote:
> If you want to know my preference .. I would prefer to drop the
> session rather then to deal with peer which can not even build the
> BGP Update message properly.
> 

Me too. If a peer is unable to build a valid basic BGP packet it may do
even worse things. The session is not trustworthy and it must be the
direct neighbor at fault, so throwing a NOTIFICATION is the right way to
handle this.

Sometimes it is ok to error out instead of building insane fail saves
(which make the system less stable and can backfire later on).
-- 
:wq Claudio

---

From: Jakob Heitz <[email-masked]>
Date: Fri, 16 Dec 2011 17:36:50 -0500

Points made at the Taipei meeting:

Old thinking: No service is better than unreliable service.
New thinking: Partial service is better than no service.

All errors will be logged for debugging and manual verification
of the routing state.

Experience has shown that using a session reset causes severe disruption
to routing, making it an inappropriate way to call attention
to a protocol error. Such errors can be handled in a much
less disruptive manner.

Another point made:

Error handling must be as simple as possible (and no simpler),
lest bugs in the error handling make the medicine worse than the disease.

I think the error handling draft as it stands now lays out
a well balanced solution to meet these criteria.

--
Jakob Heitz.

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 16 Dec 2011 23:50:33 +0100

Hello Jakob,

> New thinking: Partial service is better than no service.

I could agree with the above realizing the spectrum of applications we 
use BGP as database distributors today. But pls see below ...

> All errors will be logged for debugging and manual verification

Sorry but this is not sufficient. We have had number of discussions in 
the past both on the list and during the meetings which agreed to send 
to the peer details of the errors encountered. BGP Operational message 
proposal I have co-authored with Dave and Rob does just that. Somehow we 
are not seeing much progress there in the WG.

> to a protocol error. Such errors can be handled in a much

True. But for some strange reason no one recognized that we don't need 
to jump from one extreme to the other extreme.

Simple solution allowing for per BGP message ack would be sufficient to 
address most of the current BGP errors without impacting the critical 
mass of applications traversing over the same BGP session.

Literally I would see keeping the TCP session up (at least till someone 
smart will propose real short lived BGP database like updates), but 
having a way to rejecting the BGP message as opposed to playing colorful 
games how to cover it with various blankets to pretend nothing happened.

> lest bugs in the error handling make the medicine worse than the disease.

Somehow I don't think what we are observing here follows that point.

> a well balanced solution to meet these criteria.

I would disagree. I would call well balanced a solution which would 
allow to accept or reject granularity of BGP Messages rather then 
tampering with them. And making error-handling on a per BGP Message 
boundaries seems to me like a very simple proposal.

Best regards,
R.

---

From: Ilya Varlashkin <[email-masked]>
Date: Sat, 17 Dec 2011 00:41:17 +0100

> (which make the system less stable and can backfire later on).

An alternative between these black and white behaviours could be to throw away whole update but inform the neighbour that it has send nonsense, same time applying some penalty points. If neighbour had once-in-life-time mistake it can resend proper update, everybody will be happy. If on the other hand neighbour persists sending rubbish, it will get more penalty points and eventually session can be supressed/dropped, possible also (automatically) putting neighbour into a quarantine group which will prevent this session from coming up again any time soon.

Having had recent experience with malformed attributes and "offending" sessions coming up over and over again just to disappear few moments later, I just don't see any help (in this particular context) from either dropping offending duplicate attributes or sending NOTIFICATION and dropping the session.

/iLya

---

From: Robert Raszuk <[email-masked]>
Date: Sat, 17 Dec 2011 00:58:49 +0100

Hi Ilya,

> An alternative between these black and white behaviours could be to
> throw away whole update but inform the neighbour that it has send
> nonsense, same time applying some penalty points.

Glad you have said so. This would be per BGP message action + as you 
suggest signalling to a peer as well as accumulating the penalty. I 
think this would be good error handling tactics.

> If neighbour had once-in-life-time mistake

I think I am a bit stuck and just do not understand the genesis of those 
"once-in-life-time mistakes". Routers are not humans who make mistakes 
.. routers are just programs which usually give the same output if the 
input is still the same.

> If on the other hand neighbour persists sending rubbish, it
> will get more penalty points and eventually session can be
> supressed/dropped, possible also (automatically) putting neighbour
> into a quarantine group which will prevent this session from coming
> up again any time soon.

This will be really good IMHO.

Rather then only seeing tier 1s and 2s and optimizing for their view of 
BGP to try to cure infection in flight without crashing the entire plane 
what you are proposing would actually help to eliminate and stop the 
issue at it's source .. before it takes off again.

Best,
R.

---

From: Enke Chen <[email-masked]>
Date: Fri, 16 Dec 2011 16:36:59 -0800

Robert and Claudio,

Resetting the session is undesirable as we can not recover from the 
situation without a new version of the software.

-- Enke

---

From: Jakob Heitz <[email-masked]>
Date: Fri, 16 Dec 2011 22:16:12 -0500

Either draft does not preclude the other.

--
Jakob Heitz. x25475. 510-566-2901

-----Original Message-----
From: Robert Raszuk [mailto:[email-masked]] 
Sent: Friday, December 16, 2011 2:51 PM
To: Jakob Heitz
Cc: [email-masked]
Subject: Re: [Idr] I-D Action: draft-ietf-idr-error-handling-01.txt

BGP Operational message

---

From: Claudio Jeker <[email-masked]>
Date: Sat, 17 Dec 2011 11:39:52 +0100

> situation without a new version of the software.

I hope that the people writing that software are well aware of this and
that they fix the problem before it actually goes out into the wild.
Do you trust the software that is unable to follow the very basic rules of
the RFC of not duplicating an attribute to be bug free in other regards?
You think it will not mangle the ASPATH in ways that will suddenly direct
all your traffic to that router where it is dropped or looped back to you?

I understand that the error handling with optional transitive attributes
needs to be very careful and should not result in NOTIFICATIONS and
session drops. I do not understand that we try to keep direct neighbor
routers that act flaky (either because of dying HW or bad SW) up until the
situation is so bad that it may result in a much bigger issue.

Yes, the connection between the two neighbors in question will be down,
there will be no traffic exchanged over it (but we have a dynamic routing
protocol here and maybe it is able to find an alternate route to that site
that is not disrupted). Plus the rest of the network is not influenced by
this incident (because a good design would add enough redundancy into it).
On the other hand if bad routing information is spread through the network
it can cause a massive blackout (and this is what you actually try to
avoid in the first place).

Btw. I think the draft needs further explanation for the following cases:
- missing well-known attribute
   I guess this should be a treat-as-withdraw (but it is not said so)
- unrecognized well-known attribute
   Are all unknown attributes automatically optional?
   In other words is this draft killing the option of having new
   well-known attributes?
   Also what is the correct value of a unknown attribute?
- leftmost AS check in ASPATH (enforce neighbor-as)
   Shouldn't we allow the "enforce neighbor-as" to send a NOTIFICATION as
   it is doing so now? Because else we can remove that from the RFC and
   tell people to filter on the leftmost AS (has the same effect).

-- 
:wq Claudio

---

From: Ilya Varlashkin <[email-masked]>
Date: Sat, 17 Dec 2011 16:35:36 +0100

NOTIFICATION as it is now won't prevent broken neighbour from coming up again. Session going down or up can rise CPU utilisation (==affect control plane) for several minutes. And not only on the router that shuts broken session but potentially on many other due to constant flapping of the routes.

Bottom line: simple resetting session with broken neighbour is not a solution.

/iLya

---

From: <[email-masked]>
Date: Mon, 19 Dec 2011 11:37:27 +0000

As some of the recent comments are more on the requirement side, it may be useful to remind that there is a requirement draft on BGP error handling in the GROW WG: draft-ietf-grow-ops-reqs-for-bgp-error-handling-02 Might want to read and comment.

More inline (a bit long)

>From: Claudio Jeker Sent: Saturday, December 17, 2011 11:40 AM
>> Robert and Claudio,
>>

I don't believe tests will ever be able to detect all bugs. So we need to be prepare to handle them.

>situation is so bad that it may result in a much bigger issue.

I don't believe in the "so bad SW" argument:
- Yes, something is bad, there is no way to figure it out in live, it would be safer to shut to not trust this router anymore and shut the BGP session. But to keep it safe, going that way, we should shut the BGP session forever, shut the IGP adjacencies if any (it's so bad I can't trust it/it's FIB any more), flood -at least in the AS- the OS version to warn all routers that that OS (HW) should not be trusted and that all routers should shut their BGP sessions with this bad SW... It may locally be seem as "safer", but this has consequences in term of network availability. Another valid path, the "cautious path" is to try to limit the consequences of the error (as defined in draft-ietf-idr-error-handling). Frankly, between the "safe path" and the "cautious path" the latter also seems a very reasonable option (and definitely the safer in term of security / DOS consideration). Yes, sometimes it will result in a bad decision with a bad neighbor still being used. But let's try this new path because we have tried the first one and we don't like it.
- we should also remember that we are mostly talking about events on routers from majors companies doing significant effort to carefully implement and test it, with SP doing furthers tests and then OS/routers running for months without any issue. Seems strange to suddenly assume they are bad and should not been talked to anymore.

I agree that optional transitive attributes are the most most likely to cause issues. However I would prefer that no single BGP update cause the BGP session to be shut down. Including well known mandatory attributes. 
e.g. let's take the AS_PATH, probably the most well known mandatory one. If I were the first AS to deploy draft-ietf-idr-as0, if one remote AS advertise an update with AS 0 in the AS_PATH, If I don't treat that error as withdraw (but as notification) I would end up tearing down all my BGP sessions (nominal, and _all_ backup ones) toward that NRLI and all the NLRI advertised over those sessions. That could be lots of NLRI (>50%), with no backup path, and forever until someone find the issue and the solution.

>that is not disrupted). 

You are assuming a local error with your neighbor. Chances are that errors have been trigger by a specific state and that all backup paths also got that state. Hence you can't assume that there would be a backup path if you choose to shut down the session.

However, to better take your valid case (a single bad neighbor) into account one could:
- keep the BGP session up (as per draft-ietf-idr-error-handling-01)
- in addition, flag all routes received (and not treated as withdraw) over that session with an IANA community. Then SP would be free to lower the LOCAL_PREF (or cost community) of those routes so that others routers in the AS would prefer another path if another good path were available.

That would probably better cover both cases (single bad neighbor, global issue with all BGP session affected), at the cost of creating BGP churn at probably a bad moment (but just like the existing RFC 4271 does).

Bruno

>Plus the rest of the network is not influenced by

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 19 Dec 2011 15:25:06 +0100

Hi Bruno,

> You are assuming a local error with your neighbor. Chances are that
> errors have been trigger by a specific state and that all backup
> paths also got that state. Hence you can't assume that there would be
> a backup path if you choose to shut down the session.

Reasonable networks have many neighbors. The like hood that due to 
mandatory BGP attribute error we will experience an outage from all 
neighbors is rather quite unlikely IMHO.

However I am not opposing recommendations of 
draft-ietf-idr-error-handling. What I am not quite comfortable with is 
to assume that this should be the new default behaviour.

You say that "we don't like it" what is happening today. One may argue 
that "some do not like it some do". Those who do not like it do they 
really use already available tools like multisession deployed in 
production to limit one SAFI impacting other SAFIs ? Could they just not 
use validating layer before the BGP update hits RP ?

The fact that draft-ietf-idr-error-handling seems to be targetting to 
update 4271 means that when I boot router with new software the 
behaviour will change and now all of the operations teams worldwide need 
to develop new syslog parsing scripts to catch the new error categories. 
Today they see it easily as session is going down.

--

Also note that if this is hacker on the CE trying to mess with his EBGP 
session towards PE to perhaps destroy other VPNs on such PE keeping the 
session up is only helping him to better attack the PE. Is this the 
aspect of "improvement" you have all considered ?

--

Last is I think the point that Ilya brought. While we are at the topic 
let's address the flapping session problem (assume that non of 
conditions in the new draft apply) and that session is getting resetted. 
It seems valid to me to accumulate penalty on such consecutive resets 
and keep the session shut when such penalty reaches some value. While 
some may say that this is implementation thing some may also see 
reasonable to define what the behaviour should be.

Best regards,
R.

---

From: <[email-masked]>
Date: Mon, 19 Dec 2011 15:43:15 +0000

Hi Robert,

>From: Robert Raszuk [mailto:[email-masked]]>Sent: Monday, December 19, 2011 3:25 PM
>
>Hi Bruno,
>

Please refer to my previous example: I'm the first AS (on the path) to implement draft-ietf-idr-as0. AS_X originates route Z/z with an AS 0 in the AS_PATH.
The number of neighbors is of little relevance as from all neighbors advertising Z/z, I will receive an invalid AS_PATH.
For a tier-1 like AS, the more neighbor ASes advertising a path to Z/z, the more sessions will go done.
For a lower tier AS, typically less BGP session will go done but each session would carry more NLRI (e.g. the whole Internet as advertised by its upstream transit).

And that's a single example. I'm sure reality will be much more creative.

BTW, I agree with you that this is quite unlikely. However when this happen, how do we want to react?

>that "some do not like it some do". 

Sure. But again, that's a discussion for draft-ietf-grow-ops-reqs-for-bgp-error-handling-02

>Those who do not like it do they
>production to limit one SAFI impacting other SAFIs ? 

Bringing down a whole AFI/SAFI is already too much. E.g. 1/1 (Internet) or 128/1 (L3 VPN) especially on an iBGP session with your BGP Route Reflector.

>Could they just not

Sorry, I don't see what you are refering to.

>Today they see it easily as session is going down.

I agree. I already brought that point both during the meeting and on the mailing list: http://www.ietf.org/mail-archive/web/idr/current/msg04710.html

Ideally _current_ operations teams/tools should have a chance to learn about the issue (degraded behavior) even if they have not been "upgraded". Something using the same news channel that when BGP sessions are reported down. (e.g. a new "bgp4V2PeerState" value "established-UPDATES in error (7)" which would probably also trigger "bgp4V2BackwardTransitionNotification NOTIFICATION-TYPE", but I'm not really into the MIB stuff so I would prefer that some else do the job).

Could authors of draft-ietf-idr-error-handling consider addressing the above point? Many thanks.

IMHO, having the ability to trigger a BGP session down further in the network is a bigger attack vector.

Would DampPeerOscillations (§8.1.1. of RFC 4271) do the job or are you thinking about something else?

Best regards,
Bruno

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 19 Dec 2011 17:08:50 +0100

Hi Bruno,

> Please refer to my previous example: I'm the first AS (on the path)
> to implement draft-ietf-idr-as0. AS_X originates route Z/z with an AS
> 0 in the AS_PATH. The number of neighbors is of little relevance as
> from all neighbors advertising Z/z, I will receive an invalid
> AS_PATH. For a tier-1 like AS, the more neighbor ASes advertising a
> path to Z/z, the more sessions will go done. For a lower tier AS,
> typically less BGP session will go done but each session would carry
> more NLRI (e.g. the whole Internet as advertised by its upstream
> transit).
>

draft-ietf-idr-as0 implementation should generate "warning-only" in the 
syslog rather then attempting to reset the session. Then your NOC can 
deal with it.

It is pretty bad to upgrade the router with new image supporting 
draft-ietf-idr-as0 and suddenly see session resets or treat-as-withdraws 
going on left and right.

Then you could use "replace-as knob" which is available in most of the 
BGP implementations and replace AS 0 with your AS in the AS_PATH in 
ingress. Warning will disappear from syslog giving you enough time to 
talk to those who inject it and perhaps letting them know that IDR came 
with new draft/rfc ;)

>> You say that "we don't like it" what is happening today. One may
>> argue that "some do not like it some do".
>
> Sure. But again, that's a discussion for
> draft-ietf-grow-ops-reqs-for-bgp-error-handling-02

Well there are many ways to meet requirements. One way as I suggested 
was to treat errors on a per Update message not per attributes. Add 
sequence numbers to Update Messages and notify peers that some of those 
have contained the errors. Note that due to BGP nature treat-as-withdraw 
may still be applicable.

> Bringing down a whole AFI/SAFI is already too much. E.g. 1/1
> (Internet) or 128/1 (L3 VPN) especially on an iBGP session with your
> BGP Route Reflector.

See above.

>> The fact that draft-ietf-idr-error-handling seems to be targetting
>> to update 4271 means that when I boot router with new software the
>> behaviour will change and now all of the operations teams worldwide
>> need to develop new syslog parsing scripts to catch the new error
>> categories. Today they see it easily as session is going down.
>
> I agree. I already brought that point both during the meeting and on
> the mailing list:
> http://www.ietf.org/mail-archive/web/idr/current/msg04710.html
>
> Ideally _current_ operations teams/tools should have a chance to
> learn about the issue (degraded behavior) even if they have not been
> "upgraded". Something using the same news channel that when BGP
> sessions are reported down. (e.g. a new "bgp4V2PeerState" value
> "established-UPDATES in error (7)" which would probably also trigger
> "bgp4V2BackwardTransitionNotification NOTIFICATION-TYPE", but I'm not
> really into the MIB stuff so I would prefer that some else do the
> job).
>

Glad we agree here !

>> Also note that if this is hacker on the CE trying to mess with his
>> EBGP session towards PE to perhaps destroy other VPNs on such PE
>> keeping the session up is only helping him to better attack the PE.
>> Is this the aspect of "improvement" you have all considered ?
>

True - but former is a targeted one the latter seems like a random one. 
I think we should try to avoid both.

>> Last is I think the point that Ilya brought. While we are at the
>> topic let's address the flapping session problem (assume that non
>> of conditions in the new draft apply) and that session is getting
>> resetted. It seems valid to me to accumulate penalty on such
>> consecutive resets and keep the session shut when such penalty
>> reaches some value. While some may say that this is implementation
>> thing some may also see reasonable to define what the behaviour
>> should be.
>

I am not sure if DampPeerOscillations kicks in due to error triggered 
sessions resets. It seems reading 4271 it is to catch session 
connect/session disconnect case. I think Ilya observed recently that it 
does not. But it is better that he comments here.

Best,
R.

---

From: Ilya Varlashkin <[email-masked]>
Date: Mon, 19 Dec 2011 20:29:18 +0100

On 19Dec 2011, at 17:08, Robert Raszuk wrote:

> I am not sure if DampPeerOscillations kicks in due to error  
> triggered sessions resets. It seems reading 4271 it is to catch  
> session connect/session disconnect case. I think Ilya observed  
> recently that it does not. But it is better that he comments here.
>

My experience was with an implementation sending malformed NLRI. As  
expected, this was causing session reset, flushing all the good routes  
as well, then coming up and "marry go 'round". Although logic  
described for DampPeerOscillations is similar to what could have  
remedied such behaviour, the actual values specified in rfc4271 won't  
catch the exact problem that I've encountered. 4271 specifies 10 times  
within 5 minutes and max IdleHoldTime of only 2 minutes, whereas in my  
case getting getting all routes was taking 5-20 minutes.

The behaviour I've observed isn't unique to my network and not even to  
single vendor. Few months ago one could observe constant session flaps  
in many networks due to malformed attribute, then coming up again and  
on and on, and DampPeerOscillations provided no significant protection.

Kind regards,
iLya

---

From: <[email-masked]>
Date: Tue, 27 Dec 2011 11:33:55 +0100

[Robert]
>If you want to know my preference .. I would prefer to drop the session
rather then to deal with peer which can not even >build the BGP Update
message properly.

Dropping session is bad and is really impacting the service, if we can
deal errors more smoothly it would be better ...

[Jakob & Bruno]
   >discarded and the UPDATE message continue to be processed.

IMHO, as you mentioned Bruno, the behavior must be in relation with the
behavior when the attribute is bad.
We consider here that both attributes are valid (not malformed), so the
"treat-as-withdraw"/"attribute-discard" doesn't apply, the only applying
thing currently is the "Malformed attribute List" notification.

I see this as follows : if the duplicate (but valid) attribute is a
"treat-as-withdraw" attribute, the NLRIs must be withdrawn (otherwise
this could cause routing issues).  if the duplicate (but valid)
attribute is a "attribute-discard" attribute , I would agree on keeping
only one (first or last, it doesn't really matter as we are not able to
choose !), the important thing is then to inform the peer and operator
of what we done (see next lines ...). 

[iLya] 
>An alternative between these black and white behaviours could be to
throw away whole update but inform the neighbour 
>that it has send nonsense, same time applying some penalty points. 

I agree with the point to not go to a total opposite direction.
Moreover,it's not the case in the draft, as all attributes that are
impacting the routing must use "treat-as-withdraw" behavior.
About the way to inform neighbor about a potential bad update (because
some attributes are stripped), maybe we can use some mechanism like
"STALE" community proposed in BGP persistence draft to inform peer that
this path is maybe bad, so prefer another path if there s some other
available (it should be a policy based decision by the SP). And if there
is no path, this "STALE" path could be used.
Globally this marking approach could be used each time we are discarding
an attribute (duplicate case, or bad attribute with "attribute-discard"
behavior).

[Robert]
>You have received Update message with two MP_UNREACH attributes first
containing 5 NLRIs and the second 50 I don't think >it is safe to drop
the second MP_UNREACH and continue processing the message as usual.

A total out of scope suggestion/discussion, why  are we not able to put
multiple MP_UNREACH/MP_REACH_NLRI in a BGP message ? Today it's not
permitted by the BGP RFC, as these are BGP  attributes (easier to
implement than recoding a BGP per AFI/SAFI), so same behavior for all
attributes (one per type per update) but globally we are transporting
prefixes (and NH) only in these "attributes". So being able to transport
multiple occurence of MP_UNR/MP_R (one per AFI/SAFI), would permit a
better packing of BGP updates when for example a VRF with common
policies is using IPv4,IPv6,mVPN , multiple prefixes could be packed
with same attribute sequence. Easy to say, but maybe hard to implement.
There's topics about AFI/SAFI separations/fault isolation, and this
would go in an opposite direction ...

I just wanted to have some opinions on that (I was too young when BGP MP
extensions were defined :) ).

---

From: Jakob Heitz <[email-masked]>
Date: Tue, 27 Dec 2011 06:01:00 -0500

When a speaker sends a malformed attribute and the receiver
has to guess at the intention, SOMETHING WENT WRONG and
MUST BE FIXED.

We are here just trying to define how to limp along
instead of needlessly spreading damage.

It doesn't matter much whether we keep one or the other
or neither or withdraw.

Personally, I would treat a repeated attribute as a
malformation of that attribute: the receiver can only
guess at the intent and the problem needs to be fixed.
However, I'm agreeable to the current draft wording too.

--
Jakob Heitz.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Tuesday, December 27, 2011 2:34 AM
To: Ilya Varlashkin
Cc: [email-masked]
Subject: Re: [Idr] I-D Action: draft-ietf-idr-error-handling-01.txt

[Robert]

[Jakob & Bruno]

[iLya] 

[Robert]

---

From: <[email-masked]>
Date: Tue, 27 Dec 2011 15:38:59 +0100

Hi,

Are we sure that DampPeerOscillations has been implemented in vendors code ?
We globally have many session flap , especially PE-CE (sometimes PE-RR), with high flapping rate leading to high CPU on nodes, and we were wondering if the functionnality was implemented or not ...
Maybe vendors can answer ...
The values given in RFC 4271 seems to be only examples, if there is implementation of DampPeerOscillations, the values may differs.

Thanks,

Stephane

-----Message d'origine-----
De : [email-masked] [mailto:[email-masked]] De la part de Ilya Varlashkin
Envoyé : lundi 19 décembre 2011 20:29
À : [email-masked]
Cc : DECRAENE Bruno RD-CORE; [email-masked]
Objet : Re: [Idr] I-D Action: draft-ietf-idr-error-handling-01.txt

On 19Dec 2011, at 17:08, Robert Raszuk wrote:

> connect/session disconnect case. I think Ilya observed recently that 
> it does not. But it is better that he comments here.
>

My experience was with an implementation sending malformed NLRI. As expected, this was causing session reset, flushing all the good routes as well, then coming up and "marry go 'round". Although logic described for DampPeerOscillations is similar to what could have remedied such behaviour, the actual values specified in rfc4271 won't catch the exact problem that I've encountered. 4271 specifies 10 times within 5 minutes and max IdleHoldTime of only 2 minutes, whereas in my case getting getting all routes was taking 5-20 minutes.

The behaviour I've observed isn't unique to my network and not even to single vendor. Few months ago one could observe constant session flaps in many networks due to malformed attribute, then coming up again and on and on, and DampPeerOscillations provided no significant protection.

Kind regards,
iLya

---

From: Claudio Jeker <[email-masked]>
Date: Tue, 27 Dec 2011 15:41:20 +0100

On Mon, Dec 19, 2011 at 11:37:27AM +0000, [email-masked] wrote:
> As some of the recent comments are more on the requirement side, it may
> be useful to remind that there is a requirement draft on BGP error
> handling in the GROW WG:
> draft-ietf-grow-ops-reqs-for-bgp-error-handling-02 Might want to read
> and comment.
> More inline (a bit long)

Me too.
 
> to be prepare to handle them.

No, they will not find all bugs but some of the obvious ones that
everybody suffered from. Maybe I'm to naive...

> - Yes, something is bad, there is no way to figure it out in live, it
> would be safer to shut to not trust this router anymore and shut the BGP
> session. But to keep it safe, going that way, we should shut the BGP
> session forever, shut the IGP adjacencies if any (it's so bad I can't
> trust it/it's FIB any more), flood -at least in the AS- the OS version
> to warn all routers that that OS (HW) should not be trusted and that all
> routers should shut their BGP sessions with this bad SW... It may
> locally be seem as "safer", but this has consequences in term of network
> availability. Another valid path, the "cautious path" is to try to limit
> the consequences of the error (as defined in
> draft-ietf-idr-error-handling). Frankly, between the "safe path" and the
> "cautious path" the latter also seems a very reasonable option (and
> definitely the safer in term of security / DOS consideration). Yes,
> sometimes it will result in a bad decision with a bad neighbor still
> being used. But let's try this new path because we have tried the first
> one and we don't like it.

Actually that was tried already long time ago and resulted in nice
meltdowns. Yes, throwing a NOTIFICATION and closing the session is often
too extrem but it is not OK to ever accept and maybe propagate a possible
faulty attribute. Which is exactly the case when a router sends an
attribute twice.

> - we should also remember that we are mostly talking about events on
> routers from majors companies doing significant effort to carefully
> implement and test it, with SP doing furthers tests and then OS/routers
> running for months without any issue. Seems strange to suddenly assume
> they are bad and should not been talked to anymore.

So if they are tested so carefully why has the problem not been found
already?
Since the problem was not found until now how can you assume that because
the system was fine for a few month it will still be fine? You don't know
the cause of the problem, it may be harmless or it may be something that
will take the network down.
 
> I agree that optional transitive attributes are the most most likely to
> cause issues. However I would prefer that no single BGP update cause the
> BGP session to be shut down. Including well known mandatory attributes. 
> e.g. let's take the AS_PATH, probably the most well known mandatory one.
> If I were the first AS to deploy draft-ietf-idr-as0, if one remote AS
> advertise an update with AS 0 in the AS_PATH, If I don't treat that
> error as withdraw (but as notification) I would end up tearing down all
> my BGP sessions (nominal, and _all_ backup ones) toward that NRLI and
> all the NLRI advertised over those sessions. That could be lots of NLRI
> (>50%), with no backup path, and forever until someone find the issue
> and the solution.
>

This is a bad example. You deploy something that extends the error
handling and makes the interpretation of messages more strict. Sure you
may end up in that situation, it was your decision to deploy it.
I guess a more valid point would be the handling of more then 255 elements
in an AS PATH (which caused some issues some time ago). In in that case
treating that UPDATE as a withdraw is probably the better option than
shutting down the session.
 
> errors have been trigger by a specific state and that all backup paths
> also got that state. Hence you can't assume that there would be a backup
> path if you choose to shut down the session.

I was talking about duplicate attributes in an UPDATE message. I wonder
how that can propagate beyond the two systems. I guess we need to assume
that an external event caused this bug and yes it could affact other
systems to. But why does the system that found the error propagate a
possible faulty attribute? What assumtion tells us that the first
attribute was the correct one? Why not the last? We're guessing here and
that is bad for an error-handling RFC. IMO the correct behaviour would be
"attribute discard" or "treat-as-withdraw" depending on the attribute.

> - in addition, flag all routes received (and not treated as withdraw)
> over that session with an IANA community. Then SP would be free to lower
> the LOCAL_PREF (or cost community) of those routes so that others
> routers in the AS would prefer another path if another good path were
> available.

Nice idea.

> 
> That would probably better cover both cases (single bad neighbor, global
> issue with all BGP session affected), at the cost of creating BGP churn
> at probably a bad moment (but just like the existing RFC 4271 does).

-- 
:wq Claudio

---

From: Claudio Jeker <[email-masked]>
Date: Tue, 27 Dec 2011 15:49:12 +0100

> However, I'm agreeable to the current draft wording too.

I can agree with most of draft-ietf-idr-error-handling-01
but this is one point I can't agree. I'm more on your line here. Repeated
attributes are a malformation and the receiver should never guess about
the intent esp. when handling errors. If in doubt ignore all instead of
possibly distributing the wrong attribute.

-- 
:wq Claudio

---

From: <[email-masked]>
Date: Tue, 27 Dec 2011 15:52:27 +0100

Hmmm ... Here are some findings from RFC 4276 :

3.28.156.  Event6: AutomaticStart_with_DampPeerOscillations

       Functionality/Description: The AllowAutomaticStart attribute
       SHOULD be set to TRUE

       RFC2119: SHOULD

       Alcatel Y/N/O/Comments: N
       Cisco   Y/N/O/Comments: O    Don't support DampPeerOscillations
                                    attribute.
       Laurel  Y/N/O/Comments: Y

       NextHop Y/N/O/Comments: Y

Alcatel/Cisco were not supporting DampPeerOscillations when the survey was done. Maybe it changed, maybe not (RFC is old ...).
No feedback about Juniper.

-----Message d'origine-----
De : LITKOWSKI Stephane DTF/DERX 
Envoyé : mardi 27 décembre 2011 15:39
À : 'Ilya Varlashkin'; [email-masked]
Cc : DECRAENE Bruno RD-CORE; [email-masked]
Objet : RE: [Idr] I-D Action: draft-ietf-idr-error-handling-01.txt

Hi,

Thanks,

Stephane

-----Message d'origine-----
De : [email-masked] [mailto:[email-masked]] De la part de Ilya Varlashkin Envoyé : lundi 19 décembre 2011 20:29 À : [email-masked] Cc : DECRAENE Bruno RD-CORE; [email-masked] Objet : Re: [Idr] I-D Action: draft-ietf-idr-error-handling-01.txt

Kind regards,
iLya

---

From: <[email-masked]>
Date: Thu, 29 Dec 2011 09:12:49 +0100

Hi Jakob,

>When a speaker sends a malformed attribute and the receiver has to guess at the intention, SOMETHING WENT WRONG and MUST BE FIXED.
>We are here just trying to define how to limp along instead of needlessly spreading damage.

Totally agree !

But ot sure to understand your meaning, when you said :

"I would treat a repeated attribute as a malformation of that attribute"
So basically, with the current proposed behavior in the draft, it means that the NLRIs are withdraw if attribute is "treat-as-withdraw", or attributes (all occurences) are discarded if attribute is "attribute-discard" type. 

But you say that you agree with the current wording which say that we keep/process the first occurrence of the attribute. So there is a contradiction here, maybe I misunderstood something ... Please clarify.

I totally agree on "the receiver can only guess at the intent and the problem needs to be fixed.", so if we decide to keep one attribute in some cases, the receiver must be informed that there s something wrong with the path (the way to inform need to be defined -> see my proposal to reuse STALE community).

Regards

Stephane

-----Message d'origine-----
De : Jakob Heitz [mailto:[email-masked]] 
Envoyé : mardi 27 décembre 2011 12:01
À : LITKOWSKI Stephane DTF/DERX; Ilya Varlashkin
Cc : [email-masked]
Objet : RE: [Idr] I-D Action: draft-ietf-idr-error-handling-01.txt

When a speaker sends a malformed attribute and the receiver has to guess at the intention, SOMETHING WENT WRONG and MUST BE FIXED.

We are here just trying to define how to limp along instead of needlessly spreading damage.

It doesn't matter much whether we keep one or the other or neither or withdraw.

Personally, I would treat a repeated attribute as a malformation of that attribute: the receiver can only guess at the intent and the problem needs to be fixed.
However, I'm agreeable to the current draft wording too.

--
Jakob Heitz.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Tuesday, December 27, 2011 2:34 AM
To: Ilya Varlashkin
Cc: [email-masked]
Subject: Re: [Idr] I-D Action: draft-ietf-idr-error-handling-01.txt

[Robert]
>If you want to know my preference .. I would prefer to drop the session
rather then to deal with peer which can not even >build the BGP Update message properly.

Dropping session is bad and is really impacting the service, if we can deal errors more smoothly it would be better ...

[Jakob & Bruno]
   >discarded and the UPDATE message continue to be processed.

IMHO, as you mentioned Bruno, the behavior must be in relation with the behavior when the attribute is bad.
We consider here that both attributes are valid (not malformed), so the "treat-as-withdraw"/"attribute-discard" doesn't apply, the only applying thing currently is the "Malformed attribute List" notification.

I see this as follows : if the duplicate (but valid) attribute is a "treat-as-withdraw" attribute, the NLRIs must be withdrawn (otherwise this could cause routing issues).  if the duplicate (but valid) attribute is a "attribute-discard" attribute , I would agree on keeping only one (first or last, it doesn't really matter as we are not able to choose !), the important thing is then to inform the peer and operator of what we done (see next lines ...). 

[iLya] 

I agree with the point to not go to a total opposite direction.
Moreover,it's not the case in the draft, as all attributes that are impacting the routing must use "treat-as-withdraw" behavior.
About the way to inform neighbor about a potential bad update (because some attributes are stripped), maybe we can use some mechanism like "STALE" community proposed in BGP persistence draft to inform peer that this path is maybe bad, so prefer another path if there s some other available (it should be a policy based decision by the SP). And if there is no path, this "STALE" path could be used.
Globally this marking approach could be used each time we are discarding an attribute (duplicate case, or bad attribute with "attribute-discard"
behavior).

[Robert]
>You have received Update message with two MP_UNREACH attributes first
containing 5 NLRIs and the second 50 I don't think >it is safe to drop the second MP_UNREACH and continue processing the message as usual.

A total out of scope suggestion/discussion, why  are we not able to put multiple MP_UNREACH/MP_REACH_NLRI in a BGP message ? Today it's not permitted by the BGP RFC, as these are BGP  attributes (easier to implement than recoding a BGP per AFI/SAFI), so same behavior for all attributes (one per type per update) but globally we are transporting prefixes (and NH) only in these "attributes". So being able to transport multiple occurence of MP_UNR/MP_R (one per AFI/SAFI), would permit a better packing of BGP updates when for example a VRF with common policies is using IPv4,IPv6,mVPN , multiple prefixes could be packed with same attribute sequence. Easy to say, but maybe hard to implement.
There's topics about AFI/SAFI separations/fault isolation, and this would go in an opposite direction ...

I just wanted to have some opinions on that (I was too young when BGP MP extensions were defined :) ).

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, France Telecom - Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci

This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorization.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, France Telecom - Orange shall not be liable if this message was modified, changed or falsified.
Thank you.

---

From: <[email-masked]>
Date: Thu, 29 Dec 2011 09:41:48 +0100

Hi Authors,

In the draft, there is a work started about describing expected behavior
for some attribute : Aggregator, Community, Extcommunity.

How to you want to proceed for all other existing attributes ? And for
futures (the draft has a precision for futures) ? 
Potential solutions :
- make this document as a reference for all attributes (so document is
appended each time a new attribute is defined -> it's not what has been
agreed today)
- make this document as a reference for all already existing attributes
and for futures, the behavior will be described in the associated
draft/RFC
- update all existing draft/RFCs with attribute definition to reflect
the behavior (treat-as-withdraw or attribute-discard).

Currently the document says : "A document which specifies a new
attribute MUST provide specifics
   regarding what constitutes an error for that attribute and how that
   error is to be handled."

So we must deal with all old ones...

IMHO, we should not leave any ambiguity on per attribute behavior. If
only a generic rule is defined as it's done in the draft, vendors may
have a different view about Routing impact of attributes and we will be
exposed to different behavior per vendor and so potential routing issues
...

Current generic rule :"  When a path attribute in an UPDATE message is
determined to be
   malformed, the UPDATE message containing that attribute MUST be
   treated as though all contained routes had been withdrawn just as if
   they had been listed in the WITHDRAWN ROUTES field (or in the
   MP_UNREACH_NLRI attribute [RFC4760bis] if appropriate) of the UPDATE
   message, thus causing them to be removed from the Adj-RIB-In
   according to the procedures of [RFC4271].  In the case of an
   attribute which has no effect on route selection or installation, the
   malformed attribute MAY instead be discarded and the UPDATE message
   continue to be processed.  For the sake of brevity, the former
   approach is termed "treat-as-withdraw", and the latter as "attribute
   discard"."

I would prefer to describe behavior for all existing attributes to
suppress any ambiguity, especially for attributes like Cluster List,
Originator ID ... (I hope a treat as withdraw behavior to prevent
routing loops). Then all others : ATTR_SET, PMSI Tunnel, IPv6 EXT CT ...
The job is not huge, attribute list is not too long.

Thanks to clarify,

Stephane