Subject: [Idr] I-D Action: draft-ietf-idr-error-handling-02.txt

From: [email-masked]
Date: Sun, 17 Jun 2012 13:44:51 -0700

A New Internet-Draft is available from the on-line Internet-Drafts directories.
 This draft is a work item of the Inter-Domain Routing Working Group of the IETF.

	Title           : Revised Error Handling for BGP UPDATE Messages
	Author(s)       : John G. Scudder
                          Enke Chen
                          Pradosh Mohapatra
                          Keyur Patel
	Filename        : draft-ietf-idr-error-handling-02.txt
	Pages           : 10
	Date            : 2012-06-17

Abstract:
   According to the base BGP specification, a BGP speaker that receives
   an UPDATE message containing a malformed attribute is required to
   reset the session over which the offending attribute was received.
   This behavior is undesirable as a session reset would impact not only
   routes with the offending attribute, but also other valid routes
   exchanged over the session.  This document partially revises the
   error handling for UPDATE messages, and provides guidelines for the
   authors of documents defining new attributes.  Finally, it revises
   the error handling procedures for several existing attributes.

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-idr-error-handling

There's also a htmlized version available at:
http://tools.ietf.org/html/draft-ietf-idr-error-handling-02

A diff from previous version is available at:
http://tools.ietf.org/rfcdiff?url2=draft-ietf-idr-error-handling-02

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Mon, 18 Jun 2012 00:29:16 +0000

I took a read on this ( I will read more thoroughly) but one section jumped out at me.  "Operational Considerations".

   Although the "treat-as-withdraw" error-handling behavior defined in
   Section 2 makes every effort to preserve BGP's correctness, we note
   that if an UPDATE received on an IBGP session is subjected to this
   treatment, inconsistent routing within the affected Autonomous System
   may result.  The consequences of inconsistent routing can include
   long-lived forwarding loops and black holes.  While lamentable, this
   issue is expected to be rare in practice, and more importantly is
   seen as less problematic than the session-reset behavior it replaces.

"   When a malformed attribute is indeed detected over an IBGP session,
   we RECOMMEND that routes with the malformed attribute be identified
   and traced back to the ingress router in the network where the routes
   were sourced or received externally, and then a filter be applied on
   the ingress router to prevent the routes from being sourced or
   received.  This will help maintain routing consistency in the
   network."

I don't think the terminology "lamentable" to describe Long lived forwarding loops and black holes is appropriate as it would probably be more than regrettable. Could you expand on the conditions and how BGP would eventually recover. That would be a good addition. I am a bit confused, how does this type of error handling which restarts GR ( As of my last read ) and which BGP persistence lives through change the paradigm of these solutions? There is a notion of treat as withdraw for some attrs, but don't tear down session?? How many malformed updates before the session is torn down? Ever? 

According to the next paragraph ( See below ) there is an expectation that the offending ingress router should be identified, and then a filter should be applied for those paths in a given updated with the malformed attrs :) What is the recommendation for doing this? Will there be a draft whereby the detecting router somehow figures out the offending ingress router, builds routing policy and then sends it there to dynamically create the filter. This seems non-trivial, I guess Flowspec like technology may be used not sure.. Is the ingress router assumed to be in the same AS domain as the detecting router? Administrative domain may span multiple AS domains..

Jim Uttaro

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Sunday, June 17, 2012 4:45 PM
To: [email-masked]
Cc: [email-masked]
Subject: [Idr] I-D Action: draft-ietf-idr-error-handling-02.txt

Abstract:

_______________________________________________
Idr mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/idr

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 18 Jun 2012 18:00:15 +0200

Jim,

> I am a bit confused, how does this type of error handling which
> restarts GR ( As of my last read ) and which BGP persistence lives
> through change the paradigm of these solutions?

Can you please expand on what do you mean that "BGP persistence lives 
through"?

Are you expecting that in the case of malformed update message where 
treat-as-withdraw would be applicable BGP speaker MUST NOT withdraw 
prefixes under the described error condition because neighbor is marked 
as "persistent" ?

Thx,
R.

---

From: "John G. Scudder" <[email-masked]>
Date: Mon, 18 Jun 2012 15:48:46 -0400

Hi Jim,

On Jun 17, 2012, at 8:29 PM, UTTARO, JAMES wrote:

> I don't think the terminology "lamentable" to describe Long lived forwarding loops and black holes is appropriate as it would probably be more than regrettable.

My dictionary says "(of circumstances or conditions) deplorably bad or unsatisfactory" which seems about right to me. We could have said "it really sucks a lot" but the style seems wrong. :-) But this sentence could be rewritten as needed, it's not a big deal.

> Could you expand on the conditions

One very simple case:

  A--B--C--D

The four routers depicted form an IBGP full mesh (sessions not shown). The links shown are the physical topology. Standard IP forwarding is used. Routers A and D are ASBRs and both advertise a path for prefix X into IBGP. Router C prefers the path from A, for example due to a shorter AS Path. However, router B decides that the path from A is malformed, so it selects the path from D. Now we have C's next hop to reach X as B, and B's next hop to reach X as C. This is a stable forwarding loop. (Why do B and C have different conclusions about whether the update is malformed? Different code bases with different assumptions, as we have seen more than once in the field.)

> and how BGP would eventually recover.

It wouldn't :-( without operator intervention. 

> That would be a good addition. I am a bit confused, how does this type of error handling which restarts GR ( As of my last read ) and which BGP persistence lives through change the paradigm of these solutions?

I don't understand this question.

> There is a notion of treat as withdraw for some attrs, but don't tear down session??

Yes. You're right to find this shocking. I find it shocking, but have been browbeaten :-)/2 into agreeing that the cure may be a little less bad than the disease. However, the floor is still definitely open for debate!

> How many malformed updates before the session is torn down? Ever? 

This was discussed a while ago, inconclusively. To date, I don't recall having seen any suggestions I really liked for how to decide a session should be torn down. It actually seems to make things more complicated (to implement, to debug in operation) to have two error-handling modes you flip between according to some heuristic. The floor is still open for more discussion.

> According to the next paragraph ( See below ) there is an expectation that the offending ingress router should be identified, and then a filter should be applied for those paths in a given updated with the malformed attrs :) What is the recommendation for doing this?

The router should log (and possibly trap, alarm, etc) the offending update(s). The operator should examine the logs and by so doing, figure it out from there. Sorry this seems glib but that's what it comes down to. Remember that the alternate (present-day) scenario is that the BGP session to the affected router is going to flap, which is likely even worse operationally, possibly much worse.

> Will there be a draft whereby the detecting router somehow figures out the offending ingress router, builds routing policy and then sends it there to dynamically create the filter.

I sincerely hope not and have spoken against this idea in the past.

> This seems non-trivial,

That's why. When you start building machinery to automatically patch around bugs in your primary routing machinery, IMO you have officially Jumped The Shark and started building a Rube Goldberg router.

> I guess Flowspec like technology may be used not sure.. Is the ingress router assumed to be in the same AS domain as the detecting router?

Since the paragraph is talking about IBGP, yes. Of course this doesn't mean the offending update was sourced within the same AS, but the paragraph you're picking on is just talking about what happens within IBGP.

Note that draft-ietf-grow-ops-reqs-for-bgp-error-handling-04.txt is very applicable to this discussion and is currently in WGLC in GROW. Since you have opinions about this topic I strongly suggest you review it and comment on the GROW WGLC. It runs until June 25.

--John

> Administrative domain may span multiple AS domains..
> 
> Jim Uttaro
> 
> 
> 
>

---

From: Jakob Heitz <[email-masked]>
Date: Mon, 18 Jun 2012 16:08:19 -0400

On Monday, June 18, 2012 12:49 PM, John G. Scudder <> wrote:

>   A--B--C--D
> 
> The four routers depicted form an IBGP full mesh (sessions not
> shown). The links shown are the physical topology. Standard IP
> forwarding is used. Routers A and D are ASBRs and both advertise a
> path for prefix X into IBGP. Router C prefers the path from A, for
> example due to a shorter AS Path. However, router B decides that the
> path from A is malformed, so it selects the path from D. Now we have
> C's next hop to reach X as B, and B's next hop to reach X as C. This
> is a stable forwarding loop. (Why do B and C have different
> conclusions about whether the update is malformed? Different code
> bases with different assumptions, as we have seen more than once in
> the field.)          

The draft is not changing the way that a "malformed" decision is
made. It is changing the resulting action: reset session or drop route.

Either way, your problem occurs.

-- 
Jakob Heitz.

---

From: "John G. Scudder" <[email-masked]>
Date: Mon, 18 Jun 2012 16:19:13 -0400

[Trimmed i-d-announce and internet-drafts from cc list]

> Either way, your problem occurs.

Yes, exactly. The difference is in the consequences, and likelihood of detection. The consequences of the session flap are likely more dire, but it's a sure bet that the operator will notice. The consequences of the forwarding loop may be bad but possibly still remain below the radar. It's our responsibility to make sure the operator is able to notice what's going on even without having sessions bounce like a ball.

--John

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Mon, 18 Jun 2012 21:13:04 +0000

John,

	The notion of this draft is to change the actual behavior of the protocol in terms of the resultant action due to a update error malformed attr.. Currently the session is torn down.. Another behavior would be timer expiry, in that case the actual NH of the update is correct, but a timer expiry between a RR and the egress PE also tears down the session. Should we change the specification to deal with a bug where a RR cannot maintain sessions? Are there addl bugs in the protocol implementation that are being considered ?

 	The BGP persistence draft and GR to a lesser extent as it does not survive as many error conditions accepts the base specification but simply allows paths to persist without changing the fundamental nature of the protocol.. Why do we need this solution also? If the egress PE used BGP Persistence or GR ( Assume modified to persist through mal-formed attr ), then the current specification continues unchanged, and valid paths persist..

Another observation is that the draft attempts to provide a finer level of granularity in terms of specification for some errors. The resultant action is no longer at a session but at a path level. 

If one path has a mal-formed update I believe that all of the paths in that update are treated as withdrawn . If so, then this fix discards possibly different sets of good paths at different egress PEs? This may create inconsistent routing topologies as different "good paths" may be discarded based on update packing at the ingress PE. So not an inconsistency for the path with the mal-formed attr but possibly all the others packed in..

Comments In-Line..

Thanks,
	Jim Uttaro

-----Original Message-----
From: John G. Scudder [mailto:[email-masked]] 
Sent: Monday, June 18, 2012 3:49 PM
To: UTTARO, JAMES
Cc: [email-masked]'; [email-masked]; [email-masked]
Subject: Re: [Idr] I-D Action: draft-ietf-idr-error-handling-02.txt

Hi Jim,

My dictionary says "(of circumstances or conditions) deplorably bad or unsatisfactory" which seems about right to me. We could have said "it really sucks a lot" but the style seems wrong. :-) But this sentence could be rewritten as needed, it's not a big deal.
[Jim U>] Agreed it is bad/unsatisfactory. I guess I was addressing the premise.

One very simple case:

  A--B--C--D

The four routers depicted form an IBGP full mesh (sessions not shown). The links shown are the physical topology. Standard IP forwarding is used. Routers A and D are ASBRs and both advertise a path for prefix X into IBGP. Router C prefers the path from A, for example due to a shorter AS Path. However, router B decides that the path from A is malformed, so it selects the path from D. Now we have C's next hop to reach X as B, and B's next hop to reach X as C. This is a stable forwarding loop. (Why do B and C have different conclusions about whether the update is malformed? Different code bases with different assumptions, as we have seen more than once in the field.)
[Jim U>] Ouch. We could get some big flows going back and forth in our topology based upon the path in question... would this be trading forwarding loops within a network for routing churn amongst networks and/or traffic re-direction.. It sort of feels that way to me..

[Jim U>] Ouch +1.. 

I don't understand this question.
[Jim U>] Assume BGP current behavior.. The sessions would be torn down, BGP Persist/GR ( Assume GR is modified to persist through a mal-formed attr ) would become active.. In this model for a subset of failure modes the session would not be torn down. I guess the issue is that for different error conditions we see different solutions/behaviors.. If malformed then no session tear down, treat as withdrawn, If timer expiry than if GR tear session down flush paths, BGP Persistence session tear down maintain paths. I am war of the lack of consistency..

Yes. You're right to find this shocking. I find it shocking, but have been browbeaten :-)/2 into agreeing that the cure may be a little less bad than the disease. However, the floor is still definitely open for debate!
[Jim U>] Ouch.. I don't like the notion of changing the fundamental nature of the protocol.. 

This was discussed a while ago, inconclusively. To date, I don't recall having seen any suggestions I really liked for how to decide a session should be torn down. It actually seems to make things more complicated (to implement, to debug in operation) to have two error-handling modes you flip between according to some heuristic. The floor is still open for more discussion.
[Jim U>] Maybe determine the exposure to inconsistency.. 

The router should log (and possibly trap, alarm, etc) the offending update(s). The operator should examine the logs and by so doing, figure it out from there. Sorry this seems glib but that's what it comes down to. Remember that the alternate (present-day) scenario is that the BGP session to the affected router is going to flap, which is likely even worse operationally, possibly much worse.
[Jim U>] If we could persist the state than the damage from the flap should be mitigated.. Of course this is different based on application i.e internet, VPNV4, vPNV2 etc... We would need to think about how to automate this..

[Jim U>] LOL

That's why. When you start building machinery to automatically patch around bugs in your primary routing machinery, IMO you have officially Jumped The Shark and started building a Rube Goldberg router.
[Jim U>] Yes. If there is a bug that should be fixed.. There are solution that allow state to persist.. I would prefer this as opposed to modifying the specification to deal with this one type of bug? BTW Is it extensible to other Update Message Errors?? 

Since the paragraph is talking about IBGP, yes. Of course this doesn't mean the offending update was sourced within the same AS, but the paragraph you're picking on is just talking about what happens within IBGP.
[Jim U>] Yup. Assuming same AS then like flowspec there may be a way but again it is patching a bug in the machinery.. 

Note that draft-ietf-grow-ops-reqs-for-bgp-error-handling-04.txt is very applicable to this discussion and is currently in WGLC in GROW. Since you have opinions about this topic I strongly suggest you review it and comment on the GROW WGLC. It runs until June 25.[Jim U>] I will.. Thx...

--John

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Mon, 18 Jun 2012 21:32:29 +0000

Jakob,

	I think it is a question of inflicted the least amount of damage. Actually brings up an interest question.. Is this service specific, is the same tradeoff work for internet and VPNV2 solutions?

Jim Uttaro

-----Original Message-----
From: Jakob Heitz [mailto:[email-masked]] 
Sent: Monday, June 18, 2012 4:08 PM
To: John G. Scudder; UTTARO, JAMES
Cc: [email-masked]; [email-masked]'; [email-masked]
Subject: RE: [Idr] I-D Action: draft-ietf-idr-error-handling-02.txt

-- 
Jakob Heitz.

---

From: <[email-masked]>
Date: Tue, 19 Jun 2012 07:47:56 +0000

Hi John, all,

>From: John G. Scudder >Sent: Monday, June 18, 2012 9:49 PM

[...]
>The four routers depicted form an IBGP full mesh (sessions not shown). The
>links shown are the physical topology. Standard IP forwarding is used. Routers
>A and D are ASBRs and both advertise a path for prefix X into IBGP. Router C
>prefers the path from A, for example due to a shorter AS Path. However, router
>B decides that the path from A is malformed, so it selects the path from D. Now
>we have C's next hop to reach X as B, and B's next hop to reach X as C. This is
>a stable forwarding loop. (Why do B and C have different conclusions about
>whether the update is malformed? Different code bases with different
>assumptions, as we have seen more than once in the field.)

draft-ietf-idr-error-handling-02: "The consequences of inconsistent routing can include long-lived forwarding loops and black holes."

IMHO I would favor expanding a bit on this as this is an important point. E.g.
1) adding your simple and very illustrative example in appendix.
2) discussing the applicability of such consequences. E.g. I feel that they are different for AS/applications using tunnels between ASBR vs hop by hop routing ones(*). Whether I'm right or wrong, IMHO a clarification is always helpful.

[...]

"For any malformed attribute which is handled by the "attribute
   discard" instead of the "treat-as-withdraw" approach, it is critical
   to consider the potential impact of doing so."

IMHO I would rather move that paragraph from the "4. Operational Considerations" to "2. Revision to Base Specification" just below "A document which specifies a new attribute MUST provide specifics
   regarding what constitutes an error for that attribute and how that error is to be handled." (unless this choice is left to operators, which does not seem to be the case)

[...]

>Yes. You're right to find this shocking. I find it shocking, but have been
>browbeaten :-)/2 into agreeing that the cure may be a little less bad than the
>disease. However, the floor is still definitely open for debate!

IMHO 
- we should be cautious when changing the "fundamental nature of the protocol"
- but we should be able to question (, discuss and exchange arguments on) _any_ aspects of the protocol.

And on a side note, the notion of "fundamental nature of the protocol" is personal (not to say religious). E.g. the fundamental nature of BGP is Internet IDR. Using it for L3 VPN -not to mention L2 VPN :-) - is against nature.

Regards,
Bruno

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
France Telecom - Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, France Telecom - Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Tue, 19 Jun 2012 13:56:38 +0000

Bruno,

	Comments In-Line

Thanks,
	Jim Uttaro

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Tuesday, June 19, 2012 3:48 AM
To: John G. Scudder; UTTARO, JAMES
Cc: [email-masked]
Subject: RE: [Idr] I-D Action: draft-ietf-idr-error-handling-02.txt

Hi John, all,

[...]

[...]

[...]

IMHO 

And on a side note, the notion of "fundamental nature of the protocol" is personal (not to say religious). E.g. the fundamental nature of BGP is Internet IDR. Using it for L3 VPN -not to mention L2 VPN :-) - is against nature.
[Jim U>] My thought here was that the resultant action would affect paths by "treat as withdraw", on other cases the resultant action would be at the session level.. I know of no other change in other AFs that changes the resultant action in this way.. Could be but I am not aware...

Regards,
Bruno

---

From: "John G. Scudder" <[email-masked]>
Date: Wed, 20 Jun 2012 18:48:35 -0400

[trimmed cc]

Jim,

>        The notion of this draft is to change the actual behavior of the protocol in terms of the resultant action due to a update error malformed attr.. Currently the session is torn down.. Another…

No. If you or another WG member would like to propose such, feel free to bring it up. 

>        The BGP persistence draft and GR to a lesser extent as it does not survive as many error conditions accepts the base specification but simply allows paths to persist without changing the fundamental nature of the protocol.. Why do we need this solution also? If the egress PE used BGP Persistence or GR ( Assume modified to persist through mal-formed attr ), then the current specification continues unchanged, and valid paths persist.

There are several answers to this but the most fundamental is that if you've got a poison route in the system then when you restart your session (whether with GR, persistence, or what-have-you) you'll eventually get the poison route again, reset the session again, repeat forever. In the mean time your routing will be badly degraded and anything that would arrive after the poison route in the initial convergence will be completely stuck since you'll always reset before seeing it. 

By ignoring the poison route you allow the rest of your routes to continue getting service. You may even be able to use an alternate path toward the prefixes in the poison route. 

> If one path has a mal-formed update I believe that all of the paths in that update are treated as withdrawn . If so, then this fix discards possibly different sets of good paths at different egress PEs?

I guess? Though if you're distributing a consistent set of routes within your AS I think you'll normally end up discarding them consistently too because they'll arrive with the same (presumed to be malformed) attributes, whether packed into one update or spread across several. OTOH if you're not distributing a consistent set of routes within your AS then yes, you are hoist on your own petard. 

Anything can happen of course if you posit just the right (wrong) set of bugs, but that way lies madness. I think in reasonably foreseeable circumstances things will tend to go as I've described above. Even in the case where inconsistency ensues, well, this is already broadly speaking a known side-effect of the proposal and the notion is it's acceptable if the other alternative is session reset. 

> This may create inconsistent routing topologies as different "good paths" may be discarded based on update packing at the ingress PE. So not an inconsistency for the path with the mal-formed attr but possibly all the others packed in..

By definition all the prefixes packed in the same update share the same (presumed to be malformed) attributes, so there is zero chance of an "innocent" route catching a drive-by bullet as you describe. 

> Comments In-Line..

Ditto. 

> [Jim U>] Ouch. We could get some big flows going back and forth in our topology based upon the path in question... would this be trading forwarding loops within a network for routing churn amongst…

Potentially, yes. Devil, deep blue sea. One thing to observe is that a flapping bgp session in the middle of your network is the other alternative in this scenario. Does that sound better to you? Most reviewers have thought it sounded worse. 

> [Jim U>] Assume BGP current behavior.. The sessions would be torn down, BGP Persist/GR ( Assume GR is modified to persist through a mal-formed attr ) would become active..

See discussion above. 

> In this model for a subset of failure modes the session would not be torn down. I guess the issue is that for different error conditions we see different solutions/behaviors.. If malformed then no session tear down, treat as withdrawn, If timer expiry than if GR tear session down flush paths, BGP Persistence session tear down maintain paths. I am war of the lack of consistency..

I think this is a legitimate case of different diseases having different cures, although as Enke (I think) mentioned in a different context, if we can come up with a single, simpler, unified solution to these problems then that's great. Big "if", though. 

> [Jim U>] If we could persist the state than the damage from the flap should be mitigated..

Again, see above. 

> Of course this is different based on application i.e internet, VPNV4, vPNV2 etc... We would need to think about how to automate this..
> 

The idea is to cover pretty much every message error, as long as the prefixes still can be extracted and the update boundaries determined, yes. 

Thanks,  

--John

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Fri, 22 Jun 2012 17:09:12 +0000

John,

	Comments In-Line

Thanks,
	Jim Uttaro

-----Original Message-----
From: John G. Scudder [mailto:[email-masked]] 
Sent: Wednesday, June 20, 2012 6:49 PM
To: UTTARO, JAMES
Cc: John Scudder; [email-masked]
Subject: Re: [Idr] I-D Action: draft-ietf-idr-error-handling-02.txt

[trimmed cc]

Jim,

[Jim U>] I commented on Rob's draft in GROW and I feel that this is a great opportunity to consider a number of session failures beyond the Update variety.. I will pursue with Rob

[Jim U>] I think two cases here. The first is if the speaker is initially coming up, then all paths up to the one with the mal-formed would be retained and valid, the poison path would cause a failure of the session, all subsequent paths are not received..The second is a speaker getting updates in response to normal behavior or a soft refresh.. In either of these cases the paths already learned are programmed and not affected or better said stale... IMO it is best to consider these cases independently, in the former I do not want the session established s there is something fundamentally wrong with the machinery, in the latter I have all paths learned already and am in steady state..

[Jim U>] See Above

I guess? Though if you're distributing a consistent set of routes within your AS I think you'll normally end up discarding them consistently too because they'll arrive with the same (presumed to be malformed) attributes, whether packed into one update or spread across several. OTOH if you're not distributing a consistent set of routes within your AS then yes, you are hoist on your own petard. 
[Jim U>] Have to think about this some more...

Anything can happen of course if you posit just the right (wrong) set of bugs, but that way lies madness. I think in reasonably foreseeable circumstances things will tend to go as I've described above. Even in the case where inconsistency ensues, well, this is already broadly speaking a known side-effect of the proposal and the notion is it's acceptable if the other alternative is session reset. 
[Jim U>] Do not agree. Session reset is fine if forwarding is retained.. That is the end goal not maintain a session and thus forwarding is retained. As I commented on Rob's draft, we need to maintain forwarding through this and other failure modes of a session, we cannot have different bottom up solutions for different errors..

By definition all the prefixes packed in the same update share the same (presumed to be malformed) attributes, so there is zero chance of an "innocent" route catching a drive-by bullet as you describe. 
[Jim U>] Ok. I thought the packing paradigm did not require all the exact same attributes.. 

Ditto. 

See discussion above. 

Again, see above. 

Thanks,  

--John

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Fri, 22 Jun 2012 22:58:53 +0000

John,

	Apologies, I missed the section below. Comments In-line

Jim Uttaro

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of UTTARO, JAMES
Sent: Friday, June 22, 2012 1:09 PM
To: 'John G. Scudder'
Cc: [email-masked]
Subject: Re: [Idr] I-D Action: draft-ietf-idr-error-handling-02.txt

*** Security Advisory: This Message Originated Outside of AT&T ***.
Reference http://cso.att.com/EmailSecurity/IDSP.html for more information.

John,

	Comments In-Line

Thanks,
	Jim Uttaro

-----Original Message-----
From: John G. Scudder [mailto:[email-masked]] 
Sent: Wednesday, June 20, 2012 6:49 PM
To: UTTARO, JAMES
Cc: John Scudder; [email-masked]
Subject: Re: [Idr] I-D Action: draft-ietf-idr-error-handling-02.txt

[trimmed cc]

Jim,

[Jim U>] See Above

Ditto. 

>>[Jim U>] If my forwarding remained stable ( GR, Persistence ) I would prefer the session flap to the network flows as the network is still stable in both the forwarding and control planes ( Control Plane is Working ).. The traffic flow size, QOS etc is not deterministic this is of concern to me as classes of service may be prioritized along certain paths in the network... Too be honest again this may service specific..

See discussion above. 

>>[Jim U>] Yes agreed. But philosophically I think there is a big change with this draft. It takes the position that a subset of error codes should be addressed at a path vs session level.. This leads to big complexity in understanding what is happening in the network, and notifying/correcting/fixing part of the equation.

Again, see above. 

>>[Jim U>] Ok..

Thanks,  

--John

---

From: "Simpson, Adam (Adam)" <[email-masked]>
Date: Thu, 28 Jun 2012 10:02:20 -0500

I have a few comments on the new version of this draft.

My first comment is a general one: I think the draft is clear in terms of goals and overall framework but still lacking important details...important if we want different BGP implementations following this draft to react more or less the same way to the same error(s) in an Update message. I think minor differences in how certain errors are handled is inevitable and probably acceptable but if the same basic error causes a session reset in some part of the network and 'treat as withdraw' somewhere else that could really complicate operations and troubleshooting. 

I think someone suggested, in their feedback to draft-01, that a table giving the expected handling for each category of error would be better than this alone: "The error handling of all other cases described in Section 6.3 of [RFC4271] that specify a session reset is revised as follows..." (section 2 of the draft). I agree with that. In fact, what I think would be even better would be some sort of flowchart in text form....i.e. receive an Update, do these length checks {a, b, c}, if they all pass then parse the NLRI, if all NLRI are decipherable then check other attributes for errors, for attribute X its length should be N or else it is an error...

However the extra detail is provided, here are some of the points/questions I would like to see better explained in the next version:

1. The only length check that seems to be a 'critical' error in the draft is this: "if Withdrawn Routes Length + Total Attribute Length + 23 exceeds the message Length), then the Error Subcode MUST be set to Malformed Attribute List" [causing a session reset.] But what should happen if these other length checks don't work out:

(a) Message length <> withdrawn routes length + $B-t(B length of each individual attribute (as reported in attribute length) + $B-t(B length of each individual IPv4 NLRI (at the end of the Update message)
(b) Total attribute length <> $B-t(B length of each individual attribute (as reported in attribute length)
(c) Length of a TLV encoded attribute <> $B-t(B length of each individual TLV (as reported in TLV length)

To me all of these errors introduce uncertainty about whether we have identified and parsed the NLRI correctly, but if the authors disagree, then they should at least discuss why in the draft.

2. The draft still states that we should ignore all but the first attribute with a specific type code if the attribute is duplicated in the message. Surely that does not apply to MP_REACH_NLRI. There was also debate last time around about whether we should instead select the first attribute with a specific type *that appears correct*. Was that suggestion considered?

3. Given the importance of the MP_REACH_NLRI attribute to the treat-as-withdrawn approach I think it would be appropriate to state what errors in that attribute specifically are expected to be recoverable and which ones are not. For example, is a next-hop-length that is incorrect for the AFI/SAFI recoverable using treat-as-withdraw or should we assume that perhaps the NLRI fields are not starting where we think they are. And if treat-as-withdraw should be applied when there is an error within an MP_REACH_NLRI should we really withdraw all the NLRI in the Update message (as currently implied by the draft) or should we only withdraw the NLRI in the MP_REACH_NLRI?

Thanks,
Adam

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Sunday, June 17, 2012 4:45 PM
To: [email-masked]
Cc: [email-masked]
Subject: [Idr] I-D Action: draft-ietf-idr-error-handling-02.txt

Abstract:

---

From: Jakob Heitz <[email-masked]>
Date: Thu, 28 Jun 2012 12:14:27 -0400

All but the trivial length errors can not be verified, so no attempt is made. If a length is wrong, it will manifest itself as an error in subsequent parsing. The draft should explain this.

--
Jakob Heitz.

On Jun 28, 2012, at 8:02 AM, "Simpson, Adam (Adam)" <[email-masked]> wrote:

> (a) Message length <> withdrawn routes length + ㊥ length of each individual attribute (as reported in attribute length) + ㊥ length of each individual IPv4 NLRI (at the end of the Update message)
> (b) Total attribute length <> ㊥ length of each individual attribute (as reported in attribute length)
> (c) Length of a TLV encoded attribute <> ㊥ length of each individual TLV (as reported in TLV length)
>

---

From: Enke Chen <[email-masked]>
Date: Thu, 28 Jun 2012 11:48:43 -0700

Hi, Adam:

Thanks for your review and comments.  We will work on addressing yours 
and Bruno's comments.

-- Enke

On 6/28/12 8:02 AM, Simpson, Adam (Adam) wrote:
> (a) Message length<>  withdrawn routes length + ? length of each individual attribute (as reported in attribute length) + ? length of each individual IPv4 NLRI (at the end of the Update message)
> (b) Total attribute length<>  ? length of each individual attribute (as reported in attribute length)
> (c) Length of a TLV encoded attribute<>  ? length of each individual TLV (as reported in TLV length)
>

---

From: Brian Dickson <[email-masked]>
Date: Thu, 28 Jun 2012 16:56:41 -0400

On Mon, Jun 18, 2012 at 3:48 PM, John G. Scudder <[email-masked]> wrote:

> links shown are the physical topology. Standard IP forwarding is used.
> Routers A and D are ASBRs and both advertise a path for prefix X into IBGP.
> Router C prefers the path from A, for example due to a shorter AS Path.
> However, router B decides that the path from A is malformed, so it selects
> the path from D. Now we have C's next hop to reach X as B, and B's next hop
> to reach X as C. This is a stable forwarding loop. (Why do B and C have
> different conclusions about whether the update is malformed? Different code
> bases with different assumptions, as we have seen more than once in the
> field.)

This is a very useful example, and has prompted me to think about the
problem space. See below for discussion (by me).

>
> > Will there be a draft whereby the detecting router somehow figures out
> the offending ingress router, builds routing policy and then sends it there
> to dynamically create the filter.
>
> That's why. When you start building machinery to automatically patch
> around bugs in your primary routing machinery, IMO you have officially
> Jumped The Shark and started building a Rube Goldberg router.
>

Patch around bugs, no. However, achieving AS-wide loop-free routing should
be considered an explicit goal, and to that end, how such errors get
handled is pretty important.

That may mean a first-order method of achieving consistency is needed, i.e.
one that involves very little semantics, and doesn't attempt to overload
existing per-AFI/SAFI machinery.

Here are what I see as problems when a subset of IBGP speakers think a
given update is malformed:

   - In order to prevent a routing INFORMATION loop, malformed updates need
   to be matched against previous corresponding non-malformed updates (if any)
   - A router that wants to avoid forwarding packets inside a routing loop
   (where the loop was induced by treat-as-withdrawn), would possibly need to
   basically do RPF-checks on the interface it chose as next-hop for those
   packets
   - Any attempt to infer whether a given router should selectively also
   withdraw a given route from its Adj-IN-RIB, to avoid a routing loop induced
   by treat-as-withdrawn by an IBGP peer, requires combining IGP and IBGP
   routing information, need to consider this on a hop-by-hop basis, and can
   only identify route-loops involving an immediately adjacent (topologically
   speaking) router. This requires hop-by-hop, route-by-route, iterative
   action, which clearly scales, in arbitrary topologies, arbitrarily badly.

Unless all route instances are collectively marked as malformed, there
should be one or more non-malformed routes available for use.

The easiest way to match updates that _some_ router inside a given AS
thinks are malformed, against the original non-malformed announcement, is
to have the iBGP speaker do the "treat-as-withdrawn" action itself.

This would accomplish several things:

   - Every router in the AS would receive the withdrawal (from the original
   iBGP speaker, i.e. with correct behavior from the stateful perspective)
   - Every router in the AS would have the same BGP information from which
   to choose best path (and is more likely to converge loop-free)
   - Only the original iBGP speaker would need to keep state regarding the
   route that is now under "treat-as-withdrawn"
   - Logging from the two iBGP speakers (the one that believes the update
   was malformed, and the one that sent the malformed update) is enough to
   isolate the problem quickly
   - Malformed updates would not propagate (this is a big plus)
   - Only updates that all vendors/code-bases believe are non-malformed can
   propagate beyond the AS (a huge plus - trap the error at the ingress
   boundaries)
   - No flapping sessions (a huge plus)

The absolute minimum needed to achieve this basically is, for the
receiving/detecting/reporting iBGP speaker to send back the UPDATE message,
wrapped in some other message type, to the original sender (or targeted at
the sender but relayed to the RR, if an RR cluster list is present).

In order to make this simple and to scale implementation, the idea I have
is to use a single, new AFI/SAFI, which is "Malformed", and whose message
includes the AFI/SAFI and everything else from the original UPDATE.

Upon receipt of such a Malformed message, the (presumed) ASBR would then
send do "treat-as-withdrawn", mark the original ingress NLRI as "reported
malformed", in such a way as to require operator intervention to clear the
"malformed" state (and thus not repeatedly re-propagate updates and
contribute churn related to malformed updates).

The motivation for this is - don't create persistent routing loops (and
don't check for an error condition you don't know how to handle :-)).

This is only for the iBGP case, which is IMHO the most important one. Not
sure if this logic can or should extend to the eBGP environment, but I
think it can be inferred from the above that this would not be strictly
necessary. Routing updates that are seen by an entire AS as well-formed
means reachability is fine for that AS. If other AS neighbors don't agree
then they won't use the affected routes, so things still result in
loop-free forwarding and reachability is maximized.

Thoughts?

Does this make sense, and would this fit in the proposal, or should it go
in a separate draft?

Brian

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 28 Jun 2012 23:11:37 +0200

Hi Brian,

Excellent analysis. But let me ask a very basic question.

You said:

 > This is only for the iBGP case, which is IMHO the most important one.

How much from what you have suggested do you think applies if one uses 
IP (or MPLS) encapsulation within the AS ingress to egress ?

Optionally one could make sure next hop self is not used on the ASBRs 
and perform direct encapsulation to the external AS (no IP lookup at the 
egress ASBR).

In those cases do you think you still see real value for the proposed 
below error handling ?

Best regards,
R.

 > On Mon, Jun 18, 2012 at 3:48 PM, John G. Scudder <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     Hi Jim,
>
>      > Will there be a draft whereby the detecting router somehow
>     figures out the offending ingress router, builds routing policy and
>     then sends it there to dynamically create the filter.
>
>     around bugs in your primary routing machinery, IMO you have
>     officially Jumped The Shark and started building a Rube Goldberg router.
>
>
> Patch around bugs, no. However, achieving AS-wide loop-free routing
> should be considered an explicit goal, and to that end, how such errors
> get handled is pretty important.
>
> That may mean a first-order method of achieving consistency is needed,
> i.e. one that involves very little semantics, and doesn't attempt to
> overload existing per-AFI/SAFI machinery.
>
>   * In order to prevent a routing INFORMATION loop, malformed updates
>     need to be matched against previous corresponding non-malformed
>     updates (if any)
>   * A router that wants to avoid forwarding packets inside a routing
>     loop (where the loop was induced by treat-as-withdrawn), would
>     possibly need to basically do RPF-checks on the interface it chose
>     as next-hop for those packets
>   * Any attempt to infer whether a given router should selectively also
>     withdraw a given route from its Adj-IN-RIB, to avoid a routing loop
>     induced by treat-as-withdrawn by an IBGP peer, requires combining
>     IGP and IBGP routing information, need to consider this on a
>     hop-by-hop basis, and can only identify route-loops involving an
>     immediately adjacent (topologically speaking) router. This requires
>     hop-by-hop, route-by-route, iterative action, which clearly scales,
>     in arbitrary topologies, arbitrarily badly.
>
>
> thinks are malformed, against the original non-malformed announcement,
> is to have the iBGP speaker do the "treat-as-withdrawn" action itself.
>
>   * Every router in the AS would receive the withdrawal (from the
>     original iBGP speaker, i.e. with correct behavior from the stateful
>     perspective)
>   * Every router in the AS would have the same BGP information from
>     which to choose best path (and is more likely to converge loop-free)
>   * Only the original iBGP speaker would need to keep state regarding
>     the route that is now under "treat-as-withdrawn"
>   * Logging from the two iBGP speakers (the one that believes the update
>     was malformed, and the one that sent the malformed update) is enough
>     to isolate the problem quickly
>   * Malformed updates would not propagate (this is a big plus)
>   * Only updates that all vendors/code-bases believe are non-malformed
>     can propagate beyond the AS (a huge plus - trap the error at the
>     ingress boundaries)
>   * No flapping sessions (a huge plus)
>
> receiving/detecting/reporting iBGP speaker to send back the UPDATE
> message, wrapped in some other message type, to the original sender (or
> targeted at the sender but relayed to the RR, if an RR cluster list is
> present).
>
> In order to make this simple and to scale implementation, the idea I
> have is to use a single, new AFI/SAFI, which is "Malformed", and whose
> message includes the AFI/SAFI and everything else from the original UPDATE.
>
> send do "treat-as-withdrawn", mark the original ingress NLRI as
> "reported malformed", in such a way as to require operator intervention
> to clear the "malformed" state (and thus not repeatedly re-propagate
> updates and contribute churn related to malformed updates).
>
> This is only for the iBGP case, which is IMHO the most important one.
> Not sure if this logic can or should extend to the eBGP environment, but
> I think it can be inferred from the above that this would not be
> strictly necessary. Routing updates that are seen by an entire AS as
> well-formed means reachability is fine for that AS. If other AS
> neighbors don't agree then they won't use the affected routes, so things
> still result in loop-free forwarding and reachability is maximized.
>
> Thoughts?
>
> Does this make sense, and would this fit in the proposal, or should it
> go in a separate draft?
>
> Brian
>
>

---

From: "John G. Scudder" <[email-masked]>
Date: Fri, 29 Jun 2012 17:55:46 -0400

Hi Brian,

This is a good analysis. I'd like to point out that this topic was discussed briefly at our IETF-82 meeting. In particular, see slide 4 of http://www.ietf.org/proceedings/82/slides/idr-5.pdf:

	• Not pursuing automation of the procedures in protocol 
		- Issues from EBGP typically dominate
		- Require significant changes to the protocol

Your analysis is a good illustration of "require significant changes to the protocol". If the pathology in question were considered to be either sufficiently high-probability, or sufficiently grave even if low-probability, that could motivate work on an approach such as you describe. However I think that is not the case. Recall that the types of errors we are mitigating against have only been seen in the field rarely. The motivation for proposing the changes in idr-error-handling is that the consequences can be quite grave. On the flip side, the cost of implementing idr-error-handling is relatively modest and equally important, it can be deployed piecemeal. On the other hand, the potential forwarding loops that we're talking about would occur only in a (possibly small) subset of these incidents, which themselves only occur with low probability. Furthermore, as Robert points out, in a network that uses edge-to-edge tunneling, as is quite common, looping will not occur. The cost of implementing the solution appears quite a bit higher and it must be deployed in a coordinated fashion. Finally, it's the case that if my argument above turns out to be wrong, a proposal such as yours could be implemented in the future -- lack of it shouldn't impede deployment of idr-error-handling.

For these reasons I would prefer not to pursue this approach right now.

Regards,

--John

P.S.: It should go without saying, but I'm speaking as a WG member.

On Jun 28, 2012, at 4:56 PM, Brian Dickson wrote:

> This is a very useful example, and has prompted me to think about the problem space. See below for discussion (by me).
>  
> 
> Patch around bugs, no. However, achieving AS-wide loop-free routing should be considered an explicit goal, and to that end, how such errors get handled is pretty important.
> 
> That may mean a first-order method of achieving consistency is needed, i.e. one that involves very little semantics, and doesn't attempt to overload existing per-AFI/SAFI machinery. 
> 
> Here are what I see as problems when a subset of IBGP speakers think a given update is malformed:
> 
> 	• In order to prevent a routing INFORMATION loop, malformed updates need to be matched against previous corresponding non-malformed updates (if any)
> 	• A router that wants to avoid forwarding packets inside a routing loop (where the loop was induced by treat-as-withdrawn), would possibly need to basically do RPF-checks on the interface it chose as next-hop for those packets
> 	• Any attempt to infer whether a given router should selectively also withdraw a given route from its Adj-IN-RIB, to avoid a routing loop induced by treat-as-withdrawn by an IBGP peer, requires combining IGP and IBGP routing information, need to consider this on a hop-by-hop basis, and can only identify route-loops involving an immediately adjacent (topologically speaking) router. This requires hop-by-hop, route-by-route, iterative action, which clearly scales, in arbitrary topologies, arbitrarily badly.
> 
> Unless all route instances are collectively marked as malformed, there should be one or more non-malformed routes available for use.
> 
> The easiest way to match updates that _some_ router inside a given AS thinks are malformed, against the original non-malformed announcement, is to have the iBGP speaker do the "treat-as-withdrawn" action itself.
> 
> 	• Every router in the AS would receive the withdrawal (from the original iBGP speaker, i.e. with correct behavior from the stateful perspective)
> 	• Every router in the AS would have the same BGP information from which to choose best path (and is more likely to converge loop-free)
> 	• Only the original iBGP speaker would need to keep state regarding the route that is now under "treat-as-withdrawn"
> 	• Logging from the two iBGP speakers (the one that believes the update was malformed, and the one that sent the malformed update) is enough to isolate the problem quickly
> 	• Malformed updates would not propagate (this is a big plus)
> 	• Only updates that all vendors/code-bases believe are non-malformed can propagate beyond the AS (a huge plus - trap the error at the ingress boundaries)
> 	• No flapping sessions (a huge plus)
> The absolute minimum needed to achieve this basically is, for the receiving/detecting/reporting iBGP speaker to send back the UPDATE message, wrapped in some other message type, to the original sender (or targeted at the sender but relayed to the RR, if an RR cluster list is present).
> 
> In order to make this simple and to scale implementation, the idea I have is to use a single, new AFI/SAFI, which is "Malformed", and whose message includes the AFI/SAFI and everything else from the original UPDATE.
> 
> Upon receipt of such a Malformed message, the (presumed) ASBR would then send do "treat-as-withdrawn", mark the original ingress NLRI as "reported malformed", in such a way as to require operator intervention to clear the "malformed" state (and thus not repeatedly re-propagate updates and contribute churn related to malformed updates).
> 
> The motivation for this is - don't create persistent routing loops (and don't check for an error condition you don't know how to handle :-)).
> 
> This is only for the iBGP case, which is IMHO the most important one. Not sure if this logic can or should extend to the eBGP environment, but I think it can be inferred from the above that this would not be strictly necessary. Routing updates that are seen by an entire AS as well-formed means reachability is fine for that AS. If other AS neighbors don't agree then they won't use the affected routes, so things still result in loop-free forwarding and reachability is maximized.
> 
> Thoughts?
> 
> Does this make sense, and would this fit in the proposal, or should it go in a separate draft?
> 
> Brian