Subject: [Idr] Error Handling for BGP-LS with Segment Routing

From: Alvaro Retana <[email-masked]>
Date: Tue, 18 Dec 2018 13:09:07 -0800

Dear idr and spring WGs:

tl;dr  I don't think that BGP-LS, with error handling as specified
("attribute discard"), can provide the robustness that an application (like
SR), with direct impact on the forwarding in the network, needs.  [Jump to
the bottom for discussion.]

The BGP-LS extensions for SR (e.g.
draft-ietf-idr-bgp-ls-segment-routing-ext) are, as explained in that draft,
used so that "an external component (e.g., a controller) then can collect
SR information from across an SR domain and construct the end-to-end path
(with its associated SIDs) that need to be applied to an incoming packet to
achieve the desired end-to-end forwarding."

To me, that obviously implies that use of BGP-LS for SR has a direct effect
on how traffic is forwarded in the network.  Does any one see it
differently?

The error handling mechanism specified in rfc7752 is "attribute discard"
[rfc7606].  If an error is detected, then the information in the controller
may be, at best, incomplete, but it could also be out of date...resulting
in "segment routes" that don't follow the best available path or that may
even end in a black hole.

It seems clear to me that this is one of the cases that rfc7606 warned
about:

   o  Attribute discard: In this approach, the malformed attribute MUST
      be discarded and the UPDATE message continues to be processed.
      This approach MUST NOT be used except in the case of an attribute
      that has no effect on route selection or installation.

      ...
   For any malformed attribute that is handled by the "attribute
   discard" instead of the "treat-as-withdraw" approach, it is critical
   to consider the potential impact of doing so.  In particular, if the
   attribute in question has or may have an effect on route selection or
   installation, the presumption is that discarding it is unsafe unless
   careful analysis proves otherwise.  The analysis should take into
   account the tradeoff between preserving connectivity and potential
   side effects.

There was a related discussion as a result of my AD review of
draft-ietf-idr-ls-distribution (= rfc7606) [1][2].  At that time (2015),
the consensus on the list was (paraphrasing): if there's a malformed
attribute we won't be able to recover, but that's ok because BGP-LS is
"purely application-level data that has no immediate corresponding
forwarding state impact", and there won't be an impact on critical AFI/SAFI
for network operations.   No one else argued against that...so I ended up
in the rough...

I think the situation has now changed because BGP-LS is carrying SR
information that is used to define paths in the network -- even if
isolation exists, as described in rfc7752:

                 ...    Furthermore, it is anticipated that
   distribution of this NLRI will be handled by dedicated route
   reflectors providing a level of isolation and fault containment
   between different NLRI types.

...the BGP-LS information could still be incomplete, stale, etc..

After all that...  I don't think that BGP-LS, with error handling as
specified ("attribute discard"), can provide the robustness that an
application (like SR), with direct impact on the forwarding in the network,
needs.

What now?  I see several potential paths forward (there are probably more):

(1) "fix" BGP-LS to mandate (MUST) isolation and change the error handling
approach

(2) change the error handling approach...maybe just when used with SR

(3) the controller should only use the SR information received from routing
protocols (IGP/BGP, e.g. draft-ietf-idr-bgp-prefix-sid)

(4) ..??

I didn't find a specific discussion about this topic in the archive...but I
may have missed it in between other related ones.  If I did, please point
me to it.

Thoughts/ideas/comments?

Thanks!

Alvaro.

[1] https://mailarchive.ietf.org/arch/msg/idr/FomvQV2DqjaaRiAcLYLn3LcIdYM
[2] https://mailarchive.ietf.org/arch/msg/idr/wbPNQ-HM2NeR75gR2Or948J9o1I

---

From: <[email-masked]>
Date: Tue, 18 Dec 2018 23:10:19 +0000

Alvaro,

Speaking as an individual IDR contributor,

Interesting discussion.
But error handling in routing is a difficult topic and sometimes a controversial one.

> Thoughts/ideas/comments?

1) shouldn’t BGP-LS error handling be also discussed in the LSVR WG?
https://tools.ietf.org/html/draft-ietf-lsvr-bgp-spf-03#section-5.7 does not seem to cover this.
And this document was under WGLC till yesterday.

2) Regarding BGP-LS error handling, it’s not clear to me that “treat as withdraw” would be “safer” than “Attribute Discard”. “Session reset” is safer from an inconsistency standpoint but definitely also “has a direct effect on how traffic is forwarded in the network” and a sever one.

3)
> The BGP-LS extensions for SR (e.g. draft-ietf-idr-bgp-ls-segment-routing-ext) are, as explained in that draft, used so that "an external component (e.g., a controller) then can collect SR information from across an SR domain and construct the end-to-end path (with its associated SIDs) that need to be applied to an incoming packet to achieve the desired end-to-end forwarding."

> To me, that obviously implies that use of BGP-LS for SR has a direct effect on how traffic is forwarded in the network.  Does any one see it differently?

a) IMHO that implication would be the same without SR, e.g., with RSVP-TE. In fact, the effect on how traffic is forwarded is coming from the PCE computation using partial/incorrect topology information, not how the forwarding is enforced.

b) IMHO RFC7606 was more concerned about forwarding loops/black holing –especially for IBGP-, rather than changing the path of the traffic. (as ““treat as withdraw “ or “sessions reset” would also have “a direct effect on how traffic is forwarded in the network”.) Note that the latter quote is not from RFC760 which uses the terms  “no effect on route selection or installation” which is a bit different.

c) Coming back to SR, quickly looking at the ToC, the discard of the SID simply means that the SID can”t be used by the SR source/ingress node. The discard of the SR node attribute means that the node can’t be used to forward a global segment. The use of flex-algo is a bit more touchy as discarding the support for a flex algo will change the routing along this flex algo. But only from the perspective of the BGP-LS consumer, so this would not create forwarding loops/black hole, but only a non expected routing path.

4)  I haven’t checked but it’s not clear to me that IS-IS has a perfect (better?) error handling.
e.g.,
> “the procedure used to choose which copy shall be used
> is undefined.”
> https://datatracker.ietf.org/doc/html/draft-ietf-isis-rfc4971bis-04

Les or Stefano could comment on IS-IS error handling and there has been a recent related discussion on this https://tools.ietf.org/html/draft-ginsberg-lsr-isis-invalid-tlv
In theory, IS-IS would have a way to signal the error back to the sender/whole network by purging the LSP, which BGP-LS has not. But it”s not signaling the error so the ingress can’t try removing it.
5) BGP-LS and IS-IS have chosen a different granularity to advertise the LSDB (per link/node vs oer LSP) which very likely will result in a different error handling hence a different vision of the topology. This looks like day 1 design choice for BGP-LS, so difficult to address.

Thanks,
--Bruno

From: spring [mailto:[email-masked]] On Behalf Of Alvaro Retana
Sent: Tuesday, December 18, 2018 10:09 PM
To: idr@ietf. org; SPRING WG
Subject: [spring] Error Handling for BGP-LS with Segment Routing

Dear idr and spring WGs:

tl;dr  I don't think that BGP-LS, with error handling as specified ("attribute discard"), can provide the robustness that an application (like SR), with direct impact on the forwarding in the network, needs.  [Jump to the bottom for discussion.]

The BGP-LS extensions for SR (e.g. draft-ietf-idr-bgp-ls-segment-routing-ext) are, as explained in that draft, used so that "an external component (e.g., a controller) then can collect SR information from across an SR domain and construct the end-to-end path (with its associated SIDs) that need to be applied to an incoming packet to achieve the desired end-to-end forwarding."

To me, that obviously implies that use of BGP-LS for SR has a direct effect on how traffic is forwarded in the network.  Does any one see it differently?

The error handling mechanism specified in rfc7752 is "attribute discard" [rfc7606].  If an error is detected, then the information in the controller may be, at best, incomplete, but it could also be out of date...resulting in "segment routes" that don't follow the best available path or that may even end in a black hole.

It seems clear to me that this is one of the cases that rfc7606 warned about:

      ....

There was a related discussion as a result of my AD review of draft-ietf-idr-ls-distribution (= rfc7606) [1][2].  At that time (2015), the consensus on the list was (paraphrasing): if there's a malformed attribute we won't be able to recover, but that's ok because BGP-LS is "purely application-level data that has no immediate corresponding forwarding state impact", and there won't be an impact on critical AFI/SAFI for network operations.   No one else argued against that...so I ended up in the rough...

I think the situation has now changed because BGP-LS is carrying SR information that is used to define paths in the network -- even if isolation exists, as described in rfc7752:

After all that...  I don't think that BGP-LS, with error handling as specified ("attribute discard"), can provide the robustness that an application (like SR), with direct impact on the forwarding in the network, needs.

(1) "fix" BGP-LS to mandate (MUST) isolation and change the error handling approach

(3) the controller should only use the SR information received from routing protocols (IGP/BGP, e.g. draft-ietf-idr-bgp-prefix-sid)

(4) ..??

I didn't find a specific discussion about this topic in the archive...but I may have missed it in between other related ones.  If I did, please point me to it.

Thoughts/ideas/comments?

Thanks!

Alvaro.

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 19 Dec 2018 00:23:08 +0100

Hi Alvaro,

What comes as #1 question to your points is a comparison of SR controller
with regular BGP RR.

I think it is safe to assume that error handling on SR controller would be
no more aggressive then on RRs. So if there is error the updates may be
dropped on the RRs itself, logged and proper NOC alarm generated.

IMO this is no different regardless if you use SR with BGP-LS or just plane
regular BGP routing.

So unless your goal here is to point out the deficiency of BGP error
handling RFC I am not sure what is so specific to BGP-LS and SR.

Yes I am not big proponent of using BGP-LS all together, but not because it
has inherited BGP error handling.

Regards,
R.

On Tue, Dec 18, 2018 at 10:09 PM Alvaro Retana <[email-masked]>
wrote:

> To me, that obviously implies that use of BGP-LS for SR has a direct
> effect on how traffic is forwarded in the network.  Does any one see it
> differently?
>
>
> (3) the controller should only use the SR information received from
> routing protocols (IGP/BGP, e.g. draft-ietf-idr-bgp-prefix-sid)
>
> (4) ..??
>
>
> I didn't find a specific discussion about this topic in the archive...but
> I may have missed it in between other related ones.  If I did, please point
> me to it.
>
> Thoughts/ideas/comments?
>
> Thanks!
>
> Alvaro.
>
> _______________________________________________
> spring mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/spring
>

---

From: Alvaro Retana <[email-masked]>
Date: Wed, 19 Dec 2018 09:57:34 -0800

On December 18, 2018 at 6:10:21 PM, [email-masked] (
[email-masked]) wrote:

Bruno:

Hi!

...

https://tools.ietf.org/html/draft-ietf-lsvr-bgp-spf-03#section-5.7 does not
seem to cover this.

Yes, good point.

I wanted to focus on SR’s use, but I think you’re right to point out that
other applications may have the same needs.  I think/hope that people on
the lsvr list are also on the idr list (at least), so I’ll forward a
pointer to this thread just in case.

2) Regarding BGP-LS error handling, it’s not clear to me that “treat as
withdraw” would be “safer” than “Attribute Discard”. “Session reset” is
safer from an inconsistency standpoint but definitely also “has a direct
effect on how traffic is forwarded in the network” and a sever one.

[Not sure about the ending “…and a sever one”.]

I agree.  I don’t want to rehash the discussion from rfc7606 about the
types of approached and whether there should be more or not (or what those
could be)…. I’m just pointing out that I think the current approach is not
the right one for all applications.

3)

a) IMHO that implication would be the same without SR, e.g., with RSVP-TE.
In fact, the effect on how traffic is forwarded is coming from the PCE
computation using partial/incorrect topology information, not how the
forwarding is enforced.

b) IMHO RFC7606 was more concerned about forwarding loops/black holing
–especially for IBGP-, rather than changing the path of the traffic.
(as ““treat as withdraw “ or “sessions reset” would also have “a
direct effect on how traffic is forwarded in the network”.) Note that
the latter quote is not from RFC760 which uses the terms  “no effect
on route selection or installation” which is a bit different.

Interpreting the difference between “a direct effect on how traffic is
forwarded in the network” and  “no effect on route selection or
installation” is part of the reason this topic is not straight forward.

To me, in the BGP-LS+SR context, because the controller *installs* the
source route at the ingress router, the two phrases apply.

However, other interpretations are possible…which is one of the reasons for
this thread.  For example, during the rfc7752 discussion, a point was made
that the controller (being at the receiving end of the BGP session) would
not have to worry about the effects of attribute discard because any loss
of information would not have an effect on how it (the controller) selected
or installed routes.  That argument is not completely flawed (the
controller does not use the BGP-LS itself for routing), but (my personal
opinion) is that the use of the information (in later programming the
network) is what is important.

c) Coming back to SR, quickly looking at the ToC, the discard of the
SID simply means that the SID can”t be used by the SR source/ingress
node. The discard of the SR node attribute means that the node can’t
be used to forward a global segment. The use of flex-algo is a bit
more touchy as discarding the support for a flex algo will change the
routing along this flex algo. But only from the perspective of the
BGP-LS consumer, so this would not create forwarding loops/black hole,
but only a non expected routing path.

Which ToC?

You’re right…but only if the information is discarded when it was initially
learned.  If the error occurs later, when the information was changing for
example, there is the possibility that the controller will want to use a
node that shouldn’t be used any more….or be calculating
not-the-best-routes.  Sub-optimal routes are not great and may not matter
too much (compared to loops, for example), but some users may have specific
business objectives (application performance, for instance) tied to the
definition of the paths…it will be important to them.

4)  I haven’t checked but it’s not clear to me that IS-IS has a perfect
(better?) error handling.

If you want to discuss this, please do it in the lsr WG. :-)

...

5) BGP-LS and IS-IS have chosen a different granularity to advertise the
LSDB (per link/node vs oer LSP) which very likely will result in a
different error handling hence a different vision of the topology. This
looks like day 1 design choice for BGP-LS, so difficult to address.

Yes…

Thanks!

Alvaro.

---

From: Alvaro Retana <[email-masked]>
Date: Wed, 19 Dec 2018 10:13:32 -0800

On December 18, 2018 at 6:23:19 PM, Robert Raszuk ([email-masked]) wrote:

Robert:

Hi!

In general, I agree that error handling should be the same regardless of
the type of BGP speaker (RR, controller, PE, whatever).

No, the goal is not to point at any deficiency in the error handling RFC.
I just replied to Bruno saying: " I don’t want to rehash the discussion
from rfc7606 about the types of approached and whether there should be more
or not (or what those could be)…. I’m just pointing out that I think the
current approach is not the right one for all applications.”

When BGP-LS was defined, it was noted that the "information present in this
document carries purely application-level data that has no immediate
corresponding forwarding state impact.”  I think that SR has a direct
impact on the forwarding state of the network.  That is what is specific
about BGP-LS+SR.

To be clear, this thread is about using BGP-LS with applications that have
an impact on forwarding/route selection in the network, like SR (Bruno
pointed at lsvr and there may be others).  It is not about about the error
handling approaches (rfc7606) or BGP sessions in general…just that specific
application.

Thanks for helping me clarify what I mean.  Hopefully this makes more
sense. ;-)

Alvaro.

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 19 Dec 2018 20:27:10 +0100

Hi Alvaro,

> When BGP-LS was defined, it was noted that the "information present in
> this document carries purely application-level data that has no immediate
> corresponding forwarding state impact..”  I think that SR has a direct
Ok let's bring this section you partially quoted:

6.1.1 <https://tools.ietf.org/html/rfc7752#section-6.1.1>.  Operations

   Existing BGP operational procedures apply.  No new operation
   procedures are defined in this document.  It is noted that the
NLRI*   information present in this document carries purely
application-level
   data that has no immediate corresponding forwarding state impact.  As
   such, any churn in reachability information has a different impact
*   than regular BGP updates, which need to change the forwarding state
   for an entire router.  Furthermore, it is anticipated that
   distribution of this NLRI will be handled by dedicated route
   reflectors providing a level of isolation and fault containment
   between different NLRI types.

The way I interpret author's intention above was only to say that if you
experience a BGP-LS churn on any BGP speaker which is used to carry it the
impact will not result in RIB/FIB install - as for such BGP speakers the
carried data is of opaque nature as NLRIs are by definition not to be
processed by BGP subsystem.

> handling approaches (rfc7606) or BGP sessions in general…just that specific

Well I think you are mixing the potential impact of BGP speakers which
happen to carry BGP-LS from overall impact to the network which may be
fully or partially derived from content of BGP-LS update messages. To the
best of my knowledge BGP-LS always assumed that carried data will be used
on controllers for something real  :)

I think we all agree that BGP-LS keeps the original promise and its NLRIs
do not interfere with local BGP reachability in the domain. The deployments
may be done not in "anticipated" way of using dedicated RRs but I guess
this was expected from the beginning too.

So now the real bigger question is why one SAFI in BGP would be allowed to
carry forwarding information and other SAFI would not ? Say SAFI 4 defined
in RFC3107 piggybacks labels (clearly forwarding state) as part of IPv4
NLRI. SAFI 128 adds VPN labels (also pretty much forwarding information).
SAFI 71 carries some opaque information which are to be used by
controllers. Yet say BFD SAFI may be used in new ways on RS-ers also to
modify set of paths used in data plane etc ... bottom line is that every
extension we are putting in routing protocols have some impact on
forwarding state in the network be it directly installed by routers, by
controllers, by route servers or even by applications which decided based
on the received info to install new paths to new alternative application
level destinations.

And as always I must say that I would like to see BGP-LS being removed from
BGP4 port 179 and replaced by alternative ways to get IGP information
distributed to controllers, but I just do not see how stated in the subject
of this thread "Error Handling of BGP" could be used as a good reason for
it.

Best wishes,
Robert

---

From: <[email-masked]>
Date: Fri, 21 Dec 2018 14:28:03 +0000

Hi Alvaro,

From: Alvaro Retana [mailto:[email-masked]]
Sent: Wednesday, December 19, 2018 6:58 PM
To: DECRAENE Bruno TGI/OLN
Cc: idr@ietf. org; SPRING WG
Subject: RE: [spring] Error Handling for BGP-LS with Segment Routing

On December 18, 2018 at 6:10:21 PM, [email-masked]<mailto:[email-masked]> ([email-masked]<mailto:[email-masked]>) wrote:

Bruno:

Hi!

...

Yes, good point.

I wanted to focus on SR’s use, but I think you’re right to point out that other applications may have the same needs.  I think/hope that people on the lsvr list are also on the idr list (at least), so I’ll forward a pointer to this thread just in case.

Sorry, I meant « more severe” (serious). IOW the impact on the traffic is more important.

I agree.  I don’t want to rehash the discussion from rfc7606 about the types of approached and whether there should be more or not (or what those could be)…. I’m just pointing out that I think the current approach is not the right one for all applications.

3)

Interpreting the difference between “a direct effect on how traffic is forwarded in the network” and  “no effect on route selection or installation” is part of the reason this topic is not straight forward.

I think RFC 7606 had primarily IP/Internet routes in mind. Within an AS, different IBGP nodes could have a different opinion regarding the error (especially if the error is on the receiver side) which would create difference in route installation and forwarding inconsistencies such as persistent forwarding loops

To me, in the BGP-LS+SR context, because the controller *installs* the source route at the ingress router, the two phrases apply.

However, other interpretations are possible…which is one of the reasons for this thread.  For example, during the rfc7752 discussion, a point was made that the controller (being at the receiving end of the BGP session) would not have to worry about the effects of attribute discard because any loss of information would not have an effect on how it (the controller) selected or installed routes.  That argument is not completely flawed (the controller does not use the BGP-LS itself for routing), but (my personal opinion) is that the use of the information (in later programming the network) is what is important.

Which ToC?

Table of content of https://tools.ietf.org/html/draft-ietf-idr-bgp-ls-segment-routing-ext

You’re right…but only if the information is discarded when it was initially learned.  If the error occurs later, when the information was changing for example, there is the possibility that the controller will want to use a node that shouldn’t be used any more….or be calculating not-the-best-routes.  Sub-optimal routes are not great and may not matter too much (compared to loops, for example), but some users may have specific business objectives (application performance, for instance) tied to the definition of the paths…it will be important to them.

I have already been involved in discussions regarding error handling… I know that the subject is difficult and the opinions are diverse (partly because perspectives are different). One have to choose its battles. I’m happy to share the load with you ;-)

--Bruno

...

Yes…

Thanks!

Alvaro.

---

From: Rob Shakir <[email-masked]>
Date: Fri, 21 Dec 2018 08:23:04 -0800

Alvaro,

I think this is one of the difficulties of overloading a protocol like BGP
with different datasets -- it's not simple to say how particular attributes
are actually going to be used within a protocol deployment. This was one of
the things that was noted in 7606 -- i.e., I can make *any* attribute
really affect forwarding if I write a policy that accepts/rejects some
UPDATE based on the presence of that attribute.

In general, any topology discovery mechanism (whether used in real-time or
not) needs to define how it handles cases where it might end up with
missing information. Let's consider what the different mechanisms for
discovery we have are today:

   - IGP listening -- in this case, if we have some malformed IS-IS TLV,
   then we might end up discarding this information (whether it be at the
   listening node, or a device that didn't flood it earlier in the chain) --
   meaning that we know that we have some potential gap in the topology.
   - Streaming telemetry -- speaking particularly to gNMI for LSDB
   streaming encoded using the OpenConfig model, here, we are tolerant to
   getting as much information as can be parsed, and have a way to carry
   unknown TLVs (which might include those that cannot be successfully parsed)
   as binary data to the external consumer. This means that the approach is
   "as complete data as possible", but has the same characteristic that we can
   also end up having the potential to lose data.
   - BGP-LS with attribute discard -- this has some information loss, since
   we'll have some attributes that could be malformed in the input data, and
   we discard them at the receiver.

It doesn't seem to me that, given the source of the data is the IGP, and we
might have information discarded there -- that we can really guarantee
strong consistency of an off-box view of the network, since we can't
guarantee strong consistency across the IGP domain itself.

Thus, I'm not sure that the issue that is being highlighted here actually
makes a difference when we're considering the overall system design -- we
always need to deal with the fact that the view of the network at the path
computing node might not match exactly the network's current state in the
presence of malformed protocol messages. One motivation for having the LSDB
via streaming telemetry is the ability to provide such validation ("do all
nodes within my IGP domain, including listeners, have a consistent view of
the state of the network?").

If the discussion is "should we adopt treat-as-withdraw vs. attribute
discard?" -- I don't think that from the system perspective there is really
any difference between the two in this situation. We still have the same
potentially inconsistent view of the network.

For these reasons, I'd err on leaving this unchanged in the current
specification(s).

Cheers,
r.

On Wed, Dec 19, 2018 at 10:13 AM Alvaro Retana <[email-masked]>
wrote:

> IMO this is no different regardless if you use SR with BGP-LS or just
> plane regular BGP routing.
>