Subject: [Idr] draft-ietf-idr-bgp-enhanced-route-refresh

From: <[email-masked]>
Date: Thu, 10 Oct 2013 12:45:03 +0000

Hi all,

The draft may have two different goals:
-a- systematic but light checks for refreshing purpose
-b- "on demand" but extensive checks for troubleshooting/debugging purpose

Because If we want to do a full check ("b"), i.e. determine which BGP routing information has not been reflected, we may need to define a real "snapshot" and only sends the update related to this snapshot between the 2 new demarcation messages.
The drawback is that during this time, the BGP convergence is slow down because "real time" BGP updates needs to wait for the end of this snapshot. And this can be long (minutes, ten of minutes). Hence IMO, this is not advisable for all (existing) Route Refresh. I would rather limit such behavior to specific "on demand" request (e.g. through a new RR request sub-type)

OTOH, if we just want to "refresh" the routes ("a"), i.e. know when to purge the non-received one, we can interleave live BGP updates during the refresh. This does not delay BGP convergence but we won't be able to detect if there were inconsistencies and what they were. 

IMO, the draft would need to take a position between both options. Eventually, it could be extended to cover both, if there is a desire for this.
If only "a" is needed, I'm not sure to see the value of the "begin" demarcation message, as the sender of RR could locally mark its route when sending the RR, rather than when receiving the "RR Start"

Thank you,
Bruno

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 10 Oct 2013 18:10:09 +0200

Hi Bruno,

Let me observe that BGP Operation Message has been proposed to address
"a" http://tools.ietf.org/html/draft-ietf-idr-operational-message-00

I would leave bgp enhanced route refresh to address "b" regardless how
heavy weight it may sound .. IMHO it is required.

>From implementation point of view the demarcation point is only
required at the receiver of the updates replay. The sender is free to
send new updates during the replay the same way in regular route
refresh as with enhanced route refresh .. I see no difference there at
all. There is no need for any "snapshot" on the BGP speaker which
replays the updates.

In fact the draft says the above in this sentence already giving an
implementation hint:

   "When a route entry in the "ADJ-RIB-Out" changes, the advertisement of
    the modified route entry (instead of the snapshot entry) would suffice.

Best,
R.

On Thu, Oct 10, 2013 at 2:45 PM,  <[email-masked]> wrote:
> _______________________________________________
> Idr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idr

---

From: <[email-masked]>
Date: Fri, 11 Oct 2013 08:15:31 +0000

Robert,

>From: Robert Raszuk >Sent: Thursday, October 10, 2013 6:10 PM
>
>Hi Bruno,
>

Speaking of requirements, it would be good for the draft to document them, in order for the WG to be able to understand the tradeoffs.
Current text is not specific enough:
- "The enhancement can be used to facilitate on-line, non-disruptive consistency validations of BGP routing updates."
- "It is sometimes necessary to perform routing consistency validations such as checking for possible missing withdraws between BGP speakers"

e.g.:
- is checking for missing updates or missing/incorrect attributes a requirement?
- is delaying routing convergence during all Route Refresh acceptable? As a reminder, in BGP/MPLS environment, Route Refresh are pretty common and are typically automatically issued by a PE when a new VRF/VPN/RT is configured. 

>all. There is no need for any "snapshot" on the BGP speaker which

What is needed or not depend on the requirements. Cf above point.
If you want to be able to "perform routing consistency validations" and detect a missed update, you need to exclude "new" updates during the route refresh process. Otherwise, the receiver won't be able to know whether this new/unexpected update (i.e. not locally present when he received the RR START) is due to a missed one (i.e. you have detected an routing inconsistency) or a new normal one.

>    the modified route entry (instead of the snapshot entry) would suffice.

Yes, and as a consequence, we this exclude the ability to detect missed update.
Also, given nothing is said about withdraw, I read that the sender is not allowed to send live withdraw between RR start and RR stop, which delay the routing convergence by minutes. As a reminder, in BGP/MPLS environment, explicit withdraw are required as implicit withdraw is not an option (with a unique RD policy) and neither advertising the external interface in the IGP.

So to sum up, although I understand that trade-offs are required, the current text seems to get the drawbacks of both option a & b:
- delaying routing convergence by minutes during normal network operation time (i.e. not only during troubleshooting)
- not allowing to detect missing updates and missing or incorrect attributes

So I'd like to see what the requirements were/are. At least, that does not seem to match mine.

At the very minimum, the document needs to clearly spell out such limitations. Because if it's difficult to troubleshoot where/why a BGP update has been missed, it will probably be harder to troubleshoot where/why a BGP convergence has been delayed by minutes (harder because at troubleshooting time, the problematic states have been erased).

Thank you.
Bruno

>> Because If we want to do a full check ("b"), i.e. determine which BGP
>routing information has not been reflected, we may need to define a real
>"snapshot" and only sends the update related to this snapshot between the 2
>new demarcation messages.
>> The drawback is that during this time, the BGP convergence is slow down
>because "real time" BGP updates needs to wait for the end of this snapshot.
>And this can be long (minutes, ten of minutes). Hence IMO, this is not
>advisable for all (existing) Route Refresh. I would rather limit such
>behavior to specific "on demand" request (e.g. through a new RR request sub-
>type)
>>
>> OTOH, if we just want to "refresh" the routes ("a"), i.e. know when to
>purge the non-received one, we can interleave live BGP updates during the
>refresh. This does not delay BGP convergence but we won't be able to detect
>if there were inconsistencies and what they were.
>>
>> IMO, the draft would need to take a position between both options.
>Eventually, it could be extended to cover both, if there is a desire for
>this.
>> If only "a" is needed, I'm not sure to see the value of the "begin"
>demarcation message, as the sender of RR could locally mark its route when
>sending the RR, rather than when receiving the "RR Start"
>>
>> Thank you,
>> Bruno
>>
>>
>____________________________________________________________________________
>_____________________________________________
>>
>> Ce message et ses pieces jointes peuvent contenir des informations
>confidentielles ou privilegiees et ne doivent donc
>> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez
>recu ce message par erreur, veuillez le signaler
>> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
>electroniques etant susceptibles d'alteration,
>> This message and its attachments may contain confidential or privileged
>information that may be protected by law;

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 11 Oct 2013 14:40:31 +0200

Hi Bruno,

I understand your perspective, but one needs to keep in mind also BGP
implementation side. Imagine BGP implementation which does not have a
concept of per peer BGP_RIB_Out.

So such implementation in order to be able to replay a frozen snapshot
would need to queue incoming updates and withdraws to the global bgp
table.

That effectively means that one RR client requesting enhanced route
refresh towards such RR will delay convergence of BGP to 1000 of other
clients of the same RR which never needed route refresh. Is this what
you would accept in your option "a" even if it would be a requirement
?

Cheers,
R.

---

From: <[email-masked]>
Date: Fri, 11 Oct 2013 13:14:14 +0000

Hi Robert,

>From: Robert Raszuk >Sent: Friday, October 11, 2013 2:41 PM
>
>Hi Bruno,
>

I was ready to keep silent (hidden?) the potential effect on peers, assuming that some people would say that this is "an implementation issue".

However, your comment is very valid and I now believe that the draft should also explicit the consequences of this new procedures on such peers.
So many thanks for your comment.

>would need to queue incoming updates and withdraws to the global bgp

I would suggest to defer further discussion on this point as there are too much pending open questions and hence too many options to consider. So let's wait for answers from the authors or a revised version.

>From a requirement perspective, I have precisely the opposite requirements (i.e. do no delay BGP convergence in normal network operation time).
>From a solution perspective, that's one of the concern I have regarding the solution defined in this document. I'm waiting about clarification from the authors/document about the intended behavior is. Then we'll know what side of the drawbacks we have to face and we'll be able to discuss whether we can live with that or not (in the latter case a different solution or an extension may be needed)

Thank you.
Cheers,
Bruno

>> Speaking of requirements, it would be good for the draft to document them,
>in order for the WG to be able to understand the tradeoffs.
>> - "The enhancement can be used to facilitate on-line, non-disruptive
>consistency validations of BGP routing updates."
>> - "It is sometimes necessary to perform routing consistency validations
>such as checking for possible missing withdraws between BGP speakers"
>>
>> e.g.:
>> - is delaying routing convergence during all Route Refresh acceptable? As
>a reminder, in BGP/MPLS environment, Route Refresh are pretty common and are
>typically automatically issued by a PE when a new VRF/VPN/RT is configured.
>>
>> If you want to be able to "perform routing consistency validations" and
>detect a missed update, you need to exclude "new" updates during the route
>refresh process. Otherwise, the receiver won't be able to know whether this
>new/unexpected update (i.e. not locally present when he received the RR
>START) is due to a missed one (i.e. you have detected an routing
>inconsistency) or a new normal one.
>>
>> Also, given nothing is said about withdraw, I read that the sender is not
>allowed to send live withdraw between RR start and RR stop, which delay the
>routing convergence by minutes. As a reminder, in BGP/MPLS environment,
>explicit withdraw are required as implicit withdraw is not an option (with a
>unique RD policy) and neither advertising the external interface in the IGP.
>>
>> So to sum up, although I understand that trade-offs are required, the
>current text seems to get the drawbacks of both option a & b:
>> - delaying routing convergence by minutes during normal network operation
>time (i.e. not only during troubleshooting)
>> At the very minimum, the document needs to clearly spell out such
>limitations. Because if it's difficult to troubleshoot where/why a BGP
>update has been missed, it will probably be harder to troubleshoot where/why
>a BGP convergence has been delayed by minutes (harder because at
>troubleshooting time, the problematic states have been erased).
>>
>> Thank you.
>> Bruno
>>
>>>
>>>Best,
>>>R.
>>>
>>>
>>>
>>>
>>>

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 11 Oct 2013 15:34:17 +0200

Hi Bruno,

I am not sure what questions are not answered. The proposed "feature"
is shipping so it is very easy to test it in your own lab :)

Moreover IDR has just recently accepted
http://tools.ietf.org/html/draft-keyupate-idr-enhanced-refresh-impl-00
to be an IDR WG document so if you need additional questions answered
in this document I think the best would be to propose those and allow
implementations to answer in next revision.

Also I think there is already third BGP implementation exaBGP which is
open source which also supports this draft.

Best,
R.

---

From: Thomas Mangin <[email-masked]>
Date: Fri, 11 Oct 2013 15:01:35 +0100

Hello Bruno, Robert,

ExaBGP does indeed implement the feature. I tested interoperability with Cisco successfully - the feature is considered stable in 3.2.16 ( thank you to Eric Rosen for providing me a way to test my code against Cisco's implementation and Hideaki HAYASHI who was very helpful and helped me a lot with behaviour correctness).

The code buffers new announcements / withdrawals for the peer while the Adj-RIB-OUT is being flushed. That said, I have no objection to change the way the implementation should it to shown to be preferable.

Thomas

---

From: <[email-masked]>
Date: Fri, 11 Oct 2013 14:51:05 +0000

Hi Robert,

>From: Robert Raszuk >Sent: Friday, October 11, 2013 3:34 PM
>
>Hi Bruno,
>
>I am not sure what questions are not answered. 

Please refer to the questions in my previous emails on the IDR mailing list (e.g. during WG last call)
I don't recall any answer from the authors nor an updated document. Hence they remained "not answered".

>The proposed "feature"

I guess that this is a joke.
But just in case, IMO, the behavior is defined as per the IETF spec, not as per an implementation. (otherwise, the implementation would be always right, bug free and interoperable from vendor A point of view... Idem from vendor B point of view, even if they turned out to be non interoperable in lab or worst in the network. And otherwise, there is no point in this IDR WG: let's just have vendor B write code and have other vendors reverse engineer the spec)

>implementations to answer in next revision.

My questions are related to the specification, not the implementation report.

But speaking about the implementation report, I'm wondering what was the exact TSS (Test Suite Structure), since some of my questions are related to interop points which are not crystal clear in the spec. e.g.

> 5) not receiving "Demarcation of the ending of a route refresh
> How is the receiver supposed to react?
> - Never ever send a new RR on this session? This would be problematic as RR are typically used often, e.g. when a new VPN is configured. But it's not clear if sending a new RR is, strictly speaking, allowed by the current text. Hence the receiver would take the risk of "upsetting" the speaker and receiving a NOTIFICATION. 

> - And if it sends a new RR, and receive a new "Demarcation of the beginning of a route refresh", should it wait for two " Demarcation of the ending of a route refresh" or is single one. (same question for the speaker: should it send two or one).

So, was has been the result of the above test during the interop test?

Best,
Bruno

>> I was ready to keep silent (hidden?) the potential effect on peers,
>assuming that some people would say that this is "an implementation issue".
>>
>> However, your comment is very valid and I now believe that the draft
>should also explicit the consequences of this new procedures on such peers.
>> I would suggest to defer further discussion on this point as there are too
>much pending open questions and hence too many options to consider. So let's
>wait for answers from the authors or a revised version.
>>
>> From a requirement perspective, I have precisely the opposite requirements
>(i.e. do no delay BGP convergence in normal network operation time).
>> From a solution perspective, that's one of the concern I have regarding
>the solution defined in this document. I'm waiting about clarification from
>the authors/document about the intended behavior is. Then we'll know what
>side of the drawbacks we have to face and we'll be able to discuss whether
>we can live with that or not (in the latter case a different solution or an
>extension may be needed)
>>
>>
>> Thank you.
>> Cheers,
>> Bruno
>>
>>>
>>>Cheers,
>>>R.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>> So I'd like to see what the requirements were/are. At least, that does
>not
>>>seem to match mine.
>>>>
>>>>>"snapshot" and only sends the update related to this snapshot between
>the
>>>2

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Fri, 11 Oct 2013 15:23:21 +0000

Comments In-Line..

Jim Uttaro

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Friday, October 11, 2013 9:14 AM
To: Robert Raszuk
Cc: idr wg
Subject: Re: [Idr] draft-ietf-idr-bgp-enhanced-route-refresh

Hi Robert,

>From a solution perspective, that's one of the concern I have regarding the solution defined in this document. I'm waiting about clarification from the authors/document about the intended behavior i…
[Jim U>] Our customers expect the fastest convergence possible. In my network I have taken steps to do this. May be good to get external customers to weigh in on the reqs also..

Thank you.
Cheers,
Bruno

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 11 Oct 2013 21:28:34 +0200

> [Jim U>] Our customers expect the fastest convergence possible. In my
> network I have taken steps to do this.

By recommending deployment of "BGP Persistence" ? Note that not so far
back you have wrote in version -01 of "BGP Persistence" draft:

   The lack of convergence within a Layer 3 topology during the
   persistent state SHOULD be considered since it may adversely affect
   services, however, an assumption is made that a degraded service is
   preferable to a complete service outage during a large-scale BGP
   control plane failure.

Best,
R.

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Fri, 11 Oct 2013 19:34:12 +0000

And your point? 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Robert Raszuk
Sent: Friday, October 11, 2013 3:29 PM
To: UTTARO, JAMES
Cc: [email-masked]; idr wg
Subject: Re: [Idr] draft-ietf-idr-bgp-enhanced-route-refresh

Best,
R.

---

From: "Keyur Patel (keyupate)" <[email-masked]>
Date: Fri, 11 Oct 2013 20:24:08 +0000

Bruno,

Apologies for the delayed response. Comments are inlined #Keyur.
>> -b- "on demand" but extensive checks for troubleshooting/debugging

#Keyur: Sure.

>>the 2 new demarcation messages.

#Keyur: I am not sure if I agree with the above comment. Why can't an
implementation interleave routes (refresh with "real time updates")?

>>behavior to specific "on demand" request (e.g. through a new RR request
>>sub-type)

#Keyur: Ack.

Regards,
Keyur
>>purge the non-received one, we can interleave live BGP updates during
>>the refresh. This does not delay BGP convergence but we won't be able to
>>detect if there were inconsistencies and what they were.
>>
>>demarcation message, as the sender of RR could locally mark its route
>>when sending the RR, rather than when receiving the "RR Start"
>>
>> Thank you,
>> Bruno
>>
>> 
>>_________________________________________________________________________
>>________________________________________________
>>
>> a l'expediteur et le detruire ainsi que les pieces jointes. Les
>>messages electroniques etant susceptibles d'alteration,

---

From: "Keyur Patel (keyupate)" <[email-masked]>
Date: Fri, 11 Oct 2013 23:36:59 +0000

Hi Bruno,

Comments inlined #Keyur

On 10/11/13 1:15 AM, "[email-masked]"
<[email-masked]> wrote:

>Speaking of requirements, it would be good for the draft to document
>them, in order for the WG to be able to understand the tradeoffs.

#Keyur: IMHO, the draft tries to ensure that BGP tables exchanged between
BGP peers are consistent.  Furthermore, the mechanism described in the
draft assists in identifying and fixing missing withdraws. Current Route
Refresh mechanism does not achieve this functionality.

The draft does NOT try to address missing prefix (reachability)
announcements as they cannot be solved by Route Refresh mechanisms.
Furthermore, in accordance with RFC4271 and RFC2918, the draft DOES NOT
alter re-advertisement rules upon a receipt of a Route Refresh message.

If you like, we can augment the draft text to reflect it.

>- is checking for missing updates or missing/incorrect attributes a

#Keyur: No (assuming you meant missing reachability updates). It does
check for missing withdraws.

>a reminder, in BGP/MPLS environment, Route Refresh are pretty common and
>are typically automatically issued by a PE when a new VRF/VPN/RT is
>configured.

#Keyur: No. I don't believe the draft tries to add any convergence delay.
As I mentioned in my earlier email, there is no reason to not interleave
updates (new or old). The rules mentioned in RFC2918 for re-advertisment
of routes are not changed.

Best Regards,
Keyur

>detect a missed update, you need to exclude "new" updates during the
>route refresh process. Otherwise, the receiver won't be able to know
>whether this new/unexpected update (i.e. not locally present when he
>received the RR START) is due to a missed one (i.e. you have detected an
>routing inconsistency) or a new normal one.
>
>allowed to send live withdraw between RR start and RR stop, which delay
>the routing convergence by minutes. As a reminder, in BGP/MPLS
>environment, explicit withdraw are required as implicit withdraw is not
>an option (with a unique RD policy) and neither advertising the external
>interface in the IGP.
>
>not seem to match mine.
>update has been missed, it will probably be harder to troubleshoot
>where/why a BGP convergence has been delayed by minutes (harder because
>at troubleshooting time, the problematic states have been erased).
>
>Thank you.
>Bruno
>
>>
>>Best,
>>R.
>>
>>
>>
>>
>>
>>sub-
>>when
>>___
>>>messages
>>> Orange decline toute responsabilite si ce message a ete altere,
>>>deforme ou
>>falsifie. Merci.
>>>
>>> As emails may be altered, Orange is not liable for messages that have
>>>been
>>modified, changed or falsified.
>>> Thank you.
>>>
>__________________________________________________________________________

---

From: Thomas Mangin <[email-masked]>
Date: Sat, 12 Oct 2013 12:34:13 +0100

> On 12 Oct 2013, at 00:36, "Keyur Patel \(keyupate\)" <[email-masked]> wrote:
> 

It would be nice to be able to differentiate between re-emitted routes and new routes.
An attribute could be used for this purpose, tagging the new routes within the RR window, but I fear it would change the draft a lot / too much.

Thomas

---

From: heasley <[email-masked]>
Date: Sun, 13 Oct 2013 07:23:38 +0000

> It would be nice to be able to differentiate between re-emitted routes and new routes.

For what use(s)/reason(s)?

---

From: Thomas Mangin <[email-masked]>
Date: Sun, 13 Oct 2013 13:10:07 +0100

> For what use(s)/reason(s)?

Hello Heasley,

Short answer:  

As it would allow better snapshotting of the network state without causing any delay in convergence, which may or may not be required when debugging the network.
If you are asking me when this would be advisable to what is in the current draft, I can not answer as I have no use case in mind for the feature myself.

Long answer:

The draft says that it wants to help to "perform routing consistency validations". People on this list stated that any implementation increasing convergence time are undesirable, therefore, new updates should not be hold back while the RR process is taking place. In practice this mean that during a RR refresh window, new and old updates need to be interleaved to allow FIB update on the receiving router.

In that case, the RR window is "non exclusive" and will have a mix of old and current updates. NLRI changes could be made explicit by sending first the previous NLRI and then the updated one, route withdrawal could be made explicit the same way. However nothing allows to differentiate new updates from "old" ones. It will not be possible to know if the NLRI in an update was received during the RR window or was preceding it.

Another consequence is that the synchronisation time will be defined by the EoRR message and and not the BoRR.
In the case of a snapshot, One can assume that the state of the updates sent was calculated between the moment the RR request was sent and the BoRR is received, which narrows the time to a small interval (hopefully).
In the case where new updates are sent during the RR window, the state of the remote peer received is the state of the peer when the EoRR is received and the route transfer is completed.  The interval is therefore more likely to be much longer and vary from peer to peer. Performing check between several peer may then become more difficult if one is trying to check the state of a whole IBGP mesh to make sure all the announcements are consistent. 

Having the updates containing the old (or new) routes tagged with a attribute to differentiate them would allow to not delay any convergence but still know what was the status of the peer at when the BoRR is received.

Depending on the problem being debugged, this may or may not be important. From the draft authors' point of view, it seems that no one came back with a use case where this ambiguity is a problem.

I am not advocating for any change to allow such feature, nothing prevent the creation of a new draft which would including a new RR type with this behaviour later on.

Thomas

---

From: Randy Bush <[email-masked]>
Date: Sun, 13 Oct 2013 15:19:10 +0200

> If you are asking me when this would be advisable to what is in the
> current draft, I can not answer as I have no use case in mind for the
> feature myself.

QED

---

From: Jakob Heitz <[email-masked]>
Date: Sun, 13 Oct 2013 15:24:46 +0000

You can't do snapshot.

Not even "show bgp route" is a snapshot unless it's a really small router.
The only way to get a snapshot is to coredump the bgp process and even then, it's not a snapshot of what's in the FIB.

--
Jakob Heitz.

---

From: Thomas Mangin <[email-masked]>
Date: Sun, 13 Oct 2013 18:03:40 +0100

Hello Jakob,

You will need to help me understand why you consider that ExaBGP does not perform a snapshot of the Adj-RIB-out with the current code. Are we not agreeing on the meaning of snapshot or is it something else ?

Sincerely,

Thomas

Sent from my iPad

> On 13 Oct 2013, at 16:24, Jakob Heitz <[email-masked]> wrote:
> 
> You can't do snapshot.
>

---

From: Jakob Heitz <[email-masked]>
Date: Sun, 13 Oct 2013 17:51:42 +0000

I think I see what you want:
Create a snapshot of an adj-rib-out at the "Demarcation of the beginning of a route refresh" at the sending router and have the receiving router create a similar snapshot of the corresponding adj-rib-in when it receives that demarcation.

This can be potentially expensive if lots of change is happening. It is worse on the receiving router, because it has no control of when it must create the snapshot. Worse, it may need to do it for several incoming snapshots from different neighbors. To deal with this, we must invent messages and procedures to abort the snapshot. I think this is a lot harder than enhanced-route-refresh and deserves a separate draft and capability if we want to do it.

I would suggest a two step procedure: first, send a hash of the snapshot in a single message and then send a real snapshot if the hash doesn't match at the receiving router. That way, you could send cheap hashes frequently and only send the expensive snapshot if an error was detected.

Is there wider interest in this?

--
Jakob Heitz.

---

From: Thomas Mangin <[email-masked]>
Date: Sun, 13 Oct 2013 20:31:38 +0100

Jakob,

I will have to admit that it is more than I had in mind :)

As I have no use for the feature myself - I just implemented it and my users are more likely to request a RR than have ExaBGP respond to one - I would rather hear from potential users about the problem they are trying to solve before trying to solve an non-issue.

My comment was more to highlight the current limitation(s) of the draft, with a throw away idea of attribute, so it could be established if the trade off were acceptable or warrant more work on the document.

For the moment, it is clear that more can be done to improve the feature but it is not clear at all that if it should be ..

QED ?

Thomas

Sent from my iPad

> On 13 Oct 2013, at 18:51, Jakob Heitz <[email-masked]> wrote:
>

---

From: Robert Raszuk <[email-masked]>
Date: Sun, 13 Oct 2013 21:44:43 +0200

> QED ?

quod erat demonstrandum
SRC: http://en.wikipedia.org/wiki/Q.E.D.

;-)

Cheers,
R.

---

From: Thomas Mangin <[email-masked]>
Date: Sun, 13 Oct 2013 20:55:17 +0100

Thank you Robert,

I was just trying to say that what we need is demonstrate is user demand :)
The QED ? was for that as it is what Randy was saying so concisely .. Perhaps not most clear :)

Sent from my iPad

---

From: Randy Bush <[email-masked]>
Date: Mon, 14 Oct 2013 06:05:00 +0300

thomas,

> .. Perhaps not most clear :)

two factors.  first, unneeded code complexity leads to bugs, code bloat,
spec bloat, ...  second, complexity often increases attack surface.
third (yes, you get an extra for free!) , it could get in the way of a
future really needed feature.

btw, we're (actually david hauweele) starting to use exabgp for
injection in some experiments.  thanks for the code.

randy

---

From: <[email-masked]>
Date: Mon, 14 Oct 2013 09:59:21 +0000

Hi Keyur,

Many thanks for the feedback. Comments are inlined #Bruno

>From: Keyur Patel >Sent: Friday, October 11, 2013 10:24 PM
>
>Bruno,
>

#Bruno: If the sender interleaves routes, I don't see how the receiver could detect BGP routing inconsistencies (as stated in the abstract and introduction of the draft).

Let's take the case of a BGP withdraw:
- if they are delayed, the L3 VPN convergence is delayed.
- if they are not delayed, but interleaved with the during the Route Refresh, the receiver will detect an difference (between the RR START state and the RR END state) and will log an error "for further analysis". Hence we have lots of error logged which are falls positive (hopefully all of them). Honestly, given the ratio of falls information (>99%), I'm not sure that people will do analysis of even track such logs. Even during a specific troubleshooting, it's a bit misleading to have logs that you can't really rely on.

Let's take the case of BGP update:
- since they are not delayed, the receiver is equally not really capable of detecting inconsistencies. Most/all detected inconsistencies, will be regular BGP updates.

So, if we don't delay BGP messages, what the draft really does is purging routes which should have been previously withdrawn. It does not detect inconsistencies as the receiver can't make the distinction between an inconsistency (e.g. due to a missed BGP messages) and a normal BGP message sent during the route refresh.
Note that this goal would be fine for me because although the gain is more limited, it is still interesting and the cost is null. In such case, IMO the draft should:
- specifically states that BGP messages SHOULD NOT be delayed
- update the introduction/requirements/abstract to reflect the reduced goal
- add a text to explain that the logged messages do not necessarily represent an error or BGP inconsistency. Quite the contrary, the vast majority of logs are due to normal BGP operation.

Thank you,
Bruno

>>>been modified, changed or falsified.
>>> Thank you.
>>>

---

From: Jakob Heitz <[email-masked]>
Date: Mon, 14 Oct 2013 12:01:25 +0000

> - add a text to explain that the logged messages do not necessarily represent an error or BGP inconsistency. Quite the contrary, the vast majority of logs are due to normal BGP operation.

I agree with this, except I would not even log those "normal" inconsistencies, just the ones after the end marker.

Kind Regards,
Jakob.

---

From: "Keyur Patel (keyupate)" <[email-masked]>
Date: Mon, 14 Oct 2013 15:33:05 +0000

Hi Bruno,

Comments inlined ##Keyur. :)

On 10/14/13 2:59 AM, "[email-masked]"
<[email-masked]> wrote:

>#Bruno: If the sender interleaves routes, I don't see how the receiver
>could detect BGP routing inconsistencies (as stated in the abstract and
>introduction of the draft).
>
>- if they are not delayed, but interleaved with the during the Route
>Refresh, the receiver will detect an difference (between the RR START
>state and the RR END state) and will log an error "for further analysis".
>Hence we have lots of error logged which are falls positive (hopefully
>all of them). Honestly, given the ratio of falls information (>99%), I'm
>not sure that people will do analysis of even track such logs. Even
>during a specific troubleshooting, it's a bit misleading to have logs
>that you can't really rely on.
>
>- since they are not delayed, the receiver is equally not really capable
>of detecting inconsistencies. Most/all detected inconsistencies, will be
>regular BGP updates.
>
>So, if we don't delay BGP messages, what the draft really does is purging
>routes which should have been previously withdrawn.

##Keyur: Yes. Like I said in my earlier emails, the draft tries to ensure
that BGP tables exchanged between BGP Peers are consistent. Furthermore,
the mechanism described in the draft assists in identifying and fixing
missing withdraws.

> It does not detect inconsistencies as the receiver can't make the
>distinction between an inconsistency (e.g. due to a missed BGP messages)
>and a normal BGP message sent during the route refresh.

##Keyur: The inconsistencies in context of both, the withdraws and the
final route state announcements are detected. The draft does NOT try to
address missing prefix (reachability) announcements as they cannot be
solved by Route Refresh mechanisms. Furthermore, most implementations
can't do snapshot for the reasons that Jakob and Randy explained in their
emails.

>Note that this goal would be fine for me because although the gain is
>more limited, it is still interesting and the cost is null. In such case,
>IMO the draft should:
>- add a text to explain that the logged messages do not necessarily
>represent an error or BGP inconsistency. Quite the contrary, the vast
>majority of logs are due to normal BGP operation.

##Keyur: Super. Will do. :)

Best Regards,
Keyur

>>>>deforme
>>>>ou falsifie. Merci.

---

From: "[email-masked]" <[email-masked]>
Date: Mon, 14 Oct 2013 19:04:01 +0000

> For the moment, it is clear that more can be done to improve the feature but it is not clear at all that if it should be ..

and find the value to complexity & expense?  Jakob hints to the indeterminate
expense from queueing updates.  can't BMP be used to create a snapshot?

I suppose an implementation could offer a configuration knob to queue updates.

> > Is there wider interest in this?

presumably for research/debugging.

---

From: Jakob Heitz <[email-masked]>
Date: Mon, 14 Oct 2013 19:17:07 +0000

The enhanced route refresh as it stands will fix all routes that were either dropped or not withdrawn.
This is good value already.
The additional value that a snapshot gives is to tell you exactly which routes were not withdrawn.
Because the snapshot adds extra memory and processing requirements, if we were to do this, I would like to see it with a separate capability and a separate draft.

Thanks,
Jakob.

> From: [email-masked] [mailto:[email-masked]]
> Sent: Monday, October 14, 2013 12:04 PM
> To: Thomas Mangin
> Cc: Jakob Heitz; [email-masked]; idr wg; [email-masked]
> > As I have no use for the feature myself - I just implemented it and my users
> are more likely to request a RR than have ExaBGP respond to one - I would
> rather hear from potential users about the problem they are trying to solve
> before trying to solve an non-issue.
> >
> > My comment was more to highlight the current limitation(s) of the draft,
> with a throw away idea of attribute, so it could be established if the trade off
> were acceptable or warrant more work on the document.
> >
> > For the moment, it is clear that more can be done to improve the feature
> but it is not clear at all that if it should be ..
> 
> and find the value to complexity & expense?  Jakob hints to the
> indeterminate expense from queueing updates.  can't BMP be used to
> create a snapshot?
> 
> > > Create a snapshot of an adj-rib-out at the "Demarcation of the beginning
> of a route refresh" at the sending router and have the receiving router
> create a similar snapshot of the corresponding adj-rib-in when it receives
> that demarcation.
> > >
> > > This can be potentially expensive if lots of change is happening. It is
> worse on the receiving router, because it has no control of when it must
> create the snapshot. Worse, it may need to do it for several incoming
> snapshots from different neighbors. To deal with this, we must invent
> messages and procedures to abort the snapshot. I think this is a lot harder
> than enhanced-route-refresh and deserves a separate draft and capability if
> we want to do it.
> > >
> > > I would suggest a two step procedure: first, send a hash of the snapshot
> in a single message and then send a real snapshot if the hash doesn't match
> at the receiving router. That way, you could send cheap hashes frequently
> and only send the expensive snapshot if an error was detected.
> > >

---

From: Jared Mauch <[email-masked]>
Date: Mon, 14 Oct 2013 20:36:46 -0400

> Because the snapshot adds extra memory and processing requirements, if we were to do this, I would like to see it with a separate capability and a separate draft.

This sounds like a poor excuse for an original bug leaking by.  Not every defect needs to be fixed with a spec change.

This seems to be unnecessary.

- Jared

---

From: <[email-masked]>
Date: Tue, 15 Oct 2013 10:17:34 +0000

Hi Keyur,

Comments inlined ##Bruno

>From: Keyur Patel (keyupate) [mailto:[email-masked]] >Sent: Monday, October 14, 2013 5:33 PM
>
>Hi Bruno,
>

##Bruno: Good. In think we are in sync now. 2 more minor points:
- improving the consistency is good. But this assumes that the bug which create the inconsistency does not show up during the RR. Which is debatable (as RR as likely stressing both the speaker and the receiver) :-) Also, compared to current usage of RR, the requirement on the speaker to not forget to refresh a route is more important. Indeed, in current usage (policy change, VRF provisioning) missing a route/update has likely no consequences as most of the routes are already known to the receiver. While with this draft, any missed route with result in a erroneous withdraw. Let's hope that this will not reveal new bugs.

- I don't really agree with "the draft assists in identifying and fixing missing withdraws.". Or at least, IMHO, the level of "assistance" (or Signal / Noise) is too low to be really useful. 

>final route state announcements are detected. 

##Bruno: :s/detected/corrected
(They are not really detected. Cf above)

> The draft does NOT try to

##Bruno: Great. Thank you.

Best regards,
Bruno