Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-00

From: "UTTARO, JAMES" <[email-masked]>
Date: Tue, 1 Nov 2011 15:19:44 +0000

Enke,

                Comments in-Line..

Thanks,
                Jim Uttaro

From: Enke Chen [mailto:[email-masked]]
Sent: Friday, October 28, 2011 2:19 AM
To: UTTARO, JAMES
Cc: [email-masked]; [email-masked] List; Enke Chen
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-00

Jim,

My comments are inlined.

On 10/27/11 1:17 PM, UTTARO, JAMES wrote:

Enke,

GR is a solution that is essentially local in scope it does not have the ability to inform downstream speakers of the viability of routing state from the point of possible control plane failure. OTOH Persistence does propagate the condition of state. This provides distinct advantages in terms of customers awareness of the SPs control plane. One could imagine a customer receiving a STALE path and responding by selecting a backup. Some of the extensions to this draft that I have considered in colouring of STALE to inform if the condition arises from a local ( PE ) or internal iBGP ( RR ) failures..

GR makes no distinction from STALE state and ACTIVE state.. This can lead to the STALE path still being preferred throughout the topology. IMO this is incorrect behavior regardless of the comparison.

PERSISTENCE allows for a customer to indicate which paths should be candidates. Customers may want to immediately failover to the backup for some paths and not for others. GR is not capable of doing this it is all or nothing. The granularity is not sufficient. It needs to be at the path level. There may even be a case for having even more granularity i.e a per path timer.. GR is not capable of being extended for either of these cases.

I am not sure how this path level persistence would work operationally.   Without the detailed information of a provider's network, how would a customer know what kind of failures and recovery that they might experience?   Consider the example of the simultaneous RR failures in the draft,  why would
dn't any customer not to want to protect against such failures?   The end result could be that the PERSISTENCE flag is always set, thus losing its significance.
[Jim U>] One ex would be customers who create multiple VPNs over different SPs.. A customer may want to take advantage of the knowledge that a control plane failure has occurred and migrate the traffic to the backup. This could be done at a path granularity by use of the DO_NOT_PERSIST CV. . We as SPs want to provide our customers with the tools needed to manage their VPNs and not prescribe a one size fits all solution.

Regarding the use of the STALE state vs ACTIVE state, clearly there is a tradeoff.   GR uses the stale routes in order to avoid forwarding churns, which has been a critical requirement for a long time.   If there is a real need for favoring a ACTIVE one over a STALE one in GR, it can be done by a simple knob.
[Jim U>] The current draft has no ability to inform downstream speakers of whether or not a path is STALE or ACTIVE. The knob may be simple but a lot of machinery would have to be built. This is one of the big reasons for the PERSIST draft. I do not understand the routing churn part in the context of vpnv4, vpnv2, 3107 etc... maybe the GR solution was constructed as a solution that primarily speaks to eBGP IPV4 connections for the IPV4 AF ( Internet ).. I could understand that..

As you know, BGP is full of knobs that adjust behaviors for different needs :-)
[Jim U>] More Knobs..

GR does not provide protection through successive restarts of the session. I believe that if this occurs the state will be invalidated. So for a session that is bouncing due to overload condition GR will not provide the required protection

This can be addressed by a simple knob to set the min stale timer for GR.
[Jim U>] And yet more knobs

GR does not employ a make before break strategy. All state is invalidated first then the newly learned state is processed. This leads to routing churn especially if the majority of the state is the same which I am pretty sure is the case

Such behavior would be an implementation bug that needs to be fixed.  But it is not an issue with the protocol itself.

This is what we have in 4.2. Procedures for the Receiving Speaker, RFC 4724:

---

   The Receiving Speaker MUST replace the stale routes by the routing

   updates received from the peer.  Once the End-of-RIB marker for an

   address family is received from the peer, it MUST immediately remove

   any routes from the peer that are still marked as stale for that

   address family.
[Jim U>] This does not address the lack of clarity about make before break.. it only states that must immediately remove routes marked as stale. It should state that any paths that are learned which are the same as the STALE paths should not force the forwarding plane to be re-programmed for those paths.. This should be made clear and in general is good practice to avoid churn..

There are several possibilities for the premature purge of the stale routes. For example, the "Forwarding state" flag was somehow not set after the session was re-established, or the the EOR was sent prematurely.   Further investigation will be needed in order to identify any possible implementation or config issues involved in your setup.
[Jim U>] More moving parts to worry about..

GR invalidates state due to the case of protocol error i.e A malformed update will invalidate all of the state. This is not the desired behavior.

It has been addressed by the following extension:

http://datatracker.ietf.org/doc/draft-keyupate-idr-bgp-gr-notification/

[Jim U>] A few comments here.. I do not understand, the draft does not clarify that the only thing that will force a tear down is the cease subcode and a hard reset error code.. is the intention that this is the only thing that will tear it down? I guess I would like to see which things will and will not force a session termination in the original draft.. Like

-          Holdtime Expiration

-          Malformed Update

-          Consecutive Restarts.. So what does this exactly mean   "As part of this extension, possible consecutive restarts SHOULD NOT
   delete a route (from the peer) previously marked as stale, until
   required by rules mentioned in [RFC4724]." Possible consecutive restarts means what? I really need clarity on this whole notion of when is a session truly invalidated.

Why is the purpose of the following text?

   Once the session is re-established, both BGP speakers MUST set their
   "Forwarding State" bit to 1 if they want to apply planned graceful
   restart.  The handling of the "Forwarding State" bit should be done
   as specified by the procedures of the Receiving speaker in [RFC4724]
   are applied.

GR is not specific as to which events invoke it or not. From my read on the draft it is not clear if holdtime expiration invokes GR or not.. The draft is unclear.

I think that it is covered by the above extension.  If not, it should be clarified.
[Jim U>] I did not see it..

It is not clear to me how RRs and PEs differ in using GR.

I think that there is a main difference when a RR is not in the forwarding path.  In that case, the RR should always set the F bit in the GR Capability so that its clients will continue forwarding after they lose the sessions with RR.  It is a deployment issue, though.
[Jim U>] Yes.. Again from an operations perspective I have to deploy technology differently in different parts of the network across multiple vendors. This is generally not a desired starting point for the successful deployment of new technology.. I want solutions that are generic and simple to deploy.

The time that state can persist is limit to about 1 hour max.

I think that you are talking about the "Restart time" field which has 12 bits and amount to about 68 minutes.  The "Restart time" is for the session re-establishment.  It does not impact the duration for holding stale routes after the session is re-established.
[Jim U>]  But if the session does not become re-established then the state is invalidated as the session terminates with an error code that GR will not persist through..

If the session does not get re-established in 68 minutes, the stale routes would be purged.  That is a long time, isn't it?   However, if one really wants to extend the session re-establishment time and continue to hold stale routes, it can be done by a simple knob.
[Jim U>] And yet even more knobs

GR does detail the behavior where convergence is not achieved between restarts.. Similar to above..

The min stale timer knob can cover it (see above).

But do you meant "does not"?  We can certainly clarify in 4724bis if that is the case.
[Jim U>] If convergence is not achieved what is the behavior. I could not determine from the draft..

I do not believe that the current GR paradigm can be extended to cover the majority of the cases above.

Except for the path level persistence you mentioned, I believe the GR will be able to address all other persistence requirements you listed, with some simple knobs and some implementation enhancements.
[Jim U>] IMO GR was originally designed to prevent churn due to intermittent failure on an eBGP session for the IpV4 AF.. I do not want to have different knobs and implementation enhancements to solve the basics of persistence.. Regardless of that it does not inform the topology of the state of a path in re the control plane it was learned over so there can be no independent decisions about the value of a given path by different customers/providers.. This is required for my applications..

Thanks,

        Jim Uttaro

Thanks.   -- Enke

-----Original Message-----

From: Enke Chen [mailto:[email-masked]]

Sent: Wednesday, October 26, 2011 8:43 PM

To: UTTARO, JAMES

Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]> List; Enke Chen

Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-00

Hi, folks:

I have a hard time in understanding what new problems (beyond the GR)

the draft try to solve :-(

If the concern is about the simultaneous RR failure as shown in the

examples in Sect. 6 Application, that can be addressed easily using GR.

As the RRs are not in the forwarding path, it means that the forwarding

is not impacted (thus is preserved) during the restart of a RR.   The

Forwarding State bit (F) in the GR capability should always be set by

the RR when it is not in the forwarding path.

Also in the case of simultaneous RR failure, I do not see why one would

want to retain some routes, but not others, using the communities

specified in the draft.  As the RRs are not in the forwarding path,

wouldn't be better to retain all the routes on a PE/client?

As you might be aware, efforts have been underway to address issues with

GR found during implementation and deployment. They include the spec

respin, notification handling, and implementations.  If there are issues

in the GR area that are not adequately addressed,  I suggest that we try

to address them in the GR respin if possible, instead of creating

another variation unnecessarily.

Thanks.   -- Enke

On 10/26/11 10:24 AM, Robert Raszuk wrote:

Jim,

When one during design phase of a routing protocol or routing protocol

extension or modification to it already realizes that enabling such

feature may cause real network issue if not done carefully - that

should trigger the alarm to rethink the solution and explore

alternative approaches to the problem space.

We as operators have already hard time to relate enabling a feature

within our intradomain boundaries to make sure such rollout is network

wide. Here you are asking for the same level of awareness across ebgp

boundaries. This is practically unrealistic IMHO.

Back to the proposal ... I think that if anything needs to be done is

to employ per prefix GR with longer and locally configurable timer.

That would address information persistence across direct IBGP sessions.

On the RRs use case of this draft we may perhaps agree to disagree,

but I do not see large enough probability of correctly engineered RR

plane to experience simultaneous multiple ibgp session drops. If that

happens the RR placement, platforms or deployment model should be

re-engineered.

Summary .. I do not think that IDR WG should adopt this document. Just

adding a warning to the deployment section is not sufficient.

Best regards,

R.

Robert,

The introduction of this technology needs to be carefully evaluated

when being deployed into the network. Your example clearly calls out

how a series of independent design can culminate in incorrect

behavior. Certainly the deployment of persistence on a router that

has interaction with a router that does not needs to be clearly

understood by the network designer. The goal of this draft is to

provide a fairly sophisticated tool that will protect the majority of

customers in the event of a catastrophic failure.. The premise being

the perfect is not the enemy of the good.. I will add text in the

deployment considerations section to better articulate that..

Thanks, Jim Uttaro

-----Original Message----- From: [email-masked]<mailto:[email-masked]>

[mailto:[email-masked]] On Behalf Of Robert Raszuk Sent:

Sunday, October 23, 2011 5:32 PM To: [email-masked]<mailto:[email-masked]> List Subject: [Idr]

draft-uttaro-idr-bgp-persistence-00

Authors,

Actually when discussing this draft a new concern surfaced which I

would like to get your answer on.

The draft in section 4.2 says as one of the forwarding rules:

o  Forwarding to a "stale" route is only used if there are no other

paths available to that route.  In other words an active path always

wins regardless of path selection.  "Stale" state is always

considered to be less preferred when compared with an active path.

In the light of the above rule let's consider a very simple case of

dual PE attached site of L3VPN service. Two CEs would inject into

their IBGP mesh routes to the remote destination: one marked as STALE

and  one not marked at all. (Each CE is connected to different PE and

each PE RT imports only a single route to a remote hub headquarter to

support geographic load balancing).

Let me illustrate:

VPN Customer HUB

PE3      PE4 SP PE1      PE2 |        | |        | CE1      CE2 |

| 1|        |10 |        | R1 ------ R2 1

CE1,CE2,R1,R2 are in IBGP mesh. IGP metric of CE1-R1 and R1-R2 are 1

and R2-CE2 is 10.

Prefix X is advertised by remote hub in the given VPN such that PE1

vrf towards CE1 only has X via PE3 and PE2's vrf towards CE2 only has

X via PE4.

Let's assume EBGP sessions PE3 to HUB went down, but ethernet link

is up, next hop is in the RIB while data plane is gone. Assume no

data plane real validation too. /* That is why in my former message

I suggested that data plane validation would be necessary */.

R1 has X via PE1/S (stale) and X via PE2/A (active) - it understands

STALE so selects in his forwarding table path via CE2.

R2 has X via PE1/S (stale) and X via PE2/A (active) - it does not

understand STALE, never was upgraded to support the forwarding rule

stated above in the draft and chooses X via CE1 (NH metric 2 vs 10).

R1--R2 produce data plane loop as long as STALE paths are present in

the system. Quite fun to troubleshoot too as the issue of PE3

injecting such STALE paths may be on the opposite site of the world.

The issue occurs when some routers within the customer site will be

able to recognize STALE transitive community and prefer non stale

paths in their forwarding planes (or BGP planes for that matter)

while others will not as well as when both stale and non stale paths

will be present.

Question 1: How do you prevent forwarding loop in such case ?

Question 2: How do you prevent forwarding loop in the case when

customer would have backup connectivity to his sites or connectivity

via different VPN provider yet routers in his site only partially

understand the STALE community and only partially follow the

forwarding rules ?

In general as the rule is about mandating some particular order of

path forwarding selection what is the mechanism in distributed

systems like today's routing to be able to achieve any assurance that

such rule is active and enforced across _all_ routers behind EBGP

PE-CE L3VPN boundaries in customer sites ?

Best regards, R.

-------- Original Message -------- Subject: [Idr]

draft-uttaro-idr-bgp-persistence-00 Date: Sat, 22 Oct 2011 00:23:55

+0200 From: Robert Raszuk<[email-masked]><mailto:[email-masked]> Reply-To:

[email-masked]<mailto:[email-masked]> To: [email-masked]<mailto:[email-masked]> List<[email-masked]><mailto:[email-masked]>

Hi,

I have read the draft and have one question and one observation.

Question:

What is the point of defining DO_NOT_PERSIST community ? In other

words why not having PERSIST community set would not mean the same as

having path marked with DO_NOT_PERSIST.

Observation:

I found the below statement in section 4.2:

o  Forwarding must ensure that the Next Hop to a "stale" route is

viable.

Of course I agree. But since we stating obvious in the forwarding

section I think it would be good to explicitly also state this in

the best path selection that next hop to STALE best path must be

valid.

However sessions especially those between loopbacks do not go down

for no reason. Most likely there is network problem which may have

caused those sessions to go down. It is therefor likely that LDP

session went also down between any of the LSRs in the data path and

that in spite of having the paths in BGP and next hops in IGP the LSP

required for both quoted L2/L3VPN applications is broken. That may

particularly happen when network chooses to use independent control

mode for label allocation.

I would suggest to at least add the recommendation statement to the

document that during best path selection especially for stale paths

a validity of required forwarding paradigm to next hop of stale

paths should be verified.

For example using techniques as described in:

draft-ietf-idr-bgp-bestpath-selection-criteria

Best regards, R.

_______________________________________________ Idr mailing list

[email-masked]<mailto:[email-masked]> https://www.ietf.org/mailman/listinfo/idr

_______________________________________________ Idr mailing list

[email-masked]<mailto:[email-masked]> https://www.ietf.org/mailman/listinfo/idr

_______________________________________________

Idr mailing list

[email-masked]<mailto:[email-masked]>

https://www.ietf.org/mailman/listinfo/idr

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 01 Nov 2011 19:35:12 +0100

Jim,

> [Jim U>] One ex would be customers who create multiple VPNs over
> different SPs.. A customer may want to take advantage of the
> knowledge that a control plane failure has occurred and migrate the
> traffic to the backup. This could be done at a path granularity by
> use of the DO_NOT_PERSIST CV. . We as SPs want to provide our
> customers with the tools needed to manage their VPNs and not
> prescribe a one size fits all solution.

I would recommend not to generalize "We as SPs" requirements as above.

Some SPs may prefer to build their control plane in a robust and multi 
vendor way for any application they offer to customers in order to 
prevent from any control plane failures to impact the services.

Other SPs may actually focus on delivering the actual services even 
during a transient control plane failures they may encounter.

I do not think that DO_NOT_PERSIST community is the best way to indicate 
that peering network is multihomed and that it would prefer the normal 
BGP withdraw as opposed to receiving STALE routes from their service 
provider during said service provider encountering a control plane failure.

It is clearly a very new requirement and justification for this proposal 
to develop a tool in order to communicate to your customers about your 
network's control plane failures even though data plane and services 
could continue to be active.

With this in mind what is the mechanism build into this tool for your 
customers to indicate interest is receiving such information ?

Best regards,
R.

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Tue, 1 Nov 2011 19:24:41 +0000

Comments In-Line..

Thanks
	Jim Uttaro

-----Original Message-----
From: Robert Raszuk [mailto:[email-masked]] 
Sent: Tuesday, November 01, 2011 2:35 PM
To: UTTARO, JAMES
Cc: 'Enke Chen'; [email-masked] List
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-00

Jim,

I would recommend not to generalize "We as SPs" requirements as above.
[Jim U>] s/'We as SPs'/'As an SP'/

Some SPs may prefer to build their control plane in a robust and multi 
vendor way for any application they offer to customers in order to 
prevent from any control plane failures to impact the services.
[Jim U>] That may or may not protect you as people architect/design/write code.

[Jim U>] DO_NOT_PERSIST does not indicate that a network is multi-homed simply allows for a speaker to be informed as to the viability of a control plane. It does allow a peer/speaker/customer to make a decision as to whether or not they want to continue using that network or to migrate away from it in a graceful fashion

With this in mind what is the mechanism build into this tool for your 
customers to indicate interest is receiving such information ?
[Jim U>] ? I do not understand the question.. 

Best regards,
R.

---

From: <[email-masked]>
Date: Wed, 2 Nov 2011 12:37:56 +0100

Hi Robert,

Sorry for the week delay.
More inline.

>From: Robert Raszuk >Sent: Sunday, October 23, 2011 11:32 PM
>
>Authors,
>
would
>like to get your answer on.
>
>       paths available to that route.  In other words an active path
>       always wins regardless of path selection.  "Stale" state is
always
>In the light of the above rule let's consider a very simple case of
dual
>PE attached site of L3VPN service. Two CEs would inject into their IBGP
>mesh routes to the remote destination: one marked as STALE and  one not
>marked at all. (Each CE is connected to different PE and each PE RT
>imports only a single route to a remote hub headquarter to support
>geographic load balancing).
>
>Let me illustrate:
>
>  VPN Customer HUB
>
>    PE3      PE4
>         SP
>    PE1      PE2
>     |        |
>     |        |
>    CE1      CE2
>     |        |
>    1|        |10
>     |        |
>    R1 ------ R2
>          1
>
and
>R2-CE2 is 10.
>
>Prefix X is advertised by remote hub in the given VPN such that PE1 vrf
>towards CE1 only has X via PE3 and PE2's vrf towards CE2 only has X via
PE4.
>
>Let's assume EBGP sessions PE3 to HUB went down, but ethernet link is
>up, next hop is in the RIB while data plane is gone. Assume no data
>plane real validation too. /* That is why in my former message I
>suggested that data plane validation would be necessary */.
>

Nope. In your above case, I would expect the failure to be concealed
within the SP AS:

PE1 has X via PE1/S (stale) and X via PE2/A (active) - it understands
STALE so selects in his forwarding table path via PE2.

--> The customer does not see the "STALE" information.

However, you bring very valid points, so let's continue (more comments
below)

>R1--R2 produce data plane loop as long as STALE paths are present in
the
>system. Quite fun to troubleshoot too as the issue of PE3 injecting
such
>STALE paths may be on the opposite site of the world.
>
able
>to recognize STALE transitive community and prefer non stale paths in
>their forwarding planes (or BGP planes for that matter) while others
>will not as well as when both stale and non stale paths will be
present.
>Question 1: How do you prevent forwarding loop in such case ?

You bring the issue of the incremental deployment of a feature which
change the (BGP) routing decision. That's a valid point (not necessarily
specific to the persistence draft).
One answer is careful deployment of such features.
However I the persistence draft could probably be improved to handles
incremental deployment. This could be addressed in the next revision.
A first tentative could be:
- over an iBGP session, the router detecting the BGP session failure,
flag the routes with a low (e.g. 0) LOCAL_PREF plus the STALE community.
Within the AS, the STALE community does not influence routing decision
but inform possible downstream BGP routers of the cause of the low
local_pref. In particular that the LOCAL_PREF should probably not be
further changed (some AS handles load balancing by rewriting the Local
Pref on the ingress PE)
- inform downstream AS of the STALE condition. First eBGP router could
translate this community into a low local_pref. With the iBGP mesh, cf
above.

>Question 2: How do you prevent forwarding loop in the case when
customer
>would have backup connectivity to his sites or connectivity via
>different VPN provider yet routers in his site only partially
understand
>the STALE community and only partially follow the forwarding rules ?

Cf above.

>In general as the rule is about mandating some particular order of path
>forwarding selection what is the mechanism in distributed systems like
>today's routing to be able to achieve any assurance that such rule is
>active and enforced across _all_ routers behind EBGP PE-CE L3VPN
>boundaries in customer sites ?

Cf above.

Best regards,
Bruno

>Best regards,
>R.
>
>
>-------- Original Message --------
>Subject: [Idr] draft-uttaro-idr-bgp-persistence-00
>Date: Sat, 22 Oct 2011 00:23:55 +0200
>From: Robert Raszuk <[email-masked]>
>Reply-To: [email-masked]
>To: [email-masked] List <[email-masked]>
>
>Hi,
>
>What is the point of defining DO_NOT_PERSIST community ? In other words
>why not having PERSIST community set would not mean the same as having
>path marked with DO_NOT_PERSIST.
>
>Observation:
>
>section I think it would be good to explicitly also state this in the
>best path selection that next hop to STALE best path must be valid.
>
>However sessions especially those between loopbacks do not go down for
>no reason. Most likely there is network problem which may have caused
>those sessions to go down. It is therefor likely that LDP session went
>also down between any of the LSRs in the data path and that in spite of
>having the paths in BGP and next hops in IGP the LSP required for both
>quoted L2/L3VPN applications is broken. That may particularly happen
>when network chooses to use independent control mode for label
allocation.
>document that during best path selection especially for stale paths a
>validity of required forwarding paradigm to next hop of stale paths
>should be verified.
>
>[email-masked]
>[email-masked]

---

From: <[email-masked]>
Date: Wed, 2 Nov 2011 14:34:45 +0100

Hi Robert,

>From: Robert Raszuk, Sent: Tuesday, October 25, 2011 8:16 AM
>
>Hi Jim,
>
>> [Jim U>] Why do you believe that? BGP can
>> and does fail independently of the underlying LDP session.. I do
>> believe that viability of a data path between two endpoints should be
>> taken into account when validating paths to NHs but that effort is
>> clearly a much bigger effort.. This draft would take advantage of
>> this mechanism but I am not looking to define that in this draft..
>
>In my opinion  there is a need to recognize the difference of possibly
>locally using a stale path (and that is btw exactly what GR spec does
..
>except here it would be GR per prefix with longer timer) versus keep
>advertising paths which may be invalid.

There may be a need to distinguish both. However, I wish you could
explain why you see this need.

BTW GR does both: locally keep a stale path _and_ implicitly keep
advertising a path which may be invalid. 

>> paths should be verified. [Jim U>] Why would this not be something
>> that needs to be done as a general rule for paths.. I mean the
>> underlying LSP could fail independently of BGP.. If this needs to be
>> done in should be done as a general rule...
>
>Respectfully disagree.
>
>I am sure you very well know how many customers are relaying on using
>BGP keepalives to determine PE-CE link up and PE-CE reachability.
Recall
>how many times SP customers including your own customers were asking to
>reduce holdtime on their ebgp session to 3/9 ... those customers just
do
>not have a better way to determine VPN site reachability.

Regarding the activation of persistence over an eBGP PE-CE session, I
agree both with you and Jim that you would need to evaluate how much the
forwarding will share fate with the BGP session. And if there is a
doubt, you would need to check forwarding status (e.g. BFD if link level
info is not reliable).
Note however, that this is not new. GR has the same issue (e.g. if on
the CE the forwarding share fate with the control plane, you should not
enable BG GR). Note also that even if persistence take the wrong
decision (i.e. advertise the route while the forwarding is down) if the
site is not dual homed, the issue is limited as traffic is dropped in
all cases. And if the site is dual homed, persistence will favor the
backup link. Hence the status of the forwarding is less important that
for GR, which do use the link/BGP session with an uncertain status.

>Of course such reduction of keepalives is a bad idea (even if their
>processing is offloaded or prioritized on main RP) so BFD was invented
>to address this issue. And either BFD if present or EBGP session up
>state was _the_only_ probe to determine if CE over frame relay PVC or
>multiaccess media is live. That is the key difference.
>
>So when you drop session you remove prefixes advertised from the site
>and backup can be activated. Now you are proposing to keep advertising
>them as STALE which really means MAYBE_REACHABLE. I am not sure why
>validation of next hop reachability by BFD or even OBJECT_TRACKING or
>SCRIPT would be at all difficult.
>
>Anyway I think this proposal as pointed out in the other email has much
>bigger deployment issue so I think there is not much point on this
>validation debate at this point before the loop problem is solved due
to
>inconsistent forwarding plane decisions.
>
>Best regards,
>R.

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 02 Nov 2011 14:39:02 +0100

Hi Bruno,

 > --> The customer does not see the "STALE" information.

Nope - it sure does.

As I have obviously foreseen you are going to say so I specifically 
pointed out that there is no symmetry on the PEs due to the specific RT 
configuration. See below quote:

"Prefix X is advertised by remote hub in the given VPN such that PE1 vrf
towards CE1 only has X via PE3 and PE2's vrf towards CE2 only has X via
PE4."

> specific to the persistence draft).

I think any draft modifying routing decision should discuss such issues.

> One answer is careful deployment of such features.

I am afraid this is very difficult to achieve across EBGP boundaries. If 
you would just limit STALE to IBGP I would perhaps have a bit lighter 
objection :)

> translate this community into a low local_pref. With the iBGP mesh, cf

Translating STALE to Local Pref over EBGP and not using STALE for local 
decision makes the proposal much less dangerous indeed.

Is any action associated with STALE going to be disabled by default in 
all BGP implementations ? If yes I would see it acceptable - provided 
the above changes are introduced.

However if any implementation is going to enable auto translation of 
STALE to local pref = 0 across EBGP boundary, or make best path decision 
based on STALE presence -  we have the same problem as soon as any two 
routers are not upgraded in the same time to the same vendor's code 
version.

--

Regardless how novel and appealing is the idea of telling your customer 
that you can possibly forward, but your control plane is undergoing 
recovery - I still think that churn introduced by this sort of 
signalling on a per path basis is a pretty bad idea. If anything I would 
rather see this in the BGP Operational Message not in the actual routing 
UPDATE messages.

Cheers,
R.

---

From: <[email-masked]>
Date: Wed, 2 Nov 2011 14:47:42 +0100

Robert,

>feature may cause real network issue if not done carefully - that
should
>trigger the alarm to rethink the solution and explore alternative
>approaches to the problem space.
>
to
>employ per prefix GR with longer and locally configurable timer. That
>would address information persistence across direct IBGP sessions.
>
>On the RRs use case of this draft we may perhaps agree to disagree, but
>I do not see large enough probability of correctly engineered RR plane
>to experience simultaneous multiple ibgp session drops. If that happens
>the RR placement, platforms or deployment model should be
re-engineered.

The low chance / probability of the failure of both RR planes is indeed
a good point. I can also agree that this is expected to be low. I
disagree that it never happens. E.g. with current BGP error handling,
the same BGP attribute in "error" would cause both iBGP plane to go
down. Hopefully work is being done to improve this. But I would not bet
that this will cover all future cases. In such catastrophic cases, some
routing is better than no routing.
On a side note, considering that the RR and hence their failure are
independent and therefore will quite never fail at the same time, is
questionable. To start with, they process they mostly process the same
input (BGP updates) and are part of the same network. 

>> -----Original Message----- From: [email-masked]
>> Sunday, October 23, 2011 5:32 PM To: [email-masked] List Subject: [Idr]
>> +0200 From: Robert Raszuk<[email-masked]> Reply-To:
>> [email-masked] To: [email-masked] List<[email-masked]>
>>
>> Hi,
>>
>> [email-masked] https://www.ietf.org/mailman/listinfo/idr
>>
>>
>> [email-masked] https://www.ietf.org/mailman/listinfo/idr
>>
>>
>

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 02 Nov 2011 14:55:41 +0100

Bruno,

> that this will cover all future cases.

If trigger is due to malformed update why are we so concerned about RRs? 
Wouldn't equally susceptible be also PEs/ASBRs ?

And the cure that those is GR, not persistence ;).

> In such catastrophic cases, some routing is better than no routing.

I don't know about that.

> input (BGP updates) and are part of the same network.

I don't think I used the term "never". But with dual or triple vendor 
control plane the probability of RR network wide outage drops 
significantly lower.

Cheers,
R.

---

From: <[email-masked]>
Date: Wed, 2 Nov 2011 15:15:27 +0100

Hi Enke,

>From: Enke Chen; Sent: Thursday, October 27, 2011 2:43 AM
>
>Hi, folks:
>

That's a good comment. Do you think the draft should put both GR and
persistence into perspective? (e.g. either in the introduction or in
appendix)

>the RR when it is not in the forwarding path.

GR has limitations: limited duration, does not handle consecutive
session restarts, (currently) does not handle BGP failures, does not
advertise that the STALE path may be less trusted.

>wouldn't be better to retain all the routes on a PE/client?

I find interesting that you considered that all routes would be kept
STALE for a long time. I was rather anticipating that some people would
be uncomfortable with this, especially in the VPN cases as correct and
up to date routing information are required to ensure isolation between
VPNs. (cf security section of the persistence draft).
Regarding your point, some route could be fairly static and hence kept
STALE for a long time. Other routes may be more dynamic and more
problematic to keep STALE for a long time. Hence the ability for a
different response. Also, some route may be more critical than others.

>another variation unnecessarily.

Indeed, possibly the persistence requirements could be inserted in the
GR draft. But how much are you / IDR WG ready to change that much the GR
procedures? That would basically force all GR implementations to
implement the persistence behavior.

Besides, IMHO, GR and persistence make different assumptions and hence
are rather different:
- GR assumes that the forwarding is preserved during the BGP session
restart and hence decides to not advertise this event in the network.
For such assumption, you need to believe pretty strongly that the
forwarding is indeed preserved. IMHO this calls for a short timer
duration and quick fall back to session failure if you suspect this may
not be the case (e.g. in case of consecutive restart of the BGP session,
GR is aborted).
- "persistence" considers that in case of issues, some stale information
is better than no information, but worst than correct information. The
downside is BGP churn to advertise the event. The good side is that this
is a safer assumption hence more likely to be valid over time or events.

Regards,
Bruno

---

From: <[email-masked]>
Date: Wed, 2 Nov 2011 15:44:56 +0100

Enke,

Please find some comments inlined [BD]

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Enke Chen
Sent: Friday, October 28, 2011 8:19 AM
To: UTTARO, JAMES
Cc: [email-masked] List; [email-masked]
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-00

Jim,

My comments are inlined.

Enke,
 
GR is a solution that is essentially local in scope it does not have the
ability to inform downstream speakers of the viability of routing state
from the point of possible control plane failure. OTOH Persistence does
propagate the condition of state. This provides distinct advantages in
terms of customers awareness of the SPs control plane. One could imagine
a customer receiving a STALE path and responding by selecting a backup.
Some of the extensions to this draft that I have considered in colouring
of STALE to inform if the condition arises from a local ( PE ) or
internal iBGP ( RR ) failures.. 
 
GR makes no distinction from STALE state and ACTIVE state.. This can
lead to the STALE path still being preferred throughout the topology.
IMO this is incorrect behavior regardless of the comparison. 
 
PERSISTENCE allows for a customer to indicate which paths should be
candidates. Customers may want to immediately failover to the backup for
some paths and not for others. GR is not capable of doing this it is all
or nothing. The granularity is not sufficient. It needs to be at the
path level. There may even be a case for having even more granularity
i.e a per path timer.. GR is not capable of being extended for either of
these cases.

I am not sure how this path level persistence would work operationally.
Without the detailed information of a provider's network, how would a
customer know what kind of failures and recovery that they might
experience?   Consider the example of the simultaneous RR failures in
the draft,  why wouldn't any customer not to want to protect against
such failures?   The end result could be that the PERSISTENCE flag is
always set, thus losing its significance.

Regarding the use of the STALE state vs ACTIVE state, clearly there is a
tradeoff.

[BD] clearly. Cf my previous email.

   GR uses the stale routes in order to avoid forwarding churns, which
has been a critical requirement for a long time.   If there is a real
need for favoring a ACTIVE one over a STALE one in GR, it can be done by
a simple knob.

[BD] That "simple knob" would contradict the GR spec "The router MUST
NOT  differentiate between stale and other routing information during
forwarding."

As you know, BGP is full of knobs that adjust behaviors for different
needs :-)

[BD] sure. But I fail to see why a set of knobs independently defined by
each implementation would be better than a draft describing the expected
behavior.

GR does not provide protection through successive restarts of the
session. I believe that if this occurs the state will be invalidated. So
for a session that is bouncing due to overload condition GR will not
provide the required protection

This can be addressed by a simple knob to set the min stale timer for
GR.

[BD] I'm not sure to understand how changing the min stale timer would
address this. Again, this "simple knob" would contradict the GR spec "To
deal with possible consecutive restarts, a route (from the peer)
previously marked as stale MUST be deleted."

GR does not employ a make before break strategy. All state is
invalidated first then the newly learned state is processed. This leads
to routing churn especially if the majority of the state is the same
which I am pretty sure is the case

Such behavior would be an implementation bug that needs to be fixed.
But it is not an issue with the protocol itself.

This is what we have in 4.2. Procedures for the Receiving Speaker, RFC
4724:

---

---

There are several possibilities for the premature purge of the stale
routes. For example, the "Forwarding state" flag was somehow not set
after the session was re-established, or the the EOR was sent
prematurely.   Further investigation will be needed in order to identify
any possible implementation or config issues involved in your setup.

 
GR invalidates state due to the case of protocol error i.e A malformed
update will invalidate all of the state. This is not the desired
behavior.

http://datatracker.ietf.org/doc/draft-keyupate-idr-bgp-gr-notification/
<http://datatracker.ietf.org/doc/draft-keyupate-idr-bgp-gr-notification/

[BD] I would rather have said that this should be addressed by
draft-ietf-idr-optional-transitive-04. (as in case of a malformed
update, draft-keyupate-idr-bgp-gr-notification, would mostly GR restart
the BGP session again and again indefinitely). However, as hard as we
try, I would not bet that draft-ietf-idr-optional-transitive will cover
all possible future cases. This is mostly a (good IMHO) reaction to past
issue, rather than a solution to all future issues BGP could have to
face. OTOH, persistence could be the safety net to address the
unexpected issues.

GR is not specific as to which events invoke it or not. From my read on
the draft it is not clear if holdtime expiration invokes GR or not.. The
draft is unclear.

I think that it is covered by the above extension.  If not, it should be
clarified.

[BD] Indeed. I think this should be clarify in the GR spec. Could you
please clarify this in draft-chen-idr-rfc4724bis-00.txt?

I think that there is a main difference when a RR is not in the
forwarding path.  In that case, the RR should always set the F bit in
the GR Capability so that its clients will continue forwarding after
they lose the sessions with RR.  It is a deployment issue, though.

 

I think that you are talking about the "Restart time" field which has 12
bits and amount to about 68 minutes.  The "Restart time" is for the
session re-establishment.  It does not impact the duration for holding
stale routes after the session is re-established.

If the session does not get re-established in 68 minutes, the stale
routes would be purged.  That is a long time, isn't it?   However, if
one really wants to extend the session re-establishment time and
continue to hold stale routes, it can be done by a simple knob.

[BD] It's advertised in the GR capability and bounded by the 12 bits
fields. Eventually, you could have local knob to override every GR
behavior, but I would not call this GR anymore.

 
GR does detail the behavior where convergence is not achieved between
restarts.. Similar to above.. 

But do you meant "does not"?  We can certainly clarify in 4724bis if
that is the case.

 
I do not believe that the current GR paradigm can be extended to cover
the majority of the cases above.

Except for the path level persistence you mentioned, I believe the GR
will be able to address all other persistence requirements you listed,
with some simple knobs and some implementation enhancements. 
[BD] What about calling "persistence" this set of "simple knobs and
implementation enhancements"?

Regards,
Bruno
 
Thanks,
       Jim Uttaro

Thanks.   -- Enke

 
-----Original Message-----
From: Enke Chen [mailto:[email-masked]] 
Sent: Wednesday, October 26, 2011 8:43 PM
To: UTTARO, JAMES
Cc: [email-masked]; [email-masked] List; Enke Chen
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-00
 
Hi, folks:

 

 
Thanks.   -- Enke

	Jim,
	 
	When one during design phase of a routing protocol or routing
protocol 
	extension or modification to it already realizes that enabling
such 
	feature may cause real network issue if not done carefully -
that 
	should trigger the alarm to rethink the solution and explore 
	alternative approaches to the problem space.
	 
	We as operators have already hard time to relate enabling a
feature 
	within our intradomain boundaries to make sure such rollout is
network 
	wide. Here you are asking for the same level of awareness across
ebgp 
	boundaries. This is practically unrealistic IMHO.
	 
	Back to the proposal ... I think that if anything needs to be
done is 
	to employ per prefix GR with longer and locally configurable
timer. 
	That would address information persistence across direct IBGP
sessions.
	 
	On the RRs use case of this draft we may perhaps agree to
disagree, 
	but I do not see large enough probability of correctly
engineered RR 
	plane to experience simultaneous multiple ibgp session drops. If
that 
	happens the RR placement, platforms or deployment model should
be 
	re-engineered.
	 
	Summary .. I do not think that IDR WG should adopt this
document. Just 
	adding a warning to the deployment section is not sufficient.
	 
	Best regards,
	R.

		Robert,
		 
		The introduction of this technology needs to be
carefully evaluated
		when being deployed into the network. Your example
clearly calls out
		how a series of independent design can culminate in
incorrect
		behavior. Certainly the deployment of persistence on a
router that
		has interaction with a router that does not needs to be
clearly
		understood by the network designer. The goal of this
draft is to
		provide a fairly sophisticated tool that will protect
the majority of
		customers in the event of a catastrophic failure.. The
premise being
		the perfect is not the enemy of the good.. I will add
text in the
		deployment considerations section to better articulate
that..
		 
		Thanks, Jim Uttaro
		 
		-----Original Message----- From: [email-masked]
		[mailto:[email-masked]] On Behalf Of Robert Raszuk
Sent:
		Sunday, October 23, 2011 5:32 PM To: [email-masked] List
Subject: [Idr]
		draft-uttaro-idr-bgp-persistence-00
		 
		Authors,
		 
		Actually when discussing this draft a new concern
surfaced which I
		would like to get your answer on.
		 
		The draft in section 4.2 says as one of the forwarding
rules:
		 
		o  Forwarding to a "stale" route is only used if there
are no other
		paths available to that route.  In other words an active
path always
		wins regardless of path selection.  "Stale" state is
always
		considered to be less preferred when compared with an
active path.
		 
		In the light of the above rule let's consider a very
simple case of
		dual PE attached site of L3VPN service. Two CEs would
inject into
		their IBGP mesh routes to the remote destination: one
marked as STALE
		and  one not marked at all. (Each CE is connected to
different PE and
		each PE RT imports only a single route to a remote hub
headquarter to
		support geographic load balancing).
		 
		Let me illustrate:
		 
		VPN Customer HUB
		 
		PE3      PE4 SP PE1      PE2 |        | |        | CE1
CE2 |
		| 1|        |10 |        | R1 ------ R2 1
		 
		CE1,CE2,R1,R2 are in IBGP mesh. IGP metric of CE1-R1 and
R1-R2 are 1
		and R2-CE2 is 10.
		 
		Prefix X is advertised by remote hub in the given VPN
such that PE1
		vrf towards CE1 only has X via PE3 and PE2's vrf towards
CE2 only has
		X via PE4.
		 
		Let's assume EBGP sessions PE3 to HUB went down, but
ethernet link
		is up, next hop is in the RIB while data plane is gone.
Assume no
		data plane real validation too. /* That is why in my
former message
		I suggested that data plane validation would be
necessary */.
		 
		R1 has X via PE1/S (stale) and X via PE2/A (active) - it
understands
		STALE so selects in his forwarding table path via CE2.
		 
		R2 has X via PE1/S (stale) and X via PE2/A (active) - it
does not
		understand STALE, never was upgraded to support the
forwarding rule
		stated above in the draft and chooses X via CE1 (NH
metric 2 vs 10).
		 
		R1--R2 produce data plane loop as long as STALE paths
are present in
		the system. Quite fun to troubleshoot too as the issue
of PE3
		injecting such STALE paths may be on the opposite site
of the world.
		 
		The issue occurs when some routers within the customer
site will be
		able to recognize STALE transitive community and prefer
non stale
		paths in their forwarding planes (or BGP planes for that
matter)
		while others will not as well as when both stale and non
stale paths
		will be present.
		 
		Question 1: How do you prevent forwarding loop in such
case ?
		 
		Question 2: How do you prevent forwarding loop in the
case when
		customer would have backup connectivity to his sites or
connectivity
		via different VPN provider yet routers in his site only
partially
		understand the STALE community and only partially follow
the
		forwarding rules ?
		 
		In general as the rule is about mandating some
particular order of
		path forwarding selection what is the mechanism in
distributed
		systems like today's routing to be able to achieve any
assurance that
		such rule is active and enforced across _all_ routers
behind EBGP
		PE-CE L3VPN boundaries in customer sites ?
		 
		Best regards, R.

		-------- Original Message -------- Subject: [Idr]
		draft-uttaro-idr-bgp-persistence-00 Date: Sat, 22 Oct
2011 00:23:55
		+0200 From: Robert Raszuk<[email-masked]>
<mailto:[email-masked]>  Reply-To:
		[email-masked] To: [email-masked] List<[email-masked]>
<mailto:[email-masked]> 
		 
		Hi,
		 
		I have read the draft and have one question and one
observation.
		 
		Question:
		 
		What is the point of defining DO_NOT_PERSIST community ?
In other
		words why not having PERSIST community set would not
mean the same as
		having path marked with DO_NOT_PERSIST.
		 
		Observation:

		o  Forwarding must ensure that the Next Hop to a "stale"
route is
		viable.
		 
		Of course I agree. But since we stating obvious in the
forwarding
		section I think it would be good to explicitly also
state this in
		the best path selection that next hop to STALE best path
must be
		valid.
		 
		However sessions especially those between loopbacks do
not go down
		for no reason. Most likely there is network problem
which may have
		caused those sessions to go down. It is therefor likely
that LDP
		session went also down between any of the LSRs in the
data path and
		that in spite of having the paths in BGP and next hops
in IGP the LSP
		required for both quoted L2/L3VPN applications is
broken. That may
		particularly happen when network chooses to use
independent control
		mode for label allocation.
		 
		I would suggest to at least add the recommendation
statement to the
		document that during best path selection especially for
stale paths
		a validity of required forwarding paradigm to next hop
of stale
		paths should be verified.

		Best regards, R.

		_______________________________________________ Idr
mailing list
		[email-masked] https://www.ietf.org/mailman/listinfo/idr

		_______________________________________________ Idr
mailing list
		[email-masked] https://www.ietf.org/mailman/listinfo/idr

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 02 Nov 2011 15:46:50 +0100

Bruno,

 > GR draft.

Really ? I find this totally opposite.

GR can be very useful without persistence draft. But how persistent 
draft be of any use without assuring even at the given router proper 
forwarding ?

So correct me if I am not mistaken. Imagine following sequence of events:

- PE operates fine GR is not enable and persistence is
- PE looses session to RRs as RRs goes down
- PE keeps all routes in control plane and data plane (no GR involved)
- PE re-advertises all routes (for given AFI/SAFI) to all 1000 of CEs
   marked as STALE
- PE re-establishes IBGP sessions with RRs and effectively performs GR
   invalid path removal

Is this correct ?

If so then end result is double churn to 1000 CEs and reuse of the main 
GR re-sync specification without enabling GR.

Moreover with this you are relaxing the need for any capability 
negotiation. Perhaps GR authors could clarify why did we need in the 
first place per SAFI GR capability negotiation. Going by your draft just 
global GR / EOR marker would be sufficient. I don't recall any 
discussion on that point in persistent draft too.

Cheers,
R.

---

From: <[email-masked]>
Date: Wed, 2 Nov 2011 16:35:00 +0100

Hi Robert,

>From: Robert Raszuk [mailto:[email-masked]] >Sent: Wednesday,
November 02, 2011 2:39 PM
>towards CE1 only has X via PE3 and PE2's vrf towards CE2 only has X via

Ok. I thought you were using local pref to influence routing decision.
If you actually remove the other route by RT configuration, this is
indeed different and the customer will see the STALE information.
Looks to me a very specific case as in you topology, a
failure/maintenance in the remote hub site and in the MH site could lead
to network unavailability (e.g. failure of PE3/CE3/PE3-CE3 link and
PE2/CE2/PE2-Ce link)

  VPN Customer HUB

    PE3      PE4
         SP
    PE1      PE2
     |        |

However, that does not change the below discussion as I had not
dismissed your argument.

>the above changes are introduced.

I'm happy that you would find the persistence proposal acceptable. We'll
work on this on the next revision.
Many thanks for bringing this valid point.

---

From: <[email-masked]>
Date: Wed, 2 Nov 2011 17:33:58 +0100

Robert,

>If trigger is due to malformed update why are we so concerned about
RRs?

We are concerned about RR-PE sessions because if the sessions are down,
all routes are down. PEs become essentially useless.

>Wouldn't equally susceptible be also PEs/ASBRs ?

Indeed, inter AS VPN eBGP session are also important. (although you
would one use the inter AS routes and hence PE/VPN are still useable.

>And the cure that those is GR, not persistence ;).

Cf Enke's thread.

>I don't think I used the term "never". 

Ok.

>But with dual or triple vendor

Sure. But given the small number of RR boxes, and the high requirements
on these boxes, it's harder to justify a two or three vendor sourcing.

Cheers,
Bruno

---

From: <[email-masked]>
Date: Wed, 2 Nov 2011 17:51:08 +0100

Robert,

>GR can be very useful without persistence draft.

That was mostly my point. Sorry if I didn't make it clear.

> But how persistent

For an eBGP session, I agree forwarding should be more or less trusted as running. (even if this is not strictly required given all routers will select the other path. Cf my previous email)

>- PE re-establishes IBGP sessions with RRs and effectively performs GR

No.
Interaction between GR and persistence is currently not described in this version. It will be in the next version.
In short, if GR & persistence were enabled both on a BGP session, GR would first run. Then if GR does not succeed, persistence would come into play.

>GR re-sync specification without enabling GR.

No.
If GR success, there would be no churn.
Otherwise, single churn

>global GR / EOR marker would be sufficient. 

Indeed, no capability negotiation is required. We only need EOR. In v00, it has been assumed that EOR is now a standard behavior. Otherwise we could explicitly signal this using the GR capability.

>I don't recall any

" A speaker
   configures the ability to persist independently of it's peer.  There
   is no negotiation between the peers.  A timer must be configured
   indicating the time to persist stale state from a peer where the
   session is no longer viable. "

§3 http://tools.ietf.org/html/draft-uttaro-idr-bgp-persistence-00#page-6

"   o  The Receiving Speaker MUST replace the stale routes by the routing
      updates received from the peer.  Once the End-of-RIB marker for an
      address family is received from the peer, it MUST immediately
      remove any paths from the peer that are still marked as stale for
      that address family."

§4.2 http://tools.ietf.org/html/draft-uttaro-idr-bgp-persistence-00#section-4.2

Cheers,
Bruno

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 02 Nov 2011 18:06:27 +0100

> on these boxes, it's harder to justify a two or three vendor sourcing.

I would think just the very opposite. Small number of RRs and their high 
requirements IMHO fully justify more then one vendor sourcing.

Of course fun starts where it is hard to find such diverse RRs with all 
required feature sets. Those concerned about such issues are looking 
actively in solving the problem fundamentally at it's roots and not 
cover it with yet another curtain.

Cheers,
R.

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 02 Nov 2011 18:15:46 +0100

Bruno,

> No. Interaction between GR and persistence is currently not described
> in this version. It will be in the next version. In short, if GR&
> persistence were enabled both on a BGP session, GR would first run.
> Then if GR does not succeed, persistence would come into play.

Ohh this is news.

How do you define GR success ? Do you mean that persistance would work 
till GR timer expires then if session did not come up re-advertise all 
routes as stale as opposed to withdraw them ?

> No. If GR success, there would be no churn. Otherwise, single churn

Sorry ... double. When session comes up and such session was there I 
assume for a reason so it is likely to come up you need to wipe out the 
STALE color.

Btw ... I assume next version will also include discussion that SHUT 
community is mutually exclusive with persistence draft - correct ? ;)

> Indeed, no capability negotiation is required. We only need EOR. In
> v00, it has been assumed that EOR is now a standard behavior.
> Otherwise we could explicitly signal this using the GR capability.

As much as we all want to have it as standard behaviour it is not yet so 
mentioning about it will not hurt.

> "   o  The Receiving Speaker MUST replace the stale routes by the
> routing updates received from the peer.  Once the End-of-RIB marker
> for an address family is received from the peer, it MUST immediately

And that is precisely what GR spec recommends. This is also why I 
started by saying that persistence draft copies existing control plane 
functionality from GR RFC.

Best,
R.

---

From: <[email-masked]>
Date: Wed, 2 Nov 2011 18:22:05 +0100

>requirements IMHO fully justify more then one vendor sourcing.

Somehow we disagree on this. 
Regarding the small number of RR, if you need 10 RR, it's hard to go to
the buying, testing, integration and ops teams asking for a new vendor
for 5 only boxes.
Regarding dual sourcing, to keep redundancy, you would need to restrict
yourself to the smaller intersection of features and performance. This
can be limiting.

>required feature sets.

My point. Both features and scale.

> Those concerned about such issues are looking

Dual sourcing helps but does not guaranty that both RR will never fail.
You said it yourself. So this does not solve the fundamental problem.

However, again, given the low probability of double failure, it's harder
to justify work on this.

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 02 Nov 2011 18:26:52 +0100

> You said it yourself. So this does not solve the fundamental problem.

When I mentioned to solve the fundamental problem I did not mean to 
continue to use current model in particular to continue to use RRs at all.

Btw this is topic not for this thread ;) I will start new one perhaps in 
few weeks to discuss it.

Cheers,
R.

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Wed, 2 Nov 2011 17:53:26 +0000

I believe that the what is being missed in this discussion is impact... Yes Dual RR failures are rare but when they happen it is analogous to an asteroid killer, you know, front page NYT etc.. So although it is a low frequency event the impact is beyond the pale.. Without getting into specifics I know this has happened. It needs to be defended against in a way that does not rely on probabilities of different RRs from different vendors on different platforms.. All of that may lessen the chance ( maybe ) but it comes with a huge testing/certification/inter-operability etc... challenge and does not guarantee protection. 

Thanks,
	Jim Uttaro

---

From: Jakob Heitz <[email-masked]>
Date: Wed, 2 Nov 2011 16:05:45 -0400

> the draft try to solve :-(

Me too.

The persisting routers will persistently send labeled packets
into the core. If the intended destination really has disappeared,
and restarted, what is the chance that such labeled packets
interfere with other unrelated services, just because of labels
being reused?

Quote from 3.1 of the draft:
The persist-timer
      should be set to a large value on the order of days to infinity.

Customers rely on the separation between VPN's. The "P" means private.
Anything that threatens that "P" should not be taken lightly.

I'm starting to imagine my video stream intrespersed with dzzt, zzt
from random packets being injected into it. How real is that?

-- 
Jakob Heitz.

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Wed, 2 Nov 2011 20:20:40 +0000

Jakob,

	See the security section of the draft.

Jim Uttaro

-----Original Message-----
From: Jakob Heitz [mailto:[email-masked]] 
Sent: Wednesday, November 02, 2011 4:06 PM
To: Enke Chen; UTTARO, JAMES
Cc: [email-masked] List; [email-masked]
Subject: RE: [Idr] draft-uttaro-idr-bgp-persistence-00: Security Considerations

Me too.

-- 
Jakob Heitz.

---

From: Jakob Heitz <[email-masked]>
Date: Wed, 2 Nov 2011 16:29:18 -0400

> 	See the security section of the draft.

I did, but it doesn't answer my questions.

It diverts the reader's attention with talk of
malicious disturbance.

Accidental disturbance seems much more real to me.
The chance of such an accidental disturbance seems to
me to be high enough to sink this proposal cold.

However, I don't actually know. Am I scared of ghosts
or is it real? Some credible research results might
relieve my anxiety.

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 02 Nov 2011 23:35:02 +0100

Hi Jakob,

Let me try to understand the issue you are describing a bit better.

If we have a below scenario:

           / PE2
PE1 -- RR
           \ PE3

When PE2 disappears RR will re-advertise his routes as persistent to PE1 
with some labels - for simplicity let's assume those are L3VPN labels.

There are two options ...

* PE2 went down for good - RR by advertising persistent routes may at 
most cause to blackhole the traffic - that is why I started by 
recommending a MUST for next hop reachability

* PE2 restarted - here I assume RR would withdraw previous STALE routes 
by both implicit withdraw and explicit withdraw - when new session comes 
up between RR and PE2. RR knows the same rtr_id/peer_ip (session got 
reestablished). I really can not imagine otherwise regardless of 
persistence timer value setting.

Labels are of local significance. They alone mean nothing without the 
next hop. So I fail to see the case you describe or for that matter 
described by security section that even transiently labels advertised 
with BGP NLRIs are going to cause harm or inter-VPN disturbance.

En-light me pls ! I want to see the ghost ;-)

Cheers,
R.

> On Wednesday, October 26, 2011 5:43 PM, Enke Chen<>  wrote:
>
>> Hi, folks:
>>

---

From: Eric Rosen <[email-masked]>
Date: Wed, 02 Nov 2011 22:49:39 -0400

> with some labels - for simplicity let's assume those are L3VPN labels.

In the case being discussed (I think), the RR is down at the time PE2
disappears, but PE1 is still using the routes previously distributed to it
by the RR.

> by both implicit withdraw and explicit withdraw

The RR can't withdraw any routes because it is down.

If PE1 happens to have noticed that PE2 blinked out, it may be wise for PE1
to invalidate the routes for which PE2 is the next hop.  If PE1 did not
notice that PE2 blinked out, it will continue to make use PE2's old VPN-IP
routes.  Unfortunately, all these routes will now have incorrect labels, and
packets from one VPN may get forwarded by PE2 to another VPN.

As far as I can tell, until the RR comes back up, there is no assurance that
PE1 will notice PE2's restart, so random cross-connection of VPNs is
certainly possible.

But it's not obvious that random cross-connections of this sort are any
worse than a complete loss of service.  While datagrams from one VPN might
get sprayed into the other, it is hard to see how a TCP connection between
the two VPNs could be set up or maintained, since there is not going to be a
working reverse path.

---

From: Jakob Heitz <[email-masked]>
Date: Wed, 2 Nov 2011 23:51:19 -0400

On Wednesday, November 02, 2011 7:50 PM, Eric Rosen <mailto:[email-masked]> wrote:

>> When PE2 disappears RR will re-advertise his routes as persistent to
>> PE1 with some labels - for simplicity let's assume those are L3VPN
>> labels. 
> 
>> * PE2 restarted - here I assume RR would withdraw previous STALE
>> routes by both implicit withdraw and explicit withdraw
> 

I didn't think a stray packet could set up a TCP connection
either. I was wondering if stray packets could cause
disruptions of other kinds. Also, as a customer of a VPN
service, I would not like the possibility of my packets being
sprayed into other VPNs.

-- 
Jakob Heitz.

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 03 Nov 2011 07:06:07 +0100

Eric,

> routes.

So when RR is down PE2's BGP process restarts while IGP not - PE1 has no 
chance to notice any next hop blink - hence till RR comes up and cleans 
up the mess pretty much network is in a bad state.

How long RR will wait after coming up to get his previous IBGP peering 
reestablished before starting the cleanup ?

Thx,
R.

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 03 Nov 2011 07:15:03 +0100

Extending the question ...

 > How long RR will wait after coming up to get his previous IBGP
 > peering reestablished before starting the cleanup ?

Actually RR can not do any cleanup if RR - PE2 never comes up can it ? 
It seems that PE1 will continue in this case to keep zombie paths PE2 
via RR forever.

The possible solution would be for PE1 to not only invalidate paths with 
"blinked" next hop, but also when previously lost session given paths 
were received over got reestablished. But I do not think draft mandates 
such behaviour.

R.

---

From: <[email-masked]>
Date: Thu, 3 Nov 2011 09:45:03 +0100

Jakob,

>From: Jakob Heitz >Sent: Wednesday, November 02, 2011 9:06 PM
>

If by "really has disappeared", you mean the egress PE (PE2 in Robert's
picture, copied below) is down, then BGP routes whose BGP Next Hop are
PE2 are invalidated (by all ingress PE and also by the persistent RR).
No problem.
If you actually mean that PE2 is alive but has lost its BGP session to
both RR, then this is the case discussed in the security section of the
draft. If you're not happy with it, please be more specific and say why
(because " It diverts the reader's attention with talk of malicious
disturbance." is not helping in addressing your original comment).

            / PE2
 PE1 -- RR
            \ PE3

---

From: <[email-masked]>
Date: Thu, 3 Nov 2011 09:45:11 +0100

Robert,

>From: Robert Raszuk >Sent: Thursday, November 03, 2011 7:06 AM
>
>Eric,
>

I've lost the kind of failures you are discussing. I don't recognize
Jakob's case any more.
Robert, are you talking about the following case:
- RR1 is down
- RR2 is down
- PE2's BGP process restarts while IGP not.

If yes, that seems a pretty specific case to me, but nonetheless, in
such case, I believe the security section of the draft is applicable to
this case. In short, at the time of the failure, forwarding (labels in
FIB) are still correct. They will start becoming incorrect when PE2
change them (as ingress PE won't be aware of that change). If it's only
a BGP restart on PE2 (everything else working fine) I assume PE2 should
try to reuse the same label for the same route. Cf RFC 4781. If it's
others events (e.g. customers route churn), I believe the security
section of the draft precisely discuss this.

If no, could you describe more precisely the case you are considering?

>>> When PE2 disappears RR will re-advertise his routes as persistent to
PE1
>>> * PE2 restarted - here I assume RR would withdraw previous STALE
routes

---

From: <[email-masked]>
Date: Thu, 3 Nov 2011 09:49:47 +0100

Robert,

>From: Robert Raszuk [mailto:[email-masked]] Sent: Wednesday,
November 02, 2011 6:16 PM
>> Then if GR does not succeed, persistence would come into play.

This is a teaser. To save cycles, I suggest we wait for the draft to
describe interaction with GR, before we discuss by emails.

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 03 Nov 2011 13:37:19 +0100

> If it's only
> try to reuse the same label for the same route.

I don't think so. Usually BGP asks for label block from label manager 
and allocates labels as seems fit.

Only when a router would support RFC4781 which is BGP GR with labels 
that may not be the case.

So are you saying that persistence draft now requires support of enabled 
RFC4781 as prerequisite ?

Cheers,
R.

---

From: <[email-masked]>
Date: Thu, 3 Nov 2011 15:21:31 +0100

>RFC4781 as prerequisite ?

No, this is not required.
But I have the impression that the possible use of RFC 4781 should be
discussed as part of the security section of the draft.

Cheers,
Bruno

---

From: Eric Rosen <[email-masked]>
Date: Thu, 03 Nov 2011 12:06:36 -0400

> I was wondering if stray packets could cause disruptions of other kinds.

If PE2 connects to two sites of VPN1, then after PE2 restarts, proper VPN
service between those two sites can be established without the aid of the
RR.  However, if those sites are now getting stray packets from VPN2 (via
PE1), it does seem possible that these packets will disrupt service in
VPN1.  In such a case, the "persistence" strategy disrupts a service that
would otherwise have been fine.  This may or may not be deemed acceptable by
a SP, but probably should be discussed in the draft.

> as a customer of a VPN service, I would not like the possibility of my
> packets being sprayed into other VPNs.

This is the issue that the Security Considerations attempts to address by
pointing out that there is no way to cause this spraying maliciously, and
hence no way for someone to actually exploit the situation.  The assumption
is that a bit of accidental spraying is not actually going to result in
damages, and is unlikely to even be noticed.

Well, it is unlikely to be noticed unless the customer does a traceroute,
and the ICMP error messages somehow make it back to the customer (via the
Internet, say).  

Whether damages will result may depend on whether someone with poor ethics
just happens to be running a LAN analyzer at an opportune moment, and just
happens to see a packet from the other VPN that just happens to have some
useful information in it.

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Tue, 15 Nov 2011 03:08:55 +0000

All,

                First let me apologize for not being able to attend the ietf..

                After watching jabber it seems that the presentation of persistence degraded to something in re how we can apply multiple Band-Aids, patches, knobs etc... to extend GR to cover the Persistence draft.. IMO GR is a subset of the Persistence functionality and may be therefore subsumed into the Persistence draft. Philosophically GR makes the session the invariant and all behavior that follows is based upon this premise.. As stated in the slides, many of the new services have much looser association between the control and forwarding planes i.e L2VN, L3VPN, 3107 etc...

 Again let me clarify, when I looked at GR earlier this year it became clear that the basic philosophy, and rules stating when to persist, how long, triggers to stop would not meet the requirements I have.. From a philosophical point GR assumes that the paths that are persisting in forwarding should maintain their preference throughout the routing topology. Why? This seems beyond prescriptive as it does not provide the network operators with any flexibility as to how state learned  over a compromised control may be viewed/used by the downstream routing topology including customers. From my reading of the draft there is no way to allow operators to persist, not-persist, or de pref the state (stale). Do I have this wrong?  I do not think so.. So if we want to incorporate this ability into GR it would require a basic philosophical change.. of course even if this is done there is no way for a peer to pre-program  paths it has advertised to be treated if there is a failure.. So, even if we perform major surgery there is still no way to accomplish this beyond unique filters and maybe CVs.. WE could call them STALE, PERSIST and DO_NOT_PERSIST and then build routing policy across all of our BGP speakers.. Then we can spends lots of time making sure that this bottom approach is consistent across the entire topology, and coordinate with customers also..

As I mentioned in a previous note ( see below ) GR does not meet requirements in terms of triggers that terminate GR persistence, timers etc...  personally I am not that interested in  a solution that fundamentally does not meet my requirements.. Most of the responses I have received is that GR can be extended. I do not think it can ever meet the requirements above and will require major surgery to fix the triggers, timers etc.. So the approach should be to fold the subset of GR functionality into the larger Persistence draft..

Jim Uttaro

Enke,

Thanks,
        Jim Uttaro

From: UTTARO, JAMES
Sent: Tuesday, November 01, 2011 11:20 AM
To: 'Enke Chen'
Cc: [email-masked]; [email-masked] List
Subject: RE: [Idr] draft-uttaro-idr-bgp-persistence-00

Enke,

                Comments in-Line..

Thanks,
                Jim Uttaro

Jim,

My comments are inlined.

Enke,

---

   address family.

-          Holdtime Expiration

-          Malformed Update

Thanks,

        Jim Uttaro

Thanks.   -- Enke

-----Original Message-----

To: UTTARO, JAMES

Hi, folks:

Thanks.   -- Enke

Jim,

re-engineered.

Best regards,

R.

Robert,

Thanks, Jim Uttaro

Authors,

Let me illustrate:

VPN Customer HUB

and R2-CE2 is 10.

X via PE4.

will be present.

forwarding rules ?

Best regards, R.

Hi,

Question:

Observation:

viable.

valid.

Best regards, R.

Idr mailing list

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 15 Nov 2011 08:10:26 +0100

Hi Jim,

You are correct that GR does not allow to propagate information by 
default on what paths should be called as "suspicious". However as 
mentioned at the mic during the session we need to have a way to modify 
best path selection criteria universally as opposed to continue to 
invent new tools to solve point problems and punch holes in the various 
implementations of BGP best path selection.

Chairs in fact responded that we will work on this when the right time 
comes.

Back to the persistence draft ... GR can address 90% of the persistence 
draft requirements. Propagating information about "suspicious" paths if 
at all we all agree if this is a good idea can be done today with either 
setting lowest local pref, using cost community or marking with 
community so your ebgp peers can interpret it correctly.

Why do we need new way to signal this ?

Also I think current -00 version has serious issues as already pointed 
on the list. We need to wait till -01 is published in order to see if 
those issues have been fixed.

Best regards,
R.

> After watching jabber it seems that the presentation of persistence
> degraded to something in re how we can apply multiple Band-Aids,
> patches, knobs etc... to extend GR to cover the Persistence draft..
> IMO GR is a subset of the Persistence functionality and may be
> therefore subsumed into the Persistence draft. Philosophically GR
> makes the session the invariant and all behavior that follows is
> based upon this premise.. As stated in the slides, many of the new
> services have much looser association between the control and
> forwarding planes i.e L2VN, L3VPN, 3107 etc...
>
> Again let me clarify, when I looked at GR earlier this year it became
> clear that the basic philosophy, and rules stating when to persist,
> how long, triggers to stop would not meet the requirements I have..
> From a philosophical point GR assumes that the paths that are
> persisting in forwarding should maintain their preference throughout
> the routing topology. Why? This seems beyond prescriptive as it does
> not provide the network operators with any flexibility as to how
> state learned  over a compromised control may be viewed/used by the
> downstream routing topology including customers. From my reading of
> the draft there is no way to allow operators to persist, not-persist,
> or de pref the state (stale). Do I have this wrong?  I do not think
> so.. So if we want to incorporate this ability into GR it would
> require a basic philosophical change.. of course even if this is done
> there is no way for a peer to pre-program  paths it has advertised to
> be treated if there is a failure.. So, even if we perform major
> surgery there is still no way to accomplish this beyond unique
> filters and maybe CVs.. WE could call them STALE, PERSIST and
> DO_NOT_PERSIST and then build routing policy across all of our BGP
> speakers.. Then we can spends lots of time making sure that this
> bottom approach is consistent across the entire topology, and
> coordinate with customers also..
>
> As I mentioned in a previous note ( see below ) GR does not meet
> requirements in terms of triggers that terminate GR persistence,
> timers etc...  personally I am not that interested in  a solution
> that fundamentally does not meet my requirements.. Most of the
> responses I have received is that GR can be extended. I do not think
> it can ever meet the requirements above and will require major
> surgery to fix the triggers, timers etc.. So the approach should be
> to fold the subset of GR functionality into the larger Persistence
> draft..
>
> Jim Uttaro
>
>
> Enke,
>
> GR is a solution that is essentially local in scope it does not have
> the ability to inform downstream speakers of the viability of routing
> state from the point of possible control plane failure. OTOH
> Persistence does propagate the condition of state. This provides
> distinct advantages in terms of customers awareness of the SPs
> control plane. One could imagine a customer receiving a STALE path
> and responding by selecting a backup. Some of the extensions to this
> draft that I have considered in colouring of STALE to inform if the
> condition arises from a local ( PE ) or internal iBGP ( RR )
> failures..
>
> candidates. Customers may want to immediately failover to the backup
> for some paths and not for others. GR is not capable of doing this it
> is all or nothing. The granularity is not sufficient. It needs to be
> at the path level. There may even be a case for having even more
> granularity i.e a per path timer.. GR is not capable of being
> extended for either of these cases.
>
> session. I believe that if this occurs the state will be invalidated.
> So for a session that is bouncing due to overload condition GR will
> not provide the required protection
>
> invalidated first then the newly learned state is processed. This
> leads to routing churn especially if the majority of the state is the
> same which I am pretty sure is the case
>
> GR invalidates state due to the case of protocol error i.e A
> malformed update will invalidate all of the state. This is not the
> desired behavior.
>
> GR is not specific as to which events invoke it or not. From my read
> on the draft it is not clear if holdtime expiration invokes GR or
> not.. The draft is unclear.
>
> From: UTTARO, JAMES Sent: Tuesday, November 01, 2011 11:20 AM To:
> 'Enke Chen' Cc: [email-masked]; [email-masked] List Subject: RE:
> [Idr] draft-uttaro-idr-bgp-persistence-00
>
> Enke,
>
> Comments in-Line..
>
> Thanks, Jim Uttaro
>
> From: Enke Chen [mailto:[email-masked]] Sent: Friday, October 28,
> 2011 2:19 AM To: UTTARO, JAMES Cc: [email-masked]; [email-masked]
> List; Enke Chen Subject: Re: [Idr]
> GR is a solution that is essentially local in scope it does not have
> the ability to inform downstream speakers of the viability of routing
> state from the point of possible control plane failure. OTOH
> Persistence does propagate the condition of state. This provides
> distinct advantages in terms of customers awareness of the SPs
> control plane. One could imagine a customer receiving a STALE path
> and responding by selecting a backup. Some of the extensions to this
> draft that I have considered in colouring of STALE to inform if the
> condition arises from a local ( PE ) or internal iBGP ( RR )
> failures..
>
>
>
> candidates. Customers may want to immediately failover to the backup
> for some paths and not for others. GR is not capable of doing this it
> is all or nothing. The granularity is not sufficient. It needs to be
> at the path level. There may even be a case for having even more
> granularity i.e a per path timer.. GR is not capable of being
> extended for either of these cases.
>
> I am not sure how this path level persistence would work
> operationally.   Without the detailed information of a provider's
> network, how would a customer know what kind of failures and recovery
> that they might experience?   Consider the example of the
> simultaneous RR failures in the draft,  why would dn't any customer
> not to want to protect against such failures?   The end result could
> be that the PERSISTENCE flag is always set, thus losing its
> significance. [Jim U>] One ex would be customers who create multiple
> VPNs over different SPs.. A customer may want to take advantage of
> the knowledge that a control plane failure has occurred and migrate
> the traffic to the backup. This could be done at a path granularity
> by use of the DO_NOT_PERSIST CV. . We as SPs want to provide our
> Regarding the use of the STALE state vs ACTIVE state, clearly there
> is a tradeoff.   GR uses the stale routes in order to avoid
> forwarding churns, which has been a critical requirement for a long
> time.   If there is a real need for favoring a ACTIVE one over a
> STALE one in GR, it can be done by a simple knob. [Jim U>] The
> current draft has no ability to inform downstream speakers of whether
> or not a path is STALE or ACTIVE. The knob may be simple but a lot of
> machinery would have to be built. This is one of the big reasons for
> the PERSIST draft. I do not understand the routing churn part in the
> context of vpnv4, vpnv2, 3107 etc... maybe the GR solution was
> constructed as a solution that primarily speaks to eBGP IPV4
> connections for the IPV4 AF ( Internet ).. I could understand that..
>
>
> needs :-) [Jim U>] More Knobs..
>
>
>
>
>
>
>
> session. I believe that if this occurs the state will be invalidated.
> So for a session that is bouncing due to overload condition GR will
> not provide the required protection
>
> GR. [Jim U>] And yet more knobs
>
>
>
>
>
>
>
> invalidated first then the newly learned state is processed. This
> leads to routing churn especially if the majority of the state is the
> same which I am pretty sure is the case
>
> address family. [Jim U>] This does not address the lack of clarity
> about make before break.. it only states that must immediately remove
> routes marked as stale. It should state that any paths that are
> learned which are the same as the STALE paths should not force the
> forwarding plane to be re-programmed for those paths.. This should be
> made clear and in general is good practice to avoid churn..
>
>
> prematurely.   Further investigation will be needed in order to
> identify any possible implementation or config issues involved in
> your setup. [Jim U>] More moving parts to worry about..
>
>
>
>
>
>
>
> GR invalidates state due to the case of protocol error i.e A
> malformed update will invalidate all of the state. This is not the
> desired behavior.
>
>  [Jim U>] A few comments here.. I do not understand, the draft does
> not clarify that the only thing that will force a tear down is the
> cease subcode and a hard reset error code.. is the intention that
> this is the only thing that will tear it down? I guess I would like
> to see which things will and will not force a session termination in
> the original draft.. Like
>
>
> -          Holdtime Expiration
>
> -          Malformed Update
>
> -          Consecutive Restarts.. So what does this exactly mean
> "As part of this extension, possible consecutive restarts SHOULD NOT
> required by rules mentioned in [RFC4724]." Possible consecutive
> restarts means what? I really need clarity on this whole notion of
> when is a session truly invalidated.
>
> GR is not specific as to which events invoke it or not. From my read
> on the draft it is not clear if holdtime expiration invokes GR or
> not.. The draft is unclear.
>
> I think that it is covered by the above extension.  If not, it should
> be clarified. [Jim U>] I did not see it..
>
>
>
>
>
> [Jim U>] Yes.. Again from an operations perspective I have to deploy
> technology differently in different parts of the network across
> multiple vendors. This is generally not a desired starting point for
> the successful deployment of new technology.. I want solutions that
> are generic and simple to deploy.
>
>
>
>
>
>
>
> I think that you are talking about the "Restart time" field which has
> 12 bits and amount to about 68 minutes.  The "Restart time" is for
> the session re-establishment.  It does not impact the duration for
> holding stale routes after the session is re-established. [Jim U>]
> But if the session does not become re-established then the state is
> invalidated as the session terminates with an error code that GR will
> not persist through..
>
>
> continue to hold stale routes, it can be done by a simple knob. [Jim
> U>] And yet even more knobs
>
>
>
>
>
>
>
> that is the case. [Jim U>] If convergence is not achieved what is the
> behavior. I could not determine from the draft..
>
>
>
>
>
>
>
> will be able to address all other persistence requirements you
> listed, with some simple knobs and some implementation enhancements.
> [Jim U>] IMO GR was originally designed to prevent churn due to
> intermittent failure on an eBGP session for the IpV4 AF.. I do not
> want to have different knobs and implementation enhancements to solve
> the basics of persistence.. Regardless of that it does not inform the
> topology of the state of a path in re the control plane it was
> learned over so there can be no independent decisions about the value
> of a given path by different customers/providers.. This is required
> for my applications..
>
>
>
>
>
>
>
> Thanks,
>
> Jim Uttaro
>
> Thanks.   -- Enke
>
>
>
>
>
>
> Cc: [email-masked]<mailto:[email-masked]>;
> [email-masked]<mailto:[email-masked]>  List; Enke Chen
>
> -----Original Message----- From:
> [email-masked]<mailto:[email-masked]>
>
> Sunday, October 23, 2011 5:32 PM To:
> [email-masked]<mailto:[email-masked]>  List Subject: [Idr]
>
> +0200 From: Robert
> Raszuk<[email-masked]><mailto:[email-masked]>  Reply-To:
>
> [email-masked]<mailto:[email-masked]>  To:
> [email-masked]<mailto:[email-masked]>
> List<[email-masked]><mailto:[email-masked]>
>
>
>
> Hi,
>
>
>

---

From: <[email-masked]>
Date: Tue, 15 Nov 2011 09:01:24 +0100

Hi Robert,

>default on what paths should be called as "suspicious".

Yes, that's part of its original DNA: to locally conceal a BGP session
failure which is expected to restart soon (or has already restarted)

> However as

Ah! If your concern is about the vehicle used to convey the information
(to de-preference the route) I think we are pretty open to this. As a
matter of fact, as per a previous email sent to you and IDR, I said that
the use of local_pref could be envisaged within the AS. No new tools /
holes punched. And on eBGP, I proposed to use the STALE community: no
new attribute. If you're concerned about the community value being new,
I guess we could re-use the g-shut (graceful restart) community.

http://www.ietf.org/mail-archive/web/idr/current/msg05646.html

>Why do we need new way to signal this ?

Cf my above point: no new way required. Point closed (again).

>those issues have been fixed.

I somewhat disagree with the level of seriousness. As you did not
express comments, I take that the list of point to be addressed in -01
as listed in the slides this morning are fine for you.

Bruno

---

From: <[email-masked]>
Date: Tue, 15 Nov 2011 09:06:07 +0100

> If you're concerned about the community value being new,

I meant graceful shutdown. Sorry.

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Tue, 15 Nov 2011 14:11:10 +0000

Robert,

Comments In-Line..

Jim Uttaro

-----Original Message-----
From: Robert Raszuk [mailto:[email-masked]] 
Sent: Tuesday, November 15, 2011 2:10 AM
To: UTTARO, JAMES
Cc: 'Enke Chen'; [email-masked] List'
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-00

Hi Jim,

You are correct that GR does not allow to propagate information by 
default on what paths should be called as "suspicious". However as 
mentioned at the mic during the session we need to have a way to modify 
best path selection criteria universally as opposed to continue to 
invent new tools to solve point problems and punch holes in the various 
implementations of BGP best path selected
[Jim U>] So two distinct items here.. 1) GR cannot propagate information in re paths. From the draft there seems to be no way to do this for all paths or some subset of paths without major surgery. So, do you agree with this statement?  2) Punching holes?? So here are a few examples where holes have been punched RT-Constrain, AIGP, L2VPN, etc... So, are all of these going to be addressed? Will all further drafts that "tweak" best path selection be parked until this done.. The tools serve a real business/service/network need so cannot afford to wait..

Chairs in fact responded that we will work on this when the right time 
comes.
[Jim U>] That is fine.. In the meantime work will continue..

Back to the persistence draft ... GR can address 90% of the persistence 
draft requirements.
[Jim U>] Disagree.. See my prior note..
 Propagating information about "suspicious" paths if 
at all we all agree if this is a good idea can be done today with either 
setting lowest local pref, using cost community or marking with 
community so your ebgp peers can interpret it correctly.
[Jim U>] Need to be careful when we use customer tools i.e LP to accomplish provider goals. This type of approach may conflict with RFC1998.. Marking paths with CVs does not influence best path decision unless policy is applied at every point. This is not a workable solution in a large network

[Jim U>] ? See Above.

Also I think current -00 version has serious issues as already pointed 
on the list. We need to wait till -01 is published in order to see if 
those issues have been fixed.
[Jim U>] Disagree.. I do believe that the draft needs to evolve in how it deals with different use cases and the peculiarities of each.. IMO the purpose of IETF specifications is to provide tools for designers. We cannot take an approach that we will not develop tools as designers may not understand how to use them and could screw up their networks.. if this is the approach then we probably need to retire lots of technology that can be used incorrectly.. The argument that designers can screw up their network is specious at best..

Best regards,
R.

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 15 Nov 2011 16:54:45 +0100

Jim,

[Jim U>] So
> two distinct items here.. 1) GR cannot propagate information in re
> paths. From the draft there seems to be no way to do this for all
> paths or some subset of paths without major surgery. So, do you agree
> with this statement?

Yes I do. First we need to agree if such introduction of propagation of 
information is a feature or a bug.

I am personally completely not convinced that there is any value in 
informing my peers that one of my BGP sessions went down. You either 
have reachability to next hop and can attract traffic or you do not and 
if so you withdraw.

Telling peers that "I may be perhaps used to reach prefix X as last 
resort" is of highly questionable value.

Then as it has been said GR could be instrumented to upon session down 
event in case some/all best paths were on this session they could be re 
advertised with low local pref or with cost community. This is your own 
personal decision and I do not have anything against you configuring 
your network this way - if anything blackholed customers will seek new 
SP - so I am supportive my competitors to enable such knob.

2) Punching holes?? So here are a few examples
> where holes have been punched RT-Constrain, AIGP, L2VPN, etc...

I think you are mixing things ... RT-Constrain is a new SAFI and you can 
define for each new SAFI rules how best path is computed. Same for 
L2VPNs - the key is that you define it along the new SAFI definition.

AIGP as far as I know is in the IDR holding pattern till we figure out 
how to consistently modify best path selection with external influences.

If it comes after you are much better to use standard best path 
modification which guarantees best path correctness across your network.

> Back to the persistence draft ... GR can address 90% of the
> persistence draft requirements. [Jim U>] Disagree.. See my prior
> note..

So what else is missing ? Propagation part that path is "suspicious" and 
all routers in the given AFI/SAFI reachability in the Internet should 
sort of dislike it ? Is that it ?

I think it would be great for all to understand the delta. So if this is 
the delta I think in fact that Bruno who presented the draft is very 
flexible in this space.

Other co-authors I spoke with were not even aware some communities were 
added to the draft like STALE .....

  [Jim U>]
> Need to be careful when we use customer tools i.e LP to accomplish
> provider goals. This type of approach may conflict with RFC1998..

Local pref is AS tool - not customer tool. It is not propagated across 
AS boundaries.

Besides your STALE community already is said to overwrite any other 
preference so I see completely no substantial point of your above comment.

> Marking paths with CVs does not influence best path decision unless
> policy is applied at every point. This is not a workable solution in
> a large network

Local pref is applied at every router as far as I know. STALE will also 
not be magically enabled everywhere in the network till the network is 
upgraded and knob is configured. Is this workable in large network ???

> Also I think current -00 version has serious issues as already
> pointed on the list. We need to wait till -01 is published in order
> to see if those issues have been fixed. [Jim U>] Disagree.. I do
> believe that the draft needs to evolve in how it deals with different
> use cases and the peculiarities of each.. IMO the purpose of IETF
> specifications is to provide tools for designers. We cannot take an
> approach that we will not develop tools as designers may not
> understand how to use them and could screw up their networks.. if
> this is the approach then we probably need to retire lots of
> technology that can be used incorrectly.. The argument that designers
> can screw up their network is specious at best..

I think this proves you are missing the loop case I described in the 
other email. It is fine to provide rope and count that you use it wisely 
for climbing the cliff.

However it is IMHO wrong thing to provide a tool which has no way of 
working correctly ... in this particular case STALE CV across EBGP 
session - simply as you as operator are not aware about your peers 
topology and support of routing platform in their ASes.

Bruno offered to change this to G-SHUT which in turn is designed to be 
mapped to local-pref. I have no problem with this approach. So you can 
achieve exactly what you need safely.

To summarize .. I do recommend the same as Enke said at the mic .. It 
would be much better to turn this current document into requirements 
document so WG could provide opinions and proposals on how to solve them 
(provided they are valid) rather then propose a solution and keep 
arguing that this is the best one to solve your yet to be well defined 
problem.

Best,
R.

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Tue, 15 Nov 2011 16:48:23 +0000

Robert 

Comments In-line..

Jim Uttaro

-----Original Message-----
From: Robert Raszuk [mailto:[email-masked]] 
Sent: Tuesday, November 15, 2011 10:55 AM
To: UTTARO, JAMES
Cc: 'Enke Chen'; [email-masked] List'
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-00

Jim,

[Jim U>] So

Yes I do. First we need to agree if such introduction of propagation of 
information is a feature or a bug.
[Jim U>] Ok.. Then you should revisit GR not propagating the failure and determine if it is a bug or a feature..

I am personally completely not convinced that there is any value in 
informing my peers that one of my BGP sessions went down. You either 
have reachability to next hop and can attract traffic or you do not and 
if so you withdraw.
[Jim U>] Ok so GR is still propagating some path from a BGP session that went down or is compromised.. So what is the point here? Ok for GR but not ok for Persistence??

Telling peers that "I may be perhaps used to reach prefix X as last 
resort" is of highly questionable value.
[Jim U>] As opposed to not telling peers that X, that may be of questionable value is still the best path? Just because you are not literally informing your peers does not mean that you are still essentially maintaining an X of questionable value throughout the routing context. The difference between the two is GR hides this information Persistence does not. GR is opaque, Persistence is transparent.. It seems arrogant to hide the information from your customers especially when it is their VPN routing context..

I think you are mixing things ... RT-Constrain is a new SAFI and you can 
define for each new SAFI rules how best path is computed.
[Jim U>] The point is that RT-Constrain modifies the best path selection by essentially punching holes.. So I guess your point is it is ok for this AFI/SAFI but not other.. From the RT-Constrain draft:

     " ii.  When advertising a RT membership NLRI to a non client peer,
      if the best path as selected by path selection procedure described
      in section 9.1 of the base BGP specification [5] is a route
      received from a non-client peer, and there is an alternative path
      to the same destination from a client, the attributes of the
      client path are advertised to the peer. "

This seems like a punched hole.. NO? 

 Same for 

[Jim U>] Really..

If it comes after you are much better to use standard best path 
modification which guarantees best path correctness across your network.
[Jim U>] so, I guess your point is if Best Path changes are required by some drafts deemed by folks in the know then they are warranted. I think a consistent message is needed if you want to use Best path "Hole Punching" as a rationale..

So what else is missing ? Propagation part that path is "suspicious" and 
all routers in the given AFI/SAFI reachability in the Internet should 
sort of dislike it ? Is that it ?
[Jim U>] Internet?? I think that you need to realize that BGP is being used for many different applications.. So, GR makes a decision in the internet that a path is still valid even though a session went down. This is deterministic in that a decision has been made to not inform, Persistence allows a customer to choose. There is a lot of value in this from a customer point of view..

Local pref is AS tool - not customer tool. It is not propagated across 
AS boundaries.
[Jim U>] Do not agree.. It has nothing to do with propagation across AS Boundaries.. VPNV4 routing context belongs to the customer.. They use LP to accomplish their routing designs in our network. RFC1998 is the agreed upon method. 

Local pref is applied at every router as far as I know. STALE will also 
not be magically enabled everywhere in the network till the network is 
upgraded and knob is configured. Is this workable in large network ???
[Jim U>] Knob?? STALE is appended to a path when the session is was learned over has been compromised, that path is then advertised.. Part of the implementation..

I think this proves you are missing the loop case I described in the 
other email. It is fine to provide rope and count that you use it wisely 
for climbing the cliff.
[Jim U>] Your example although contrived does illustrate a case where a customer who does not understand the technology implements incorrectly.. Again this can happen with many of the technologies as I mentioned before.. i.e RT-Constrain, Addpath, etc.. So let's either be consistent in how much rope is allotted for different technical enhancements or do away with it completely.. We cannot have some drafts get lots of rope and other get some twine..

[Jim U>] I do not understand.. Why does it have no way of working correctly? Because someone implemented it incorrectly.. More importantly I want the customers to know about the state of their VPN when it transits the routing context they have purchased in our network..

Bruno offered to change this to G-SHUT which in turn is designed to be 
mapped to local-pref. I have no problem with this approach. So you can 
achieve exactly what you need safely.
[Jim U>] We will consider all approaches..

[Jim U>] Do not agree.. The assumption that GR can be manipulated to accomplish the requirements is incorrect.. It is not right to select a solution that has not before considered the requirements and then reverse engineer a bottom up solution in attempt to patch it.. 

Best,
R.

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 16 Nov 2011 03:03:22 +0100

>[Jim U>] Ok so GR is still propagating some
> path from a BGP session that went down or is compromised.. So what is
> the point here? Ok for GR but not ok for Persistence??

GR is not propagating anything .. it is just not withdrawing nor 
re-advertising it for GR timer which is much much shorter anyway then 
what persistent draft calls as a examples of timer.

Side note:

If the BGP is used for pushing configuration information which are 
stable and should stay in the network till operator decides to remove it 
- regardless of session state they were distributed by - I have no 
problem with that.

However I am of the opinion that to accomplish that you should not just 
stuck new community and re-advertise when the session goes down. You 
should up front advertise such routes with either persistent community 
or perhaps new SAFI capability flag.

Thx,
R.

---

From: Russ White <[email-masked]>
Date: Tue, 15 Nov 2011 22:21:26 -0500

> resort" is of highly questionable value.

I would go farther --this isn't questionable, it's really bad. If you
have a route you know exists, but you don't want people to use, set
things so it's a "last resort" (wait for BGP, wait for LDP, etc). If you
don't know whether or not you really have reachability, don't advertise it.

The possible problems involved in saying, "I might have a route to x,
just in case you don't have any other path there," will end up being
really, really ugly.

:-)

Russ

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Wed, 16 Nov 2011 12:26:50 +0000

Robert,

Comments In-Line..

Jim Uttaro

-----Original Message-----
From: Robert Raszuk [mailto:[email-masked]] 
Sent: Tuesday, November 15, 2011 9:03 PM
To: UTTARO, JAMES
Cc: 'Enke Chen'; [email-masked] List'
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-00

GR is not propagating anything .. it is just not withdrawing nor 
re-advertising it for GR timer which is much much shorter anyway then 
what persistent draft calls as a examples of timer.
[Jim U>] Ok.. let me be clearer.. Due to the fact that GR is "not" withdrawing a path that was learned over a session which may be compromised that path is still valid in the routing topology.. The difference between GR and Persistence is that GR does not provide the ability to inform the topology that the path is of questionable value.. 

Side note:

[Jim U>] Read the Draft... The PERSIST community is attached up front.. In fact that identification is used when the session goes down, at the point the STALE CV is set and paths re-advertised..

"2.1. PERSIST

   This memo defines a new transitive BGP community, PERSIST, with value
   TBD (to be assigned by IANA).  Attaching of the PERSIST community
   SHOULD be controlled by configuration.  Attaching the PERSIST
   community indicates that the peer should maintain forwarding in the
   case of a session failure.  The functionality SHOULD default to being
   disabled.
"

Thx,
R.

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 16 Nov 2011 17:40:27 +0100

Jim,

> The difference between
> GR and Persistence is that GR does not provide the ability to inform
> the topology that the path is of questionable value..

You are apparently missing many pieces ...

GR maintains the routes .. but GR does not overwrite NHT (Next Hop 
Tracking) ... that means that path's next hop is still valid. Persistent 
draft however re-advertises routes as STALE without per spec any 
considerations of the next hop liveness. That is bad !

>[Jim U>] Read the
> Draft... The PERSIST community is attached up front.. In fact that
> identification is used when the session goes down, at the point the
> STALE CV is set and paths re-advertised..

You are persistently missing the point. My point was to mark the paths 
as persistent up front meaning keep them regardless of session state 
which they were received on. No need for any stale cv at all.

As I and others already explicitly said on the list re-advertising the 
paths with a marker you may use me as last resort is just a terrible idea.

Best regards,
R.

---

From: <[email-masked]>
Date: Tue, 22 Nov 2011 15:48:23 +0100

Russ,

>From: Russ White, Sent: Wednesday, November 16, 2011 4:21 AM
>

As you don't comment on Graceful Restart (GR), I assume that you are
fine with GR.
Are you?

Then I think we need to make a distinction between the information we
have on a route, and the routing decision we make based on those
information.

a) failure assumption:
let's assume a PE can lose both its iBGP session toward its redundant RR
Note: the assumption is the same for GR and persistence.

b) information available:
Once the BGP session(s) are down, we have the following information
available on the BGP router: cause of failure, route tags, time (t)
elapsed since the session failure, local configuration/preference
Note: idem for GR and persistence.

At that point, I would say that once t>0, we have no certainty on the
validity of the route (from the BGP peer/BGP Next-Hop and from
downstream BGP routers). And the longer the duration, the less
certainty.
Note: for a given "t", this is the same for GR and persistence. Hence I
don't get how you claim that "it's really bad" for persistence while
it's ok for GR.

c) routing decision:
Given the above information and level of uncertainty, at a given "t"
time, currently 2 routing decisions are possible:
- withdraw the route.
	- This is the regular BGP behavior. 
	- The reasoning is that the level of uncertainty is too high to
use that route.
- keep the route.
	- This is the BGP Graceful Restart behavior. 
	- The reasoning is that the level of uncertainty is low enough
so that the route can still be considered perfectly valid.

But the router making the above decision is the BGP advertising router
hence the egress router. It's lacking some information only available on
the ingress router. Namely the availability of alternate paths/Next Hop
on the ingress. I call that this is an important information to make the
routing decision. Because in the end, each router/AS tries to take the
_best_ decision among _available_ options. This is not a binary decision
between "right/good" and "bad". So compared to GR, BGP persistence
proposes to give some additional information and routing decision on
upstream ingress BGP routers. (e.g. using "STALE" community, low BGP
local pref. But other vehicles could be considered).

Could you elaborate on why this would be "really bad"?

Thanks,
Regards,
Bruno

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 22 Nov 2011 19:06:13 +0100

Hi Bruno,

Great comparison between GR and persistent !

However there is significant two points which need to be highlighted ...

1.

- time t for GR is max expressed as 12 bits in seconds which is 68 min 
... if session does not come up for that long I think it is wise to 
clean up

- time t for persistence is indicated in the draft to be in case of 
L2VPN: "The persist-timer should be set to a large value on the order of 
days to infinity." In case of L3VPN: "The persist-timer should be set to 
a large value on the order of hours to a few days."

2.

In both GR and persistance the key to using the path on BGP speakers in 
the data plane is in next hop liveness detection. Be it in scope of the 
draft or out of scope does not really matter. Ingress as you call it 
should not care if RR says this path is suspicious if he can reach just 
fine the next hop of the prefix.

So this is not that we like GR and we do not like persistence. All we 
are saying is that the only real delta between GR and persistance is 
this new signalling informing that path is "perhaps broken". This is 
this "perhaps" which many folks have a problem with.

Best regards,
R.

---

From: <[email-masked]>
Date: Wed, 23 Nov 2011 12:18:31 +0100

Hi Robert,

>a large value on the order of hours to a few days."

If you believe 68 minutes is the right max value,
- IMHO Max timer duration should be AS and application specific. There
is no right value. That's also the position taken by GR. Especially as
some routes may be rather static.
- Where does 68 minutes come from? Why would this be the right value?
Why are you more restrictive than the BGP RFC (Hold Time)?
- for consistency, I guess you will also argue against defining a long
timer value in the GR respin draft as has been proposed during the
meeting.

>fine the next hop of the prefix.

No.
BGP Next-Hop liveliness is important and need to be checked. But this
only one part of the route validity problem as this only checks the BGP
Next-Hop /ASBR. Any downstream BGP routers could withdraw the route.
You/egress/ingress would not know as the BGP session is down. Hence it's
important to mark this path (as "stale" or "not refreshed" if you
prefer) and ingress should care.

>>> don't know whether or not you really have reachability, don't
advertise
>> it.