Subject: [Idr] draft-uttaro-idr-bgp-persistence-00

From: Robert Raszuk <[email-masked]>
Date: Sat, 22 Oct 2011 00:23:55 +0200

Hi,

I have read the draft and have one question and one observation.

Question:

What is the point of defining DO_NOT_PERSIST community ? In other words 
why not having PERSIST community set would not mean the same as having 
path marked with DO_NOT_PERSIST.

Observation:

I found the below statement in section 4.2:

    o  Forwarding must ensure that the Next Hop to a "stale" route is
       viable.

Of course I agree. But since we stating obvious in the forwarding 
section I think it would be good to explicitly also state this in the 
best path selection that next hop to STALE best path must be valid.

However sessions especially those between loopbacks do not go down for 
no reason. Most likely there is network problem which may have caused 
those sessions to go down. It is therefor likely that LDP session went 
also down between any of the LSRs in the data path and that in spite of 
having the paths in BGP and next hops in IGP the LSP required for both 
quoted L2/L3VPN applications is broken. That may particularly happen 
when network chooses to use independent control mode for label allocation.

I would suggest to at least add the recommendation statement to the 
document that during best path selection especially for stale paths a 
validity of required forwarding paradigm to next hop of stale paths 
should be verified.

For example using techniques as described in: 
draft-ietf-idr-bgp-bestpath-selection-criteria

Best regards,
R.

---

From: Robert Raszuk <[email-masked]>
Date: Sun, 23 Oct 2011 23:32:20 +0200

Authors,

Actually when discussing this draft a new concern surfaced which I would 
like to get your answer on.

The draft in section 4.2 says as one of the forwarding rules:

    o  Forwarding to a "stale" route is only used if there are no other
       paths available to that route.  In other words an active path
       always wins regardless of path selection.  "Stale" state is always
       considered to be less preferred when compared with an active path.

In the light of the above rule let's consider a very simple case of dual 
PE attached site of L3VPN service. Two CEs would inject into their IBGP 
mesh routes to the remote destination: one marked as STALE and  one not 
marked at all. (Each CE is connected to different PE and each PE RT 
imports only a single route to a remote hub headquarter to support 
geographic load balancing).

Let me illustrate:

  VPN Customer HUB

    PE3      PE4
         SP
    PE1      PE2
     |        |
     |        |
    CE1      CE2
     |        |
    1|        |10
     |        |
    R1 ------ R2
          1

CE1,CE2,R1,R2 are in IBGP mesh. IGP metric of CE1-R1 and R1-R2 are 1 and 
R2-CE2 is 10.

Prefix X is advertised by remote hub in the given VPN such that PE1 vrf 
towards CE1 only has X via PE3 and PE2's vrf towards CE2 only has X via PE4.

Let's assume EBGP sessions PE3 to HUB went down, but ethernet link is 
up, next hop is in the RIB while data plane is gone. Assume no data 
plane real validation too. /* That is why in my former message I 
suggested that data plane validation would be necessary */.

R1 has X via PE1/S (stale) and X via PE2/A (active) - it understands 
STALE so selects in his forwarding table path via CE2.

R2 has X via PE1/S (stale) and X via PE2/A (active) - it does not 
understand STALE, never was upgraded to support the forwarding rule 
stated above in the draft and chooses X via CE1 (NH metric 2 vs 10).

R1--R2 produce data plane loop as long as STALE paths are present in the 
system. Quite fun to troubleshoot too as the issue of PE3 injecting such 
STALE paths may be on the opposite site of the world.

The issue occurs when some routers within the customer site will be able 
to recognize STALE transitive community and prefer non stale paths in 
their forwarding planes (or BGP planes for that matter) while others 
will not as well as when both stale and non stale paths will be present.

Question 1: How do you prevent forwarding loop in such case ?

Question 2: How do you prevent forwarding loop in the case when customer 
would have backup connectivity to his sites or connectivity via 
different VPN provider yet routers in his site only partially understand 
the STALE community and only partially follow the forwarding rules ?

In general as the rule is about mandating some particular order of path 
forwarding selection what is the mechanism in distributed systems like 
today's routing to be able to achieve any assurance that such rule is 
active and enforced across _all_ routers behind EBGP PE-CE L3VPN 
boundaries in customer sites ?

Best regards,
R.

-------- Original Message --------
Subject: [Idr] draft-uttaro-idr-bgp-persistence-00
Date: Sat, 22 Oct 2011 00:23:55 +0200
From: Robert Raszuk <[email-masked]>
Reply-To: [email-masked]
To: [email-masked] List <[email-masked]>

Hi,

Question:

Observation:

Best regards,
R.

_______________________________________________
Idr mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/idr

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Mon, 24 Oct 2011 15:13:00 +0000

Robert,

Comments In-Line

Thanks,
	Jim Uttaro

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Robert Raszuk
Sent: Friday, October 21, 2011 6:24 PM
To: [email-masked] List
Subject: [Idr] draft-uttaro-idr-bgp-persistence-00

Hi,

Question:

Observation:

[Jim U>] Why do you believe that? BGP can and does fail independently of the underlying LDP session.. I do believe that viability of a data path between two endpoints should be taken into account when validating paths to NHs but that effort is clearly a much bigger effort.. This draft would take advantage of this mechanism but I am not looking to define that in this draft..

I would suggest to at least add the recommendation statement to the 
document that during best path selection especially for stale paths a 
validity of required forwarding paradigm to next hop of stale paths 
should be verified.
[Jim U>] Why would this not be something that needs to be done as a general rule for paths.. I mean the underlying LSP could fail independently of BGP.. If this needs to be done in should be done as a general rule...

Best regards,
R.

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 25 Oct 2011 08:15:40 +0200

Hi Jim,

> [Jim U>] Why do you believe that? BGP can
> and does fail independently of the underlying LDP session.. I do
> believe that viability of a data path between two endpoints should be
> taken into account when validating paths to NHs but that effort is
> clearly a much bigger effort.. This draft would take advantage of
> this mechanism but I am not looking to define that in this draft..

In my opinion  there is a need to recognize the difference of possibly 
locally using a stale path (and that is btw exactly what GR spec does .. 
except here it would be GR per prefix with longer timer) versus keep 
advertising paths which may be invalid.

> document that during best path selection especially for stale paths
> a validity of required forwarding paradigm to next hop of stale
> paths should be verified. [Jim U>] Why would this not be something
> that needs to be done as a general rule for paths.. I mean the
> underlying LSP could fail independently of BGP.. If this needs to be
> done in should be done as a general rule...

Respectfully disagree.

I am sure you very well know how many customers are relaying on using 
BGP keepalives to determine PE-CE link up and PE-CE reachability. Recall 
how many times SP customers including your own customers were asking to 
reduce holdtime on their ebgp session to 3/9 ... those customers just do 
not have a better way to determine VPN site reachability.

Of course such reduction of keepalives is a bad idea (even if their 
processing is offloaded or prioritized on main RP) so BFD was invented 
to address this issue. And either BFD if present or EBGP session up 
state was _the_only_ probe to determine if CE over frame relay PVC or 
multiaccess media is live. That is the key difference.

So when you drop session you remove prefixes advertised from the site 
and backup can be activated. Now you are proposing to keep advertising 
them as STALE which really means MAYBE_REACHABLE. I am not sure why 
validation of next hop reachability by BFD or even OBJECT_TRACKING or 
SCRIPT would be at all difficult.

Anyway I think this proposal as pointed out in the other email has much 
bigger deployment issue so I think there is not much point on this 
validation debate at this point before the loop problem is solved due to 
inconsistent forwarding plane decisions.

Best regards,
R.

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Wed, 26 Oct 2011 16:44:33 +0000

Robert,

	The introduction of this technology needs to be carefully evaluated when being deployed into the network. Your example clearly calls out how a series of independent design can culminate in incorrect behavior. Certainly the deployment of persistence on a router that has interaction with a router that does not needs to be clearly understood by the network designer. The goal of this draft is to provide a fairly sophisticated tool that will protect the majority of customers in the event of a catastrophic failure.. The premise being the perfect is not the enemy of the good.. I will add text in the deployment considerations section to better articulate that..

Thanks,
	Jim Uttaro

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Robert Raszuk
Sent: Sunday, October 23, 2011 5:32 PM
To: [email-masked] List
Subject: [Idr] draft-uttaro-idr-bgp-persistence-00

Authors,

Let me illustrate:

  VPN Customer HUB

    PE3      PE4
         SP
    PE1      PE2
     |        |
     |        |
    CE1      CE2
     |        |
    1|        |10
     |        |
    R1 ------ R2
          1

Best regards,
R.

-------- Original Message --------
Subject: [Idr] draft-uttaro-idr-bgp-persistence-00
Date: Sat, 22 Oct 2011 00:23:55 +0200
From: Robert Raszuk <[email-masked]>
Reply-To: [email-masked]
To: [email-masked] List <[email-masked]>

Hi,

Question:

Observation:

Best regards,
R.

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 26 Oct 2011 19:24:36 +0200

Jim,

When one during design phase of a routing protocol or routing protocol 
extension or modification to it already realizes that enabling such 
feature may cause real network issue if not done carefully - that should 
trigger the alarm to rethink the solution and explore alternative 
approaches to the problem space.

We as operators have already hard time to relate enabling a feature 
within our intradomain boundaries to make sure such rollout is network 
wide. Here you are asking for the same level of awareness across ebgp 
boundaries. This is practically unrealistic IMHO.

Back to the proposal ... I think that if anything needs to be done is to 
employ per prefix GR with longer and locally configurable timer. That 
would address information persistence across direct IBGP sessions.

On the RRs use case of this draft we may perhaps agree to disagree, but 
I do not see large enough probability of correctly engineered RR plane 
to experience simultaneous multiple ibgp session drops. If that happens 
the RR placement, platforms or deployment model should be re-engineered.

Summary .. I do not think that IDR WG should adopt this document. Just 
adding a warning to the deployment section is not sufficient.

Best regards,
R.

> Robert,
>
> The introduction of this technology needs to be carefully evaluated
> when being deployed into the network. Your example clearly calls out
> how a series of independent design can culminate in incorrect
> behavior. Certainly the deployment of persistence on a router that
> has interaction with a router that does not needs to be clearly
> understood by the network designer. The goal of this draft is to
> provide a fairly sophisticated tool that will protect the majority of
> customers in the event of a catastrophic failure.. The premise being
> the perfect is not the enemy of the good.. I will add text in the
> deployment considerations section to better articulate that..
>
> Thanks, Jim Uttaro
>
> -----Original Message----- From: [email-masked]
> [mailto:[email-masked]] On Behalf Of Robert Raszuk Sent:
> Sunday, October 23, 2011 5:32 PM To: [email-masked] List Subject: [Idr]
> draft-uttaro-idr-bgp-persistence-00
>
> Authors,
>
> Actually when discussing this draft a new concern surfaced which I
> would like to get your answer on.
>
> paths available to that route.  In other words an active path always
> wins regardless of path selection.  "Stale" state is always
> In the light of the above rule let's consider a very simple case of
> dual PE attached site of L3VPN service. Two CEs would inject into
> their IBGP mesh routes to the remote destination: one marked as STALE
> and  one not marked at all. (Each CE is connected to different PE and
> each PE RT imports only a single route to a remote hub headquarter to
> support geographic load balancing).
>
> Let me illustrate:
>
> VPN Customer HUB
>
> PE3      PE4 SP PE1      PE2 |        | |        | CE1      CE2 |
> | 1|        |10 |        | R1 ------ R2 1
>
> CE1,CE2,R1,R2 are in IBGP mesh. IGP metric of CE1-R1 and R1-R2 are 1
> and R2-CE2 is 10.
>
> Prefix X is advertised by remote hub in the given VPN such that PE1
> vrf towards CE1 only has X via PE3 and PE2's vrf towards CE2 only has
> X via PE4.
>
> Let's assume EBGP sessions PE3 to HUB went down, but ethernet link
> is up, next hop is in the RIB while data plane is gone. Assume no
> data plane real validation too. /* That is why in my former message
> I suggested that data plane validation would be necessary */.
>
> R1--R2 produce data plane loop as long as STALE paths are present in
> the system. Quite fun to troubleshoot too as the issue of PE3
> injecting such STALE paths may be on the opposite site of the world.
>
> The issue occurs when some routers within the customer site will be
> able to recognize STALE transitive community and prefer non stale
> paths in their forwarding planes (or BGP planes for that matter)
> while others will not as well as when both stale and non stale paths
> will be present.
>
> Question 2: How do you prevent forwarding loop in the case when
> customer would have backup connectivity to his sites or connectivity
> via different VPN provider yet routers in his site only partially
> understand the STALE community and only partially follow the
> forwarding rules ?
>
> In general as the rule is about mandating some particular order of
> path forwarding selection what is the mechanism in distributed
> systems like today's routing to be able to achieve any assurance that
> such rule is active and enforced across _all_ routers behind EBGP
> PE-CE L3VPN boundaries in customer sites ?
>
> Best regards, R.
>
>
> -------- Original Message -------- Subject: [Idr]
> draft-uttaro-idr-bgp-persistence-00 Date: Sat, 22 Oct 2011 00:23:55
> +0200 From: Robert Raszuk<[email-masked]> Reply-To:
> [email-masked] To: [email-masked] List<[email-masked]>
>
> Hi,
>
> What is the point of defining DO_NOT_PERSIST community ? In other
> words why not having PERSIST community set would not mean the same as
> having path marked with DO_NOT_PERSIST.
>
> Observation:
>
> section I think it would be good to explicitly also state this in
> the best path selection that next hop to STALE best path must be
> valid.
>
> However sessions especially those between loopbacks do not go down
> for no reason. Most likely there is network problem which may have
> caused those sessions to go down. It is therefor likely that LDP
> session went also down between any of the LSRs in the data path and
> that in spite of having the paths in BGP and next hops in IGP the LSP
> required for both quoted L2/L3VPN applications is broken. That may
> particularly happen when network chooses to use independent control
> mode for label allocation.
>
> paths should be verified.
>
> Best regards, R.
> _______________________________________________ Idr mailing list
> [email-masked] https://www.ietf.org/mailman/listinfo/idr
>
>
> _______________________________________________ Idr mailing list
> [email-masked] https://www.ietf.org/mailman/listinfo/idr
>
>

---

From: Enke Chen <[email-masked]>
Date: Wed, 26 Oct 2011 17:43:08 -0700

Hi, folks:

I have a hard time in understanding what new problems (beyond the GR) 
the draft try to solve :-(

If the concern is about the simultaneous RR failure as shown in the 
examples in Sect. 6 Application, that can be addressed easily using GR.  
As the RRs are not in the forwarding path, it means that the forwarding 
is not impacted (thus is preserved) during the restart of a RR.   The 
Forwarding State bit (F) in the GR capability should always be set by 
the RR when it is not in the forwarding path.

Also in the case of simultaneous RR failure, I do not see why one would 
want to retain some routes, but not others, using the communities 
specified in the draft.  As the RRs are not in the forwarding path, 
wouldn't be better to retain all the routes on a PE/client?

As you might be aware, efforts have been underway to address issues with 
GR found during implementation and deployment. They include the spec 
respin, notification handling, and implementations.  If there are issues 
in the GR area that are not adequately addressed,  I suggest that we try 
to address them in the GR respin if possible, instead of creating 
another variation unnecessarily.

Thanks.   -- Enke

On 10/26/11 10:24 AM, Robert Raszuk wrote:
> feature may cause real network issue if not done carefully - that 
> should trigger the alarm to rethink the solution and explore 
> alternative approaches to the problem space.
>
> Back to the proposal ... I think that if anything needs to be done is 
> to employ per prefix GR with longer and locally configurable timer. 
> That would address information persistence across direct IBGP sessions.
>
> On the RRs use case of this draft we may perhaps agree to disagree, 
> but I do not see large enough probability of correctly engineered RR 
> plane to experience simultaneous multiple ibgp session drops. If that 
> happens the RR placement, platforms or deployment model should be 
> re-engineered.
>

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Thu, 27 Oct 2011 20:17:23 +0000

Enke,

GR is a solution that is essentially local in scope it does not have the ability to inform downstream speakers of the viability of routing state from the point of possible control plane failure. OTOH Persistence does propagate the condition of state. This provides distinct advantages in terms of customers awareness of the SPs control plane. One could imagine a customer receiving a STALE path and responding by selecting a backup. Some of the extensions to this draft that I have considered in colouring of STALE to inform if the condition arises from a local ( PE ) or internal iBGP ( RR ) failures.. 

GR makes no distinction from STALE state and ACTIVE state.. This can lead to the STALE path still being preferred throughout the topology. IMO this is incorrect behavior regardless of the comparison. 

PERSISTENCE allows for a customer to indicate which paths should be candidates. Customers may want to immediately failover to the backup for some paths and not for others. GR is not capable of doing this it is all or nothing. The granularity is not sufficient. It needs to be at the path level. There may even be a case for having even more granularity i.e a per path timer.. GR is not capable of being extended for either of these cases.

GR does not provide protection through successive restarts of the session. I believe that if this occurs the state will be invalidated. So for a session that is bouncing due to overload condition GR will not provide the required protection

GR does not employ a make before break strategy. All state is invalidated first then the newly learned state is processed. This leads to routing churn especially if the majority of the state is the same which I am pretty sure is the case

GR invalidates state due to the case of protocol error i.e A malformed update will invalidate all of the state. This is not the desired behavior.

GR is not specific as to which events invoke it or not. From my read on the draft it is not clear if holdtime expiration invokes GR or not.. The draft is unclear.

It is not clear to me how RRs and PEs differ in using GR. 

The time that state can persist is limit to about 1 hour max. 

GR does detail the behavior where convergence is not achieved between restarts.. Similar to above..  

I do not believe that the current GR paradigm can be extended to cover the majority of the cases above.

Thanks,
	Jim Uttaro

-----Original Message-----
From: Enke Chen [mailto:[email-masked]] 
Sent: Wednesday, October 26, 2011 8:43 PM
To: UTTARO, JAMES
Cc: [email-masked]; [email-masked] List; Enke Chen
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-00

Hi, folks:

Thanks.   -- Enke

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Thu, 27 Oct 2011 20:31:24 +0000

Robert,

	To be honest I do not agree and am confused by your comments.. There are many specifications that demand correct deployment in a network to prevent unwanted behavior. Some immediate examples come to mind:

L3VPN. If the operator mis-configures RTs or stitches incorrectly VPN pollution can easily occur. There is no protection that I know of in the draft to prevent this.

RT Constrain. If incorrectly deployed in a network with devices that do and don't support the VPN distribution graph will be incorrectly built. This will result in customers VPN being compromised. The draft does not prevent this.

It is the responsibility of the network designers to determine how to deploy powerful technology into their networks. Part of this is to ensure that we do not create a "real network issue if not done carefully ". We do not have the luxury of not being careful. Generally speaking when introducing technology with new capability the expectation is that the network architects/designers understand the technology and deploy it correctly. 

Thanks,
	Jim Uttaro

-----Original Message-----
From: Robert Raszuk [mailto:[email-masked]] 
Sent: Wednesday, October 26, 2011 1:25 PM
To: UTTARO, JAMES
Cc: [email-masked] List
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-00

Jim,

Best regards,
R.

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 27 Oct 2011 23:49:30 +0200

Hello Jim,

L3VPN RT assignment is how you define a VPN policy. Of course if you 
misconfigure policy the system will take it and execute accordingly. 
That is in fact a L3VPN drawback not something to be proud of. Customers 
subscribing to L3VPN service must trust provider how their routes are 
distributed by SP and to whom else are exported. I do know for a fact 
that some customers just do not trust SP to handle their route 
distribution. Those customers either ask for L2VPN service or purchase 
their transit links directly.

RT Constrain - I do not see how this can cause wrong construction of VPN 
distribution graph when deployed partially. Basically PEs which do not 
support RT-Constrain will get full set of VPNv4 routes and RRs which 
such PEs are connected to will need to keep all of them. I do not see 
any issue if you just enable RT constrain on one PE and push RT routes 
from such PE to RR(s). Please illustrate a scenario where partially 
deployed RT Constrain results in "VPN distribution graph will be 
incorrectly built". Maybe we have missed something ...

 > It is the responsibility of the network designers to determine how to
 > deploy powerful technology into their networks.

I could perhaps agree with that statement in general. However when you 
are injecting STALE routes to your EBGP peers you have zero control over 
their network.

Best regards,
R.

> Robert,
>
> To be honest I do not agree and am confused by your comments.. There
> are many specifications that demand correct deployment in a network
> to prevent unwanted behavior. Some immediate examples come to mind:
>
> L3VPN. If the operator mis-configures RTs or stitches incorrectly VPN
> pollution can easily occur. There is no protection that I know of in
> the draft to prevent this.
>
> RT Constrain. If incorrectly deployed in a network with devices that
> do and don't support the VPN distribution graph will be incorrectly
> built. This will result in customers VPN being compromised. The draft
> does not prevent this.
>
> It is the responsibility of the network designers to determine how to
> deploy powerful technology into their networks. Part of this is to
> ensure that we do not create a "real network issue if not done
> carefully ". We do not have the luxury of not being careful.
> Generally speaking when introducing technology with new capability
> the expectation is that the network architects/designers understand
> the technology and deploy it correctly.
>
> Thanks, Jim Uttaro
>
> -----Original Message----- From: Robert Raszuk
> [mailto:[email-masked]] Sent: Wednesday, October 26, 2011 1:25 PM
> To: UTTARO, JAMES Cc: [email-masked] List Subject: Re: [Idr]
> When one during design phase of a routing protocol or routing
> protocol extension or modification to it already realizes that
> enabling such feature may cause real network issue if not done
> carefully - that should trigger the alarm to rethink the solution and
> explore alternative approaches to the problem space.
>
> within our intradomain boundaries to make sure such rollout is
> network wide. Here you are asking for the same level of awareness
> across ebgp boundaries. This is practically unrealistic IMHO.
>
> Summary .. I do not think that IDR WG should adopt this document.
> Just adding a warning to the deployment section is not sufficient.
>
> Best regards, R.

---

From: Enke Chen <[email-masked]>
Date: Thu, 27 Oct 2011 23:19:24 -0700

Jim,

My comments are inlined.

> PERSISTENCE allows for a customer to indicate which paths should be candidates. Customers may want to immediately failover to the backup for some paths and not for others. GR is not capable of doin…

I am not sure how this path level persistence would work 
operationally.   Without the detailed information of a provider's 
network, how would a customer know what kind of failures and recovery 
that they might experience?   Consider the example of the simultaneous 
RR failures in the draft,  why wouldn't any customer not to want to 
protect against such failures?   The end result could be that the 
PERSISTENCE flag is always set, thus losing its significance.

Regarding the use of the STALE state vs ACTIVE state, clearly there is a 
tradeoff.   GR uses the stale routes in order to avoid forwarding 
churns, which has been a critical requirement for a long time.   If 
there is a real need for favoring a ACTIVE one over a STALE one in GR, 
it can be done by a simple knob.

As you know, BGP is full of knobs that adjust behaviors for different 
needs :-)

> GR does not provide protection through successive restarts of the session. I believe that if this occurs the state will be invalidated. So for a session that is bouncing due to overload condition G…

This can be addressed by a simple knob to set the min stale timer for GR.

> GR does not employ a make before break strategy. All state is invalidated first then the newly learned state is processed. This leads to routing churn especially if the majority of the state is the…

Such behavior would be an implementation bug that needs to be fixed.  
But it is not an issue with the protocol itself.

This is what we have in 4.2. Procedures for the Receiving Speaker, RFC 4724:

---

    The Receiving Speaker MUST replace the stale routes by the routing
    updates received from the peer.  Once the End-of-RIB marker for an
    address family is received from the peer, it MUST immediately remove
    any routes from the peer that are still marked as stale for that
    address family.

---

There are several possibilities for the premature purge of the stale 
routes. For example, the "Forwarding state" flag was somehow not set 
after the session was re-established, or the the EOR was sent 
prematurely.   Further investigation will be needed in order to identify 
any possible implementation or config issues involved in your setup.

> GR invalidates state due to the case of protocol error i.e A malformed update will invalidate all of the state. This is not the desired behavior.

It has been addressed by the following extension:

http://datatracker.ietf.org/doc/draft-keyupate-idr-bgp-gr-notification/

> GR is not specific as to which events invoke it or not. From my read on the draft it is not clear if holdtime expiration invokes GR or not.. The draft is unclear.

I think that it is covered by the above extension.  If not, it should be 
clarified.

> It is not clear to me how RRs and PEs differ in using GR.

I think that there is a main difference when a RR is not in the 
forwarding path.  In that case, the RR should always set the F bit in 
the GR Capability so that its clients will continue forwarding after 
they lose the sessions with RR.  It is a deployment issue, though.

> The time that state can persist is limit to about 1 hour max.

I think that you are talking about the "Restart time" field which has 12 
bits and amount to about 68 minutes.  The "Restart time" is for the 
session re-establishment.  It does not impact the duration for holding 
stale routes after the session is re-established.

If the session does not get re-established in 68 minutes, the stale 
routes would be purged.  That is a long time, isn't it?   However, if 
one really wants to extend the session re-establishment time and 
continue to hold stale routes, it can be done by a simple knob.

> GR does detail the behavior where convergence is not achieved between restarts.. Similar to above..

The min stale timer knob can cover it (see above).

But do you meant "does not"?  We can certainly clarify in 4724bis if 
that is the case.

> I do not believe that the current GR paradigm can be extended to cover the majority of the cases above.

Except for the path level persistence you mentioned, I believe the GR 
will be able to address all other persistence requirements you listed, 
with some simple knobs and some implementation enhancements.

Thanks.   -- Enke