Subject: Graceful restart - inter-protocol dependencies

From: David Charlap <[email-masked]>
Date: Wed, 28 Sep 2005 14:37:09 -0400

Is there any point to implementing RSVP-TE's graceful restart without 
also implementing graceful restart for routing protocols?

On the one hand, RSVP doesn't require routing to recover LSPs.  It knows 
the next-hop interface, because of the preserved data-plane connection. 
  Whatever other information it may need, the switch can either preserve 
this information or recover it from a neighbor using the RecoveryPath 
object (draft-ietf-ccamp-rsvo-restart-ext-03).

On the other hand, nodes more than one hop upstream of the failure will 
detect the loss of routing-connectivity to the failed node if IGP 
graceful restart is not also implemented.  They may reroute the LSP away 
from the failed node, or tear it down altogether, even though the data 
plane is still active and RSVP graceful restart is recovering the 
control-plane state.

An originating node may consider the destination unreachable, as a 
result of losing the routes even though the data-plane for the LSP is 
still up (which can be confirmed via OAM.)

A transit node, when processing an loose ERO-hop, may choose to reroute 
or fail the LSP if its local topology information says that the failed 
(and restarting) node is not available.  It might even choose to do this 
for an established LSP, as a result of Path refresh processing.

My questions are:

1: Does this mean it's pointless to use RSVP graceful restart without
    also using IGP graceful restart (for whatever IGP is active).

2: Is IGP graceful restart sufficient to prevent this problem?  For
    instance, OSPF's restart procedure requires all preserved state to
    be thrown away if a topology change is detected.

3: An originating node can use OAM to validate the data plane of an
    LSP, and choose to ignore what routing tells it about the
    destination's reachability.  But what about transit nodes?  As far
    as I know, MPLS doesn't support segment-OAM, and it would be
    prohibitive for every transit node to run its own OAM streams
    to detect self-to-end connectivity.

4: Are there other solutions to this problem?

One possible solution might be route-pinning, but RSVP doesn't have a 
built-in mechanism for this.  The usual workaround (signal the LSP, 
requesting route-recording, then turn the RRO into an ERO for subsequent 
refreshes) can work, but are there situations where even this would be 
insufficient to prevent a transit node from rerouting/tearing the 
connection in this particular situation (where RSVP is doing a graceful 
restart but the IGP is not)?

-- David

---

From: <[email-masked]>
Date: Thu, 29 Sep 2005 16:37:04 +0100

Thanks David....it's not simply how they *should* work its how they
*must* work if one wants to be able to offer network services to
customers that have deterministic behaviour, ie one has precise (and it
also should be simple) control over traffic/fault/performance management
facets.

A full service operator like BT needs all 3 networking modes working in
concert...and they are for sure different.  Understanding *why* they are
different at fundamental level has taken us a while to properly figure
out.  However, the stuff we knew was intuitively right is now getting
placed on a far stronger foundation of underlying 'truths' because of
the unified modelling work I mentioned (this goes right back to Shannon
and Turing stuff...indeed a network is precisely the union of
Information Theory and a computing machine).

The GMPLS co-cs stuff is largely self-protecting of abuse.....this is
down to the nature of the constraints that define this mode (esp the
labelling).  Same can be said for the cl-ps mode.  The major risks arise
with the co-ps mode....and one can make life very complex here if one is
not careful.....I'll resist the strong temptation to flesh this out wrt
comments on MPLS and the PW stuff. 

However, we do need a correctly specified co-ps mode networking
capability because as we sun-set many of our legacy co-cs and co-ps
technologies there is a major gap opening for a lower granularity (than
SDH VC4) network builder capability....and we have many customers who
demand such a network service.

regards, Neil

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of David Charlap
> Sent: 29 September 2005 15:40
> To: [email-masked]; [email-masked]
> Subject: Re: Graceful restart - inter-protocol dependencies
> 
> 
> You're absolutely right that this is how co networks should 
> operate.  I 
> would agree that there is little point to rerouting an LSP that has 
> data-plane connectivity, even if routing says the path is invalid or 
> suboptimal.
> 
> Unfortunately, I don't think current MPLS signaling protocols 
> (RSVP-TE 
> and LDP) work this way.  LDP is explicitly designed to dynamically 
> change as routing changes.
> 
> RSVP-TE seems (to my best understanding) to also react to routing 
> changes, except for special cases (like when signaled with an ERO 
> consisting entirely of strict nodes whose addresses can be 
> resolved from 
> interface-neighbor information instead of from routing.)
> 
> -- David
> 
> 
> [email-masked] wrote:
> > David,
> > 
> > This is a very interesting/thoughtful mail.  When I read it I was 
> > immediately struck by recent discussions some of us have 
> been having 
> > on the topic of unified functional modelling within SG15, ie the 
> > underlying axiomatic truths that stem directly from 
> Shannon/Turing wrt 
> > networking and that show how the co-cs, co-ps and cl-ps networking 
> > modes all have a common root.  However, whilst I don't want to get 
> > into that here I do want to illustrate a key consequence of 
> this work 
> > that is particularly relevant to your observations.
> > 
> > In the co modes we have this rather important object called 
> a trail.  
> > A trail is, in essence, a concatenated sequence of link-connections 
> > and nodes (which are the smallest subnetwork connections).  
> However, 
> > the key observation is that at the epoch we create the 
> trail we remove 
> > all further routing choices from the trail.  You can almost 
> think of 
> > this as effectively removing the trail from its parent network 
> > topology (there is a resource accounting issue to deal 
> with, but that 
> > is a different point).....and I guess in IETF speak one would call 
> > this 'route-pinning'.
> > 
> > What this means is that it does not matter what happens to 
> the routing 
> > process after this epoch, once the trail is established there is no 
> > need to move it.  And a key example of why this is important is the 
> > case of augmenting the network topology by adding further 
> nodes/links.  
> > Because the network topology has now changed then future routing 
> > decisions are very likely to alter.  But in the co mode 
> case this does 
> > not matter....once a trail is created it remains faithful to its 
> > original routing choice unless we consciously decide to 
> move it (which 
> > may of course be the result of failures or planned works).
> > 
> > Now contrast this with the cl mode case. If we augment a cl 
> network by 
> > adding new nodes/links then the routing process will update and 
> > traffic will move in accordance with whatever is the optimal path.
> > 
> > Note a trail has other key properties, eg it allows one to 
> completely 
> > decouple a survivability semantic from the QoS semantics of client 
> > traffic units.  This property is rather important when one 
> offering a 
> > network builder service to customers.
> > 
> > regards, Neil
> > 
> > 
> > 
> > 
> >>-----Original Message-----
> >>From: [email-masked]
> >>[mailto:[email-masked]] On Behalf Of David Charlap
> >>Sent: 28 September 2005 19:37
> >>To: IETF MPLS List; IETF CCAMP List
> >>Subject: Graceful restart - inter-protocol dependencies
> >>
> >>

---

From: [email-masked]
Date: Fri, 30 Sep 2005 04:16:49 +0200

Hi David,

We should specify which version of RSVP we are talking. What I have said is 
about GMPLS RSVP-TE as used in ASON. 

When there is a failed link within the network, the ingress node will know the 
failure by transport plan OAM mechanism such as SDH path layer AIS, or by RSVP-
TE Notify message. And usually these mechanisms are faster than routing 
protocol convergence speed. After the ingress node knows which link or route 
has failed, it will do the rerouting while excluding the failed link or route. 

Hope I say it clearly :-)

Rick 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of 
David Charlap
Sent: Thursday, September 29, 2005 10:35 PM
To: 'IETF MPLS List'; 'IETF CCAMP List'
Subject: Re: Graceful restart - inter-protocol dependencies

But RSVP doesn't work this way.  It is a soft-state protocol.

If the route table changes such that the next-hop interface changes, RSVP is 
supposed to reroute the connection accordingly.

This probably won't happen if the next-hop is a strict-hop in an ERO, but it 
can easily happen if the next-hop is loose or if the LSP is signaled without 
an ERO (meaning routing is consulted to determine the next-hop to the egress
node.)

If there is a paragraph in an RFC that says RSVP-TE should ignore all route-
table changes once an LSP comes up, I missed it.

-- David

Jing Ruiquan wrote:
> I think only the  originating node should do the rerouting with 
> RSVP-TE .At the same time, all the nodes will converge their IGP 
> routing
data base.
> Rick
> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of David Charlap
> Sent: Thursday, September 29, 2005 2:37 AM
> On the one hand, RSVP doesn't require routing to recover LSPs.  It 
> knows the next-hop interface, because of the preserved data-plane
connection.
>   Whatever other information it may need, the switch can either 
> preserve this information or recover it from a neighbor using the 
> RecoveryPath object (draft-ietf-ccamp-rsvo-restart-ext-03).
> 
> On the other hand, nodes more than one hop upstream of the failure 
> will detect the loss of routing-connectivity to the failed node if IGP 
> graceful restart is not also implemented.  They may reroute the LSP 
> away from the failed node, or tear it down altogether, even though the 
> data plane is still active and RSVP graceful restart is recovering the
control-plane state.
> A transit node, when processing an loose ERO-hop, may choose to 
> reroute or fail the LSP if its local topology information says that 
> the failed (and
> restarting) node is not available.  It might even choose to do this 
> connection in this particular situation (where RSVP is doing a 
> graceful restart but the IGP is not)?
> 
> -- David

---

From: <[email-masked]>
Date: Fri, 30 Sep 2005 07:12:35 +0100

Rick....you must differentiate the co-cs mode case from the co-ps case.
They both belong to the co mode BUT there are very significant
differences....indeed, the 3 network modes arise by the way constraints
are applied to a unified networking model (and if you are active in ITU
then check out what is happening in SG15 on this topic).

In the co-cs mode case the constraints mean the following behaviours are
FORCED:
-	control/management OOB.....this means that in a co-cs network
system there is more than 1 data-plane network.  This has major
implications on the design of these networks (wrt addressing, security,
resilience, stability, etc);
-	the only traffic carrying trail constructs allowed are p2p and
p2mp, ie no merging is possible......and these are the ONLY trail
constructs that allow deterministic traffic/fault/performance management
anyway;
-	a trail MUST complete between trail source and sink points as
this is where the characteristic information is processed that looks
after the integrity of the trail (eg OAM stuff);
-	there are no QoS/traffic classes.....this looks like a throwaway
remark but it's extremely important.....especially when considering
network builder services, ie when building a client layer topology this
defines performance inheritance behaviour;
-	true client/server relationships can exist, ie complete
functional decoupling of client and server layer networks....again
absolutely critical for a whole raft of factors (security,
scalability,...) when considering network builder services.

Now add these observation to my previous ones (eg the importance of the
trail object (which I think many folks still don't really get), and how
once created it gets decoupled from the routing process) and you start
to get the picture of the required behaviour of a co mode technology.

Its jolly hard to get things wrong with the co-cs mode but this is not
true for the co-ps mode.  However, there is no reason why the co-ps mode
should not reproduce the important behaviours, and in doing so re-use
the components (eg similar OOB design, signalling), that apply to co-cs
mode technologies.....indeed when we kicked off the ASON work this was
precisely what was in our minds.  And as a more general observation
here:  When folks talk about convergence there is a lot of confusion as
to what this really means....it for sure does NOT mean that (i) 1
network mode/technology can do it all, nor does it mean (ii) one can
somehow 'functionally crunch' all 3 networking modes and treat them
alike.  What is does mean however is that we don't need lots of
different technologies belonging to the SAME mode (as this creates
duplication/waste and leads to stovepipe services and downstream
migration/interworking problems), but we do need all 3 modes working in
concert....and wrt to this latter observation we can apply sensible
component re-use, ergo the comment above wrt ASON.

And the 'service' reason this is required is because (as we sunset PDH,
lower rate SDH VC12, FR and ATM layer networks) we need a network
builder service capability at bit rates smaller than SDH VC4.

It's taking a lot longer then we expected for the above to be better
understood but I can see strong signs now emerging that we are starting
to get there.  

regards, Neil

> [mailto:[email-masked]] On Behalf Of 
> [email-masked]
> Sent: 30 September 2005 03:17
> To: [email-masked]
> Subject: RE: Graceful restart - inter-protocol dependencies
> 
> 
> 
> 
> Hi David,
> 
> failure by transport plan OAM mechanism such as SDH path 
> layer AIS, or by RSVP- TE Notify message. And usually these 
> mechanisms are faster than routing 
> If there is a paragraph in an RFC that says RSVP-TE should
> ignore all route- table changes once an LSP comes up, I missed it.
> 
> -- David
> 
> Jing Ruiquan wrote: