Subject: Summary [was: RE: Starting a discussion on graceful shutdown]

From: "Zafar Ali" <[email-masked]>
Date: Wed, 16 Feb 2005 01:34:26 -0500

Hi Adrian, 

Thanks for putting this email together. In the following I have tried to
summarize the discussion we had on this thread and next steps. 

Thanks

Regards... Zafar 

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Adrian Farrel
> Sent: Tuesday, December 07, 2004 10:39 AM
> To: [email-masked]
> Subject: Starting a discussion on graceful shutdown
> 
> Hi,
> 
> http://www.ietf.org/internet-drafts/draft-ali-ccamp-mpls-grace
> ful-shutdown-00.txt was discussed at the meeting in 
> Washington DC, and one of the questions raised was whether or 
> not we already had (multiple!) mechanisms capable of 
> performing the function described in the draft.
> 
> Kireeti quite rightly suggested that we step back and make 
> sure we understand the requirements. This is my take on those 
> requirements and I would appreciate it if the authors of the 
> draft joined in and other people on the list commented.
> 
> We wish to manage a link that needs to be taken out of 
> service in some way (data plane and/or control plane will be 
> disrupted). The link concerned has active LSPs and we wish to 
> offer upstream LSRs (in particular the ingress) the 
> opportunity to use make-before-break to re-route the LSPs.
> 
> In order to achieve this, we need to communicate to the 
> upstream nodes. Should we choose signaling or routing? Are 
> there benefits that mean we should use both, or should we 
> limit to just one?

Yes, we need both. 

> 
> There is another aspect to be considered. Should we also 
> attempt to protect new path computations from selecting the 
> link that will be taken out of service?

Yes, 

> 
> How should we consider the case of a node (data plane and/or 
> control plane) being taken out of service? 

Yes, 

>Is a node simply a 
> collection of links?

Yes, during graceful shutdown period. 

> 
> If a component link of a bundle is being taken out of service 
> (and assuming other component links are available) is this 
> just an issue for the adjacent nodes or does it need to be 
> communicated more widely? 

Let head end do the mb4b, 

> If the downstream node decides to 
> take the component link out of service, how does it inform 
> the upstream node?
> 
> Does it matter whether it is the control plane or the data 
> plane that will be taken out of service?

Yes, if data plane is going OOS, we are loosing traffic and motivation
behind mb4b to reduce traffic hit is lost. 

> 
> Opinions please.
> 
> I would like to see these issues and their answers captured 
> clearly in a requirements section of any draft. Would the 
> authors of draft-ali-ccamp-mpls-graceful-shutdown-00.txt
> be willing to take that on even though the end result might 
> be that procedures other than those they suggest will be selected?

Sure, we plan to revised the ID based on this survey as well as comments
from the WG for the upcoming IETF. 

Thanks again Adrian for your suggestion and putting this together. 

Thanks

Regards... Zafar

> Adrian

---

From: "Richard Rabbat" <[email-masked]>
Date: Thu, 17 Feb 2005 14:35:17 -0800

Hi Zafar,
First, I'd like to point out that your draft has expired; thought you might
want to put a quick version up.
Some comments inline and more generally about the draft.
Thanks a lot,
Richard.
--
- in the text, you say: MAY NOT. This is not defined. 

- Discussing PCE in this draft seems to be out of place. This draft is sent
to ccamp and should not specify what a PCE node has to do.

- This is especially confusing since you say that a PCE is receiving RSVP
Path error messages. This implies somebody sent it to the PCE. But you have
defined communication from the node where shutdown occurs to the ingress.
How does the PCE know about this message?

- You talk in the draft about FRR but FRR does not apply to all types of
switching in GMPLS. Please reflect that in your text to say: "In the case of
PSC, ..."

- You mention FSC, LSC and PSC. Is it your objective not to address L2SC and
TDM?

- I'd also like to ask you how you decommission a bundle? Does the headend
of that bundle have the responsibility of telling each headend of each
component about the shutdown?
E.g. let's say I'm shutting down a link in a SONET box that has a
server-layer circuit (say an STS-3c) that is carrying Ethernet as the client
layer. Who tells the Ethernet "headend" about the shutdown?

> [mailto:[email-masked]] On Behalf Of Zafar Ali
> Sent: Tuesday, February 15, 2005 10:34 PM
> To: 'Adrian Farrel'; [email-masked]
> Subject: Summary [was: RE: Starting a discussion on graceful shutdown]
> 
> 
> Hi Adrian, 
> 
Can you explain to me the justification for both? 
I don't think there is no need for routing extensions, since the routing
protocol will stop advertising decommissioned links. 

Can you also explain the difference between your method and the available
SONET mechanisms for shutting down resources?

In section 4, you talk about OSPF/ISIS. Do you mean OSPF-TE/ISIS-TE? 

> > link that will be taken out of service?

There is a balance between (1) telling ingress LSR's vs. (2) ingress LSR's
receiving notification through usual updates. Authors need to justify need
and advantage for (1).

---

From: "Adrian Farrel" <[email-masked]>
Date: Thu, 17 Feb 2005 21:01:12 -0000

Thanks, Zafar.

Comments in line.

Adrian

> > there benefits that mean we should use both, or should we

Well, I was kind of hoping for a reasoned debate :-)

> >Is a node simply a  collection of links?
>

So does that mean you would like to advertise the withdrawal of 10,000
links, or the withdrawal of one node. Recall that all link advertisement
is associated with the node, so node withdrawal is meaningful.

> Let head end do the mb4b,

Presumably this would not be true if the bundle was advertised as having
some form of link protection (and the ocmponent links provided this
protection).

> > take the component link out of service, how does it inform

You missed this one.

> behind mb4b to reduce traffic hit is lost.

Let me rephrase the question.
If I know that the control plane is going OOS and I want to continue to be
able to manage my LSPs I will want to move them to LSRs that will continue
to have an active control plane. The data plane through the impacted LSRs
may continue to be active (and therefore could continue to carry traffic).

Is this an important case? If so, do we manage it in the same way?

---

From: "Zafar Ali" <[email-masked]>
Date: Sun, 20 Feb 2005 12:45:41 -0500

Dear Adrian, 

Please see my reply in-line. 

Thanks again for your input.

Regards... Zafar

> Sent: Thursday, February 17, 2005 4:01 PM
> To: Zafar Ali; [email-masked]
> Subject: Re: Summary [was: RE: Starting a discussion on 
> graceful shutdown]
> 
> Thanks, Zafar.
> 
> Comments in line.
> 
> Adrian
> 
> Washington DC, 
> > > and one of the questions raised was whether or not we already had 
> > > (multiple!) mechanisms capable of performing the function 
> described 
> > > in the draft.
> > >
> > > Kireeti quite rightly suggested that we step back and 
> make sure we 
> > > understand the requirements. This is my take on those 
> requirements 
> > > and I would appreciate it if the authors of the draft 
> joined in and 
> > > other people on the list commented.
> > >
> > > We wish to manage a link that needs to be taken out of service in 
> > > some way (data plane and/or control plane will be disrupted). The 
> > > link concerned has active LSPs and we wish to offer upstream LSRs 
> > > (in particular the ingress) the opportunity to use 
> make-before-break 
> > > to re-route the LSPs.
> > >
> > > In order to achieve this, we need to communicate to the upstream 
> > > nodes. Should we choose signaling or routing? Are there benefits 
> > > that mean we should use both, or should we limit to just one?
> >
> > Yes, we need both.
> 

The reason for me not listing the motivation as I was trying to summarize
the discussion on this thread. Here are the reasons/ requirements for having
complementary mechanism in IGP and RSVP-TE (as we discussed on the list).

- It is required to prevent other network nodes to use network resources
which are about to be shutdown, should new TE LSP be set up. As RSVP
mechanisms only convey the information for the transiting LSPs to the router
along the upstream Path and not to all nodes in the network, indication of
graceful shutdown via IGP flooding is required to discourage a node from
establishing new LSPs through the resources being shut-down.

- Graceful shutdown mechanisms are required to address TE LSPs spanning
multiple domains, as well as intra domain TE LSPs. Here, a domain is defined
as either an IGP area or an Autonomous System. An IGP based solution is not
applicable when dealing with Inter-area and Inter-AS traffic engineering, as
LSA or LSP flooding is restricted to IGP areas/levels. Consequently, RSVP
based mechanisms are required to cope with TE LSPs spanning multiple
domains.

> attempt to 
> > > protect new path computations from selecting the link 
> that will be 
> > > taken out of service?
> >
> > Yes,

Please see requirements listed above. 

> >
> > > How should we consider the case of a node (data plane 
> and/or control 
> > > plane) being taken out of service?
> >
> > Yes,
> >
> So does that mean you would like to advertise the withdrawal 
> of 10,000 links, or the withdrawal of one node. Recall that 
> all link advertisement is associated with the node, so node 
> withdrawal is meaningful.

I think removal of Router Address TLV would suffice. 

> 
> > > If a component link of a bundle is being taken out of 
> service (and 
> > > assuming other component links are available) is this 
> just an issue 
> > > for the adjacent nodes or does it need to be communicated more 
> > > widely?
> >
> Presumably this would not be true if the bundle was 
> advertised as having some form of link protection (and the 
> component links provided this protection).

Yes, it is required to co-ordinate graceful shutdown with the protection
available for the TE link or the affected LSP. The draft did list the
following case, 

"When a graceful shutdown operation is performed along the path of a
protected LSP, the PLR MAY trigger Fast Reroute [FRR] for the appropriate
set of affected TE LSPs and forward a Path Error with "Tunnel locally
repaired" sub-code, as per the procedures specified in [MPLS-FRR]". 

When unbundled TE link is protected and maintenance of components within the
protected TE link is possible using means other than graceful shutdown
mechanisms (e.g., by L2), graceful shutdown may be handled by a local
switchover without informing the network of graceful shutdown condition.

In case of bundled TE links, RSVP flows are "pined" to a component link and
removal of that component link does affect the flows using it. Hence removal
of a component link has to be handled at the MPLS layer using means like
mb4b, FRR, path/ segment protection. 

In the case of Path/ segment protection, it is also possible to make use of
protection switchover mechanism (instead of mb4b) but IMO this is a matter
of local choice at the head-end LSR. The job of graceful shutdown mechanism
is to carry information about the graceful shutdown event and specify the
resource under graceful shutdown. 

> > > take the component link out of service, how does it inform the 
> > > upstream node?
> 
> You missed this one.

At present draft mentions the use of perr with "Local component link
maintenance" flag and assumes a head-end based rerouting/ mb4b. 

> 
> > > Does it matter whether it is the control plane or the data plane 
> > > that will be taken out of service?
> >
> If I know that the control plane is going OOS and I want to 
> continue to be able to manage my LSPs I will want to move 
> them to LSRs that will continue to have an active control 
> plane. The data plane through the impacted LSRs may continue 
> to be active (and therefore could continue to carry traffic).

I am not sure if I understood your comment completely. However, the ID
mentions about Grace Period and Removal of data plane associated with the
LSP under mb4b. 

> > > I would like to see these issues and their answers 
> captured clearly 
> > > in a requirements section of any draft. Would the authors of 
> > > draft-ali-ccamp-mpls-graceful-shutdown-00.txt
> > > be willing to take that on even though the end result 
> might be that 
> > > procedures other than those they suggest will be selected?
> >
> > Sure, we plan to revised the ID based on this survey as well as 
> > comments from the WG for the upcoming IETF.
> >

---

From: "Zafar Ali" <[email-masked]>
Date: Sun, 20 Feb 2005 12:16:10 -0500

Dear Richard, 

Thanks a lot for your review and detailed comments, 
 
Regards... Zafar

> [mailto:[email-masked]] On Behalf Of Richard Rabbat
> Sent: Thursday, February 17, 2005 5:35 PM
> To: 'Zafar Ali'; 'Adrian Farrel'; [email-masked]
> Subject: RE: Summary [was: RE: Starting a discussion on 
> graceful shutdown]
> 
> Hi Zafar,
> First, I'd like to point out that your draft has expired; 
> thought you might want to put a quick version up.

You will see a copy of draft reflecting all comments received thus far petty
soon. 

> Some comments inline and more generally about the draft.

Good catch, in the new version you will see revised text. 

> 
> - Discussing PCE in this draft seems to be out of place. This 
> draft is sent to ccamp and should not specify what a PCE node 
> has to do.
> 
> - This is especially confusing since you say that a PCE is 
> receiving RSVP Path error messages. This implies somebody 
> sent it to the PCE. But you have defined communication from 
> the node where shutdown occurs to the ingress.

Via PCC, 

There are places where document lists set of roles that any entity computing
a path should follow, i.e., Ingress node, intermediate nodes performing ERO
extensions or PCE. Certainly specification of (related) PCE-PCC
communication is beyond the scope of this document and in the revision we
have clearly stated that. 

> 
> - You talk in the draft about FRR but FRR does not apply to 
> all types of switching in GMPLS. Please reflect that in your 
> text to say: "In the case of PSC, ..."

Done, 

> 
> - You mention FSC, LSC and PSC. Is it your objective not to 
> address L2SC and TDM?

No, nothing intentional to avoid L2SC, TDM. We have modified the text
accordingly. 

> 
> - I'd also like to ask you how you decommission a bundle? 
> Does the headend of that bundle have the responsibility of 
> telling each headend of each component about the shutdown?

Yes, and ID talks about this. When all component links of the bundle are
being decommissioned, its like graceful shutdown of a TE link. However, if
only a subset of component links are decommissioned, procedure for graceful
shutdown of the component links applies (as listed in section 4.2 of the
last version of the ID). 

> E.g. let's say I'm shutting down a link in a SONET box that 
> has a server-layer circuit (say an STS-3c) that is carrying 
> Ethernet as the client layer. Who tells the Ethernet 
> "headend" about the shutdown?

The box/ node upstream to the link under graceful shutdown. I.e., the box/
node which is using the link under graceful shutdown as an Egress link for
the affected flows. 

> > Thanks for putting this email together. In the following I 
> have tried 
> > to summarize the discussion we had on this thread and next steps.
> > 
> > Thanks
> > 
> > Regards... Zafar
> > 
> I don't think there is no need for routing extensions, since 
> the routing protocol will stop advertising decommissioned links. 
> 
> Can you also explain the difference between your method and 
> the available SONET mechanisms for shutting down resources?
> 
> There is a balance between (1) telling ingress LSR's vs. (2) 
> ingress LSR's receiving notification through usual updates. 
> Authors need to justify need and advantage for (1).
> > 
> > >

---

From: "Adrian Farrel" <[email-masked]>
Date: Mon, 21 Feb 2005 12:18:57 -0000

Hi,

Thanks for answers. Snipped down to just one point.

> > > Yes, if data plane is going OOS, we are loosing traffic and 
> > > motivation behind mb4b to reduce traffic hit is lost.
> > 
> I am not sure if I understood your comment completely. However, 
> the ID mentions about Grace Period and Removal of data plane 
> associated with the LSP under mb4b. 

I'll try again.
The draft talks about cases where the data plane is going to be removed. No problem there.
I am asking about cases where we know that the control plane is going to be removed, but where we know that the data plane is going to stay up.
Examples include:
- going to take the software components down (perhaps for upgrade)
- going to remove the control plane CPU (to replace it, dust it, or fit nerw tires)
- going to take down the control channel (e.g. decommission the OSC laser)

In these cases, we have put a lot of effort into GMPLS to ensure that the LSP is capable of continuing to exist even when the control plane is down.

I am asking whether we want to give the ingress due warning that the LSP is about to enter this state (i.e. data plane up, but control plane disconnected). The reason why the ingress might be interested is that, once in this state, the LSP is not fully manageable.

An option is to provide an error value that says "Data plane OK, but control plane about to go away". This would allow the ingress to re-route (mb4b) if it cared.

So the question for you and the WG is: Is this an important case? Does the ingress ever care about this loss of control plane connectivity?

Cheers,
Adrian

---

From: "Zafar Ali" <[email-masked]>
Date: Mon, 21 Feb 2005 11:56:16 -0500

Hi Adrian, 
 
This time it's crystal clear on what you were referring to in your comments.
Please see my reply in-line. 
 
Thanks
 
Regards... Zafar

  _____  

From: Adrian Farrel [mailto:[email-masked]] 
Sent: Monday, February 21, 2005 7:19 AM
To: Zafar Ali; [email-masked]
Subject: Re: Summary [was: RE: Starting a discussion on graceful shutdown]

Hi,

> associated with the LSP under mb4b. 
 
I'll try again.
The draft talks about cases where the data plane is going to be removed. No
problem there.
I am asking about cases where we know that the control plane is going to be
removed, but where we know that the data plane is going to stay up.
Examples include:
- going to take the software components down (perhaps for upgrade)
- going to remove the control plane CPU (to replace it, dust it, or fit nerw
tires)
- going to take down the control channel (e.g. decommission the OSC laser) 

Yes, most certainly, this is one of the motivations behind this draft. In
fact the -00 version of the ID had following statement in the Introduction
section. 
 
"a Service Provider may desire to gracefully (temporarily or definitely)
remove a TE Link, a group of TE Links or an entire node for administrative
reasons such as link maintenance or LSR software/hardware upgrade. "
 
The revised version of the ID (-01), already in draft queue have this as one
of the main requirements. 
 
"   - Graceful shutdown mechanisms are applicable to removal a TE 
   resource (e.g., link, a group of TE Links or an entire node). Trigger 
   for the graceful shutdown event is a local matter at the node 
   initiating the graceful shutdown. Typically graceful shutdown is 
   triggered for administrative reasons, such as link maintenance or 
   software/hardware upgrade at a node.  "

In these cases, we have put a lot of effort into GMPLS to ensure that the
LSP is capable of continuing to exist even when the control plane is down. 

Certainly, but as you know these are for unplanned outages, aka graceful
restart (GR) mechanisms. This ID talks about planned outages: Graceful
Shutdown (GS), like to one you mentioned above. 

I am asking whether we want to give the ingress due warning that the LSP is
about to enter this state (i.e. data plane up, but control plane
disconnected). The reason why the ingress might be interested is that, once
in this state, the LSP is not fully manageable.
 
An option is to provide an error value that says "Data plane OK, but control
plane about to go away". This would allow the ingress to re-route (mb4b) if
it cared. 

The ID assumes that Graceful Shutdown mechanisms are agnostic to control vs.
data plan reasons for the planned outage. IMO we don't need specific codes
for one vs. the other failure. Do you or anyone else feels that we need to
have different cause code for the data vs. control plane failures (for e.g.,
some admin reasons).   

So the question for you and the WG is: Is this an important case?  

Yes, as mentioned above. 

 Does the ingress ever care about this loss of control plane connectivity? 

Yes, for the reasons you listed above, e.g., software upgrades. 

Cheers,
Adrian