Subject: [Teas] AD review: draft-ietf-teas-scheduled-resources

From: "BRUNGARD, DEBORAH A" <[email-masked]>
Date: Thu, 8 Feb 2018 22:42:00 +0000

Hi,

I've done my AD review of your draft. This draft is on a complex problem space. In reviewing it, it reminded me of the discussion around the MLN work (RFC5212, RFC5339) which discussed scheduling of resources (virtual). Most significantly, I missed discussion related to operator policy. Here's several critical items which were discussed in the MLN work:

  *   Amount of capacity available for scheduling needs to be under operator policy.
  *   The use of scheduled resources if needed during a sudden change in traffic demand event or a complex (multiple nodes/links) failure event needs to be based on operator policy so as to protect against network destabilization.
  *   The choice of reserving immediately resources for future use or to book without reserving network capacity must be under the control of an operator. And both should be supported by the database (need to be able to distinguish).
  *   Section 2.5 "allows pre-emption". This was the first mention of pre-emption of current reservations to handle a scheduled reservation. There is no additional discussion e.g. priority, etc. Pre-emption is a complex subject and if it is allowed, more description is needed and it needs to be added that it is based on operator policy.
  *   Need to include network performance factors (maximum link utilization and residual capacity of the network) with respect to supporting scheduling and subject to operator policy.
  *   When you discuss re-optimization, you need to add this needs to be under control of operator policy as the stability of the network will be impacted.
  *   The OAM status of the reserved resources (alarms) must be available. Control of recalculations for the scheduled resources and notifications of the alarms needs to be subject to policy.

In the abstract, you say "This document provides a framework..discusses the architecture..". As this document is more than an architecture, I think the title should reflect it is a "Framework..".

You listed CPU utilization, memory, buffers of interfaces as resources that can be scheduled. As this draft is on TE LSPs, I didn't see the relation to these parameters (for RSVP-TE signaling or PCE use)?

In the acknowledgements, you list chen-teas-frmwk-tts and also list it in the references. As this earlier document was dated March 2016, it would be sufficient to list the authors as you have done, saying contributed to earlier version, but don't list the no longer existing document.

There are multiple instances where the English lacks in clarity and impacts the understanding. I've suggested improvements for multiple sentences, though the document needs a more careful review.

As I said, this is a complex problem space so don't take my comments negatively. I'll send a marked up document to the authors with my suggestions for appropriate sections to add text and improving clarity.

Thanks,
Deborah

---

From: "Adrian Farrel" <[email-masked]>
Date: Sun, 11 Feb 2018 20:43:27 -0000

Hi Deborah,

Thanks for the thoughtful review.

> I’ve done my AD review of your draft. This draft is on a complex problem
space.
> In reviewing it, it reminded me of the discussion around the MLN work
(RFC5212,
> RFC5339) which discussed scheduling of resources (virtual). 

You're right. And not much has changed over the (very nearly) ten years. That
is, there is an assumption that there is some management element responsible for
handling the "booking" of network resources. Of course, 5212 and 5338
(importantly supplemented by 5623 that defined VNTM) considered multi-layered
networks, where we are focused on booking in a single layer.

> Most significantly, I missed discussion related to operator policy.

There is a slightly different nuance to this important issue. In a multi-layer
network we are concerned (one might say worried!) about how a client network
requests resources from a server network. Operator policy is, of course,
fundamental in that scenario.

But in our case, we are looking at how an operator manages their own network. Of
course, there is policy there, but (one assumes) an operator does not manage
their own network outside of their own policy. That is to say, it is the
operator's PCE that is doing the work. (No unicorns were harmed in the
production of this document!)

Nevertheless, I think we could usefully add a new section 3.3 called
"Enforcement of operator policy."  See below.

> Here’s several critical items which were discussed in the MLN work:
> • Amount of capacity available for scheduling needs to be under
>    operator policy.
> • The use of scheduled resources if needed during a sudden change
>  in traffic demand event or a complex (multiple nodes/links) failure
>  event needs to be based on operator policy so as to protect against
>  network destabilization.

There are two cases:

- If the repair is handled by the network then the fact that the resources
  are reserved is unknown by the network and that PCE will react in the
  same way that it would to a link failure and will try to recompute the
  path of the booked LSP.

- If the repair is handled by the PCE then it is aware of the booked resources
  and can apply "operator policy" to determine whether the booked LSP is
  more or less important than repair of the failed LSP.

> • The choice of reserving immediately resources for future use or to
>  book without reserving network capacity must be under the control
>   of an operator. And both should be supported by the database
>  (need to be able to distinguish).

Resources that are reserved "now" are indistinguishable from provisioned LSPs
(because they are provisioned LSPs). So, yes, an operator could service a
request for an LSP to be available next year by setting it up today (or
responding "No, come back later"). You may be suggesting that such LSPs
provisioned ahead of their actual need could be more pre-emptable than LSPs that
are in use. But I think that is crazy-talk :-) If you want to be more
pre-emptable, then don't provision the LSP - that's why you have a stateful PCE.

> • Section 2.5 “allows pre-emption”. This was the first mention of pre-emption
>  of current reservations to handle a scheduled reservation. There is no
additional
>  discussion e.g. priority, etc. Pre-emption is a complex subject and if it is
allowed,
>  more description is needed and it needs to be added that it is based on
operator
>  policy.

You're right that 2.1 and 2.2 should talk about pre-emption.
Personally, I don't think it is a particularly complex concept. But recall, the
decision is under the control of the stateful PCE.

> • Need to include network performance factors (maximum link utilization
>  and residual capacity of the network) with respect to supporting scheduling
>  and subject to operator policy.

Yes, we could make more of Objective Functions. I guess we're just too used to
seeing that as a normal, everyday factor of PCE.

> • When you discuss re-optimization, you need to add this needs to be under
>  control of operator policy as the stability of the network will be impacted.

Now you must remember whose PCE this is. Re-optimization out of the control of a
PCE is no more than a service re-provisioning request, but since the application
is unlikely to be able to see much of the TED, it is meaningless for it to make
such a request. Re-optimization under the control of the PCE is exactly
according to the operator's policy by definition.

> • The OAM status of the reserved resources (alarms) must be available. Control
of
>  recalculations for the scheduled resources and notifications of the alarms
needs to
>  be subject to policy.

It either is or it isn't!
This is no different to any other TED computation except (of course?) computing
an LSP's path for an LSP due in three years' time might be more likely to ignore
today's alarms.
 
> In the abstract, you say “This document provides a framework..discusses the
> architecture..”. As this document is more than an architecture, I think the
title
> should reflect it is a “Framework..”.

This choice of semantics is way above my pay-grade.

 > You listed CPU utilization, memory, buffers of interfaces as resources that
can be
> scheduled. As this draft is on TE LSPs, I didn’t see the relation to these
parameters
> (for RSVP-TE signaling or PCE use)?

Yup. RSVP-TE is not used to provision compute and storage resources - for which
we are suitably grateful. But some people observe that an orchestrator is a type
of PCE and that an SDN system uses a PCE at its core. 

> In the acknowledgements, you list chen-teas-frmwk-tts and also list it
>  in the references. As this earlier document was dated March 2016, it 
> would be sufficient to list the authors as you have done, saying
>  contributed to earlier version, but don’t list the no longer existing
> document.
 
The text as written is very precise. Is there a problem with an informative
reference to an expired draft?

> There are multiple instances where the English lacks in clarity and
> impacts the understanding. I’ve suggested improvements for
> multiple sentences, though the document needs a more careful review. 

Red Rag, please meet Mr. Bull.

I'll look through your comments with interest :-)
 
> As I said, this is a complex problem space so don’t take my comments
> negatively. I’ll send a marked up document to the authors with my
> suggestions for appropriate sections to add text and improving clarity.

Thanks again.

And now, proposed text...

====

3.3 Enforcement of Operator Policy

Computation requests for LSPs are serviced according to operator policy. For
example, a PCE may refuse a computation request because the application making
the request does not have sufficient permissions, or because servicing the
request might take specific resource usage over a given threshold.

Furthermore, the pre-emption and holding priorities of any particular
computation request may be subject to the operator's policies. The request could
be rejected if it does not conform to the operator's policies, or (possibly more
likely) the priorities could be set/overwritten according to the operator's
policies.

Additionally, the objective functions (OFs) of computation request (such as
maximizing residual bandwidth) are also subject to operator policies. It is
highly likely that the choice of OFs is not available to an application and is
selected by the PCE or management system subject to operator policies and
knowledge of the application.

None of these statements is new to scheduled resources. They apply to stateless,
stateful, passive, and active PCEs, and they continue to apply to scheduling of
resources.

An operator may choose to configure special behavior for a PCE that handles
resource scheduling. For example, an operator might want only a certain
percentage of any resource to be bookable. And an operator might want the
pre-emption of booked resources to be an inverse function of how far in the
future the resources are needed for the first time.

It Is a general assumption about the architecture described in Section 4 that a
PCE is under the operational control of the operator that owns the resources
that the PCE manipulates. Thus the operator may configure any amount of
(potentially complex) policy at the PCE. This configuration would also include
policy points surrounding re-optimization of existing and planned LSPs in the
event of changes in the current and future (planned) resource availability.

====

Best,
Adrian (passing through Newark and so able to throw rocks ;-)

---

From: "BRUNGARD, DEBORAH A" <[email-masked]>
Date: Fri, 16 Feb 2018 22:44:47 +0000

Hi Adrian,

Ouch😊
Proposed text looks good-
Below..[deborah]
Thanks,
Deborah

-----Original Message-----
From: Adrian Farrel [mailto:[email-masked]] 
Sent: Sunday, February 11, 2018 3:43 PM
To: BRUNGARD, DEBORAH A <[email-masked]>
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: RE: AD review: draft-ietf-teas-scheduled-resources

Hi Deborah,

space.
(RFC5212,
> RFC5339) which discussed scheduling of resources (virtual). 

You're right. And not much has changed over the (very nearly) ten years. That is, there is an assumption that there is some management element responsible for handling the "booking" of network resources. Of course, 5212 and 5338 (importantly supplemented by 5623 that defined VNTM) considered multi-layered networks, where we are focused on booking in a single layer.

> Most significantly, I missed discussion related to operator policy.

There is a slightly different nuance to this important issue. In a multi-layer network we are concerned (one might say worried!) about how a client network requests resources from a server network. Operator policy is, of course, fundamental in that scenario.

But in our case, we are looking at how an operator manages their own network. Of course, there is policy there, but (one assumes) an operator does not manage their own network outside of their own policy. That is to say, it is the operator's PCE that is doing the work. (No unicorns were harmed in the production of this document!)

Nevertheless, I think we could usefully add a new section 3.3 called "Enforcement of operator policy."  See below.

[deborah] Perhaps there was a nuance. Understand, the PCE is within an operator's domain, what I was looking for was the policy configuration knobs so operators could customize. Your suggested text below captures. If you recall, we had a very wise AD that pushed us to think about and include a manageability section (e.g. RFC6123) even when we felt it was obvious/"business as usual".

> • The use of scheduled resources if needed during a sudden change  in 
> traffic demand event or a complex (multiple nodes/links) failure  

There are two cases:

[deborah] In section 3.1, one of the issues which you identified for holding centrally, was when autonomous actions by the LSRs was allowed. And it noted "this impact can be mitigated by replanning future LSPs or through LSP preemption". As Operators have different views on this, and while it may be PCE business as usual, it would be a consideration. Your proposed text looks good.

>   of an operator. And both should be supported by the database  (need 
> to be able to distinguish).

Resources that are reserved "now" are indistinguishable from provisioned LSPs (because they are provisioned LSPs). So, yes, an operator could service a request for an LSP to be available next year by setting it up today (or responding "No, come back later"). You may be suggesting that such LSPs provisioned ahead of their actual need could be more pre-emptable than LSPs that are in use. But I think that is crazy-talk :-) If you want to be more pre-emptable, then don't provision the LSP - that's why you have a stateful PCE.

[deborah] As you say above, if doing network recovery, then the LSR's TED is not aware of the scheduling. The draft discusses the PCE's LSP-DB<->PCE's TED being sync'd but not the relationship with the LSR's TED. What I was most concerned about was the operator's ability to control - your proposed text clarifies.

> • Section 2.5 “allows pre-emption”. This was the first mention of 
> pre-emption  of current reservations to handle a scheduled 
> reservation. There is no
additional
allowed,
operator

You're right that 2.1 and 2.2 should talk about pre-emption.
Personally, I don't think it is a particularly complex concept. But recall, the decision is under the control of the stateful PCE.

> • Need to include network performance factors (maximum link 
> utilization  and residual capacity of the network) with respect to 
> supporting scheduling  and subject to operator policy.

Yes, we could make more of Objective Functions. I guess we're just too used to seeing that as a normal, everyday factor of PCE.

> • When you discuss re-optimization, you need to add this needs to be 
> under  control of operator policy as the stability of the network will be impacted.

Now you must remember whose PCE this is. Re-optimization out of the control of a PCE is no more than a service re-provisioning request, but since the application is unlikely to be able to see much of the TED, it is meaningless for it to make such a request. Re-optimization under the control of the PCE is exactly according to the operator's policy by definition.

[deborah] Ok, just looking for the words.

of
needs to

It either is or it isn't!
This is no different to any other TED computation except (of course?) computing an LSP's path for an LSP due in three years' time might be more likely to ignore today's alarms.
 
> In the abstract, you say “This document provides a 
> framework..discusses the architecture..”. As this document is more 
> than an architecture, I think the
title

This choice of semantics is way above my pay-grade.
[Deborah] It would help to be consistent😊 The abstract and introduction say "framework". Choose one or the other.

 > You listed CPU utilization, memory, buffers of interfaces as resources that can be
parameters
> (for RSVP-TE signaling or PCE use)?

Yup. RSVP-TE is not used to provision compute and storage resources - for which we are suitably grateful. But some people observe that an orchestrator is a type of PCE and that an SDN system uses a PCE at its core.

[Deborah]  But this draft is discussing LSP-DB and TED? Does it cover any/all type of database? If you want to keep, you need to identify these are not standard attributes for LSP-DB and TED.

> contributed to earlier version, but don’t list the no longer existing 
 
The text as written is very precise. Is there a problem with an informative reference to an expired draft?

[Deborah] There is no problem with an informative reference to an expired draft. My comment was based on "does this reference provide additional information for the reader?". I asked the question as it would seem more appropriate to have put it in the document. Per [RFC7322], a reference in the ack is used "to note any documents from which text was borrowed". It was not clear to me if that document was the "earlier version" of this draft? Or what was meant by "earlier version".

> impacts the understanding. I’ve suggested improvements for multiple 
> sentences, though the document needs a more careful review.

 

Thanks again.

====

Computation requests for LSPs are serviced according to operator policy. For example, a PCE may refuse a computation request because the application making the request does not have sufficient permissions, or because servicing the request might take specific resource usage over a given threshold.

Furthermore, the pre-emption and holding priorities of any particular computation request may be subject to the operator's policies. The request could be rejected if it does not conform to the operator's policies, or (possibly more
likely) the priorities could be set/overwritten according to the operator's policies.

Additionally, the objective functions (OFs) of computation request (such as maximizing residual bandwidth) are also subject to operator policies. It is highly likely that the choice of OFs is not available to an application and is selected by the PCE or management system subject to operator policies and knowledge of the application.

None of these statements is new to scheduled resources. They apply to stateless, stateful, passive, and active PCEs, and they continue to apply to scheduling of resources.

An operator may choose to configure special behavior for a PCE that handles resource scheduling. For example, an operator might want only a certain percentage of any resource to be bookable. And an operator might want the pre-emption of booked resources to be an inverse function of how far in the future the resources are needed for the first time.

It Is a general assumption about the architecture described in Section 4 that a PCE is under the operational control of the operator that owns the resources that the PCE manipulates. Thus the operator may configure any amount of (potentially complex) policy at the PCE. This configuration would also include policy points surrounding re-optimization of existing and planned LSPs in the event of changes in the current and future (planned) resource availability.

====

Best,