Subject: [nfsv4] SET_PRIORITY: new op proposal

From: "Halevy, Benny" <[email-masked]>
Date: Fri, 23 Feb 2007 09:42:27 -0500

Based on discussions during FAST 07 we'd like to propose
a SET_PRIORITY operation that allows the client to
set a relative priority for the session and/or operations.

The benefits from this operation are pretty obvious.  It can be
used by clients for example to execute metadata operations in
higher priority than data operations, or to execute operations
on specific files in a different priority, or to give different
priority to different client hosts.

I'd appreciate your prompt comments since we would like to
introduce this operation as soon as possible (draft-09)

Thanks,

Benny

Operation: SET_PRIORITY - set relative priority

SYNOPSIS

  priority_how, priority_level -> actual_priority_level

ARGUMENT

  enum priority_how4 {
    PRIORITY_SESSION    = 0,
    PRIORITY_OPERATION  = 1
  };

  enum priority_range4 {
    PRIORITY_MIN        = -50,
    PRIORITY_DEFAULT    = 0,
    PRIORITY_MAX        = 49
  };

  struct SET_PRIORITY4args {
    enum priority_how4  priority_how;
    int32_t             priority_level;
  };

RESULT

  struct SET_PRIORITY4res {
    int32_t             actual_priority_level;
  };

DESCRIPTION

  This operation may be used to set a relative priority for the current session
  or for the following operations in the compound RPC.  The default priority
  level is implicitly zero for all sessions and operations.  SET_PRIORITY for
  PRIORITY_SESSION sets the priority_level of the current session such that all
  operations following SET_PRIORITY in the same compound RPC and all operations
  related to the sessions will be executed in the given priority_level.
  SET_PRIORITY for PRIORITY_OPERATION sets the priority_level only for the
  operations following it in the compound RPC and does not affect the current
  priority_level of the related session.

  There are 100 priority levels, such that the priority_level MUST be
  greater than or equal to PRIORITY_MIN and less than or equal to PRIORITY_MAX.
  If the priority_level is outside the valid range the server should return
  an NFS4ERR_INVAL error.

  The server replies SET_PRIORITY with the actual_priority_level that was assigned
  to the session and/or following operations. Servers that do not support priorities
  MUST either return an NFS4ERR_NOTSUPP error or return zero for the
  actual_priority_level.

  Servers that support priorities SHOULD yield more resources for executing
  operations with higher priority levels than to operations with lower priority
  levels.  The amount of resources, such as network or disk bandwidth, that is
  given for each priority level is implementation specific and in some cases
  may be configurable by the system administrator.
  
IMPLEMENTATION

  If the server supports a priority system with a different scale it SHOULD map
  the protocol priority_level into its own scale and rescale the
  actual_priority_level back onto the [PRIORITY_MIN .. PRIORITY_MAX] scale.

  Care should be taken by the server to prevent starvation of lower priority
  operations.  Specifically, since setting the priority does not require any
  special access permissions, malicious clients should be detected and confined
  to prevent denial of service attacks that take advantage of the priority system.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Robert Gordon <[email-masked]>
Date: Fri, 23 Feb 2007 13:15:31 -0600

Hi Benny;

So, since SESSION may have a priority associated with it, may an  
implementer assume
that the NFS Client, may/should also support priorities (for the back- 
channel) ?
and I guess it follows for SESSIONS to data-servers as well..

It seems like EXCHANGE_ID should carry a hint in the arguments/reply  
to indicate
want support for priorities/has support for priorities.

Could we remove the need for priority_how4 by adding priority_level to
CREATE_SESSION? Hmm If _just_ SESSION was prioritized then the NFS  
Client
could pick and choose which SESSION to use for a given COMPOUND ??

Can you provide some additional usage for setting the priority within
the COMPOUND, i'm assuming PRIORITY_OPERATION scope is just for the
containing compound (or till we see another SET_PRIORITY) ?

Robert.

---

From: Benny Halevy <[email-masked]>
Date: Mon, 26 Feb 2007 14:36:14 +0200

Dan, thanks for the feedback,
answers below.

Benny

Muntz, Daniel wrote:
> Seems like a pretty significant change to include in draft-09, which is
> due out any day now.  In particular the implementation mentions some
> potentially non-trivial issues in correctly implementing priority
> queuing.
> 
> E.g., rather than leave it to the server and client to figure out some
> mapping of different priority schemes, why not define the the priority
> scheme used at the NFS layer by both clients and servers.  If the the
> server has some priority scheme implemented in lower layers (e.g., at
> physio), leave it to the server to do such mappings so as not to clutter
> the NFS protocol with extra negotiations/mappings.

The proposal allows the server to do that. Mandating the NFS priority scale
seems to be more demanding on the server, isn't it?

> 
> Also, mixing non-idempotent operations with priority queuing (e.g., the
> suggestion that metadata operation priority is > data operation
> priority), suggests to me that there might be some potentially
> significant implementation details in maintaining a correct lattice for
> execution ordering.

I agree that implementing a good priority queuing system is not trivial :)
Can you please elaborate on issues you see with non-idempotent operations
and priority queuing?  Such corner cases are good candidates to be documented
in the spec.

> 
> The last paragraph in "Implementation" seems a bit sparse, but other
> people would be better judges of what level of detail should go into an
> RFC on this topic.
> 
>  
> 
> -----Original Message-----
> From: Halevy, Benny [mailto:[email-masked]] 
> Sent: Friday, February 23, 2007 6:42 AM
> To: [email-masked]
> Cc: [email-masked]; Welch, Brent
> Subject: [nfsv4] SET_PRIORITY: new op proposal
> 
> Based on discussions during FAST 07 we'd like to propose a SET_PRIORITY
> operation that allows the client to set a relative priority for the
> session and/or operations.
> 
> The benefits from this operation are pretty obvious.  It can be used by
> clients for example to execute metadata operations in higher priority
> than data operations, or to execute operations on specific files in a
> different priority, or to give different priority to different client
> hosts.
> 
> I'd appreciate your prompt comments since we would like to introduce
> this operation as soon as possible (draft-09)
> 
> Thanks,
> 
> Benny
>

---

From: Benny Halevy <[email-masked]>
Date: Mon, 26 Feb 2007 14:46:21 +0200

> that the NFS Client, may/should also support priorities (for the back- 

Good point. The intent was to set a priority for the server only
and not for the client callback channel.

> and I guess it follows for SESSIONS to data-servers as well..

Yes.

> want support for priorities/has support for priorities.

It could, although the reply to SET_PRIORITY should be sufficient to
tell that too.

> containing compound (or till we see another SET_PRIORITY) ?

Right.  You're interpretation is exactly what I meant :)
I don't have a particular use case besides saving on sessions
and allowing a different priority for specific operations.

This is not exactly the same is having different priorities for
different sessions as the latter can have an affect at a higher layer,
such as the polling layer, e.g. executing operations off of higher
priority sessions before lower priority sessions.  Meaning
a high priority operation coming from a low priority session
may have to wait behind in the low priority session queue
before its SET_PRIORITY operation is even parsed.

That said, I'm ok with taking this mechanism out if people think the
benefit of having it is too low for the cost of implementing it.

Benny

---

From: Mike Eisler <[email-masked]>
Date: Mon, 26 Feb 2007 05:15:42 -0800 (PST)

--- Benny Halevy <[email-masked]> wrote:

> > due out any day now.  In particular the implementation mentions

I agree that the priority system as proposed, is too big for
draft-09, and hence, NFSv4.1, since draft-09 is supposed to be the
last draft for .x file changes.

I will send two more messages on this in the next few minutes. One
that responds to Benny's reponse to Robert Gordon, and another that
proposes an alternative that might be considered small enough to
put into draft-09.

---

From: Mike Eisler <[email-masked]>
Date: Mon, 26 Feb 2007 05:24:04 -0800 (PST)

> and allowing a different priority for specific operations.

Having a separate operation does complicate recovery from
partially completed COMPOUND procedures, e.g. 10 ops send,
6 executed. The client has to re-send the SET_PRIORITY with
the remaining re-sent operations.

---

From: Mike Eisler <[email-masked]>
Date: Mon, 26 Feb 2007 05:35:46 -0800 (PST)

If we were talking about NFSv4.2, SET_PRIORITY would be a natural
way to add this operation.

Since we are still working on NFSv4.1, we have an opportunity
to add this to "existing" NFSv4.1 core infrastructure.

My counter proposal is:

- add a priority field to CREATE_SESSION arguments (which the server
can
  confirm as it does various other session values). The priority is
  relative to other sessions, and represents the priority the server
  and client will take when servicing the fore and back channels of
  the session. For completeness, there can be separate fore and back
  channel priorities.

- add a priority field to SEQUENCE and CB_SEQUENCE arguments
  and results. The priority represents priority of the 
  COMPOUND/CB_COMPOUND relative to other COMPOUNDs/CB_COMPOUNDs
  executed on the session.

- There is no way to update the session's priority once the session is
  created. Just as there is no way to update the maximum request size.
  Maybe, someday, we'll need such capabilities, but a general purpose
  operation to updated session and channel attributes should be added.
  Thus far, I haven't seen a clamor for that. In the mean time, just
  create another session, and gracefully destroy the old session once
  replies for all outstanding COMPOUNDs have been received.

---

From: Benny Halevy <[email-masked]>
Date: Mon, 26 Feb 2007 15:38:52 +0200

>   replies for all outstanding COMPOUNDs have been received.

works for me.

Benny

---

From: Robert Gordon <[email-masked]>
Date: Mon, 26 Feb 2007 11:21:41 -0600

exactly what i was thinking..

Robert.

---

From: "Talpey, Thomas" <[email-masked]>
Date: Mon, 26 Feb 2007 12:38:36 -0500

But, what does the server actually do with the priority? Normally, any
operation is eligible once received at the server, does this mean the
server is required to implement strict queuing within the session, and
if so does it mean that it will need to delay the processing of requests
which are otherwise ready? If not, then what does it actually do? Benny's
text documents the field, but doesn't describe the server's requirements.

Call me confused, perhaps...

Tom.

---

From: Mike Eisler <[email-masked]>
Date: Mon, 26 Feb 2007 10:44:05 -0800 (PST)

> text documents the field, but doesn't describe the server's

I believe Chuck Lever called this a UNIX-like "nice" priority system.

The man page,
  http://www.opengroup.org/onlinepubs/007908799/xcu/renice.html
for re-nice is very interesting in that it doesn't say anything
about how the operating system is supposed to implement it.
Nothing for example about whether processes with a higher
priority are always ahead of the processes with a lower
priority.
http://www.opengroup.org/onlinepubs/007908799/xbd/glossary.html
under system scheduling priority has more weasel words, ending
with:

  Because of the use of "implementation-dependent" in 
  nice and renice, a wide range of implementation 
  strategies is possible. 

If I thught the OpenGroup copyrights allowed me to, I'd lift its
language almost verbatim. :-) Instead, I'd note the inspiration for
model,and make an informative reference.

But the point is that there is an existance proof of an underspecified
priority system having achieved success.

---

From: "Talpey, Thomas" <[email-masked]>
Date: Mon, 26 Feb 2007 14:10:04 -0500

>I believe Chuck Lever called this a UNIX-like "nice" priority system.

I must have missed some messages in this thread. The mail archive doesn't
show any of this discussion?

>about how the operating system is supposed to implement it.

I would make the observation that process schduling is very different
from NFS request scheduling. NFS requests are in effect run-to-completion.
As such, I don't see the server as having very many options when a new
request arrives, no matter what its priority. The request is by its very
nature eligible to run at its arrival.

For the server to handle the request otherwise, it would have to
begin processing the COMPOUND up to the point of decoding the
OP_SEQUENCE, then make a decision to either defer or continue
the processing. This, to me, seems very unnatural and certainly
inefficient. I would like to see the MUSTs SHOULDs and MAYs for
the server's end before committing to this proposal.

Tom.

---

From: Mike Eisler <[email-masked]>
Date: Mon, 26 Feb 2007 11:19:42 -0800 (PST)

> from NFS request scheduling. NFS requests are in effect

Implementors with thread-based repliers running on pre-emptive
kernels would disagree. One obvious implementation of priorities on
such kernels would be assign the priority based on the sessionid
priority and/or the per-SEQUENCE priority.

Event-driven implementations apparently have less obvious
implementations, but since I think event-driven can be mapped 
1-1 to threads, the obvious implementation can happen there too.

> the server's end before committing to this proposal.

I think it is all OPTIONAL.

---

From: Robert Gordon <[email-masked]>
Date: Mon, 26 Feb 2007 13:48:10 -0600

I think it should be, and had assumed it would be OPTIONAL.

Robert.

---

From: Benny Halevy <[email-masked]>
Date: Tue, 27 Feb 2007 10:22:34 +0200

> 1-1 to threads, the obvious implementation can happen there too.

In either case, the priority can also be applied for I/O scheduling
at the disk driver queue level.

> I think it is all OPTIONAL. 

Right.  My intent was to leave this open to the server since this is
highly dependent on the  server's architecture and implementation.
The clients can only express the relative importance of their requests.
If you want a stricter definition of the server MUST, SHOULD, or MAY do,
I think you're going in the direction of strictly defining many more QoS
parameters, not just priority.  The only thing I left in the proposal
is defining the "direction" so that higher priority_level values mean
"more important" stuff to do.