Subject: RE: [nfsv4] SET_PRIORITY: new op proposal

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 26 Feb 2007 12:58:40 -0500

As I understand the proposal, there are no requirements per se.
This is a case of the client proposes and the server disposes.
While there may be valid objections to leaving things so loose,
anything that imposes strict requirements seems like it will
add to implementation complexity and complicate the specification
process.  I think there would have to be a strong case made to
justify inclusion of such a feature at this stage in the 
development of v4.1.  

-----Original Message-----
From: Talpey, Thomas 
Sent: Monday, February 26, 2007 12:39 PM
To: [email-masked]; Halevy, Benny; Robert Gordon
Cc: [email-masked]; Welch, Brent; [email-masked]
Subject: Re: [nfsv4] SET_PRIORITY: new op proposal

But, what does the server actually do with the priority? Normally, any
operation is eligible once received at the server, does this mean the
server is required to implement strict queuing within the session, and
if so does it mean that it will need to delay the processing of requests
which are otherwise ready? If not, then what does it actually do?
Benny's
text documents the field, but doesn't describe the server's
requirements.

Call me confused, perhaps...

Tom.

At 12:21 PM 2/26/2007, Robert Gordon wrote:
>
>exactly what i was thinking..
>
>Robert.
>On Feb 26, 2007, at 7:35 AM, Mike Eisler wrote:
>
>> If we were talking about NFSv4.2, SET_PRIORITY would be a natural
>> way to add this operation.
>>
>> Since we are still working on NFSv4.1, we have an opportunity
>> to add this to "existing" NFSv4.1 core infrastructure.
>>
>> My counter proposal is:
>>
>> - add a priority field to CREATE_SESSION arguments (which the server
>> can
>>   confirm as it does various other session values). The priority is
>>   relative to other sessions, and represents the priority the server
>>   and client will take when servicing the fore and back channels of
>>   the session. For completeness, there can be separate fore and back
>>   channel priorities.
>>
>> - add a priority field to SEQUENCE and CB_SEQUENCE arguments
>>   and results. The priority represents priority of the
>>   COMPOUND/CB_COMPOUND relative to other COMPOUNDs/CB_COMPOUNDs
>>   executed on the session.
>>
>> - There is no way to update the session's priority once the session
is
>>   created. Just as there is no way to update the maximum request
size.
>>   Maybe, someday, we'll need such capabilities, but a general purpose
>>   operation to updated session and channel attributes should be
added.
>>   Thus far, I haven't seen a clamor for that. In the mean time, just
>>   create another session, and gracefully destroy the old session once
>>   replies for all outstanding COMPOUNDs have been received.
>>
>> _______________________________________________
>> nfsv4 mailing list
>> [email-masked]
>> https://www1.ietf.org/mailman/listinfo/nfsv4
>
>
>_______________________________________________
>nfsv4 mailing list
>[email-masked]
>https://www1.ietf.org/mailman/listinfo/nfsv4
>
>

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Benny Halevy <[email-masked]>
Date: Tue, 27 Feb 2007 10:50:49 +0200

Noveck, Dave wrote:
> process.  

agreed.

> I think there would have to be a strong case made to

I can't speak for application vendors but I can provide several
possible use cases:

* namespace related:
  - per volume: /production vs. /home
  - per subtree: /home/sales/* vs. /home/dev/*
  - per file type: *.tbl (standing for a database table) vs. *.mp3

* operation type:
  - better response time for GETATTRs and READDIRs vs. READ/WRITE (and
    from a workstation vs. from an application server).
  - lower priority for client readahead READs.
  - higher priority for WRITEs and COMMITs when client is under low memory
    pressure.

* deployment:
  - clients with different priorities: app servers in the data center
    vs. user workstations.
  - hosting farm SLAs (over shared storage ;-).
  - job scheduling in a cluster of clients (compute farm)
  - just translating the process nice priority into NFS I/O priority.

Benny

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 27 Feb 2007 22:20:57 -0500

On Tue, 2007-02-27 at 09:30 -0500, Peter Staubach wrote:
> It seems to me that most of these things can be better handled by the
> client making appropriate priority calls for itself and scheduling
> requests to the server to match its own priorities.
> 
> There are other solutions to issues such as wanting decreased latencies
> for operations such as GETATTR versus READ/WRITE, etc.  The use of a
> separate TCP connection can be used to help to avoid the blocking that a
> single TCP connection may introduce.  The metadata intensive style
> operations can be run over one connection while the data intensive
> operations could be run over another.  Even more connections could be
> used to differentiate between different client priorities if desired.
> 
> I am also somewhat dubious about the value of using these priorities
> for scheduling i/o on the server.  I/o is generally scheduled by the
> server given its own set of information, none of which is available to
> the client.  Having a client think that it can affect the i/o scheduling
> on the server for some benefit seems questionable to me.
> 
> The ability for the client to convey priorities to differentiate between
> different sessions or network connections may be interesting but still
> seems to assume a lot about the server implementation to me.

ACK. Nobody in this thread has so far presented a real-life use case
where this operation would show the "obvious benefits" that Benny talked
of in his introduction. If the client wants clever scheduling of
metadata operations, it can do so all by itself without having to
introduce new state for the server to carry around.

  Trond

---

From: Mike Eisler <[email-masked]>
Date: Tue, 27 Feb 2007 20:06:37 -0800 (PST)

--- Trond Myklebust <[email-masked]> wrote:

> > separate TCP connection can be used to help to avoid the blocking

How does a client indicate which TCP connection carries the
higher traffic? Until, NFSv4.1 there's been no way to indicate
that. With sessions a session can be tagged, and all the connections
bound it serviced in higher priority.

---

From: Benny Halevy <[email-masked]>
Date: Wed, 28 Feb 2007 14:49:48 +0200

Trond Myklebust wrote:
<snip>
> If the client wants clever scheduling of

There a couple of shortcomings with this approach:
First, the client can't schedule its operations with respect to
other clients' work and second, if the prioritization is done
only on the client side it will be very hard for the client
to achieve low latency for metadata ops while keeping the server
busy with a pipeline full of data I/O operations. Sending
metadata ops over a high priority session can allow the op
to pass I/O operations waiting in the server queues.

Benny

---

From: "Talpey, Thomas" <[email-masked]>
Date: Wed, 28 Feb 2007 07:49:15 -0500

>bound it serviced in higher priority. 

Yipes - does this proposal allow each request (or session) to *globally*
specify its priority? Holy denial of service, Batman.

I still want to see the server's MUSTs and SHOULDs w.r.t the client-specified
priority. If everything is a MAY, then I think we should ask ourselves a) what
good is it and b) why are we adding it now.

Tom.

---

From: [email-masked]
Date: Wed, 28 Feb 2007 08:26:36 -0500

Apart from the previously mentioned DOS problem, wouldn't adding a client controlled global priority also introduce issues of fairness and starvation between clients?
While the server could arbitrate that away somewhat, it seems like a good bit of added complexity.

Also wouldn't this assume that the clients are somehow aware of each other, and aware of their own status in the world?

If it was an administrative action, I can just see a sysadmin jacking the value up to maximum for all the clients.
I'm not necessarily opposed to the idea, I am only having trouble understanding the ultimate utility.

-- Tom Vanderputten

-----Original Message-----
From: Benny Halevy [mailto:[email-masked]]
Sent: Wed 2/28/2007 7:49 AM
To: Trond Myklebust
Cc: [email-masked]; Talpey, Thomas; Robert Gordon; Welch,Brent; [email-masked]; [email-masked]; Noveck, Dave
Subject: Re: [nfsv4] SET_PRIORITY: new op proposal
 
Trond Myklebust wrote:
<snip>

Benny

---

From: "Talpey, Thomas" <[email-masked]>
Date: Wed, 28 Feb 2007 08:37:50 -0500

>to pass I/O operations waiting in the server queues.

Okay, now I'm against this proposal. Your stated design goal is
in fact a denial of service - to prioritize one client's stream above
another's. It even sounds as if you envision dedicated server
threads (slots) to handle them.

Tom.

---

From: "Ellard, Daniel" <[email-masked]>
Date: Wed, 28 Feb 2007 08:54:21 -0500

> Sent: Wednesday, February 28, 2007 8:38 AM
> To: Benny Halevy
> Cc: [email-masked]
> metadata ops 
> >over a high priority session can allow the op to pass I/O operations 
> >waiting in the server queues.
> 
> Okay, now I'm against this proposal. Your stated design goal 
> is in fact a denial of service - to prioritize one client's 
> stream above another's. It even sounds as if you envision 
> dedicated server threads (slots) to handle them.

This does sound problematic.  As I believe someone pointed out a while
ago on this thread, it's the tragedy of the commons.

I'm not opposed to priorities as *hints* that the server can choose to
ignore, or normalize in whatever way it sees fit.

-Dan

---

From: Benny Halevy <[email-masked]>
Date: Wed, 28 Feb 2007 16:24:11 +0200

> ignore, or normalize in whatever way it sees fit.

Right.  That's exactly what is proposed.

---

From: Mike Eisler <[email-masked]>
Date: Wed, 28 Feb 2007 07:48:56 -0800 (PST)

> specify its priority? Holy denial of service, Batman.

Servers don't have to allow clients to set higher, service
denying priorities, if they don't want to.

Besides which, AUTH_SYS is the ultimate in denial of service,
and who cares? :-) 
> good is it and b) why are we adding it now.

How about this:

Priority is a 32 bit signed integer.

Zero means the peer doesn't care.  1 is the base
priority. Priorities other than "don't care" are in the
set { ..., -3, -1, 1, 3, ... } The higher the number,
the higher the priority.

Clients and servers MAY implement per-session priority. If
they do not, they express priority with the zero value.

At CREATE_SESSION time, the client offers two priorities,
one for the back channel, one for the fore channel, and
the server responds with confirmed values. The server MAY
override the fore channel priority with any value. The
server MUST NOT override the back channel priority, except
if it is not implementing priority, in which case it MAY
set the back channel priority to zero.

Clients and Servers MAY support per-message priorities.
At CREATE_SESSION time the client offers to support
per-message priorities on the fore and/or back channels,
and the server confirms the choices. If the client wants
per message priority on the fore channel the server MAY
deny it. If the client wants no per message priority on the
back channel, the server MUST agree.  If the per-session
priority of the channel is zero, then no per-message
priority on the channel is allowed, and the CREATE_SESSION
results will indicate that.  Each SEQUENCE and CB_SEQUENCE
operation takes a 32 bit priority formatted the same as
the session priority field.

If the client offers non-zero values for fore and/or
back channel priorities, and the server confirms (in
CREATE_SESSION results) non-zero values for fore and/or
back channel priorities, then the receiver MUST implement
priority If there are two sessions bound to the same
client_owner/server_owner paid, each with two different
non-zero priorities, then the receiver MUST give priority
to requests send over the higher priority session. The
means for how this is done is not specified.

If the same server_owner has sessions with two different
client_owners with different non-zero fore channel
priorities, then the server MUST give higher priority to
the requests with the high session priority. Similarly if
the same client_owner has sessions with two different
server_owners with different non-zero back channel
priorities, then the client MUST give higher priority to
the callback requests sent over the the higher priority
session.

For per message priorities, the receiver SHOULD give
higher priority processing to a message with a higher
priority than that of the session, and lower priority
processing to a message with lower priority than of the
session. The receiver SHOULD give messages with a higher
priority in one session higher priority processing over
messages with a lower priority (either per-message or
per-session) in another session that shares the same
client_owner/server_owner paid.  The receiver MAY give
messages with a higher priority in one session higher
priority processing over messages with a lower priority
(either per-message or per-session) in another session
that has a different client_owner/server_owner pair.

This document does not specify strategies for avoiding
starvation of serverive to lower priority sessions or
messages.

A server_owner might allow one client_owner to set a higher
priority than another, based on an implementation specific
configured priority control policy that considers client name,
client network address, client principal, etc. A server 
might also provide a way for server administrators to force
certain client_owners to have specific priorities on the
fore channel. For e.g. a particular client is consuming
most of the server's processing capability, and the sysadmin
forcibly terminates the client's session and
indicates that any subsequent sessions the client creates
will have a lower priority than over (well behaved)
clients.

[BTW, I have customers mention the above use case all the
time. There are other ways to implement it, but sessions
seem the most natural approach.]