Subject: [nfsv4] Request for an additional recommended attribute

From: [email-masked]
Date: Wed, 3 Oct 2007 17:26:08 -0400

In describing how the pNFS block client will implement fencing, it
became necessary to define the maximum time that an I/O can be
outstanding.  The client needs to communicate its value to the pNFS
server.  The most appropriate means of doing so seems to be to provide a
per server R/W recommended attribute, which clients can write.

I propose updating section 5.6 "Recommended Attributes - Definitions"
with the following

maximum_io_time  74 uint32    R/W   The default maximum time that a
client

                                    I/O to a data server can be active.

                                    Clients may update this value.

This parameter is defined in the next version of the pNFS block layout
spec as follows:

--

"maximum_io_time" is the maximum time it can take for a client I/O to
the storage system to either complete or fail; this value is often 30
seconds or 60 seconds, but may be longer in some environments. .  If the
maximum client I/O time cannot be bounded, this timed lease mechanism
MUST NOT be used.  The client can use GETATTR to query the server's
default setting of "maximum_io_time".  The server must respond with the
maximum I/O time in seconds.  If the client's maximum I/O time is
greater than the server's default, then the client MUST use SETATTR to
inform the server of its maximum_I/O time.

--

Are there comments on introducing this new attribute into the NFSv4
minor version 1 spec?  Is this a valid means of communicating a client
specific value to the server?

-Jason Glasgow

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "Talpey, Thomas" <[email-masked]>
Date: Thu, 04 Oct 2007 07:34:24 -0400

It's not clear if this text is making a requirement that every NFSv4.1 server
implement a bounded maximum I/O time, or if the client is able to request
one. Also, it is not clear to me whether this time is global across all I/Os
at the server, all I/Os for this clientid or sessionid, or whether it's per-file.

Can you clarify? If it's a server-global mandatory requirement, I have some
strong concerns. 

Is this really fencing btw? Usually I think of fencing as a way for a server
to deny access to clients which fall out of a cluster.

Tom.

At 05:26 PM 10/3/2007, [email-masked] wrote:
>Content-class: urn:content-classes:message
>Content-Type: multipart/alternative;
>        boundary="----_=_NextPart_001_01C80604.03F942D3"
>
>In describing how the pNFS block client will implement fencing, it became necessary to define the maximum time that an I/O can be outstanding.  The client needs to communicate its value to the pNFS server.  The most appropriate means of doing so seems to be to provide a per server R/W recommended attribute, which clients can write.
> 
>I propose updating section 5.6 "Recommended Attributes – Definitions" with the following
> 
>maximum_io_time  74 uint32    R/W   The default maximum time that a client
>This parameter is defined in the next version of the pNFS block layout spec as follows:
> 
>--
>"maximum_io_time" is the maximum time it can take for a client I/O to the storage system to either complete or fail; this value is often 30 seconds or 60 seconds, but may be longer in some environments. .  If the maximum client I/O time cannot be bounded, this timed lease mechanism MUST NOT be used.  The client can use GETATTR to query the server's default setting of "maximum_io_time".  The server must respond with the maximum I/O time in seconds.  If the client's maximum I/O time is greater than the server's default, then the client MUST use SETATTR to inform the server of its maximum_I/O time.
>--
> 
>Are there comments on introducing this new attribute into the NFSv4 minor version 1 spec?  Is this a valid means of communicating a client specific value to the server?
> 
>-Jason Glasgow
> 
>

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 4 Oct 2007 10:16:28 -0400

My understanding is that this is to take the place of fencing, i.e. to
provide to the protocol implementation time-based promises which can be
relied upon to avoid a requirement for actual fencing.  David Black's
concern has been that requiring actual fencing support would limit the
deployment of pnfs for blocks which limitation he believes would be a
bad thing.

-----Original Message-----
From: Talpey, Thomas 
Sent: Thursday, October 04, 2007 7:34 AM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [nfsv4] Request for an additional recommended attribute

It's not clear if this text is making a requirement that every NFSv4.1
server
implement a bounded maximum I/O time, or if the client is able to
request
one. Also, it is not clear to me whether this time is global across all
I/Os
at the server, all I/Os for this clientid or sessionid, or whether it's
per-file.

Can you clarify? If it's a server-global mandatory requirement, I have
some
strong concerns. 

Is this really fencing btw? Usually I think of fencing as a way for a
server
to deny access to clients which fall out of a cluster.

Tom.

with the following
client

---

From: "Everhart, Craig" <[email-masked]>
Date: Thu, 4 Oct 2007 11:22:57 -0400

How is it that the client will know this and the server will not,
anyway?  Is there something that puts the client in the driver's seat
here, or might it be the case that the server will know and will need to
tell the client?
		Craig 

> From: Noveck, Dave 
> Sent: Thursday, October 04, 2007 10:16 AM
> To: Talpey, Thomas; [email-masked]
> Cc: [email-masked]
> Subject: RE: [nfsv4] Request for an additional recommended attribute
> 
> My understanding is that this is to take the place of 
> fencing, i.e. to provide to the protocol implementation 
> time-based promises which can be relied upon to avoid a 
> requirement for actual fencing.  David Black's concern has 
> been that requiring actual fencing support would limit the 
> deployment of pnfs for blocks which limitation he believes 
> would be a bad thing.
>

---

From: [email-masked]
Date: Thu, 4 Oct 2007 14:51:47 -0400

Dave and Tom,

> deployment of pnfs for blocks which limitation he believes would be a

Yes, it's in support of that mechanism - the basic idea is that the
client
stops using a layout when the client's leases expire (the actual timing
is
somewhat subtler due to client-server communication latency).  The
server
needs to know how much longer it has to wait for client I/Os in flight
before it can safely take back resources.  The details are in the latest
(just posted) version of the block layout draft.

server
request
> one.

It's neither, this is the client telling the server what the max I/O
time
is for client I/O done against pNFS block layouts.  Support for it would
only be required if the pNFS block layout is supported (although I'd
have
no problem if other layouts found this useful).

> Also, it is not clear to me whether this time is global across all
I/Os
> at the server, all I/Os for this clientid or sessionid, or whether
it's per-file.

Per-clientid.  The maximum I/O time for a client I/O may depend on the
client's I/O stack, so it's better to have the client tell the server
than
have the server guess (although 60 seconds will usually be a good
guess).
This is also the answer to Craig's question - the client knows the
client's
I/O stack significantly better than the server does, especially if the
client and server aren't running the same OS.

Forcing the client to tell the server is also a way to (try to) catch
clients that may be in unbounded I/O territory and keep them from using
the block layout.  I'm not sure whether this aspect is important in
practice (it may turn out to be a blame-redirection mechanism rather
than a damage-prevention mechanism).  In the reverse direction, allowing
clients with lower I/O bounds to speed up failure processing is useful.

some
> strong concerns. 

It's only for clients and servers that implement the pNFS block layout,
unless some other layout would find this useful.

> server to deny access to clients which fall out of a cluster.

Dave Noveck's language at the start of this message is a better
description
- in essence, for the block layout, we're prepared to trust a client to
fence itself if it's lost communication with the server for long enough
to lose its leases.

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 4 Oct 2007 14:57:17 -0400

> telling the server what the max I/O time
> is for client I/O done against pNFS block layouts

"telling" doesn't really cover it.

It is making a promise or contract, and the correctness of the protocol
will depend on that promise being honored.  This is not informative in
the normal sense. 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Thursday, October 04, 2007 2:52 PM
To: Noveck, Dave; Talpey, Thomas
Cc: [email-masked]
Subject: RE: [nfsv4] Request for an additional recommended attribute

Dave and Tom,

server
request

I/Os

some

Thanks,
--David

---

From: [email-masked]
Date: Thu, 4 Oct 2007 21:29:55 -0400

I agree - it's clearly a promise.  --David 

> From: Noveck, Dave [mailto:[email-masked]] 
> Sent: Thursday, October 04, 2007 2:57 PM
> To: Black, David; Talpey, Thomas
> Cc: [email-masked]
> fencing, i.e. to
> would be a