Subject: [nfsv4] Layout stateids

From: Spencer Shepler <[email-masked]>
Date: Fri, 27 Jul 2007 00:08:55 -0500

Working from Mike's stateids in layouts proposal, the ensuing thread
of discussion and then the continued discussion during Monday's WG
meeting, I offer the following proposal for stateids in layouts.

The main purpose of this change is to ensure the client has a layout
that is correctly derived from a valid OPEN of the file and to tie the
layout to the lease held by the client.  Of secondary concern is to
increase the potential for parallelism between LAYOUTGET and
LAYOUTRETURN, and to ease the data server's validation of layouts.

Some assumptions/observations:

Layouts are similar to byte range file locks; they are range based,
can merge, can be split.  Thus, the ordering of LAYOUTGET and
LAYOUTRETURN matters greatly to the shared view the client and server
has for layout ranges.  The addition of stateids for LAYOUTGET and
LAYOUTRETURN will allow for the ordering of execution at the server to
be communicated to the client via the stateid4.seqid field just like
the OPEN and LOCK sequencing is communciated.  Note that the iomode
provide two distinct sets of layout ranges -- an upgrade path from the
READ iomode to the READ/WRITE iomode does NOT exist.

Layouts are similar to delegations in that the stateid is valid beyond
the life of the OPEN stateid (file can be closed and the client can
continue holding a valid layout stateid).  The other similarity with
delegations is that layout stateids are associated with the
filehandle/client pairing (NOT filehandle/"layout owner" like byte
range file locks).
	
The presence of layout stateids does not remove the utility of
Sessions' slot referencing mechanism on CB_SEQUENCE.  This was the
conclusion of the discussion during the NFSv4 WG meeting on Monday
(Brent Welch may send an email with a discussion of the general issues
of sequencing and race conditions).

Proposal:

LAYOUTGET will take a stateid and return one in the results.

For the first LAYOUTGET, the client MUST use a valid stateid as
returned by OPEN, OPEN_DOWNGRADE, LOCK, LOCKU, WANT_DELEGATION (open,
lock, or delegation stateids).  For subsequent LAYOUTGETs, the client
will provide a layout stateid.

The client MUST hold a valid layout for I/O to any storage
device type.

The use of special stateids for data server I/O is not allowed.

Layout stateids follow the sequencing and use rules of all other
stateid types.

Same stateid4.other value is returned for all layouts on the file
(e.g. multiple LAYOUTGETs and LAYOUTRETURNs to the same file).

LAYOUTRETURN will take a layout stateid and return a stateid.  If the
server has recalled a CB_LAYOUTRECALL and the client is returning a
layout as a result, the client will also provide the layout stateid
specified in the CB_LAYOUTRECALL.

LAYOUTCOMMIT will take a layout stateid; it will not return a stateid.

Layout stateids persist across CLOSE and DELEGRETURN.

As noted above, for the NFSv4.1 layout type, READ and WRITE to a data
server will continue to require an open, lock, or delegation stateid.
The data server SHOULD be able to map these stateids to the granted
layout to make sure the client has a layout, but in any case the data
server MUST reject I/Os to files that do not have layouts.

As with other state, layout stateids might become invalid
when the lease expires.

Spencer

-------
struct LAYOUTCOMMIT4args {
	/* CURRENT_FH: file */
	offset4			loca_offset;
	length4			loca_length;
	bool			loca_reclaim;
+	stateid4		loca_stateid;
	newoffset4		loca_last_write_offset;
	newtime4		loca_time_modify;
	newtime4		loca_time_access;
	layoutupdate4		loca_layoutupdate;
};
...
struct LAYOUTGET4args {
	/* CURRENT_FH: file */
	bool			loga_signal_layout_avail;
	layouttype4		loga_layout_type;
	layoutiomode4		loga_iomode;
	offset4			loga_offset;
	length4			loga_length;
	length4			loga_minlength;
+	stateid4		loga_stateid;
	count4			loga_maxcount;
};
struct LAYOUTGET4resok {
          bool               logr_return_on_close;
+	stateid4	   logr_stateid;
          layout4		   logr_layout;
};
union LAYOUTGET4res switch (nfsstat4 logr_status) {
case NFS4_OK:
	LAYOUTGET4resok	    logr_resok4;
case NFS4ERR_LAYOUTTRYLATER:
	bool		    logr_will_signal_layout_avail;
default:
	void;
};
+union layoutrecall_stateid switch (bool lrs_present) {
+case TRUE:
+	stateid4		lrs_stateid;
+case FALSE:
+	void;
+};
+
struct LAYOUTRETURN4args {
	/* CURRENT_FH: file */
	bool			lora_reclaim;
+	stateid4		lora_stateid;
+	recall_stateid		lora_recallstateid;
	layouttype4		lora_layout_type;
	layoutiomode4		lora_iomode;
	layoutreturn4		lora_layoutreturn;
};
struct LAYOUTRETURN4res {
	nfsstat4	    lorr_status;
+	stateid4	    lorr_stateid;
};
...
enum layoutrecall_type4 {
	LAYOUTRECALL4_FILE = 1,
	LAYOUTRECALL4_FSID = 2,
	LAYOUTRECALL4_ALL  = 3
};
struct layoutrecall_file4 {
	nfs_fh4		lor_fh;
	offset4		lor_offset;
	length4		lor_length;
+	stateid4	lor_stateid;
};
union layoutrecall4 switch(layoutrecall_type4 recalltype) {
case LAYOUTRECALL4_FILE:
	layoutrecall_file4 lor_layout;
case LAYOUTRECALL4_FSID:
	fsid4		   lor_fsid;
case LAYOUTRECALL4_ALL:
	void;
};
struct CB_LAYOUTRECALL4args {
	layouttype4		clora_type;
	layoutiomode4		clora_iomode;
	bool			clora_changed;
	layoutrecall4		clora_recall;
};

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Benny Halevy <[email-masked]>
Date: Sun, 29 Jul 2007 20:14:40 +0300

> (e.g. multiple LAYOUTGETs and LAYOUTRETURNs to the same file).

I'd restrict that per layout_type since the encapsulated layout state
per layout_type is independent and LAYOUT{GET,COMMIT,RETURN} as well
as CB_LAYOUTRECALL carry the layout_type (CB_RECALL_ANY doesn't but
it will not a have layout stateid either).

> As with other state, layout stateids might become invalid

Thanks for incorporating the feedback for Mike's proposal into this one.

To get that into the spec I think we need to provide more details about 
the following use cases (to start with):

When should the server increase the sequence part of the stated?
How strict do we want to be in the spec about this?

What layout stateid should the client use on subsequent LAYOUTGET
and LAYOUTRETURN operations?

Can the client send multiple such operations in parallel?
If so, what values should it use for the stateid?
Currently we don't allow the client to send LAYOUTGET(s)
concurrently with LAYOUTRETURN(s) but we do allow the
client to send multiple concurrent LAYOUTGETs or
multiple concurrent LAYOUTRETURNs since the order
of execution between these, respectively, should not
matter.  Sending LAYOUTGET(s) in parallel to LAYOUTRETURN(s)
may require strict ordering between the two.  Can the client
imply the order using the sent layout stateid by incrementing
the stateid4.seqid?  What should the server do if it gets
the operations out of order?

Benny

> +	recall_stateid		lora_recallstateid;

warning: unresolved symbol :)
the type above is defined as "layoutrecall_stateid"
but referred here as "recall_stateid"

> struct LAYOUTRETURN4res {

Hmm, the returned stateid should be valid only for lorr_status == NFS_OK,
like this:

union LAYOUTRETURN4res switch (nfsstat4 lorr_status) {
case NFS4_OK:
	stateid4	    lorr_stateid;
default:
	void;
};

---

From: [email-masked]
Date: Sun, 29 Jul 2007 19:29:58 -0400

Spencer,

> Thanks for incorporating the feedback for Mike's proposal 

Ditto.

> it will not a have layout stateid either).

I agree with Benny, although I'd be tempted to make use of different
layout types on the same file at the same time a "MUST NOT" on the
grounds that the opportunity for confusion probably outweighs the
benefits of this sort of cleverness.

> How strict do we want to be in the spec about this?

We need to be very specific about this, in order to avoid client
confusion about what's going to happen.  EMC's pre-pNFS protocol
[FMP] increments for every LAYOUTGET and LAYOUTRETURN operation
(actually their FMP equivalents and every recall.  The resulting
strict sequencing makes it easy for the clients to figure out
what has been processed vs. will be rejected when the callback
shows up.  In addition, it sorts out the LAYOUTGET/LAYOUTRETURN
ordering.  The alternate approach of only incrementing on callback
may require the client to wait for results of outstanding ops
on callback, which delays client processing of the callback.

> and LAYOUTRETURN operations?

With strict sequencing, it uses the value that came back from
the last operation, but ...

> Can the client send multiple such operations in parallel?

Good question.  At the very least, the ability to ship multiple
operations in a COMPOUND ought to be supported.  Multiple operations
in parallel from a single client is dangerous, as one can construct
scenarios in which the client and server get out of sync due to
not having the same understanding of which one went first when
overlapping ranges are involved.  I would hope that most of
the important situations that arise can be covered by loading
up all the operations into a single COMPOUND.

> concurrently with LAYOUTRETURN(s)

And we need to retain that as this is the situation in which
the client and server can get out of sync on layout state if
there's any overlap.  Keep in mind that the server may
create overlap when the client doesn't expect it by tracking
layout ranges at coarser granularity than the client requests
them  - the server has to be the final authority on how it
does this.

> but we do allow the client to send multiple concurrent LAYOUTGETs
> or multiple concurrent LAYOUTRETURNs since the order
> matter.  

Ok.

> Sending LAYOUTGET(s) in parallel to LAYOUTRETURN(s)
> may require strict ordering between the two.

I think it does require strict ordering - see above rationale.

> Can the client

If the client increments the seqid for concurrent ops and the
server increments the seqid on a recall, occurrence of a recall
in the middle of a bunch of concurrent client pNFS ops becomes
peculiar - I think the client may have to wait for the results of
all of its outstanding pNFS ops on that layout, as it does not
know which ops have executed or will execute (as the server may
execute an op after the recall courtesy of the client's seqid
increment [that seems wrong]).

I'd prefer to get the "multiple ops in a COMPOUND" case (where
the ops are clearly sequenced) working right before attacking the
concurrent ops case.  I'm concerned that concurrent client operations
may need to carry two sequence numbers for two purposes:
- One to indicate intended order of client operations.
- One to indicate how many recalls the client saw before
	issuing this operation.
If the client operations are strictly sequenced, these collapse
into one sequence number.  I'd prefer not to go to two sequence
numbers, with the hope that the COMPOUND case covers the
proverbial "80%" of what's needed in practice.

At a high level, if we stick to one sequence number, the resulting
design tradeoff appears to be between:
- Concurrent sets of LAYOUTGET and LAYOUTRETURN ops (but not a
	mixture - client has to not allow a mixture).
- Fast client processing of recalls when there are outstanding ops.
Strict sequencing of client operations is the simpler approach,
it gets us the latter (which is what's wanted if there's contention),
and should enable clients to mitigate the downside via use of
multiple ops in a COMPOUND.

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

> -----Original Message-----
> From: Benny Halevy [mailto:[email-masked]] 
> Sent: Sunday, July 29, 2007 1:15 PM
> To: Spencer Shepler
> Cc: [email-masked]
> Subject: Re: [nfsv4] Layout stateids
> 
> Spencer Shepler wrote:
> > 
> > The client MUST hold a valid layout for I/O to any storage device
type.

---

From: "Noveck, Dave" <[email-masked]>
Date: Sun, 29 Jul 2007 20:01:10 -0400

> benefits of this sort of cleverness. 

If we are talking about for the same client at the same time, I would
agree.

I think there may in the future be reasonable use cases for the case
where we have different layout type for the same for different clients.
Suppose you have a layout type that is an improvement-extension of an
existing one.  You very well might want the ability to provide the new
type to clients that support it while providng the old to those who
don't.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Sunday, July 29, 2007 7:30 PM
To: [email-masked]; [email-masked]
Cc: [email-masked]
Subject: RE: [nfsv4] Layout stateids

Spencer,

Ditto.

> as CB_LAYOUTRECALL carry the layout_type (CB_RECALL_ANY doesn't but it

> will not a have layout stateid either).

We need to be very specific about this, in order to avoid client
confusion about what's going to happen.  EMC's pre-pNFS protocol [FMP]
increments for every LAYOUTGET and LAYOUTRETURN operation (actually
their FMP equivalents and every recall.  The resulting strict sequencing
makes it easy for the clients to figure out what has been processed vs.
will be rejected when the callback shows up.  In addition, it sorts out
the LAYOUTGET/LAYOUTRETURN ordering.  The alternate approach of only
incrementing on callback may require the client to wait for results of
outstanding ops on callback, which delays client processing of the
callback.

> What layout stateid should the client use on subsequent LAYOUTGET and 
> LAYOUTRETURN operations?

With strict sequencing, it uses the value that came back from the last
operation, but ...

Good question.  At the very least, the ability to ship multiple
operations in a COMPOUND ought to be supported.  Multiple operations in
parallel from a single client is dangerous, as one can construct
scenarios in which the client and server get out of sync due to not
having the same understanding of which one went first when overlapping
ranges are involved.  I would hope that most of the important situations
that arise can be covered by loading up all the operations into a single
COMPOUND.

> Currently we don't allow the client to send LAYOUTGET(s) concurrently 
> with LAYOUTRETURN(s)

And we need to retain that as this is the situation in which the client
and server can get out of sync on layout state if there's any overlap.
Keep in mind that the server may create overlap when the client doesn't
expect it by tracking layout ranges at coarser granularity than the
client requests them  - the server has to be the final authority on how
it does this.

> but we do allow the client to send multiple concurrent LAYOUTGETs or 
> multiple concurrent LAYOUTRETURNs since the order of execution between

> these, respectively, should not matter.

Ok.

> Sending LAYOUTGET(s) in parallel to LAYOUTRETURN(s) may require strict

> ordering between the two.

> Can the client
> imply the order using the sent layout stateid by incrementing the 
> stateid4.seqid?  What should the server do if it gets the operations 
> out of order?

If the client increments the seqid for concurrent ops and the server
increments the seqid on a recall, occurrence of a recall in the middle
of a bunch of concurrent client pNFS ops becomes peculiar - I think the
client may have to wait for the results of all of its outstanding pNFS
ops on that layout, as it does not know which ops have executed or will
execute (as the server may execute an op after the recall courtesy of
the client's seqid increment [that seems wrong]).

I'd prefer to get the "multiple ops in a COMPOUND" case (where the ops
are clearly sequenced) working right before attacking the concurrent ops
case.  I'm concerned that concurrent client operations may need to carry
two sequence numbers for two purposes:
- One to indicate intended order of client operations.
- One to indicate how many recalls the client saw before
	issuing this operation.
If the client operations are strictly sequenced, these collapse into one
sequence number.  I'd prefer not to go to two sequence numbers, with the
hope that the COMPOUND case covers the proverbial "80%" of what's needed
in practice.

At a high level, if we stick to one sequence number, the resulting
design tradeoff appears to be between:
- Concurrent sets of LAYOUTGET and LAYOUTRETURN ops (but not a
	mixture - client has to not allow a mixture).
- Fast client processing of recalls when there are outstanding ops.
Strict sequencing of client operations is the simpler approach, it gets
us the latter (which is what's wanted if there's contention), and should
enable clients to mitigate the downside via use of multiple ops in a
COMPOUND.

Thanks,
--David

type.
> as CB_LAYOUTRECALL carry the layout_type (CB_RECALL_ANY doesn't but it

> will not a have layout stateid either).
> 
> > 
> > As with other state, layout stateids might become invalid when the 
> > lease expires.
> > 
> > Spencer
> 
> What layout stateid should the client use on subsequent LAYOUTGET and 
> LAYOUTRETURN operations?
> 
> Currently we don't allow the client to send LAYOUTGET(s) concurrently 
> with LAYOUTRETURN(s) but we do allow the client to send multiple 
> concurrent LAYOUTGETs or multiple concurrent LAYOUTRETURNs since the 
> order of execution between these, respectively, should not matter.  
> Sending LAYOUTGET(s) in parallel to LAYOUTRETURN(s) may require strict

> ordering between the two.  Can the client imply the order using the 
> sent layout stateid by incrementing the stateid4.seqid?  What should 
> the server do if it gets the operations out of order?
> 
> Benny
> 
> > 
> > -------
> > union LAYOUTGET4res switch (nfsstat4 logr_status) { case NFS4_OK:
> > +union layoutrecall_stateid switch (bool lrs_present) { case TRUE:
> > +	stateid4		lrs_stateid;
> > +case FALSE:
> > +	void;
> > +};
> > +
> Hmm, the returned stateid should be valid only for lorr_status == 
> NFS_OK, like this:
> 
> union LAYOUTRETURN4res switch (nfsstat4 lorr_status) { case NFS4_OK:
> 	stateid4	    lorr_stateid;
> default:
> 	void;
> };
> 
> > ...
> > union layoutrecall4 switch(layoutrecall_type4 recalltype) { case 
> > LAYOUTRECALL4_FILE:

---

From: [email-masked]
Date: Sun, 29 Jul 2007 22:00:42 -0400

Dave,

> type to clients that support it while providing the old to those who
> don't.

We are talking about "for the same client at the same time" as layout
stateids are necessarily per-client.  Support of different layout types
for different clients at the same time on the same file is up to the
server - it has the discretion to "just say no" to a layout request
which would create a combination that the server doesn't want to have
to deal with.

Thanks,
--David

---

From: Spencer Shepler <[email-masked]>
Date: Mon, 30 Jul 2007 16:11:20 -0500

> it will not a have layout stateid either).

This is fine with me. Seems like we have consensus with the minor
adaptation that Dave Noveck provided.

>> Layout stateids follow the sequencing and use rules of all other

Was an reference to the way file locking and open processing is
done at the server with respect to parallelism and sequencing
choices by the client.

We had a thread of discussion earlier this month about how the
server was required to update the stateid4.seqid on each
operation.  In this case, LAYOUTGET and LAYOUTRETURN in that
it signifies a change in the layout "state" and if the client
is sending these operations in parallel it can use the stateid4.seqid
as a method of sorting the execution at the server.

We also came to the conclusion that if the client doesn't care
about the sequence in which is sends the operations, it will
set the stateid4.seqid value to 0(zero) and the server will
ignore the value in essence (first come first served).  If the
client prefers to sequence the operations, then it will leave
the stateid4.seqid value present in the request's stateid and
the server will do the "old stateid" checking that is
present in NFSv4.0.

Again, modeling it after file locking (ranges and sequencing) we
should have a model that is common enough between the two to
be well understood.

As for callback processing, I intended that the server provide
the stateid4 (both seqid and other) at the time it did the callback.
The client can then look at the stateid4.seqid to determine if the
LAYOUTGET/LAYOUTRETURN was executed before or after the callback
was instantiated by the server.  The client is still obligated to
do a LAYOUTRETURN with the callback stateid4 even if it raced
with the server on a LAYOUTRETURN for the same range.  This is
done to be very clear that the client acknowledged and processed
fully the callback request.

If the server receives LAYOUTGETs for the range in callback,
it can refuse to provide them and the client will eventually
understand that the range has been under the callback request.

Spencer

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 30 Jul 2007 17:22:37 -0400

[specific details of model not copied]

> should have a model that is common enough between the two to

In fact, with the new state manegement chapter, this is only 
described once in that general chapter and it applies to all
types of stateids.

> fully the callback request.

This currently isn't stated in the state management chapter.  If 
it is a general requirement for calbacks, as opposed to something
layout-specific, it should be mentioned in the state management 
chapter.

-----Original Message-----
From: Spencer Shepler [mailto:[email-masked]] 
Sent: Monday, July 30, 2007 5:11 PM
To: [email-masked]
Subject: Re: [nfsv4] Layout stateids

Spencer

---

From: Spencer Shepler <[email-masked]>
Date: Mon, 30 Jul 2007 16:25:22 -0500

> layout-specific, it should be mentioned in the state management

It is layout specific in my view.

Spencer

---

From: Benny Halevy <[email-masked]>
Date: Tue, 31 Jul 2007 09:46:00 +0300

> which would create a combination that the server doesn't want to have

Well, the server can "just say no" to a client that has an outstanding
layout of one type and is asking for another type, if the server wishes
to do so.  Regardless if we disallow this in the protocol or not,
I think we better have a specific error to represent this condition.
This error would tell the client it needs to completely return the layout
it currently holds before asking for a new one with a different layout type.
Something in the lines of NFS4ERR_LAYOUTTYPE_CONFLICT.

---

From: Benny Halevy <[email-masked]>
Date: Tue, 31 Jul 2007 10:09:54 +0300

>> described once in that general chapter and it applies to all

Yup. This should work.

>>> fully the callback request.

I like that.  One semantic nit though: alternatively, if a previous
LAYOUTRETURN returned the whole recalled range and the client waits
for it to complete before replying to the callback, it can (and
should) return a NFS4ERR_NOMATCHING_LAYOUT error to the callback.