Subject: Re: [nfsv4] Draft 21: parallel LAYOUTGET operations and thelayoutstateid

From: "Noveck, Dave" <[email-masked]>
Date: Sat, 1 Mar 2008 14:19:46 -0500

I spoke carelessly.

The stateid is not passed to OPEN but a zero stateid-seqid field does
support parallel open in that you can issue READs and WRITEs while
parallel OPENs are going on without worry that that will invalidate the
stateid-seqid value you sent in the IO request.

The comparison with LAYOUTGET is that OPEN doesn't need the
stateid-sequence value (or any part of the stateid) on input and I can
see where it is convenient for LAYOUTGET to require the "other" field of
the stateid, but I don't see why it should depend on the "seqid" field.

The model for OPEN is that the client doesn't have to determine an order
of execution but it may need the server to communicate the order which
occurred, for which it uses the order of the seqid fields for the
stateid returned.  So I guess the question is why that sort of model
would not work just as well for LAYOUTGET.

-----Original Message-----
From: Mike Eisler [mailto:[email-masked]] 
Sent: Friday, February 29, 2008 9:27 PM
To: NFSv4
Subject: Re: [nfsv4] Draft 21: parallel LAYOUTGET operations and
thelayoutstateid

--- Spencer Shepler <[email-masked]> wrote:

> 
> On Feb 29, 2008, at 4:04 PM, Noveck, Dave wrote:
> 
> > I don't understand this.  There are all these references to parallel

> > opens as a model but parallel opens handle this by using zero seqid 
> > as a wildcard but the layout stuff explicitly rules out

???

OPEN does not take a stateid as an argument. It takes a seqid as an
argument, but the NFSv4.1 server MUST ignore it.

Here is the text parallel opens:

  Unlike the case of NFSv4.0, in which OPEN operations for the same
open-owner are inherently serialized because of the owner-based seqid,
multiple OPENs for the same open-owner may be done in parallel. When
clients do this, they may encounter situations in which, because of the
existence of hard links, two OPEN operations may turn out to open the
same file, with a later OPEN performed being an upgrade of the first,
with this fact only visible to the client once the operations complete.

   In this situation, clients may determine the order in which the OPENs
were performed by examining the stateids returned by the OPENs.
Stateids that share a common value of the "other" field can be
recognized as having opened the same file, with the order of the
operations determinable from the order of the "seqid" fields, mod any
possible wraparound of the 32-bit field.

   When the possibility exists that the client will send multiple OPENs
for the same open-owner in parallel, it may be the case that an open
upgrade may happen without the client knowing beforehand that this could
happen. Because of this possibility, CLOSEs and OPEN_DOWNGRADEs, should
generally be sent with a non-zero seqid in the stateid, to avoid the
possibility that the status change associated with an open upgrade is
not inadvertently lost. 

> 
> > zero being sent as a wildcard seqid.  It isn't clear to me why this
>  
> > is ruled out however.

Wildcard seqids are not part of the plan for detecting the issue
discussed in section 9.10. Non-zero seqids are there to detect order.
That's what seqids do for layouts too. What is there now in draft-21 is
for dealing with parallel LAYOUTGETs sent after the first LAYOUTGET. 

> I worked through the scenarios that make the wording in this section 
> plausible (even the non-zero requirement) but have not had time to get

> it written down.  Will do so as soon as I get the time.

Thanks. Do find the time soon, like before March 7.

> > -----Original Message-----
> > From: William A. (Andy) Adamson [mailto:[email-masked]]
> > Sent: Friday, February 29, 2008 4:39 PM
> > To: NFSv4
> > Subject: [nfsv4] Draft 21: parallel LAYOUTGET operations and the 
> > layoutstateid
> >
> > I brought this up with Spencer at the Linux Storage and File System
>  
> > Workshop. There is no description of how parallel LAYOUTGET 
> > operations are processed past the parallel use of a non-layout 
> > stateid (when no layout stateid has been established).

I never considered that this was a requirement to support, since unlike
parallel OPENs, LAYOUTGET only takes a filehandle, and so a client ought
to be able to serialize the first LAYOUTGET.
We are getting in true crunch time on the specification ...
so pray for a miracle from St. Spencer.

> >
> > 12.5.3.  Layout Stateid
> >
> >    As with all other stateids, the stateid consists of a "seqid"
> and
> >    "other" field.  Once a layout stateid is changed, the "other"
> field
> >    will stay constant unless the stateid is revoked, or the client
> >    returns all layouts on the file and the server disposes of the
> >    stateid.  The "seqid" field is initially set to one, and is
> never
> >    zero on any NFSv4.1 operation that uses layout stateids, whether
> it
> >    is fore channel or backchannel operation.  After the layout
> stateid
> >    is established, the "seqid" is incremented by the server in each
> >    subsequent LAYOUTGET and LAYOUTRETURN response, and in each
> >    CB_LAYOUTRECALL request.  When the client fully processes the
> >    response to a LAYOUTGET or LAYOUTRETURN, or fully processes the
> >    arguments of a CB_LAYOUTRECALL, it MUST use the seqid of the 
> > stateid
> >    of the reply from LAYOUTGET and LAYOUTRETURN, or the seqid of
> the
> >    stateid in the arguments of CB_LAYOUTRECALL, on subsequent calls
> to
> >    LAYOUTGET or LAYOUTRETURN.
> >
> > The last sentence above allows for the client layout stateid to 
> > decrease - if parallel LAYOUTGET operations are sent, the order of
> 
> > the returns could be such that a lower stateid.seqid arrives from 
> > the server after a higher stateid.seqid. Not good.

It isn't obvious to me why this is bad. However, assuming we allowed
parallel LAYOUTGETs before a layout stateid was established, what would
happen is that one LAYOUTGET would have a seqid of 2, and the other
would be 1. This would unambiguously tell the client what happenned. So
I think it is good.

> >
> >
> >                                                               The 
> > client and server use the "seqid" of
> >    the layout stateid for the following purposes:
> >
> >    o  Permit the client to send parallel LAYOUTGET operations on
> the
> >       same file.  As with parallel opens (see Section 9.10) the use
> of
> >       the sequence ID allows a client to avoid serializing
> LAYOUTGET
> >        ^^^^^^^^
> > the zero sequence ID

Again I am confused. No my intent was not to write "zero sequence ID".
(however, the use of the term "sequence ID" is very bad form here,
because it could be confused with a "sequence ID" on a slot. I fixed
that just now).

> >
> >       operations.  If LAYOUTGETs were serialized, especially non-
> >       overlapping LAYOUTGETs, then non-overlapping I/Os to storage
> >       devices would in turn be effectively serialized with each
> other.
> >       In the event parallel LAYOUTGET operations are sent with a
> non-
> >       layout stateid (because the client does not yet have a layout
> >       stateid), the successful responses MUST have the same "other"
> >       field in the LAYOUTSTATEID, and each response with a unique
> >       "seqid", where the lowest "seqid" is one, and the highest 
> > "seqid"
> >       is equal to the count of parallel LAYOUTGET operations invoked

> > on
> >       the non-layout stateid.
> >
> > What about after the layout stateid is established? Can the client
> 
> > still send LAYOUTGET operations in parallel? If so, what are the

Yes, that's the whole point of layout stateid.
 
> > rules - does the client keep higher stateid.seqids and ignore lower
>  
> > stateid.seqids? The server will receive multiple LAYOUTGET

``The client MUST always use a "seqid" that was returned by the server
in a reply to a LAYOUTGET or LAYOUTRETURN operation, or sent by the
server in a CB_LAYOUTRECALL operation. It MUST only use such a seqid
after processing the LAYOUTGET and LAYOUTRETURN results, or the
CB_LAYOUTRECALL request. Simply seeing the result or the CB_LAYOUTRECALL
request is not sufficient cause to use the seqid. ''

> > operations with the same layout stateid.seqid. Is this permitted?

Not just permitted, but REQUIRED in some cases.

BTW, the thread "layout races" that we had in December of 2007 went over
all this (except for parallel LAYOUTGETs when there is no layout
stateid). I see that only three people participated:

    * Benny Halevy 
    * Mike Eisler 
    * [email-masked] 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4