Subject: Re: [nfsv4] An issue regarding free byte-range lock state

From: "Peter Varga" <[email-masked]>
Date: Thu, 28 Jun 2007 17:22:50 -0400

> So, I would like to back up in this discussion a little.
> 
> I would like to come to a consensus of what stateid's represent and 
> how they do or do not change over the life time of the client's and 
> server's interaction.  I think once we have this agreed upon that the 
> specific issues of stateid.seqid handling will be fairly easy to 
> resolve.
> 
> In my reality, I am still of the mode that the stateid (in its
> entirety)
> represents the current state of the file.  For the case of a client 
> and server, the state returned by either and OPEN or OPEN_DOWNGRADE 
> must be used for subsequent OPEN_DOWNGRADEs or CLOSEs.  The server may

> change the stateid as a result of a OPEN_DOWNGRADE (not the same as 
> the value on the OPEN response).  However, the server is not required 
> to do this.

Apparently, the spec requires it:
"The server is required to increment the seqid field monotonically at
each transition of the stateid" 

> Using this model, the LOCK/LOCKU stateid changes also occur and then 
> each stateid represents the locking state at the point of the 
> completion of the LOCK/LOCKU.  Again, the server isn't required to do 
> this but it may do this if it wants.
> 
> I have seen the server's changing of the stateid a useful tool to 
> ensure the client and server have the correct/shared view of their 
> interaction (client returns what it received and the server keep 
> changing its value to help in overall correctness).
> 

According to the spec: "This is important since the client will inspect
the seqid in OPEN stateids to determine the order of OPEN processing
done by the server".

Why does the client care about the order?

> 
> I assume given the thread on this topic that the alternate view exists

> such that a stateid is assigned a value upon first instantiation of 
> state and it does/should/must remain the same until the sequence of 
> state events come to an end (via CLOSE, DELEGRETURN, FREE_STATEID) as 
> mentioned in the quoted text above.  True?
> 

True for me. I am having trouble understanding why the stateid.seqid is
useful to the client or server (as per Dave Noveck's arguments), aside
from, as you say helping with debugging when the client/server for
whatever reason don't have the same idea of a given open/lock owners' or
delegation's state. 

As it stands, the spec is requiring servers to increment stateid.seqid,
whereas you're saying servers don't have to. So, I suppose the debate is
about whether or not servers should have to or not in 4.1. If it is
decided that the stateid.seqid must be incremented by the server, I
would like to at least see some reasoning/explanation in the spec for
the problem it addresses. Thanks.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 28 Jun 2007 18:20:56 -0400

> Why does the client care about the order?

Say the client sends off 2 OPEN calls in parallel to 2 different hard
links to the same file (quite allowable using NFSv4.1). Which one of the
2 stateids returned by the server is currently valid?

Trond

---

From: "Peter Varga" <[email-masked]>
Date: Thu, 28 Jun 2007 21:41:20 -0400

> -----Original Message-----
> From: Trond Myklebust [mailto:[email-masked]]
> Sent: Thursday, June 28, 2007 6:21 PM
> To: Peter Varga
> Cc: [email-masked]
> Subject: Re: [nfsv4] An issue regarding free byte-range lock state
> 

OK, so the problem occurs if the server is modifying the stateid.other
part on OPEN/DOWNGRADE/LOCK/LOCKU. This has been my source of confusion.
I now understand Spencer's comment on the two views of the issue.

If a server does not change stateid.other on
OPEN/DOWNGRADE/LOCK/LOCKU/etc, then there really is no use for
stateid.seqid in the interaction with such a server.

What's a concrete example where changing stateid.other on these ops
helps avoid problems? It seems to me it just limits parallelism and
tries to keep the client/server "on the same page" a bit too rigidly. If
the server is banned from changing the stateid on OPEN,LOCK,ULOCK, the
stateid can be made entirely opaque. CLOSE,LOCKU (and perhaps others)
would not need to return a stateid. The protocol becomes simpler (if
this is even possible given it's at 489 pages ;) ).

---

From: Rick Macklem <[email-masked]>
Date: Fri, 29 Jun 2007 11:52:10 -0400 (EDT)

>> From: Trond Myklebust [mailto:[email-masked]]
[stuff snipped]
>> 2 stateids returned by the server is currently valid?
Does 4.1 allow multiple concurrent Opens on the same OpenOwner? (I've 
never read the draft to see if this is allowed?)
[Peter wrote]
> stateid.seqid in the interaction with such a server.
In my view of it, the stateid.other will be the same if (and only if)
the openowner is the same, for an open of a given file. If the 
spec says stateid4.seqid is 0, then the 2 stateids are identical.

To me, for the case of same stateid4.other, the client will end up or'ng
the access permissions to-gether and I don't see that it cares what order
the Opens happened in on the server. The client might have to wait until
all outstanding Opens on a given Openowner have replied, before using the
open stateid, if it finds the access permissions insufficient before all
replies have been received.

Note that none of this happens for 4.0, since Opens on a given Openowner
are serialized, using that other seqid (not stateid4.seqid).

> this is even possible given it's at 489 pages ;) ).
At least for 4.0, the stateid.other must be not be the same for different
files or different openowners. These cases require separate Closes and you 
can't Close the same stateid4.other multiple times. If you change this for 
4.1, the Open/Lock model changes dramatically.

There is at least one previous thread on doing concurrent Opens on the 
same openowner, but the short answer is, if you want more parallelism for 
Opens, you can use more OpenOwners. If you choose to allow/do multiple
concurrent Opens on the same openowner, then the client may find resolving
the replies difficult, but I don't believe an incrementing stateid4.seqid
helps. The only case I can think of where knowing the order the Opens
were performed on at the server might be useful is when some Opens for
a given openowner and file succeed and others fail, BUT for the failure
cases, you don't get a stateid4.seqid back.

At least that's my understanding of it, rick

---

From: "Peter Varga" <[email-masked]>
Date: Fri, 29 Jun 2007 12:40:15 -0400

I agree with what you've said. I was discussing the case where the same
open owner is used. This is allowed in v4.1 thanks to sessions removing
the need to increment the open owner's seqid.

If the client uses a different open owner, then the two OPENs will
always have result in different stateid.other. Assuming both OPENs
succeed, then both stateids are valid, and each represents a separate
[open owner,file].

Does the spec not require different stateid.other part when using
different open owners? Maybe this is the problem. Consider the case that
open owner 1 opens the file for read access, and open owner 2 opens the
file for write access, and the server returns the same stateid.other
part. If the client issues a READ request with this stateid, the server
would have no way to know if it should allow it since it only has the
stateid as an argument (it won't know which open owner is doing the
READ).

> From: Rick Macklem [mailto:[email-masked]]
> Sent: Friday, June 29, 2007 11:52 AM
> To: Peter Varga
> Cc: Trond Myklebust; [email-masked]
> Subject: RE: [nfsv4] An issue regarding free byte-range lock state
> 
> 
> 
> >> links to the same file (quite allowable using NFSv4.1). Which one
of
> >> the
> > tries to keep the client/server "on the same page" a bit too
rigidly.
> If

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 29 Jun 2007 12:48:40 -0400

> different open owners?

Given the word "unique" in the following, I would say so.  Also 
I read "identifying" as implying "different".  

  7286	   When the server grants a lock of any type (including opens,
record
  7287	   locks, delegations, and layouts) it responds with a unique
stateid,
  7288	   that represents a set of locks (often a single lock) for the
same
  7289	   file, of the same type, and sharing the same ownership
  7290	   characteristics.  Thus opens of the same file by different
open-
  7291	   owners each have an identifying stateid.

-----Original Message-----
From: Peter Varga [mailto:[email-masked]] 
Sent: Friday, June 29, 2007 12:40 PM
To: Rick Macklem
Cc: [email-masked]
Subject: RE: [nfsv4] An issue regarding free byte-range lock state

of
rigidly.

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 29 Jun 2007 12:49:15 -0500

On Jun 29, 2007, at 10:52 AM, Rick Macklem wrote:

> Does 4.1 allow multiple concurrent Opens on the same OpenOwner?  
> (I've never read the draft to see if this is allowed?)
>>>
> [Peter wrote]
>>
> the openowner is the same, for an open of a given file. If the spec  
> says stateid4.seqid is 0, then the 2 stateids are identical.

That is one view and it seems that the NFSv4.1 specification isn't
that specific about scope of the stateid (is it within the clientid,
open_owner, lock_owner or all within the clientid).  I don't have
a strong opinion at this point but it does need to be very clear given
our numerous misunderstandings in the past.  My main concern would
be one of space available in the stateid to effectively deal with
the scope chosen.

> replies have been received.

See below.

>> rigidly. If
> files or different openowners. These cases require separate Closes  
> and you can't Close the same stateid4.other multiple times. If you  
> change this for 4.1, the Open/Lock model changes dramatically.
>
> There is at least one previous thread on doing concurrent Opens on  
> the same openowner, but the short answer is, if you want more  
> parallelism for Opens, you can use more OpenOwners.

But one may choose not to do this because of the additional
state that is managed at the client and server.  We were
working from the requirement that both models (few or many openowners)
would be possible with a reasonable amount of parallelism such
that client performance would not be significantly impacted
by the use of either approach.

> If you choose to allow/do multiple

If the client choose to have more than one outstanding
OPEN/OPEN_DOWNGRADE/CLOSE for an open_owner, then the seqid is
required.  Because of hard-links, there are issues of OPEN upgrade
given that NFSv4 requires that downgrade be done in the opposite pattern
from upgrade; this means the client needs to know the ordering of the
state changing operations.  Then there is the race between OPEN/CLOSE
that can occur, again in the presence of hard-links.

Spencer

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 29 Jun 2007 12:50:29 -0500

>   7291	   owners each have an identifying stateid.

As I mentioned in my other email, we need to clarify the scope
of uniqueness (clientid or open_owner or other).  Maybe you can
address this in the updates you are working on.

Spencer

>> and
>> you
>> can't Close the same stateid4.other multiple times. If you change  
>> this
>> for

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 29 Jun 2007 14:09:58 -0400

I can make this more explicit.

I assume that there is general agreement that the scope of 
uniqueness should be the clientid.

-----Original Message-----
From: Spencer Shepler [mailto:[email-masked]] 
Sent: Friday, June 29, 2007 1:50 PM
To: [email-masked]
Subject: Re: [nfsv4] An issue regarding free byte-range lock state

Spencer

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 29 Jun 2007 14:05:06 -0500

> uniqueness should be the clientid.

Per clientid is fine with me.  Just need the clarity of
what the client can and can not do with that uniqueness
given that the stateid (in the case of open for example)
represents the open/close state for that file/openowner/clientid
combination.

Spencer

>>> (I've
>>> all outstanding Opens on a given Openowner have replied, before  
>>> using
>>> the
>>> same openowner, but the short answer is, if you want more  
>>> parallelism
>>> for

---

From: "Peter Varga" <[email-masked]>
Date: Fri, 29 Jun 2007 17:09:04 -0400

> that can occur, again in the presence of hard-links.

I see. In that case, we still need the OLD_STATEID error (spec says "
This error does not apply to and should never be generated in NFSv4.1.")

Consider:

- link1 = link2 = hard links
- stateid1 and stateid2 refer to the stateid in its entirety (seqid +
other)
- same open_owner

If the server does _not_ increment stateid.seqid:
client sends OPEN link1
client sends OPEN link2
server returns stateid1 in response to OPEN link1
client sends CLOSE stateid1
server returns stateid1 in response to OPEN link2
server closes stateid1 in response to CLOSE stateid1
client sends READ stateid1
server returns BAD_STATEID --> Problem!

If the server increments stateid.seqid:
client sends OPEN link1
client sends OPEN link2
server returns stateid1 in response to OPEN link1
client sends CLOSE stateid1
server returns stateid2 in response to OPEN link2
server returns OLD_STATEID in response to CLOSE stateid1
client figures out link1 and link2 are the same file and doens't resend
the CLOSE