Subject: RE: [nfsv4] Out of sync stateid scenario that looks difficult to resolve

From: Carl Burnett <[email-masked]>
Date: Fri, 12 Sep 2003 11:46:31 -0500

See below.

Carl Burnett
AIX Kernel Architecture - Network File System
(512) 838-8498, TL 678-8498
(please reply to [email-masked])

"Noveck, Dave" <[email-masked]>
Sent by: [email-masked]
09/12/2003 11:08 AM

        To:     Carl Burnett/Austin/IBM@IBMUS, "Trond Myklebust" 
<[email-masked]>
        cc:     <[email-masked]>, <[email-masked]>, <[email-masked]>
        Subject:        RE: [nfsv4] Out of sync stateid scenario that looks difficult to resolve

> Just as food for thought, I think removing open share reservations from 
> NFS V4 would go along ways towards allowing a significantly more 
scalable 
> protocol. Are share reservations really worth the scalability issues 
they 
> present? Think about it. Locking and delegations could still be 
supported, 
> but with a different bootstrapping mechanism.

I don't think we can go there.  While the minor versioining rules
do allow us to remove features, getting a consensus on removing them
tends to be very hard (i.e. virtually impossible).  Even if you conclude
that shaer reservations are not worth the problems, all that gets you 
is to a conclusion that they shouldn't have been put in in the first 
place.

[CARL: I know, I couldn't help mentioning it anyway - sorry]

I do believe that we can get rid of the problems that share reservation
present, while still retaining the feature.  The problem with the
current approach is that all clients, even those that have no use
for share reservations, have to pay the price.  The reason is that
open_owner's currently provide two different functions: they serve
to distinguish the different owners of share reservations (they 
perform a lock_owner-like function for share reservations), and 
they serve to identify a seqid space for the replay protection
function.

[CARL: I agree there is future hope, but it does not help the
 pains of implementing RFC 3530 as defined.]

If you don't use share reservations, you don't need the first of
these functions, since one open_owner per client would serve as far
as that funcion goes.  The problem is that you still have that 
nasty second function so you still need open_owners.

[CARL: I have often pondered if it would be better for a
 UNIX client to not use OPENS, and instead use the special
 stateids until byte range locks are required. That presents
 issues in the form of unexpected application errors if
 there happen to be open_deny modes in use somewhere. Also,
 delegations can't be exploited without OPENS, which I view
 as unfortunate.
]

If we are talking about v4.1 however, the session extensions 
proposal can rescue us here.  That puts all of the replay protection
is its own layer, and therefore you don't need open_owner's and
associated sequence id's for replay protection, so the only use 
for open_owner's would be for those clients who want and need share 
reservations.  This requires some further discussion but I'm pretty 
sure it could also be the basis for eliminating the annoying stateid 
sequence increment that causes problems with interrupted requests. 

[CARL: yes I agree. I believe the sessions stuff
 is really going to be necessary to build an NFS that can truly
 be deployed in the real world. You have to get to a model
 that allows reasonable call concurrency with a limited number of
 state anchors (owners) and light weight server management of
 the state. In the simplest case, you would like the anchors to
 reduce down to the session and probably the clientid.
]

-----Original Message-----
From: Carl Burnett [mailto:[email-masked]]
Sent: Friday, September 12, 2003 10:42 AM
To: Trond Myklebust
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [nfsv4] Out of sync stateid scenario that looks difficult
to resolve

Trond writes:

"Are you perhaps assuming that open_owner == posix file handle (as per
the recommendation in the RFC). That is a concept which doesn't appear
to scale on the server side. We have already hit problems with it
under testing against existing commercial NFSv4 servers which quickly
run out of resources...

For that reason, we'd prefer to use a limited number of open_owners
(say related to the number of allowed simultaneous RPC requests on the
wire) which we share among all the users."

I completely agree that a model where every file object has a unique 
open_owner is non-scalable. We too believe in an approach where a pool 
owners can be shared among many objects. We do however continue to find 
new challenges with this approach that keep increasing the complexity of 
the client implementation. Even with a pool model, IMHO, its going to take 

some pretty serious NFS server hardware to serve V4 at any level of scale.

Just as food for thought, I think removing open share reservations from 
NFS V4 would go along ways towards allowing a significantly more scalable 
protocol. Are share reservations really worth the scalability issues they 
present? Think about it. Locking and delegations could still be supported, 

but with a different bootstrapping mechanism.

Trond also writes:

"Now I am curious, though. Where in the RFC does it say explicitly that
OPEN will invalidate an existing stateid, and why should it be necessary?"

I think this comes from the property, that within the open stateid, there 
is a seqid that the server should increment each time it processes an 
operation that uses the stateid. Thus the stateid continuely moves forward 

in time. As Spencer hinted, this is an addtion to any info within the 
opaque portion of the id that the server is managing, which could change 
with each use of the stateid as well.

Carl Burnett
AIX Kernel Architecture - Network File System
(512) 838-8498, TL 678-8498
(please reply to [email-masked])

Trond Myklebust <[email-masked]>
Sent by: [email-masked]
09/11/2003 07:49 PM

        To:     [email-masked]
        cc:     [email-masked]
        Subject:        Re: [nfsv4] Out of sync stateid scenario that 
looks difficult to resolve

>>>>> " " == Spencer Shepler <[email-masked]> writes:

    >> So now I'm confused. Does this also mean that you cannot
    >> upgrade/downgrade the open stateid if there is already a lock
    >> stateid hanging off it?

     > For a posix locking scheme, that doesn't apply.  The first
     > close() for a file will release all file locks (if they are
     > left) and therefore the lock stateid is no longer needed.

Are you perhaps assuming that open_owner == posix file handle (as per
the recommendation in the RFC). That is a concept which doesn't appear
to scale on the server side. We have already hit problems with it
under testing against existing commercial NFSv4 servers which quickly
run out of resources...

For that reason, we'd prefer to use a limited number of open_owners
(say related to the number of allowed simultaneous RPC requests on the
wire) which we share among all the users.

Now I am curious, though. Where in the RFC does it say explicitly that
OPEN will invalidate an existing stateid, and why should it be necessary?

Cheers,
  Trond

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 16 Sep 2003 08:07:46 -0700

> If all servers are prepared to do what you propose above, then there
> is no problem, and we can drop this thread entirely.

*All* servers is tough.  In particular, servers running existing
releases (e.g. ONTAP 6.4) are going to do things the way they've 
been coded to do, which is to reject those reads with OLD_STATEID :-(.

Also, the spec allows this and even if everyone here agrees on what
the right thing is, there may be new servers out there that choose 
to do things a different way.

However, if there is a general agreement that not invalidating 
the stateid is possible and desirable, then you can avoid the 
troublesome synchronization.  You still might have to re-issue 
the reads when you get OLD_STATEID in the case of an idiosyncratic 
(but legal) server, but the idea would be that the associated 
costs would be limited to a small set of servers, and would tend
to go away as server writers were clued into what clients found 
desirable.  

-----Original Message-----
From: Trond Myklebust [mailto:[email-masked]]
Sent: Monday, September 15, 2003 5:41 PM
To: Noveck, Dave
Cc: [email-masked]
Subject: RE: [nfsv4] Out of sync stateid scenario that looks difficult
to resolve

>>>>> " " == Dave Noveck <Noveck> writes:

     > Let's take a case.  I have a file open for read and it was
     > state id X with seqid 1.  Now, asynchronously some read's are
     > issued with that stateid X:1.  Now an open for write is done
     > upgrading the open for that file (and it may be through a
     > different name) and so you get back, by hypothesis, stateid
     > X:2.

     > So if we get a read issued before the open, should we reject
     > it?  After the upgrade, we will still be able to read, as we
     > can see by looking for the current privileges recorded in the
     > object for stateid X (in this model we won't look at the seqid,
     > since the state object has been updated to include read and
     > write privileges).  Now if somehow the client issued a write,
     > before the upgrade, then we could see that fact with the seqid,
     > but is that really an advantage?  Why not just look at the
     > write and and the current options and see if it is what we are
     > currently allowed to do?  Finding that a client is issuing
     > writes when he doesn't have the privilege of doing them in a
     > situation in which they are valid when they are received on the
     > server isn't high on my list of priorities.

     > Am I missing something here?  Is there some case where this
     > sort of state invalidation really gives you a substantial
     > benefit as far as rights checking?
   
If all servers are prepared to do what you propose above, then there
is no problem, and we can drop this thread entirely.

My understanding of last week's discussion, though, was that in the
above scenario, the server may also reject those reads with stateid
X:1 if they arrive after the OPEN that upgraded the server state for
that given <open_owner/filehandle> to read/write.

To summarize:
------------
All I'm wanting is to avoid having to serialize all READ/WRITE
requests with every new OPEN request that shares the same open_owner
on the off chance that the server will change the stateid, and refuse
to accept the old one. If the server is prepared to accept my
in-flight reads with X:1==(READ,DENY_NONE) for a short period after it
has created state X:2==(READ/WRITE,DENY_NONE), then I'm a happy
camper...
If OTOH it starts returning NFS4ERR_OLD_STATEID, so I have to resend
all my reads with the new stateid, then my network admin may take
offence...

Cheers,
  Trond

---

From: Paul Mazzurana <[email-masked]>
Date: Tue, 16 Sep 2003 15:49:49 -0500

Just wanted to add a data point on how our server will be handling the 
"old" stateid issue.  Our stateids will also be changing by incrementing 
the internal sequence id of the stateid. We will also reject "old" 
stateids on all operations that take a stateid.

Dave Noveck said:
So I suppose that you are assuming that the server will shomehow keep 
the information for the previous stateid as well as the new one so that
it can act on the old one before the CONFIRM, act on the new one after
the CONFIRM, and also restore the old one if the confirm does not happen.
Since my server doesn't currently do this, I'm definitely against it :-)
[...deleted...]

Let's take a case.  I have a file open for read and it was state id X 
with seqid 1.  Now, asynchronously some read's are issued with that 
stateid X:1.  Now an open for write is done upgrading the open for that
file (and it may be through a different name) and so you get back, by
hypothesis, stateid X:2.

So if we get a read issued before the open, should we reject it?  After
the upgrade, we will still be able to read, as we can see by looking for
the current privileges recorded in the object for stateid X (in this model
we won't look at the seqid, since the state object has been updated to
include read and write privileges).  Now if somehow the client issued a
write, before the upgrade, then we could see that fact with the seqid,
but is that really an advantage?   [...deleted..]
 
> and when debugging client and server interaction it is easier to track 
> the transitions that occur with stateids.

This is true but is it worth it?  Our server does currently do this, but
if does add complexity and so if it problematic for clients, and there 
isn't
a big benefit, I'd be inclined to get rid of it. 
 
And Spencer said:

The stateid changes in the form of incrementing its internal sequence
id.  Our server implementation currently will reject "old" stateids
for all operations that take a stateid.

--
Paul D. Mazzurana
AIX - NFS group

---

From: Carl Burnett <[email-masked]>
Date: Tue, 16 Sep 2003 16:33:17 -0500

As Paul has stated below, our current approach is the more restrictive one 
that seems to be consistent with what Dave and Spencer have stated for 
Netapp and Sun. We are open to relaxing this, as has been suggested, where 
the server further looks at what access modes are allowed for the object's 
accumulated open rights held by the open owner associated with the 
stateid. It would be nice to get a consensus among the implementers. 
Unfortunately, a client has to code to and handle the most restrictive 
case unless its taken as standard convention to support the more relaxed 
server model. Ideally, while I know its not possible, the spec would be 
updated to be very specific on these aspects of the protocol and not allow 
so much latitude. Given the RFC can't be modified, its in dire need of a 
comprehensive white paper on NFS V4 state management.

Carl

Carl Burnett

Paul Mazzurana/Austin/IBM@IBMUS
Sent by: [email-masked]
09/16/2003 03:49 PM

        To:     [email-masked]
        cc: 

Dave Noveck said: 

 

And Spencer said: 

-- 
Paul D. Mazzurana 
AIX - NFS group

---

From: [email-masked]
Date: Wed, 17 Sep 2003 10:23:59 -0400

As others (everyone so far ?) have done, we also take the strict approach to
this issue.  In our implementation the stateids increment their internal
seqid as state changes, and we reject old ids.
We could certainly relax this restriction if that is the consensus.
Just to amplify what Carl mentions below, a paper to describe NFSv4 state
management may go a long way to help insure interoperable and well behaved
implementations.

If I understand the problem correctly, we are simply talking about
permitting I/O operations to forgo checking the internal stateid sequencing,
and allowing them to proceed assuming the permissions and state is still
valid.

-- Tom

Thomas Vanderputten
EMC Corporation
62 T.W. Alexander Dr
Research Triangle Park, NC 27709
(919) 248-6170

-----Original Message-----
From: Carl Burnett [mailto:[email-masked]]
Sent: Tuesday, September 16, 2003 5:33 PM
To: Paul Mazzurana
Cc: [email-masked]; [email-masked]
Subject: RE: [nfsv4] Out of sync stateid scenario that looks difficult
to resolve

Carl

Carl Burnett

        To:     [email-masked]
        cc: 
        Subject:        RE: [nfsv4] Out of sync stateid scenario that looks
difficult to resolve

Dave Noveck said: 

 

And Spencer said: 

-- 
Paul D. Mazzurana 
AIX - NFS group

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 17 Sep 2003 08:24:09 -0700

Carl Burnett wrote:
> stateid. 

One point to note.  When you say "the server further looks" above, it seems
to imply that we are talking about adding a check.  However, this is a check
that you should be making in any case, even if you are being restrictive
on the seqid thing.  If I open a file for READ, sending a WRITE request
is an error, even if the stateid (including the seqid part) does match
perfectly.
 
> It would be nice to get a consensus among the implementers. 

To be fair, I should note that we had a consensus among implementers,
which I implemented to, and I'm now arguing is sub-optimal, or at least 
that I have ceased to see the arguments for the old consensus as having 
any power.

I don't care so much about implementers agreeing to change what they
are doing.  What I'm concerned about is a consensus that what I've 
proposed doesn't have any hidden flaws.  Right now it seems OK, but 
I'd like people to think about it some more and see if anyone can find
some case in which it will not work.
  
> server model.

I think the client has to code to handle the most restrictive case, no 
matter what the (current) implementors decide.  The spec allows that
case and that isn't going to change (until maybe 4.1).

>From a coding point of view, handling this is not a big problem.  All you
have to do is check for an OLD_STATEID error on a READ, WRITE, or SETATTR
(for the truncate case).  If you get that error,  then see if the current 
seqid for that stateid is different from the one the request was issued with.
If it is, re-issue the request with the new stateid.  It's a pain but it's not 
a nightmare. 

The issue with this approach is not that it is difficult for clients to
support this but that, as Trond has noted, this can have bad consequences
for network utilization when requests are often (unnecessarily) retried.

If the clients will work with both versions and it is just a question of
efficiency then I think it is OK to leave this to the marketplace, or,
for the less commercially inclined, to the marketplace of ideas.

> Ideally, while I know its not possible, the spec would be 

We do have the option of being more specific in v4.1.  In a lot of cases
v4 servers will do different things with the different behavior being 
arguably in the realm of fs semantics, rather than within the protocol
per se.  This case is one where different sorts of behavior appear to be
allowed within something that is pretty clearly within the actual protocol 
As a general matter, agreeing to not agree (e.g. NLM which includes
RPC's and async messages to do the same things, rather than deciding 
on one or the other) is a bad thing.  So I would like us to pick one
approach to state management for v4.1, but I don't see this as crippling
for the initial v4.  It is relatively easy for the client to support both
kinds of servers.

We've had enough questions about various aspects of state and lock 
management that we need to do something.  Part of this will be clarifications
in the existing spec.  I've been meaning to propose a few small changes
based on the discussions that Rick and I and others recently had.  I think
it is probably the case that we could use a separate explanatory document 
for implementors.  The only problem is corralling people to work on it.

I'm a little bit concerned about the implications of your statement above,
"Given the RFC can't be modified, its in dire need of a comprehensive white 
paper on NFS V4 state management".  

The question I have is whether you are anticipating clarification of the
current protocol (including all the options available in it), or some 
degree of restriction thereof.  Clarification can I think be done as edits
to the existing spec (Spencer would know the rules) and I suppose that a
more extensive informal discussion would make sense in a separate white paper.
But a separate white paper is not going to be able to restrict the available
options in the protocol.  If the white paper tells the server not to do something 
and the spec allows it, then it is legal and clients have to deal with it.
The white paper can't modify the spec, for the same reason that we can't
modify the spec (in essentials) directly.

I would like the white paper to clearly lay out the current state of things
including all the options and the justifications for them.  This is partly 
to help implementors but it is also because I anticipate that this area will
be changing a lot in v4.1.  The session extensions makes a lot of the
sequencing stuff unnecessary and we (at least you and I) feel that there 
are areas where the spec allows too much leeway for things to be implemented
differently.

I think it is very important to understand the current state management model,
if we are going to be changing it.  I foresee that we are going to want to
remove lots of stuff because:

a) It is ugly and nasty
b) With the EOS features of the session extensions, there is no need for it

I think we have to be very clear about the current approach and the reasons for
it to make sure we don't remove stuff where a) but not b) is true.

Note that the state seqid change stuff (I think) falls into a different category
in which both

c) It is a big pain
d) It was never really necessary

are true, or at least that's what I believe now.

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 17 Sep 2003 08:28:24 -0700

> and allowing them to proceed assuming the permissions and state is still

I think it is more consistent with the current spec for the server to
maintain the seqid checking and simply not bump the seqid when an open
upgrade or OPEN_DOWNGRADE is done.  This has the same effect in the 
cases we are talking about, that the stateid is not unnecessarily
invalidated but the server is generating OLD_STATEID in all the cases
the spec says it should.  It is just arranging things so that those
cases don't normally arise.  

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
Sent: Wednesday, September 17, 2003 10:24 AM
To: [email-masked]
Subject: RE: [nfsv4] Out of sync stateid scenario that looks difficult
to resolve

-- Tom

Thomas Vanderputten
EMC Corporation
62 T.W. Alexander Dr

-----Original Message-----
From: Carl Burnett [mailto:[email-masked]]
Sent: Tuesday, September 16, 2003 5:33 PM
To: Paul Mazzurana
Cc: [email-masked]; [email-masked]
Subject: RE: [nfsv4] Out of sync stateid scenario that looks difficult
to resolve

Carl

Carl Burnett

        To:     [email-masked]
        cc: 

Dave Noveck said: 

 

And Spencer said: 

-- 
Paul D. Mazzurana 
AIX - NFS group

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 12 Sep 2003 09:08:22 -0700

-----Original Message-----
From: Carl Burnett [mailto:[email-masked]]
Sent: Friday, September 12, 2003 10:42 AM
To: Trond Myklebust
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [nfsv4] Out of sync stateid scenario that looks difficult
to resolve

Trond writes:

Trond also writes:

Carl Burnett

        To:     [email-masked]
        cc:     [email-masked]
        Subject:        Re: [nfsv4] Out of sync stateid scenario that looks difficult to resolve

Cheers,
  Trond