Subject: [nfsv4] question about OPEN (for truncate), and OPEN_CONFIRM

From: Carl Burnett <[email-masked]>
Date: Fri, 25 Apr 2003 10:32:58 -0500

If the server receives an OPEN for an unchecked create that indicates the 
file should be truncated (size attribute 0), and the file exists, and the 
OPEN requires an OPEN_CONFIRM, it is assumed that the truncate should not 
occur. I think this means the server (state manager) must record that the 
file should be truncated when the OPEN_CONFIRM is sent by the client.

Assuming that is correct, what should happen if the truncate fails (for 
example EACCES)?

Should the state manager cleanup up the pending open state?

What error should be returned? The OPEN_CONFIRM return values are pretty 
limited (no NFS4ERR_EACCESS).

Thanks,
Carl

Carl Burnett
AIX Kernel Architecture - Distributed File Systems
(512) 838-8498, TL 678-8498
(please reply to [email-masked])

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Mike Eisler <[email-masked]>
Date: Fri, 25 Apr 2003 09:41:09 -0700 (PDT)

> OPEN requires an OPEN_CONFIRM, it is assumed that the truncate should not 

I can't see where the spec says the truncate should not occur.

> file should be truncated when the OPEN_CONFIRM is sent by the client.

I think that makes the server implementation too difficult.
OPEN_CONFIRM is for confirming the creation of state, not
the changes to the metadata. It is reasonable for the
server to go ahead and attempt the truncate before OPEN_CONFIRM.

> limited (no NFS4ERR_EACCESS).

If your server took the hard route to implement this, I'm not sure what
error you would return, as none of them are good fit. This implies to
me that the truncate attempt MUST happen via OPEN.

A similar issue applies to when OPEN creates the file after finding
that it does not exist. There's no need for the server to delete the
created file in the event OPEN_CONFIRM fails or is never sent.

These issues have a bit of historical precedence with NFSv3. v3 has
an atomic exclusive create capabbility, but it requires the
client to clean up the metadata of the new object with a subsequent
SETATTR. The v3 spec (and the v4 spec around atomic create) does
not require the server clean up the metadata if the client never
follows up with a SETATTR. And we've managed to live with it
for 10 years.
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 25 Apr 2003 09:48:42 -0700

> occur. 

That's a desirable property but I don't think it is a required one.
I know my current code does the truncate immediately.  I think this
was discussed a bit at Connectathon.  My memory (which is probably
not all that reliable) was that Spencer said that his server does the
truncate immediately as well.  Spencer, can you confirm or deny?

The argument that the trncate should not be done immediately is
fairly clear.  If the open is not confirmed, then it is replay 
(which was not detected by the reply cache), so the open should 
not happen, including the truncate which is part of the open.

The counter-argument to that (besides the whiny "Gosh that is a
pain to do") is to ask why open-confirm and the sequence stuff 
exist and are restricted to OPEN, LOCK, etc.  For a large part 
of the protocol there is no replay protection within the protocol
itself.  People do reply caches as an implementation thing
and they seem to work most of the time, but there is no proof or
guarantee that they work.  In particular, note that a TRUNCATE via
a SETATTR has no specific replay protection within the protocol.
Only state-managing operations have this special level of protectioon.
This partly because of their special place in the protocol, i.e.
that other stuff depends on them working.  It is partly because
these are new ops that could be designed to incorporate the 
protection.  And is partly that nobody was really up to design
and debug such operations, that maintain state, when every
single request might be a replay.  It's just too hard.  So the
argument would be that doing the truncate immediately is no worse
than not having replay protection on SETATTR truncate, while 
you still maintain the sequencing as it relates to state management.

> I think this means the server (state manager) must record that the 

I think that's nice, but it leads to complexities.  One is what
happens if someone else reads or writes the file before the confirm
either happens or is timed out.   Do you have to wait to see if the 
truncate really has happened?  

Another issue in that same vein is what if the open that is being 
confirmed is a deny-write open.  Do you deny another open for write
that occurs before confirm or do you force that open to wait?  Right
now I do an immediate deny, but I'm leaning to the view that that is
bug.

Getting back to the truncate, you have the case in which the server
reboots before confirmation happens or times out.  Should the file
be truncated after the reboot?  I think the answer is probably that
it doesn't have to be (even if you go down this path) but I won't 
swear to that.  

> Assuming that is correct, what should happen if the truncate fails (for 

I don't think it can.  I think it is up the server to verify that the
truncate can be done, when he accepts the open with truncate and give
the error back at that point.  If you defer the truncate, then once 
you accept it, you are committed to doing it, and have accepted that 
he (the principal doing the open) is allowed to do it.  When the 
confirm happens, you act on that determination.

> limited (no NFS4ERR_EACCESS).

Because you are only confirming something, not doing something else
substantive.  If somethng like truncate appears to happen (because it
is now confirmed) as a result of that, it has to happen without 
a specfic error code that refers to that action.

My big regret is that v4 does not have a robust replay protection 
mechanism that works for everything uniformly, including OPEN 
(truncate or not) and SETATTR.  When we were considering the 
basic outlines of v4, it just seemed too hard to do.  The DAFS 
experience proves otherwise, however.
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Carl Burnett <[email-masked]>
Date: Fri, 25 Apr 2003 12:25:26 -0500

Dave,

I can live with your response ( also Mike E's) - that is just do the 
truncate, but I don't think this is good behavior. Basically, what could 
turn out to be an illegitimate open request has been successful in 
changing file system state. BTW: wouldn't the truncate via SETATTR case be 
somewhat handled because it requires a valid stateid.

Perhaps the protocol should have used a separate mechanism to establish 
the open owner and its associated sequence ID instead of piggy backing it 
on open. Another approach might have been for the OPEN have no effect if a 
confirm is required. The client would then do the OPEN_CONFIRM to complete 
its state handshake, and then it would make the original OPEN request 
against to get its file system work done.

Of course I guess its too late since changing the protocol doesn't seem to 
be an option. I wonder if there is something that could be done in a minor 
version. That may be hard to do without backward compatibility issues.

Carl Burnett

"Noveck, Dave" <[email-masked]>
04/25/2003 11:48 AM

        To:     Carl Burnett/Austin/IBM@IBMUS, [email-masked]
        cc: 
        Subject:        RE: [nfsv4] question about OPEN (for truncate), and OPEN_CONFIRM

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 25 Apr 2003 10:51:32 -0700

It is definitely possible in a minor version.  I don't see
any troublesome backward compatibility issues.  The only
problem is that unless the client and the server both support
the new minor version, you get the older troublesome
behavior, but that's what you have now.

-----Original Message-----
From: Carl Burnett [mailto:[email-masked]]
Sent: Friday, April 25, 2003 1:25 PM
To: Noveck, Dave
Cc: [email-masked]
Subject: RE: [nfsv4] question about OPEN (for truncate), and
OPEN_CONFIRM

Dave,

Carl Burnett

---

From: Mike Eisler <[email-masked]>
Date: Fri, 25 Apr 2003 11:58:56 -0700 (PDT)

Carl Burnett wrote,

> somewhat handled because it requires a valid stateid.

What is not legitimate? The request has a valid client id, and a valid
and authorized principal. If a file is truncated or create without open
confirmation, then that's the fault of the client implementation for
not constructing and sending a valid OPEN_CONFIRM on a timely basis.
Even if the client follows up with OPEN_CONFIRM too late (say because
of a network partition, which is certainly not the client's fault), it
can still start from the beginning with OPEN. If the OPEN was doing a
truncate, the file is still truncated. If it was doing an exclusive
create, the client's verifier will match.  So, only a broken client
implementation messes things up here.

> the open owner and its associated sequence ID instead of piggy backing it 

One issue is that the client might have not used the open owner for a
long time, and the so the server has disposed of it. The server is in a
better position to indicate whether the open owner is good or not. An
NFS4ERR_EST_OPEN_OWNER code from OPEN would be an alternative to
address this, but it would also result in more round trips. The
thinking is that because the open owner is associated with a distinct
process id, a common scenario is that the client has processes that
open one file, do their thing, close, and exit. As it is now, two round
trips are needed for that scenario. The NFS4ERR_EST_OPEN_OWNER results
in three round trips.

> against to get its file system work done.

Three round trips.

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 25 Apr 2003 12:24:04 -0700

I think the problem Carl is worried about is a situation
in which the orginal open, the one which needs to be 
confirmed is a replay, and so it should not be confirmed.

The probablity of that happening is pretty low, however.

What it requires is that you have a resent open request,
which in v4 should only happen due to connection breaks,
which for some reason languishes in the network for a
while and during that period, the server seeing that the 
open-owner has no associated state and that a considerable
time has gone by, disposes of the lockowner information.
Note that for this to happen, the client must have seen
the the open reply and done the close. Then, when this 
replayed open (with truncate) is finally seen, we
have lost the context to know what the proper sequence id
is so we request a confirm which in this case is not forthcoming,
sicne we (the client) are not looking at the response.
I can't quantify the probability here but an awful lot 
of stuff has to happen wrong for this to occur.  (You also
have to not find the request in the reply cache but given 
that we have to have a real long delay anyway, that part is
fairly likely).  By the way, one implementation trick you
might use to make this even more unlikely is when selecting
owners to recycle, bias the search against those for which
open-with-truncate has ever been done. 

-----Original Message-----
From: Mike Eisler [mailto:[email-masked]]
Sent: Friday, April 25, 2003 2:59 PM
To: Noveck, Dave; [email-masked]
Cc: [email-masked]
Subject: RE: [nfsv4] question about OPEN (for truncate), and
OPEN_CONFIRM

Carl Burnett wrote,

Three round trips.

---

From: Mike Eisler <[email-masked]>
Date: Fri, 25 Apr 2003 12:57:24 -0700 (PDT)

> From [email-masked] Fri Apr 25 12:22:45 2003

> open-with-truncate has ever been done. 

But this is attempting to use open/open_confirm for purpose
that is not intended. Let's go back to the NFSv3
days, where we had just CREATE. If you issue an unchecked CREATE,
won't the exact same scenario occur? Aren't you saying that if
if the duplicate rwquest cache has a miss, bad stuff happens? This
applies to all the non-idempotent operations, though admit that
a truncating open or setattr is particularily nasty. But
we do have VERIFY to mitigate the nastiness (in v3, we had
a guarded SETATTR to do the equivalent of VERIFY. I don't
think there was any protection for UNCHECKED creates that
did truncation).