Subject: RE: idea - atomic append for NFS writes

From: Mike Eisler <[email-masked]>
Date: Thu, 22 Mar 2001 08:50:55 -0800 (PST)

>From: Neil Brown <[email-masked]>

>On Wednesday March 21, [email-masked] wrote:
>> In summary,
>> 
>> 	- You are claiming GUARDED creates as the justification for the holes.
>
>Maybe I misunderstood the purpose of GUARDED.  I was reasoning from
>its proximity to EXCLUSIVE in the spec.  Given the lack of clear
>justification for it's existance, it seems reasonable (but apparently
>false) to assume that most obvious purpose was the intended one.
>
>Your description of the use of GUARDED in Solaris NFSv3 is interesting
>and valid, though it seems like a small plug in a large hole.
>A scenario that must be of equal concern in a stray duplicate UNLINK
>which unlinks a file that a subsequent CREATE created, and you cannot

Not of equal concern to me. GUARDED non-exclusive create
prevents a retry from the same client of the same operation from messing
the file up. The stray UNLINK messing up the CREATE is going to be
a less frequent event than a CREATE that is not preceded by an UNLINK,
because removing the file before a CREATE happens less frequently
than not.

>guard against that (though you can in v4 using VERIFY).

VERIFY doesn't cut it because there's no atomically guarantee between
operations in COMPOUND.  Interestingly, an early NFSv3 draft that Rusty
Sandberg wrote had an NFSPROC_REMOVE with arguments that included the
file handle of the object being unlinked in the arguments; the ultimate
in the guarded REMOVE. It always looked safe to me, but the NFSv3 junta
that met in 1992 to finally finish the specification dropped it.  I've
never had an explanation why. Perhaps because of the extra round trip?
	
>The WRITE_APPEND op contains a "record-length" as well as the
>"data-length", and the return value includes a "record-offset".
>The record-length indicates how much data the client wants to
>atomically append.
>The data-length indicates how much data is in the request.  This bit
>goes at the end (or maybe the beginning, whichever is easier for
>servers). The remainder of the record (if any) is filled with nuls.
>The "record-offset" indicates the file address of the start of the
>appended record.  The client can then fill in the remainder with
>normal writes.

This is a cheap form of offset reservations in stable storage
that Peter Staubach mentioned. And my objection is the same:
what if the client doesn't follow through?

>> 	- In the event of a server crash, duplicate records will appear
>> 		in the file, unless the server implements a fair
>> 		amount of state in stable storage. Currently clients do
>> 		not experience duplicate records when doing an APPEND.
>> 		
>> 	- In absence of an append delegation, append writes will be
>> 		at NFSv2 speeds. Currently they are done at NFSv3 speeds.
>
>And currently, clients can be certain that contemporaneous appends
>from separate clients will corrupt the file.

Write sharing in absence of conflicting record locks
or SHARE reservations is a fantasy conjured by the masters of
FUD at OSF and certain halls of academia. 

>So, do you want broken-semantic-A or broken-semantic-B?

I want the single client writer scenario to work as well with NFSv4 as
it does with NFSv3, so my answer is A. That's my standard for
any changes to support APPEND.

>I suspect most serious NFSv4 servers will bend over backwards to make
>delegations work because that is, I think, one of the great strengths

I still run into NFS client implementations with broken close-to-open
consistency; forgive me if I take your idealism with a large grain of
salt.

>of NFSv4.  If that is true, then the speed issue is probably moot, and
>the correctness is paramount.

I beleive if you go back to archives, you'll
find Dave Noveck entitled his caching proposal
that lead to NFSv4 delegations with the title:

	"Caching, not cache consistency"
	
NFSv4 guarantees close-to-open consistency. That's it.  If the server
finds that it cannot make callbacks back to the client (which can
happen even if the client and server are connected with a 1 metre
cross-over cat-5 cable), then there's no delegation. That means that
99.99% of the time appends will work quickly and accurately (as they
would without your proposal). 0.01 % of the time they will work
accurately and and at NFSv3 speeds without your proposal. With your
proposal in that .01 % of the time, they will work at NFSv2 speeds, and
they'll sometimes produce quirks:  nuls, duplicate records, and that's
even without write sharing. Who really wants to deal with those
customer calls?
		
>> >In this APPEND case, there server does not need to be able to
>> >call-back the client to have the delegation released, so that is one
>> >possible impediment to delegations that need not be a problem.
>> 
>> What if another client wants to write or read the file that has
>> an APPEND delegation on it?
>> 
>
>Maybe you missed that part of the proposal.

I didn't but it doesn't answer my question.

>] The "append" flag gets squeezed into OPEN somehow, it isn't important
>] how.  As the client will already have the file open, this will, in

Why will it have the file already open?

>] effect, upgrade the open to also have the "append" flag.
>] In this situation, the server is "strongly encouraged" to grant a
>] write delegation, though it may set the "recall" flag in the response
>] to say the the client should return the delegation as soon as
>] possible.
>
>The "recall" flag is the important point.  I am suggesting that the
>server issues a delegation even if there is contention, or if there is
>no-way to reclaim the delegation with a call-back.  In these cases it
>sets the "recall" flag in the delegation so that the client will
>release the delegation as soon as it has flushed it's appends.
>
>If you reject the APPEND_WRITE part, then that just leaves these two
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

I do.

>small changes to the protocol:
>  1/ adding an "append" flag to open to signal that a delegation would
>     be "a real good idea".

I'm ok with this; it meets my aforementioned standard.

>  2/ allowing the server to signal 'recall' in a non CLAIM_PREVIOUS
>     delegation, particularly in response to an "append" open.

I've no idea what that means. Recall means callback. If there is
another client that wants to read or write the file, you have to issue
a callback; return an error to the second client; or allow the access
without a callback, the latter two being incorrect. Perhaps your
intent is to include a RECALL flag in all responses to
operations that take a stateid?

	-mre

---

From: Brent Callaghan <[email-masked]>
Date: Thu, 22 Mar 2001 10:36:55 -0800

Mike Eisler wrote:
> operations in COMPOUND. 

I think the lack of inter-operation locking was a mistake.

The intent of VERIFY and NVERIFY was to simplify the protocol
via the addition of pre-conditions on ops.  Instead, we're
designing cruft into the protocol on a per-operation basis.

Since we've decided that there's no implied locking across
ops, we may as well remove VERIFY and NVERIFY.  They're
useless.

	Brent

---

From: Neil Brown <[email-masked]>
Date: Tue, 27 Mar 2001 16:45:29 +1000 (EST)

> VERIFY doesn't cut it because there's no atomically guarantee between

I guess we should scrap the example from seciont 14.2.35 of RFC3010
then....
However the intent here was protection against a stray duplicate, and
I don't think you need atomicity to protect against a stray
duplicate.

On a related note, I just realised that the lack of "rmdir" combined
with the lack of atomicity of VERIFY with the following OP means that
that you cannot create a posixly-correct rmdir op, or unlink for that
matter.
The closest you can get is:

    PUTFH (directory filehandle)
    SAVEFH
    LOOKUP (filename)
    VERIFY (ftype == DIRECTORY)
    RESTOREFH
    REMOVE (filename)

and there is still a window where another client could remove the
directory and create a file, which would then be inappropriately
deleted.

So the question is: is this a real issue?  Quite possibly not.  It is
hard to see this being a real problem in real life.  But if we agree
that is isn't an important issue, then where do we draw the line
between Posix and NFS?

> what if the client doesn't follow through?

What if a client does an EXCLUSIVE CREATE and doesn't follow through
with a SETATTR?  You get a file with very odd metadata.
What if a client does an unstable write and never follows through
with a COMMIT, (and so doesn't discover that the server restarted and
some of the data was lost).  A different client could have seen the
new data before the server restart, and then suddenly sees old data
after the restart.  Very odd.

You could avoid the issue with append by sending a sequence of "I am
going to do a big write_append soon, here is part of the data I will
be using" requests followed by a "ok, here is the last bit, do the
append now" request.

This requires extra resources on the server ofcourse but I don't see
that as a big problem, except that it might be extra code complexity
that is infrequently exercised.

> consistency; forgive me if I take your idealism with a large grain of

I confess to the charge of idealism, but I also note that RFC1813 says:

 However, caching is not required, nor is any caching policy defined
 by the protocol. 

so maybe implementers have an excuse?

> without a callback, the latter two being incorrect.

Normally a delegation is recalled with a call back and the server will
not issue a delegation without confirming that it can make a callback.
However there is (currently) one exception to this.

In section 9.2.1, the paragraph which start "When the server reboots
or restarts..."  dicusses a scenario where the server may give a
delegation without first establishing that a calback can be made.
This happens when the client opens with CLAIM_DELGATE_PREV.
In this case "a special designation is applied so that the client
treats the delegation as having been granted but recalled by the
server."

I am suggestion that in response to an OPEN(append), the server might
apply a similar designation to the delegation that is given.  The
client would then perform the write request to append the data as
necessary, and then would return the delegation immediately.
The server might do this in a situation where multiple clients
appeared to be actively appending to the file, or where the server had
not been able to establish a callback path.

> Why will it have the file already open?

I imagine the client would OPEN(write) when first gaining access to
the file.  It would not add the "append" flag until it actually had
data that it actually wanted to write.  It would then OPEN(append) to
upgrade the open.  If it didn't get a normal delegation at that point,
it would write out the data as best it could and then OPEN_DOWNGRADE
to remove the append flag until next it actually wanted to append.

And finally, in response to your:

>   Perhaps your

This is certainly something I have thought about.  I could possibly
work for read delegations, but would require an uncomfortable small
lease time to work for write delegations.
The "recall" flag would just say "there is something that is being
recalled for this client", and the result of RENEW would be extended
to include a list of delegations that have been recalled (and locks
that have been broken and ...).

Thanks for your comments.

NeilBrown