Subject: RE: idea - atomic append for NFS writes

From: Mike Eisler <[email-masked]>
Date: Wed, 21 Mar 2001 09:21:36 -0800 (PST)

In summary,

	- You are claiming GUARDED creates as the justification for the holes.
		However I don't quite see the analogy. The delta of
		server implementation burden from GUARDED to
		EXCLUSIVE is minimal, whereas the delta from no delegations
		to delegations is orders of magnitude larger in development
		effort. 
		
	- As far as I can tell, the v3 and v4 specifications don't talk 
		about GUARDED as an alternative to EXCLUSIVE create. When 
		the v3 specification came out, I figured it was at best
		redundant, and probably useless, but Peter Staubach found a
		rather nifty, yet counter-intuitive purpose for GUARDED 
		creates. He uses them in the Solaris v3 client to prevent a
		stray duplicate, truncating, non-exclusive create from stepping
		over the sequence of the successful truncating, non-exclusive
		create, followed by a write to extend the file. As far as I 
		can tell, GUARDED works perfectly for this purpose.  If you 
		are building your case for your append protocol on GUARDED, 
		then it's shakey. Besides, the only servers that cannot do 
		true EXCLUSIVE semantics are those with no arbitrarily 
		mutable attributes. That's a pretty small set; zero perhaps. 
		
	- You suggest returning EINVAL to an application that wants to
		write more bytes than the server allows, whether enforced
		by the maximum transfer size attribute, or a short write
		from the server. Applications that want to do an append
		today over NFS don't get this error today; I don't see
		the win. Again, if you are going back to GUARDED,
		the file exists error a GAURDED create might return from
		the server never reaches the
		application, because the application is doing a
		non-exclusive create.
		
	- You point out that (as did I several days ago when this thread
		started) that if the server supports delegations, then
		true, error free, append semantics are possible. However
		no enhancement to the protocol is needed in this case,
		though I see no harm in adding an append flag to
		provide a hint to the server that this delegation is
		of more importance than the client's non-append write
		delegations. 
		
	- In the event of a server crash, duplicate records will appear
		in the file, unless the server implements a fair
		amount of state in stable storage. Currently clients do
		not experience duplicate records when doing an APPEND.
		
	- In abscence of an append delegation, append writes will be
		at NFSv2 speeds. Currently they are done at NFSv3 speeds.
		
In my opinion, the above disadvantages outweigh the advantages of
your append protocol, and I don't think it should be part of the
protocol; certainly not a mandatory part.
		
>In this APPEND case, there server does not need to be able to
>call-back the client to have the delegation released, so that is one
>possible impediment to delegations that need not be a problem.

What if another client wants to write or read the file that has
an APPEND delegation on it?

>So if a server does not implement delegations, then the client can
>expect some reduction in service, particularly that O_APPEND writes
>will be synchronous and may result in EINVAL for particularly large
>O_APPEND writes.
>
>I think that this is very appropriate and is similar to the GUARDED
>mode for create.  GUARDED allows the server to provide second-best
>semantics for exclusive create if it cannot provide correct EXCLUSIVE
>semantics.  Similarly, WRITE_APPEND allows the server to provide

I'm curious if anyone has ever seen a GUARDED mode exclusive create?
The NFSv3 and NFSv4 specifications don't explain the purpose of GUARDED
well, so perhaps there are clients that do so. My suggestion to any
implementors reading this is to stop using GUARDED for exclusive creates.

	-mre

---

From: Neil Brown <[email-masked]>
Date: Thu, 22 Mar 2001 18:00:45 +1100 (EST)

> 	- You are claiming GUARDED creates as the justification for the holes.

Maybe I misunderstood the purpose of GUARDED.  I was reasoning from
its proximity to EXCLUSIVE in the spec.  Given the lack of clear
justification for it's existance, it seems reasonable (but apparently
false) to assume that most obvious purpose was the intended one.

Your description of the use of GUARDED in Solaris NFSv3 is interesting
and valid, though it seems like a small plug in a large hole.
A scenario that must be of equal concern in a stray duplicate UNLINK
which unlinks a file that a subsequent CREATE created, and you cannot
guard against that (though you can in v4 using VERIFY).

> 		the win.

Ok, that wasn't such a good idea.  Do I get a second chance?

The WRITE_APPEND op contains a "record-length" as well as the
"data-length", and the return value includes a "record-offset".
The record-length indicates how much data the client wants to
atomically append.
The data-length indicates how much data is in the request.  This bit
goes at the end (or maybe the beginning, whichever is easier for
servers). The remainder of the record (if any) is filled with nuls.
The "record-offset" indicates the file address of the start of the
appended record.  The client can then fill in the remainder with
normal writes.
Question: How easy is it to get that record-offset back atomically
though the VNODE or WIN32 interfaces?  Maybe too hard?

> 		at NFSv2 speeds. Currently they are done at NFSv3 speeds.

And currently, clients can be certain that contemporaneous appends
from separate clients will corrupt the file.
So, do you want broken-semantic-A or broken-semantic-B?
Which is more dangerous?
Which happens more often in practice?
Which really hurts the most?

I suspect most serious NFSv4 servers will bend over backwards to make
delegations work because that is, I think, one of the great strengths
of NFSv4.  If that is true, then the speed issue is probably moot, and
the correctness is paramount.

> an APPEND delegation on it?

Maybe you missed that part of the proposal.

] The "append" flag gets squeezed into OPEN somehow, it isn't important
] how.  As the client will already have the file open, this will, in
] effect, upgrade the open to also have the "append" flag.
] In this situation, the server is "strongly encouraged" to grant a
] write delegation, though it may set the "recall" flag in the response
] to say the the client should return the delegation as soon as
] possible.

The "recall" flag is the important point.  I am suggesting that the
server issues a delegation even if there is contention, or if there is
no-way to reclaim the delegation with a call-back.  In these cases it
sets the "recall" flag in the delegation so that the client will
release the delegation as soon as it has flushed it's appends.

If you reject the APPEND_WRITE part, then that just leaves these two
small changes to the protocol:
  1/ adding an "append" flag to open to signal that a delegation would
     be "a real good idea".
  2/ allowing the server to signal 'recall' in a non CLAIM_PREVIOUS
     delegation, particularly in response to an "append" open.

I should say at this point that I am to some degree playing the devil's
advocate.  I plan to use append-over-nfs to much the same degree that
I use exclusive-create-over-nfs or even locking-over-nfs. And that is
"not at all".  The only place that I really need locking at the moment
is for my mail file, and "create-unique, hardlink, test" serves fine
for that. 
I just wanted to see how far we can push the protocol to meet
perceived needs.  I am of the conclusion that designing things into
the protocol is fairly easy.  Making it easy for these features to be
implemented in existing environments (whether posix or win32) is where
the troubles come.  My response to that observation is the thought that
we should make features "possible to implement perfectly" and "easy to
implement a best-effort" and let the client and server (and user and
administrator) negotiate.

NeilBrown