Subject: RE: idea - atomic append for NFS writes

From: Mike Eisler <[email-masked]>
Date: Tue, 20 Mar 2001 13:49:30 -0800 (PST)

>  WRITE_APPEND: (cfd) count, stateid, seqid, data -> count
[...]
>If the client doesn't get a delegation, then it sends WRITE_APPEND
>requests as described above, making sure to honour the record boundry
>is multiple write-append requests have been cached-behind.

The use of stateid and seqid in the request would suggest that you are
using the same techniques as file locking to guarantee
at most once semantics for WRITE_APPEND. However, I notice
that you don't return a stateid in results. Even assuming that's an
oversight,

	(1) the amount being written by the application on the client 
	must be equal to that being written in WRITE_APPEND arguments
	and results. 

	As I noted before, what if the maximum NFS server transfer size
	is less than what the application writes? What if the count returned
	by WRITE_APPEND is less than the count in the WRITE_APPEND arguments?

	Correct me if I'm wrong, but don't POSIX append semantics guarantee
	that the a record written with write() be contiguous when it reaches
	the file?
	
	(2) You noted that WRITE_APPEND could force synchronous writes
	even when there is no sharing [and no delegation]. I suppose
	this becomes the choice of a customer, either via a knob or
	him picking an implementation that is fast. I suspect I know
	what the customer will do.
	
	(3) I did not understand recovery from a server crash
	in the event there is no delegation, there was an outstanding
	OPEN(append), the client has sent a WRITE_APPEND, the
	server has written the data, and the server dies before
	response is sent by. My guess is that the client
	re-establishes the OPEN(append), re-sends the WRITE_APPEND,
	and now there's a duplicate record in the file.

>Each append would require two round-trips if a standing delegation
>could not be granted.  This is slower than normal writes, but you
>would expect there to be some cost.

I didn't understand the second round trip.

	-mre

---

From: Neil Brown <[email-masked]>
Date: Wed, 21 Mar 2001 10:42:28 +1100 (EST)

> that you don't return a stateid in results. Even assuming that's an

There is a lot of functional overlap between the seqid and the
stateid.  You do not need both the change to ensure atmost-once
semantics.  As the file doesn't change "state" under our common
understand of "state" (which relates to locks) I chose to not have the
stateid change.  Returning a new stateid as well as implicitly
incrementing the seqid would work exactly was well.  I would be happy
either way.

> 	that the a record written with write() be contiguous when it reaches

You are right that we cannot get exact POSIX semantics in the
WRITE_APPEND case.  If the client were force to fall back to
WRITE_APPEND and could not write the whole record are once, I would
recommend EINVAL.

> 	what the customer will do.

My understanding of delegation is that it is a part of the protocol,
and it *Should* be supported.  It is understood that it may not be
easy for all servers to support it in all situations, so the server
is allowed the not give delegations.
This is, I think, an important aspect of a protocol that aims to
provide high levels of interoperability.  It should set the standard
high, but allow participants to still operate, though with reduced
functionality or performance, if they cannot meet that standard.

In this APPEND case, there server does not need to be able to
call-back the client to have the delegation released, so that is one
possible impediment to delegations that need not be a problem.

So if a server does not implement delegations, then the client can
expect some reduction in service, particularly that O_APPEND writes
will be synchronous and may result in EINVAL for particularly large
O_APPEND writes.

I think that this is very appropriate and is similar to the GUARDED
mode for create.  GUARDED allows the server to provide second-best
semantics for exclusive create if it cannot provide correct EXCLUSIVE
semantics.  Similarly, WRITE_APPEND allows the server to provide
second-best semantics for append writes if it cannot provide correct
semantics.

> 	and now there's a duplicate record in the file.

Exactly.
As I said:

>> Still, server restarts aren't very common, and the worst case scenario
>> in that face of restart would be that a few entries would be appended
>> multiple times. 

The server could conceivably log information about appends to stable
storage atomically with the actual append, and use this information to
acknowledge retry-appends without actually re-appending, but I suspect
that if there server implementor was willing to go to that much
trouble, they would go to the trouble of doing delegations.

This would probably require an extra field in the WRITE_APPEND 
op which says "I tried this before you restarted and that time the
seqid was X".

Again, I claim GUARDED as prior art.  If the server cannot provide the
required semantics, the protocol should give it the opportunity to do
the best it can (and the client must be able to choose whether to
accept second best, or to refuse to communicate).

> I didn't understand the second round trip.

The first round trip tries to get a delegation.  The second round trip
either does a WRITE_APPEND or does a write and returns the delegation.

Possibly a client might deduce, after several failed attempts, that
delegations are not to be had, and might then always use WRITE_APPEND
and thus only spend one round trip.

> 	-mre

NeilBrown