Subject: RE: idea - atomic append for NFS writes

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 16 Mar 2001 06:41:02 -0800

Brent Callaghan wrote:
> "Wittle, Mark" wrote:
> > This proposal applies to the NFS WRITE operation.  The
> > proposal adds an additional "atomic_append" flag (or enum)
> > field to the struct WRITE4args. 
>
> Why bother with a flag.  Couldn't we just define the
> maximum file offset value (all ones) as meaning the
> current end of file ?
> 
> I'm not sure how this would be used by current clients.
> They already have a "find end of file, write to that offset"
> procedure above the NFS layer.

I don't know about all of them, but lots of UNIX VFS interfaces
pass an APPEND flag into VOP_WRITE and give the responsibility
of determining the EOF at which to write to fs-dependent code.

Since VFS interfaces don't usually leave things locked
between VOP calls, it would be hard to make append mode IO
work correctly otherwise, even on local disk.

> Assuming you've got some unconventional clients that don't
> have this restriction, the worry is that two clients will
> concurrently determine the current end of file and write
> to it - one overwriting the other.  You could get pretty
> close to atomic if you included a VERIFY op for the file
> size prefixing the WRITE op.

So in the current protocol with additional effort and latency 
you might be able to get something that almost worked (i.e. 
"close to atomic").  This is why I think that append write 
performed correctly on the server would be a good thing for 
NFS to have.

---

From: Peter Staubach <[email-masked]>
Date: Fri, 16 Mar 2001 09:11:09 -0700 (MST)

> performed correctly on the server would be a good thing for 

The real problem with append mode writes is making them atomic w.r.t.
the client.  You have to implement the support which will do the write
either 0 or 1 times and you have to be able to tell which one.  The
fact that you could lose the delegation in the middle of one of these
sequences means that the original guarantees are lost.

We talked long and hard about this when we were designing the NFS
Version 3 protocol.  It seems easy, until you have to worry about the
record only getting written once.  Having the record written more than
once means that it is much less interesting to applications.

An alternative to append mode writes is reservations.  If the client
can "reserve" a number of bytes at the current end of the file, then
that client can do the write whenever it wishes and can verify that
the write happened.  Yuo do need support in the server to support these
persistent reservations though.

		ps

---

From: Mike Eisler <[email-masked]>
Date: Fri, 16 Mar 2001 09:11:09 -0800 (PST)

As Peter points out, atomic appends are probably (I say probably,
because it hasn't been proven they can't be done, its just that every
proposal has been found to have holes) impossible, although if the
server has delegated write capability to the client, as long as the
delegation is in force, appends come for free.

In Mark's proposal, what I envision occuring when (1) there is no
delegation in force, and (2) high load on the server such that the
duplicate request cache is blown, is duplicate records from the same
client written at different offsets. I think, and it has been the
consensus of the NFS community during the long NFSv3 and NFSv4 processes,
that this is worse than the current situation, where only if write
sharing is going on does the file get corrupted during appends.

Peter's notion of "reservations" at the end of the file is interesting,
but (1) as he points out, persistent reservations are required, and (2)
then there's the problem of what if the client doesn't follow through
with the write.  To me, those two issues make his append proposal less
palatable than the status quo.

The use of a COMPOUND with VERIFY (file size) followed by WRITE comes
close, but because there's no atomicity guarantee between operations,
it comes up short. Still, it might be a good idea for NFS client's
simulating append to use VERIFY to reduce corruption during
write sharing. But it's early (for me) this morning, and I'm
probably missing something.

	-mre

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 16 Mar 2001 10:32:03 -0800

> >I know I'm being heretical here, but I'm beginning to think that NFS
> >may be at the point of exhausting the universe of things that can be
> >done effectively with at-least-once semantics.  Unless we are prepared
> >to keep giving people things (like append writes) that work most of
> >the time and tell them to stop complaining, we may have to bite the
> 
> I'm not sure I've ever had a customer complaint about appends not
> working. 

Alright.  We don't have to tell them to stop complaining because
they know that complaining is useless.  A basic protocol problem
like that cannot be solved in any time frame that a customer
cares about.  He'll just do something else, besides using NFS,
I mean.

> >bullet and solve the general problem.
> >
> >     Exactly-once Semantics -- Accept No Substitutes
> 
> Maybe The Open Group and IBM will open the DCE/DFS protocol
> specifications and source code. :-)

Gosh, I hope not.

---

From: "Corbett, Peter" <[email-masked]>
Date: Fri, 16 Mar 2001 10:58:39 -0800

If multiple clients are doing appends to a log, which is a perfectly good
use of append writes, then it can't possibly be preferable to (fairly
frequently) corrupt the file, at a minimum destroying a record and worst
case leaving the file unparsable, than it is to (very occasionally) insert a
duplicate record.  The latter should be a minor problem that could be solved
at the application level. 

If the server's reply cache is blown out, if part of the workload is several
clients doing write appends to the same file, then its only going to make
the situation worse if the clients get into a {getattr, verify -> fail}
retry loop competing for the same resource with the eof moving underneath
them.  The server could blow up completely under that load.  Again, the
possibility of an occasional duplicate record seems preferable.

The file might get corrupted "only if write sharing is going on", but isn't
that the most useful canonical case of append writes?

-----Original Message-----
From: Mike Eisler [mailto:[email-masked]]
Sent: Friday, March 16, 2001 12:11 PM
To: [email-masked]'
Subject: Re: idea - atomic append for NFS writes

	-mre

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 22 Mar 2001 13:05:21 -0800

Brent Callaghan wrote:
> Mike Eisler wrote:
> > 
> > >From: Neil Brown <[email-masked]>
> > >guard against that (though you can in v4 using VERIFY).
> > 
> > VERIFY doesn't cut it because there's no atomically guarantee between
> > operations in COMPOUND. 
>
> I think the lack of inter-operation locking was a mistake.

I think it depends on what the alternative is.  I definitely
prefer what we have to a fully general mechanism that allows
arbitrary stuff to be done atomically.

> The intent of VERIFY and NVERIFY was to simplify the protocol
> via the addition of pre-conditions on ops. 

I can see that a version of VERIFY/NVERIFY that required the
decision to be made atomically with the following op might be 
a useful extension to consider for 4.1.

> Instead, we're
> designing cruft into the protocol on a per-operation basis.

I'm skeptical that any mechanism that is general enough to
eliminate what you call cruft will strike me as simple enough to
be reasonably and efficiently implementable, but I suppose
I (sort of) look forward to debating the issue, if we have
a specific proposal for 4.1.  I hope you understand I'm not 
pushing you to present one soon.  I have plenty of stuff to 
implement now.

> Since we've decided that there's no implied locking across
> ops, we may as well remove VERIFY and NVERIFY.  They're
> useless.

Are they totally useless?  What are clients using them for?  I know
the Solaris client is using one of them because it found some bugs 
in my server's implementation.