Subject: RE: [nfsv4] re: byte range delegations

From: Mike Eisler <[email-masked]>
Date: Thu, 14 Jul 2005 15:56:20 -0700 (PDT)

It's been a while since I've had to slay non-cache-coherent
APPENDs. Let's see if I can do it again. :-)

> -----Original Message-----
> From: David Robinson [mailto:[email-masked]] 
> Sent: Thursday, July 14, 2005 3:33 PM
> To: [email-masked]
> Subject: Re: [nfsv4] re: byte range delegations
> 
> 
> Gordon Waidhofer wrote:
> > I talked to somebody about an "append" flag
> > to write, or another write op, years ago.
> > I was told that the idempotency issues made
> > it pretty much impossible. Suppose the reply
> > gets lots and the server re-issues the append.
> > Does anything in NFSv4 overcome this problem?
> > 
> > That said, I still think it would be darn handy,
> > warts and all.
> 
> Append writes from a Posix client perspective is very
> hard, as Trond points out, these types of clients
> really want to know where its write actually was
> placed. Generally it can only do that by either
> dictating the offset (which is really not an append)
> or acquiring exclusive access which becomes a
> consistency protocol issue many are not willing to
> take on.

Well stated.

> However, from a server perspective append writes are
> almost trivial.  The server always knows where the
> end of a file is and can easily arbitrate two clients
> simultaneously wanting to do an append write.  As a
> case study, the Google filesystem is designed to
> do exactly this, many clients almost always appending
> to the same file. The key requirements are that it
> must be an append and the client doesn't need to know
> where it was placed.  How they use their data doesn't
> require this.

Almost, but not quite trivial.

There's the small matter of the client that sends an append request,
the session/connection breaks before the response is sent, and the
client retries.

NFSv4 sessions solve this if the server hasn't crashed, and it keeps
the replay cache around indefinitely.

If the server has crashed, but the replay cache is not kept in
stable storage, or in general, the replay cache is lost, then
upon retry, the file can have repeat records.

How does the Google file system prevent this? 

> So while I would strongly support a flag in the WRITE op to tell
> the server to perform a true append write as it will
> solve some very interesting applications (log files etc),
> it would not solve my more general Posix append
> write problem. The latter has held the former hostage
> for too long.

Assuming that many if not most servers won't keep the replay cache
in stable storage, I'm not a supporter, unless duplicate
records (read: file corruption) are acceptable. For a human
readable message log file, I could live with it. For a 
database log file, I can't.

Besides, which APIs will use this, since POSIX ones cannot?

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 14 Jul 2005 19:35:44 -0400

> How does the Google file system prevent this? 

AFAICS these appends all have to be stable writes too (even sessions
won't help you there). That will slow things down considerably on most
servers.
I'm not sure that a byte range delegation wouldn't be more efficient for
most logging purposes, as that _will_ allow you to use unstable writes,
and only commit to disk when you have to give up the delegation due to a
non-append read or write.

Finally, as I said earlier, there is the issue of implementing it on a
non-traditional architecture such as pNFS.

Cheers,
  Trond

---

From: Mike Eisler <[email-masked]>
Date: Thu, 14 Jul 2005 17:10:46 -0700 (PDT)

> won't help you there). That will slow things down considerably on most

Yes I vaguely recall making a similar point the last time this came up.
But that was before I was employed by NetApp. Now that I'm an
unbiased NetApp employee and stockholder, I now withdraw my
objection. :-) (just kidding for the emoticon impaired).

> non-append read or write.

I agree ... *if* append writes are going to be done, byte range delegations
are the only way I can see to do them efficiently and correctly on all
server architectures. Hence my statement: "It's been a while since I've had to slay
non-cache-coherent APPENDs". 

> non-traditional architecture such as pNFS.

Yep.