Subject: RE: [nfsv4] re: byte range delegations

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 14 Jul 2005 16:51:36 -0400

>  - append writes

Although you can do append writes with this facility, it doesn't
seem like the best way to do that, unless each append-writer is
going to append more than can be done in a single write or 
do many individual writes before it is likely that a different 
client will do so.

In most cases, the best way to do an append write is simply to
create an append option for WRITE and let the server put what 
you give him at the end.  It would helpful to have the byte-range
delegations to round out the support for enormous append writes but 
most people would be much better off with a simple append-
this-to-the-end-of-the-file-please facility.  A big issue with 
this used to be sequencing/duplicates but sessions solves that.

-----Original Message-----
From: Trond Myklebust [mailto:[email-masked]] 
Sent: Thursday, July 14, 2005 1:51 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [nfsv4] re: byte range delegations

to den 14.07.2005 Klokka 11:28 (-0400) skreiv
[email-masked]:

> Are people clammering for better cache coherency and arguing that byte
> range locks aren't sufficient for the cases they care about?

That's a very rhetorical question.

        The people that are happy to use NFSv2/v3 today? Probably not so
        much.

        The people that chose to go with a clustered filesystem instead?
        Yes, they probably feel quite strongly about the issue.

8-)
---

More seriously:

In the draft we list two commonly used operations that NFS currently
cannot support properly and that are directly linked to the coherency
issue:

 - shared mmap()
 - append writes

There are quite a few applications out there that rely on those working
correctly in a multi-threaded environment. If we want to help port those
applications to a clustered environment without forcing people to
redesign their entire program around the I/O quirks of the filesystem we
happen to have on offer, then we need to be able to present them with
something that goes beyond close-to-open cache consistency.

We also make the point that byte range locks cause you to take a huge
performance hit. They basically only work in an environment where you
are using uncached I/O. If you are the sole user of the file, then file
delegations help. If you are not, then irrespective of whether the
processes are behaving in a manner that causes caching conflicts, you
will still be reduced to doing synchronous I/O.

Finally, I assume that pNFS with non-files based back ends (e.g. objects
and blocks) may want to be able to do atomic read-modify-write cycles on
their data in order to support direct writes and such.

Cheers,
  Trond

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 14 Jul 2005 17:42:21 -0400

> this used to be sequencing/duplicates but sessions solves that.

AFAICS that will limit you to running on non-pNFS systems.

I don't see how you can get round the cache consistency problem the
moment you introduce the notion of stripes. The client _has_ to know to
which data server it should send the write. Furthermore, the write has
to be done atomically, making it impossible to cross a striping boundary
unless you have some form of locking.

Cheers,
  Trond

---

From: "Gordon Waidhofer" <[email-masked]>
Date: Thu, 14 Jul 2005 15:06:24 -0700

I talked to somebody about an "append" flag
to write, or another write op, years ago.
I was told that the idempotency issues made
it pretty much impossible. Suppose the reply
gets lots and the server re-issues the append.
Does anything in NFSv4 overcome this problem?

That said, I still think it would be darn handy,
warts and all.

Regards,
	-gww

> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Noveck, Dave
> Sent: Thursday, July 14, 2005 1:52 PM
> To: Trond Myklebust; [email-masked]
> Cc: [email-masked]
> Subject: RE: [nfsv4] re: byte range delegations
> 
> 
> >  - append writes
>

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 14 Jul 2005 17:16:30 -0500

> That said, I still think it would be darn handy,

I think an append flag would be wonderful.

Reply loss can be handled with an append xid -- yes, more state for the
server to keep, but state can be limited on a per-client-id/FH basis and
could be a multi-round-trip operation (i.e., require confirmation).

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 14 Jul 2005 17:27:19 -0500

> do many individual writes before it is likely that a different 

Typically the maximum append write size for which atomicity is
guaranteed is relatively small.

---

From: "Gordon Waidhofer" <[email-masked]>
Date: Thu, 14 Jul 2005 15:26:51 -0700

> could be a multi-round-trip operation (i.e., require confirmation).

You don't even have to try that hard.

The reply cache on the server can match the XID and
issue the same reply. Won't work trans-reboot, though.
That's the rub that kept it from being done before.

As soon as you defend append-write with a lot of
state management, it isn't worth doing anymore.
Existing mechanisms can handle it and no new
mechanisms need be introduced.

A best-effort append-write appeals to me. But I
certainly understand why it wasn't done before.
Print "best-effort" in 72 point type, and still
somebody somewhere will expect guaranteed behavior.

Regards,
	-gww

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 14 Jul 2005 17:32:21 -0500

> mechanisms need be introduced.

Yes, but IIRC you then end up having byte-range locking conventions that
might conflict with applications.

Nico
--

---

From: David Robinson <[email-masked]>
Date: Thu, 14 Jul 2005 17:33:14 -0500

> That said, I still think it would be darn handy,

Append writes from a Posix client perspective is very
hard, as Trond points out, these types of clients
really want to know where its write actually was
placed. Generally it can only do that by either
dictating the offset (which is really not an append)
or acquiring exclusive access which becomes a
consistency protocol issue many are not willing to
take on.

However, from a server perspective append writes are
almost trivial.  The server always knows where the
end of a file is and can easily arbitrate two clients
simultaneously wanting to do an append write.  As a
case study, the Google filesystem is designed to
do exactly this, many clients almost always appending
to the same file. The key requirements are that it
must be an append and the client doesn't need to know
where it was placed.  How they use their data doesn't
require this.

So while I would strongly support a flag in the WRITE op to tell
the server to perform a true append write as it will
solve some very interesting applications (log files etc),
it would not solve my more general Posix append
write problem. The latter has held the former hostage
for too long.

	-David