Subject: RE: [nfsv4] re: byte range delegations

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 14 Jul 2005 19:21:52 -0400

> > In most cases, the best way to do an append write is simply to
> > create an append option for WRITE and let the server put what 
> > you give him at the end.  It would helpful to have the byte-range
> > delegations to round out the support for enormous append writes but 
> > most people would be much better off with a simple append-
> > this-to-the-end-of-the-file-please facility.  A big issue with 
> > this used to be sequencing/duplicates but sessions solves that.

> AFAICS that will limit you to running on non-pNFS systems.

Not at all.  It limits you to running without using the pNFS features.
pNFS is an extension, which you can use or not, depending on whether 
it is advantageous.

Clearly if it were better, for this application, to get a byte-range
delegation, write directly to the data server, then return that 
delegation, then you would want to use the pNFS features, and anything
that didn't lend itself to that mode operation would be bad.  But 
that's not the case here.  In fact, doing this using byte-range
delegations in pNFS is much worse than using append-writes.  

If you need to go to the metadata-server and get a byte-range
delegation, write to a different server (the data server), and then
you have to release the byte-range delegation with another request 
back to the data server.  Three round-trips every time you write 
and another client was the last one to write.  Instead of the 
pNFS enhancing parallelism, you have something that is very serial,
and very low-performing.

On the other hand, if you are writing to the metadata server, there
is no wait for other machines.  Multiple cleints can append-write at
the same time.  Write can be batched up (if buffered in stable storage)
and then sent to the data server in large blocks.  While that write
is going on clients can continue to write and get responses.  You
probably have a slight hiccup as you reach the end of one stripe
but the metadata server can continue to buffer stuff through that.
The overall bandwidth is much higher.

You still have the bandwidth being limited by the throughput of the
metadata server.  But if that is insufficient and you want to do
sequential writes with a higher bandwidth, I simply don't know how
to do it.  Getting byte-range delegations from the meta-data server
is clearly not the way, except under very very special circumstances:
extremely long append writes or a single client writing many many times
before another client writes.

> I don't see how you can get round the cache consistency problem the
> moment you introduce the notion of stripes. The client _has_ to know to
> which data server it should send the write. 

Exactly.  That is the reason that the pNFS mode of operation is a real
loser here.  It guarantees multiple round-trips.  One to find out who
to take to and another to talk to him, at the very least.

> Furthermore, the write has
> to be done atomically, making it impossible to cross a striping boundary
> unless you have some form of locking.

Certainly.  You always need some form of locking.  The question is whether
you need to externalize it for the client to use.  You are assuming that
we have the required locking and we allow the client to use it.  But if 
the server has the ability to lock, then the metadata server can use it 
itself if it needs to cross a stripe boundary.  If we have the locking
facility we can choose to define the protocol to make it directly usable
by the clients, if we feel that is necessary and desirable, but that is
separate choice.

-----Original Message-----
From: Trond Myklebust [mailto:[email-masked]]
Sent: Thursday, July 14, 2005 5:42 PM
To: Noveck, Dave
Cc: [email-masked]; [email-masked]
Subject: RE: [nfsv4] re: byte range delegations

to den 14.07.2005 Klokka 16:51 (-0400) skreiv Noveck, Dave:
> >  - append writes
> 
> Although you can do append writes with this facility, it doesn't
> seem like the best way to do that, unless each append-writer is
> going to append more than can be done in a single write or 
> do many individual writes before it is likely that a different 
> client will do so.
> 
> In most cases, the best way to do an append write is simply to
> create an append option for WRITE and let the server put what 
> you give him at the end.  It would helpful to have the byte-range
> delegations to round out the support for enormous append writes but 
> most people would be much better off with a simple append-
> this-to-the-end-of-the-file-please facility.  A big issue with 
> this used to be sequencing/duplicates but sessions solves that.

AFAICS that will limit you to running on non-pNFS systems.

I don't see how you can get round the cache consistency problem the
moment you introduce the notion of stripes. The client _has_ to know to
which data server it should send the write. Furthermore, the write has
to be done atomically, making it impossible to cross a striping boundary
unless you have some form of locking.

Cheers,
  Trond

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 14 Jul 2005 21:06:37 -0400

> pNFS enhancing parallelism, you have something that is very serial,

Why would you need to write the data and release the delegation? For an
append-only file, you really only have to write the data when a reader
turns up, or when another client issues a conflicting DELEG_RANGE.

A pNFS client could, for instance, use something like the following
compound:

DELEG_RANGE WRITEW_LT, mysize, -1     // Grab a delegation starting
				      // at the estimated eof
GETATTR size			      // Read the file length
VERIFY (mysize == size)		      // Check our estimate against
				      // the true length.
SETATTR size = (mysize + length)      // Extend the file for the
				      // metadata server.
DELEG_RELEASE mysize + length, -1     // Release the delegation beyond
				      // the new eof.

A race will trip a recall, but for many error logging applications, this
should be quite sufficient.

Note. You could perhaps eliminate a few extra recalls by assigning
a special meaning to DELEG_RANGE with an offset of -1 to refer to a
range starting at the end of file. That isn't part of the current draft,
though.

Cheers,
  Trond