Subject: [nfsv4] EOF changes (truncation)

From: [email-masked]
Date: Thu, 29 Dec 2005 18:18:50 -0500

Catching up on old email (December's been a very busy month)...

> > > Dean Hildebrand
> > Garth Goodson
> Benny Halevy
David Black

> > > 2) Trond and I were talking about some odd truncate scenarios with the
> > > lastbytewritten field.
> > > s1) Client A believe the file is 20 bytes.  Client retrieves a layout
for
> > > and writes to bytes 10-14.
> > > s2) Client B truncates the file to 12 bytes.
> > > s3) Client A performs LAYOUTCOMMIT with lastbytewritten=14 (although
it
> > > still believes the file is 20 bytes)
> > > 
> > > In this scenario, it would be best for the server to update the file
size
> > > to 15.  If lastbytewritten was changed to local_file_size, it would be
set
> > > to 20, which is wrong.
> > > 
> > Remember, last_byte_written is a hint for the server so that it knows if

> > it should put in the work to figure out the file size.  At LAYOUTCOMMIT 
> > time the server is free to query the data servers to find out the real 
> > file size (at least for the file-based layout).  LAYOUTCOMMIT should not

> > be used to truncate the file (SETATTR should be used for this purpose).
> > 
> > So it depends on if the server received Client B's SETATTR for 12 bytes 
> > prior to receiving the LAYOUTCOMMIT.  Also, I believe (for file-based 
> > layouts), the server should verify the size if it is changing the file's

> > size at LAYOUTCOMMIT time (it can use the current file size and the last

> > byte written to make this judgement).
> 
> Agreed.  If the server allowed the truncate while other clients
> are holding layouts (and it didn't recall them), it doesn't know
> if the lastbytewritten refers to a write that happened before
> or after the truncate.  Therefore the server must query the storage
> servers to determine the actual file length.  That said, I'd 
> recommend that the MDS will recall write layouts when getting
> a length-changing SETATTR to force serialization across the
> truncate.

One could certainly do that, but it's rather disruptive when there
are outstanding layouts.  I agree with Benny about the underlying
principle - there needs to be a synchronized view of what has happened.

HighRoad handles this by sending a "set EOF" callback to all clients
that have a layout for the file and specifying that the server must
not handle commits that may be affected by the EOF change until all
the clients get the EOF change.  Some of this happens automagically
as the callback bumps the sequence number causing any in-flight
commit to be rejected with a stale sequence number.

The advantage of this approach is that it forces clients to resolve
whether a write occurred before vs. after the truncate.  In the above
example, both 15 and 13 could be correct answers for how long the file
is, and client A needs to make the decision as to whether its write
occurred before or after B's truncate, as communicated by the EOF
callback.  The EOF callback forces that decision to be made, and
does so without any layout recalls.  If A) and B) are communicating
out of band, pNFS as currently specified can produce incorrect behavior:

s1) Client A believe the file is 20 bytes.  Client retrieves a layout
	for and writes to bytes 10-14.
New: s1.a) Client A tells client B that it has written to the file
s2) Client B truncates the file to 12 bytes.

In this situation, the only right answer for the file length is 13,
and B's truncate must be passed to A to make this happen.

The disadvantage of this approach is that the EOF handling code can
be somewhat subtle (e.g., in the above example in the block world,
client A is responsible for zeroing out the block beyond the new
EOF because it holds the only write extent for that block).

HighRoad is specified in terms of clients passing the new EOF on a
commit (because the client always receives notification of truncates),
but the server behavior is equivalent to the pNFS last_byte_written
approach because the new EOF can only extend the file - if a client
wants to truncate, it has to issue a SETATTR to do so.

There's a new version of the pNFS block draft coming shortly - I'll
make sure this issue is noted in it.  I think we have some more
design work to do.

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "Halevy, Benny" <[email-masked]>
Date: Fri, 30 Dec 2005 07:25:07 -0500

David, this approach can be implemented with pnfs CB_SIZECHANGED
(DESCRIPTION copied below for reference). 

   DESCRIPTION

   The CB_SIZECHANGED operation is used to notify the client that the
   size pertaining to the filehandle associated with "fh", has changed.
   The new size is specified.  Upon reception of this notification
   callback, the client should update its internal size for the file.
   If the layout being held for the file is of the NFSv4 file layout
   type, then the size field within that layout should be updated (see
   Section 5.5).  For other layout types see Section 3.4.2 for more
   details.

   If the handle specified is not one for which the client holds a
   layout, an NFS4ERR_BADHANDLE error is returned.

The sequence number semantics you described below do not exist in pnfs and
I'm not sure if we absolutely need them. At the same time I'm not sure how
to resolve the race happening when the server truncates the file while other
clients may extend it by writing using the layout. How would the server know
who was the last one to set the file's length - the server or any of the
clients?  For example:

1. In parallel:
  1.a Client A writes up to offset L
  1.b Server truncates the file to offset S (S < L) on behalf of client B's
SETATTR.

2. In parallel:
  2.a Server sends CB_SIZECHANGED with size=S to client A
  2.b Client A sends LAYOUT_COMMIT with total_write_size=L

The server must reject LAYOUT_COMMIT while CB_SIZECHANGED is in flight but
can the client trust size=S?  Is the file size really S and not L?
With OSDs we can theoretically solve that by rotating the capability version
on the object atomically with truncating it in order to implicitly
invalidate any outstanding caps and this will force clients to refresh their
layouts - which we wanted to avoid with CB_SIZECHANGED :-/  Currently
Panasas simply recalls all outstanding layouts on truncate to make sure the
server is the only writer to the file.

If we want do better than that I'm inclined to add quiesce and resume
callbacks to be sent to clients with write layouts while truncating the file
(and possibly in other cases, e.g. for taking a snapshot).  When all
CB_QUIESCE calls complete successfully the server can truncate the file and
then send CB_SIZECHANGED+CB_RESUME callbacks meanwhile rejecting any
LAYOUTGET, LAYOUTCOMMIT, and LAYOUTRETURN.

Benny

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Friday, December 30, 2005 1:19 AM
To: [email-masked]; [email-masked]; [email-masked]
Cc: [email-masked]
Subject: EOF changes (truncation)

David Black

for
it
size
set

Thanks,
--David

---

From: [email-masked]
Date: Fri, 30 Dec 2005 14:52:11 -0500

Benny,

> (DESCRIPTION copied below for reference).

Right, I knew that callback was there, as I'd asked for it a long time ago.

> I'm not sure if we absolutely need them.

Right, HighRoad used sequence numbers, and I know we've been trying to
keep them out of pNFS (they were initially present as layout stateids,
but they were removed [IIRC] for simplicity).  My primary concern is
to make sure we've gone through the design space in enough detail to
understand the decisions we've made and their consequences.

In the situation where there's a commit that is concurrent with a
conflicting sizechanged callback, the sequence number approach drops
the commit on the floor if it could cause problems, whereas trying
to solve this without sequence numbers appears to run the risk of the
commit executing out-of-order (e.g., it's delayed in transit and arrives
after the sizechanged callback and reply).  Sessions don't solve this
problem, and so allowing this concurrency would appear to require a
separate SIZECHANGED_CONFIRM operation since the callback needs to be
responded to immediately, and that new operation does not seem like
a good idea.

Related note: The first paragraph in Section 7.5.3 of the 4.1 draft says
that pNFS relies on sessions for ordering.  I don't think that works,
and hence I think that text needs to be removed.  My current understanding
of sessions is that they order operations that use the same slotid within
a session, but do not globally order operations in a session.  IMHO, it
would be wrong to restrict all operations on any individual layout to
a single slotid (I can already hear Tom gagging on that idea ;-) ).

This suggests that all of us need to carefully re-check the pNFS
protocol for races based on out-of-order operation execution,
RPCs getting reordered on the wire, etc. because sessions won't help.

Returning to the EOF behavior:

> At the same time I'm not sure how

In HighRoad, the writing client resolves the race.  If I've understood
the scenario correctly, B's SETATTR gets ordered after A's writes
courtesy of CB_SIZECHANGED showing up after A's writes.

> The server must reject LAYOUT_COMMIT while CB_SIZECHANGED is in flight

It's actually worse - the LAYOUT_COMMIT must be rejected after
CB_SIZECHANGED is complete in order to avoid the wrong file size.

> but can the client trust size=S?  Is the file size really S and not L?

The client should assume that the CB_SIZECHANGED is the propagation
of the SETATTR, and then order the client's writes as having occurred
before vs. after arrival of that callback.  If all the client's writes
occurred before the callback arrived, the size is S.  If the client
has a write that extends the file to L after the arrival of the callback,
the size is L.

> server is the only writer to the file.

I think I'm headed here as well, but will need to think about it some more.
At the very least, we're going to have to write this up in Section 7.4.2
of the 4.1 draft which currently doesn't contain anything that alerts
implementers to the possible problem here.

In essence, because HighRoad ordered all client operations wrt server
operations (every client operation carries a server sequence number),
certain sequencing decisions (did the commit happen before or after
the callback) can be made at the client with the knowledge that stale
operations will get dropped on the floor at the server.  With pNFS not
having done this ordering, one has to fall back to the server doing
the sequencing.  For the truncate case, that results in a write layout
recall - as long as the "truncate in presence of conflicting write layout
at another client" case is rare, the simple brute force approach of
recalling the layout may be the better way to handle this; this also.
We should discuss this on the 4.1 draft call, and I do share your
reluctance to bring back layout stateids.

At least in the blocks world, there's another race or races
involving who is supposed to zero the remainder of the block that
contains the new EOF in the extension case because one can't ask
the storage server how long the file really is (in the blocks world,
it doesn't know).  HighRoad uses an error/status return value to 
tell the server that a client zeroed the remainder of the block
containing the new EOF.  If the server doesn't get this value back
from the client with the write layout for that block, the server has
to do the zeroing itself, which involves recalling the layout and
a read-modify[to zero]-write sequence.  I'm not sure what to do for
pNFS, as this is a nasty case where the type of layout affects what
is done in the layout-independent portion of pNFS (don't have to
zero the rest of the block in the file or object layouts, as the
storage server handles this).  Do we want to force a layout recall
whenever the EOF moves?  In any case, I'll add a section on EOF
issues to the (coming real soon now) next version of the pNFS blocks
draft.

Thanks,
--David

> From: Halevy, Benny [mailto:[email-masked]] 
> Sent: Friday, December 30, 2005 7:25 AM
> To: Black, David; Halevy, Benny; [email-masked]; 
> [email-masked]; Welch, Brent
> Cc: [email-masked]
> Subject: RE: EOF changes (truncation)
> 
> > > > s1) Client A believe the file is 20 bytes.  Client retrieves a
layout for
it
size
> > > > to 15.  If lastbytewritten was changed to local_file_size, it would
be set