Subject: RE: [nfsv4] RE: pnfs last write offset (fwd)

From: "Halevy, Benny" <[email-masked]>
Date: Tue, 6 Dec 2005 14:20:35 -0500

Dean Hildebrand wrote:
> On Mon, 5 Dec 2005, Halevy, Benny wrote:
> 
> > Garth Goodson wrote:
> > > Dean Hildebrand wrote:
> > > > I apologize to anyone who has received this post multiple
> > > times, but it
> > > > seems to not appear in the NFSv4 archives.
> > > > ...
> > > >
> > > > One question and a comment, sorry if they are obvious:
> > > >
> > > > 1)
> > > > What provisional space are you freeing if you haven't
> > > written to the file?
> > > > Is the client saying it no longer is going to use the
> > > provisional space,
> > > > in which case maybe it should return that part of the
> > > layout?  If you free
> > > > provisional space, how can the layout still be valid?  I'm
> > > just curious
> > > > about the use case here as I'm interested in what you guys
> > > are doing with
> > > > the pNFS ops.
> > > >
> > > I think provisionally allocated space should be returned via the
> > > layoutupdate field in layoutcommit.  This should be done by
> > > indicating
> > > which space has been used (e.g., for blocks: an extent map).
> >
> > right.
> But how does returning provisionally allocated space affect 
> the validity
> of the layout?  Does the client have to take special 
> precautions not to
> use the part of the layout which maps to space it returned?  
> What is the
> use case here?

[This is more typical for block storage rather than object or file
storage, but I'll describing it anyway (David B. please correct me
if I'm wrong)...]

When the client wants to write in the block world, e.g. post EOF
(or into a hole, or do copy on write) the server will provisionally
allocate storage space for the client.  To reduce allocation overhead
the server will typically do this allocation in large chunks. If the
client did not consume all provisionally allocated storage it will
eventually free the unused blocks with LAYOUTCOMMIT.  After doing that
the client must not use the freed blocks anymore.

> 
> > >
> > > > 2) Trond and I were talking about some odd truncate
> > > scenarios with the
> > > > lastbytewritten field.
> > > > s1) Client A believe the file is 20 bytes.  Client
> > > retrieves a layout for
> > > > and writes to bytes 10-14.
> > > > s2) Client B truncates the file to 12 bytes.
> > > > s3) Client A performs LAYOUTCOMMIT with lastbytewritten=14
> > > (although it
> > > > still believes the file is 20 bytes)
> > > >
> > > > In this scenario, it would be best for the server to update
> > > the file size
> > > > to 15.  If lastbytewritten was changed to local_file_size,
> > > it would be set
> > > > to 20, which is wrong.
> > > >
> > > Remember, last_byte_written is a hint for the server so that
> > > it knows if
> > > it should put in the work to figure out the file size.  At
> > > LAYOUTCOMMIT
> > > time the server is free to query the data servers to find out
> > > the real
> > > file size (at least for the file-based layout).  LAYOUTCOMMIT
> > > should not
> > > be used to truncate the file (SETATTR should be used for this
> > > purpose).
> > >
> > > So it depends on if the server received Client B's SETATTR
> > > for 12 bytes
> > > prior to receiving the LAYOUTCOMMIT.  Also, I believe 
> (for file-based
> > > layouts), the server should verify the size if it is changing
> > > the file's
> > > size at LAYOUTCOMMIT time (it can use the current file size
> > > and the last
> > > byte written to make this judgement).
> >
> > Agreed.  If the server allowed the truncate while other clients
> > are holding layouts (and it didn't recall them), it doesn't know
> > if the lastbytewritten refers to a write that happened before
> > or after the truncate.  Therefore the server must query the storage
> > servers to determine the actual file length.  That said, I'd
> > recommend that the MDS will recall write layouts when getting
> > a length-changing SETATTR to force serialization across the
> > truncate.
> 
> From Garth's comment, if extending the file requires 
> gathering the file
> size from all data servers, your performance is going to suck in the
> typical case, a linear beginning to end write of the file, 
> and with small
> files.
> 
> I agree with recalling the write layout on truncate, although 
> this does
> not match NFSv4 behavior.  In the NFSv4 file layout case, it 
> would just be
> a waste of time since the client would need to retrieve the exact same
> layout all over again.  It is very easy for a client to track the last
> byte written, we should stick with this more (but possibly 
> not intended)
> accurate value (instead of the client's knowledge of the file size).
> Some FS's might want to leave accurate file determination until it is
> formally requested.  With 3 possible values (truncated value, 
> last byte
> written, and the file size on the client), the first two have 
> at least a
> chance of being the intended value.
> 
> Dean
> 
> > Benny
> >
> > >
> > > -Garth
> > >
> > > > Dean
> > > >
> > > >
> > > > On Fri, 2 Dec 2005, Halevy, Benny wrote:
> > > >
> > > >
> > > >>I'm fine with that too.
> > > >>
> > > >>I think that the field was defined as such to stress the
> > > >>fact that the client is not setting the file's length but
> > > >>rather it's just reporting what's the highest offset it
> > > >>wrote to and server should determine the new length
> > > >>of the file based on these reports and possibly other
> > > >>information it has about the file.
> > > >>
> > > >>If we're going the way you proposed than the client
> > > >>is sending its local notion of the file size, hence the
> > > >>field should be named accordingly: e.g. local_file_size.
> > > >>
> > > >>Benny
> > > >>
> > > >>
> > > >>>-----Original Message-----
> > > >>>From: Noveck, Dave [mailto:[email-masked]]
> > > >>>Sent: Friday, December 02, 2005 8:32 PM
> > > >>>To: Halevy, Benny; Goodson, Garth; Welch, Brent; Zelenka,
> > > Jim; 'David
> > > >>>Black' (E-mail)
> > > >>>Cc: [email-masked]
> > > >>>Subject: RE: [nfsv4] pnfs last write offset
> > > >>>
> > > >>>
> > > >>>Or maybe we want the field to be one byte beyond the offset of
> > > >>>the last byte you wrote.
> > > >>>
> > > >>>Maybe this would be a problem if you are writing a 
> byte at offset
> > > >>>0xffffffffffffffff but if you did the size of the file wouldn't
> > > >>>be representable in v4 anyway.
> > > >>>
> > > >>>-----Original Message-----
> > > >>>From: Halevy, Benny [mailto:[email-masked]]
> > > >>>Sent: Friday, December 02, 2005 1:11 PM
> > > >>>To: Goodson, Garth; Welch, Brent; Zelenka, Jim; 'David
> > > Black' (E-mail)
> > > >>>Cc: [email-masked]'
> > > >>>Subject: [nfsv4] pnfs last write offset
> > > >>>
> > > >>>
> > > >>>draft-ietf-nfsv4-pnfs-00.txt says:
> > > >>>   The "last_write_offset" field specifies the offset of
> > > the last byte
> > > >>>   written by the client previous to the LAYOUTCOMMIT.  Note:
> > > >>>this value
> > > >>>   is never equal to the file's size (at most it is one
> > > byte less than
> > > >>>   the file's size).  The metadata server may use this
> > > information to
> > > >>>   determine whether the file's size needs to be 
> updated.  If the
> > > >>>   metadata server updates the file's size as the result of the
> > > >>>   LAYOUTCOMMIT operation, it must return the new size as
> > > part of the
> > > >>>   results.
> > > >>>
> > > >>>What should the client set this to if it never wrote 
> to the file
> > > >>>and issues LAYOUTCOMMIT e.g. to free up provisionally allocated
> > > >>>blocks?
> > > >>>
> > > >>>A last_write_offset value of 0 means that the file has 1
> > > byte written
> > > >>>to it so maybe last_write_offset should be of type newsize4
> > > >>>rather than
> > > >>>length4.
> > > >>>
> > > >>>Benny
> > > >>>
> > > >>>_______________________________________________
> > > >>>nfsv4 mailing list
> > > >>>[email-masked]
> > > >>>https://www1.ietf.org/mailman/listinfo/nfsv4
> > > >>>
> > > >>
> > > >>_______________________________________________
> > > >>nfsv4 mailing list
> > > >>[email-masked]
> > > >>https://www1.ietf.org/mailman/listinfo/nfsv4
> > > >>
> > > >
> > > >
> > > > _______________________________________________
> > > > nfsv4 mailing list
> > > > [email-masked]
> > > > https://www1.ietf.org/mailman/listinfo/nfsv4
> > >
> > > _______________________________________________
> > > nfsv4 mailing list
> > > [email-masked]
> > > https://www1.ietf.org/mailman/listinfo/nfsv4
> > >
> >
> 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Dean Hildebrand <[email-masked]>
Date: Tue, 6 Dec 2005 15:55:02 -0500 (EST)

> the client must not use the freed blocks anymore.

So once the blocks are freed with LAYOUTCOMMIT, the layout for those
blocks is invalid?  If the client holds on to a layout that is invalid, it
is blocking other clients from accessing those blocks.  Should we state
something in the protocol that:
a) a client must not use the layout for the blocks after it frees them
b) a client must do a LAYOUTRETURN immediately after
c) something else?
Dean