Subject: RE: [nfsv4] New pNFS ID available

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 1 Aug 2005 18:54:52 -0400

Jim Zelenka wrote:
> If client A holds a write layout on a region in a file, and client B
> does a READ to the metadata server that contains that region of that
> file, the server must ("MUST") recall the layout from client A.

I don't understand.  Are you saying:

     1) That this requirement is in the current spec.

     2) That this requirement isn't exactly in the spec but is implied
        by what is there so that you couldn't implement the spec without
        obeying this requirement.

     2') Same as 2) but only in the case of particular storage protocols,
         e.g. blocks.

     3) That this requirement is not in the spec but should be, because ...

> If client A does the read, I would expect the server to return
> bits that are not more out-of-date than the most recent LAYOUTCOMMIT
> from client A for the layout. 

I agree.

> If client A wants that read to return
> up-to-date bits, then it should LAYOUTCOMMIT or LAYOUTRETURN.

And contrariwise, if A doesn't care about such uptodateness (yet), then
it can wait until it is done (i.e. it finished with the sequence of writes
it is doing) before committing. 

I think that is the way things currently stand in the spec and the requirement
that you state in the first paragraph is not part of the spec. 

> A layout is identified by {ClientID,FileHandle,offset,length,layout_type,iomode
> }*,
> so the server should have enough information to figure out if the
> requestor holds an outstanding layout.

In your example B is the requestor isn't it?  So isn't the server trying
to figure out if some other client holds a layout for the area in question?

I agree that the server has the info to do this if it wants to but I 
don't see why it should want to and certainly don't see why it should be
required to do so.

> In the case where client A holds a read-only layout, and client B
> does a read, there is no conflict.

> In the case where the reading client hasn't set a client ID, the
> server should (IMO) assume that any/all outstanding write layouts
> are held by clients other than the reading client, and recall them
> all before servicing the read.

If the read uses a special stateid and the read is not done in the context 
of a session, then for the purposes delegation recall for example, it is 
treated as conflicting with all clients' delegations.  A similar pattern 
would hold if there were a conflict with a layout, but I don't think there 
is.  I has thought we had settled on the approach that layouts and data 
delegation/access each existed separately and independcently.  That seems 
to simplify things and I think we should stay with that approach.

-----Original Message-----
From: Jim Zelenka [mailto:[email-masked]]
Sent: Monday, August 01, 2005 5:45 PM
To: [email-masked]
Subject: Re: [nfsv4] New pNFS ID available 

If client A holds a write layout on a region in a file, and client B
does a READ to the metadata server that contains that region of that
file, the server must ("MUST") recall the layout from client A.

If client A does the read, I would expect the server to return
bits that are not more out-of-date than the most recent LAYOUTCOMMIT
from client A for the layout. If client A wants that read to return
up-to-date bits, then it should LAYOUTCOMMIT or LAYOUTRETURN.
A layout is identified by {ClientID,FileHandle,offset,length,layout_type,iomode
}*,
so the server should have enough information to figure out if the
requestor holds an outstanding layout.

In the case where client A holds a read-only layout, and client B
does a read, there is no conflict.

In the case where the reading client hasn't set a client ID, the
server should (IMO) assume that any/all outstanding write layouts
are held by clients other than the reading client, and recall them
all before servicing the read.

-jim

* Well, draft-welch-pnfs-ops-03 has iomode as an open issue, but I'm
assuming that it is included for the purposes of this e-mail.

-- 
Jim Zelenka
Software Engineer, Panasas, Inc.
Accelerating Time to Results(TM) with Clustered Storage
www.panasas.com
412.323-3500

> 
> [email-masked] wrote on 08/01/2005 03:43:10 PM:
> 
> > >I think it should be stated in the spec that
> > >a client should not issue NFSv4 I/O while it has a layout for a file.
> > 
> > I'm not sure that this is a good idea. We've discussed
> > handling most of the "hard" error cases of clients having
> > trouble writing directly to storage by having the client
> > issue a regular NFSv4 write directly to the server. I don't
> > like saying that the client does a LAYOUTRETURN before writing,
> > even in a compound request, because there's no atomicity there,
> > which raises the possibility of another client getting bogus
> > read results between the LAYOUTRETURN and the WRITE. 
> 
> So is the client responsible for cache consistency between its 
> own data buffers and the server data buffers?  How can it do 
> this?  When the client sends data to the metadata NFS server 
> to store it, but it has a layout that it could use to read 
> or write the data, should it expect that the data it could 
> read via the layout will be what it wrote? 
> 
> Or should the server recall the layout in the presence of 
> fetches and stores from the client that has the layout? 
> 
> > -jim
> > 
> > -- 
> > Jim Zelenka
> > Software Engineer, Panasas, Inc.
> > Accelerating Time to Results(TM) with Clustered Storage
> > www.panasas.com
> > 412.323-3500 
> 
>                 Craig 
> 
> Craig Everhart
> +1 919 543 2169 (tie 441 2169)
> 
> 
> 
> <br><font size=2><tt>[email-masked] wrote on 08/01/2005 03:43:10
> PM:<br>
> <br>
> &gt; &gt;I think it should be stated in the spec that<br>
> &gt; &gt;a client should not issue NFSv4 I/O while it has a layout for
> a file.<br>
> &gt; <br>
> &gt; I'm not sure that this is a good idea. We've discussed<br>
> &gt; handling most of the &quot;hard&quot; error cases of clients having<br>
> &gt; trouble writing directly to storage by having the client<br>
> &gt; issue a regular NFSv4 write directly to the server. I don't<br>
> &gt; like saying that the client does a LAYOUTRETURN before writing,<br>
> &gt; even in a compound request, because there's no atomicity there,<br>
> &gt; which raises the possibility of another client getting bogus<br>
> &gt; read results between the LAYOUTRETURN and the WRITE.</tt></font>
> <br>
> <br><font size=2><tt>So is the client responsible for cache consistency
> between its</tt></font>
> <br><font size=2><tt>own data buffers and the server data buffers? &nbsp;How
> can it do</tt></font>
> <br><font size=2><tt>this? &nbsp;When the client sends data to the metadata
> NFS server</tt></font>
> <br><font size=2><tt>to store it, but it has a layout that it could use
> to read</tt></font>
> <br><font size=2><tt>or write the data, should it expect that the data
> it could</tt></font>
> <br><font size=2><tt>read via the layout will be what it wrote?</tt></font>
> <br>
> <br><font size=2><tt>Or should the server recall the layout in the presence
> of</tt></font>
> <br><font size=2><tt>fetches and stores from the client that has the layout?</tt></font>
> <br>
> <br><font size=2><tt>&gt; -jim<br>
> &gt; <br>
> &gt; -- <br>
> &gt; Jim Zelenka<br>
> &gt; Software Engineer, Panasas, Inc.<br>
> &gt; Accelerating Time to Results(TM) with Clustered Storage<br>
> &gt; www.panasas.com<br>
> &gt; 412.323-3500</tt></font>
> <br>
> <br><font size=2><tt>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
> &nbsp; &nbsp; Craig</tt></font>
> <br>
> <br><font size=2 face="sans-serif">Craig Everhart<br>
> +1 919 543 2169 (tie 441 2169)<br>
> </font>
> 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Jim Zelenka <[email-masked]>
Date: Mon, 01 Aug 2005 19:06:35 -0400

> > from client A for the layout. 

Hmm, I think that (2) most closely matches what I am thinking. Also,
I was working with the implicit assumption that A wanted up-to-date
bits. As you point out, if A doesn't care whether the bits are up-to-date,
then it doesn't matter if it does LAYOUTCOMMIT/LAYOUTRETURN.

> that you state in the first paragraph is not part of the spec. 

Correct, and I am not saying that it should be part of the spec. Instead,
I was saying that it can be logically inferred from the spec, assuming
that A cares about the freshness of the reads.

> don't see why it should want to and certainly don't see why it should be

I don't believe that it is required to do so; I was pointing out that
it has the ability to do so if it cares. This can help the server figure
out if it needs to recall a layout held by A because a request is from B,
versus the case where A issues a request and holds a layout. This probably
matters more for some cases than others. One example of a case where this
matters is a truncating setattr on densely-packed objects.

> to simplify things and I think we should stay with that approach.

I guess I was jumping ahead of myself. I was assuming (but failing to
state) that the interesting case here is a write that changes the layout
(eg, a write in the blocks world). If the write doesn't change the layout,
then there is no need to recall it. I am not proposing conflating
data delegation semantics with layout delegations.

-jim