Subject: [nfsv4] pNFS -layouts and byte ranges

From: Todd Pisek <[email-masked]>
Date: Thu, 01 Dec 2005 14:49:13 -0600

I have a couple of questions:

1 - The pNFS draft refers to layouts and layout segments.
    If I understand correctly, a layout segment is a byte
    range within a file layout. Am I correct in assuming
    that LAYOUTGET specifies a layout range?

2 - Are layout segments recalled for range splitting?

3 - Does a layout segment carry an implicit byte range
    delegation (as opposed to a byte range lock)?

4 - How is byte range locking enforced by object/block
    data servers?

--- Todd

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Garth Goodson <[email-masked]>
Date: Thu, 01 Dec 2005 13:33:48 -0800

>     that LAYOUTGET specifies a layout range?
Correct.  LAYOUTGET specifies a byte range.

> 2 - Are layout segments recalled for range splitting?
Not sure what you mean exactly by "range splitting", but recalls can be 
done for file ranges.  These file ranges need not match the ranges 
returned by LAYOUTGET.  They can either span or sub-divide layout 
segments held by clients.  Note: layout segment is just terminology for 
a byte range within a layout.  A client may hold a layout that is 
comprised of multiple sets of byte-ranges gotten through separate 
LAYOUTGETs.

>     delegation (as opposed to a byte range lock)?
NO.  Delegations and locks are separate from layouts (at least to the 
extent possible as determined by the layout type).  E.g., for file 
layouts they are totally separate.  Also note, that byte range 
delegations don't currently exist.  Trond has proposed them, but they 
are not part of pNFS.

> 4 - How is byte range locking enforced by object/block
Check with the corresponding specs.  Objects will use capabilities to 
enforce mandatory locks.  Blocks can tie byte-ranged locks to layouts. 
Note: only mandatory locks need be enforced at the data server.

-Garth

---

From: Garth Goodson <[email-masked]>
Date: Thu, 01 Dec 2005 15:41:11 -0800

Todd Pisek wrote:
> Garth Goodson wrote On 12/01/05 03:33 PM,:
> 
>>Todd Pisek wrote:
>>
> 
> :
> :
> 
> When you refer to "mandatory locks", is this synonymous with LOCK?
> I thought the only operations between a client and the data servers
> was READ, WRITE, COMMIT, and NULL. Is there an implied operation between
> the metadata server and the data servers for the LOCK operation?
> 
All metadata ops go to the metadata server.  However, mandatory locks 
must be respected on the data server even though there is no client call 
to that data server.  Thus, there is a need for a back-end control 
protocol between metadata and data servers.  Data servers do not accept 
LOCK operations from the client.

However, for some non-intelligent devices (e.g., blocks) a control 
protocol is out of the question, thus locks are tied more closely with 
layouts.  Object devices use capabilities to control access to data, 
thus these capabilities can be tied in with locks as well.

> For OSD data servers, how are the capabilities for the data servers
> conveyed to the client?
> 
> 
I believe as part of the layout.  Check the object spec for more detail.

-Garth

---

From: Brent Welch <[email-masked]>
Date: Thu, 01 Dec 2005 17:39:09 -0800

The capability will be an opaque sent from the metadata server.
Ideally it will be encyrpted using the GSS context that the
client has established with the server, but I'm not sure the
best way to spec that yet.

>>>Garth Goodson said:

--
Brent Welch
Software Architect, Panasas Inc
Accelerating Time to Results(tm) with Clustered Storage

www.panasas.com
[email-masked]

---

From: Todd Pisek <[email-masked]>
Date: Thu, 01 Dec 2005 17:35:40 -0600

:

--- Todd

---

From: Todd Pisek <[email-masked]>
Date: Sat, 03 Dec 2005 10:25:57 -0600

:
> thus these capabilities can be tied in with locks as well.

OK, I think I now understand this. If a client wants a mandatory lock, it
needs to issue LOCK, then issue LAYOUTGET (probably as a compound). Correct?

What's the current view regarding client side caching with many writers
(think parallel applications)?

--- Todd

---

From: Garth Goodson <[email-masked]>
Date: Mon, 05 Dec 2005 09:46:29 -0800

> needs to issue LOCK, then issue LAYOUTGET (probably as a compound). Correct?
The goal was for pNFS layout ops to be separate from delegation and lock 
ops.  Thus, at least for file-based layouts, there is no correspondence 
between LOCK and LAYOUTGET.  It doesn't matter the order you do them in. 
  Whenever the LOCK completes you have the LOCK, whenever the LAYOUTGET 
completes you have the layout.  The client can pick how it makes sense 
to order them.

However, for other layouttypes, a LOCK may result in the recall of other 
layouts.

> (think parallel applications)?
pNFS does not add any client side caching features to NFSv4 (this was 
debated during pNFS meetings prior to it moving into the WG).  pNFS uses 
delegations as NFSv4 would use them.  The only difference is that dirty 
data can be flushed using the layout, or through the metadata server 
(and ideally would be flushed prior to returning a layout).

Trond has a separate byte-range delegation proposal that would allow 
byte-ranged caching.  Again, these are not (and will not) be tied to 
layouts (although could be used together).

-Garth

---

From: Todd Pisek <[email-masked]>
Date: Mon, 05 Dec 2005 12:32:47 -0600

> completes you have the layout.  The client can pick how it makes sense 

I believe the only way an OSD can enforce byte range access is through
a capability. Version 1 of the OSD spec does not specify how this is done.
My understanding is that this ability is in OSD version 2, which is not
yet finalized.

It appears that the only mechanism by which OSD capabilities can be conveyed
to the client is via the LAYOUTGET response.

I don't understand how mandatory locking can be accomplished without using
LOCK in combination with LAYOUTGET in the OSD case.

I do understand that OSD pNFS can obtain an segment that is enforced by the
OSD without using LOCK. But if I remember correctly, the result of LAYOUTGET
does not imply a range delegation or a range lock.

--- Todd

---

From: Brent Welch <[email-masked]>
Date: Mon, 05 Dec 2005 11:19:12 -0800

OSDv1 does not have byte-range granularity of its capabilities,
but we can still use capabilities to enforce mandatory byte range
locks.  However, it obviously limits the file to one writer at
a time; you cannot use OSDv1 capabilities to enforce mandatory locks
where two different clients are simultaneously writing to two
different regions of the file.  The metadata server will have to
drive this via CB_LAYOUTRECALL.

I would just note this limitation and move on.  I don't expect a lot
of mandatory lock usage, and I am content to wait for OSDv2 and
its (proposed) byte range capabilities to optimize this scenario.

I think it is much more likely that clients will use advisory byte
range locks, and that they will self-enforce those locks without
any help from the data servers, just like they do today.  The metadata
server will coordinate the lock requests, but the clients do
the enforcement.
>>>Todd Pisek said:
 > >>needs to issue LOCK, then issue LAYOUTGET (probably as a compound). 
Correc
     t?

www.panasas.com
[email-masked]

---

From: Todd Pisek <[email-masked]>
Date: Tue, 06 Dec 2005 09:04:01 -0600

I'm not so concerned about V1 versus V2 of OSD as I am about
how LAYOUTGET is being used. Let's assume V2 OSD. If a client
requests a segment using only LAYOUTGET, and the metadata
server grants the full range requested, will the data server(s)
reject client reads/writes outside this segment for both object
and file pNFS?