Subject: [nfsv4] layout race detection

From: Rick Mesta <[email-masked]>
Date: Mon, 13 Oct 2008 16:15:29 -0500

Hi folks,

	 In my first implementation attempt of the spec's layout race
	detection, I find that I'm having a hard time deciphering the
	spec, namely cases 1 and 2 of 'server considerations' section
	(12.5.5.2.1.3).

	 Given the seqid incrementation rules in section 12.5.3, I do
	not see how...

	case 1) the seqid of the LAYOUTGET could possibly be two less
		than that of the CB_LAYOUT_RECALL.

	case 2) the seqid of the LAYOUTGET would be greater than the
		seqid in the CB_LAYOUT_RECALL. (I _do_ see when they
		are the same, though).

	Is this a hole in the spec or am I misinterpreting something
	trivial ?

	thx,
		rick
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "Mike Eisler" <[email-masked]>
Date: Tue, 14 Oct 2008 04:29:10 -0700 (PDT)

> 	spec, namely cases 1 and 2 of 'server considerations' section

   1.  The client sent the LAYOUTGET before processing the
       CB_LAYOUTRECALL.  The "seqid" in the layout stateid of LAYOUTGET
       is two less than the "seqid" in CB_LAYOUTRECALL.  The server
       returns NFS4ERR_RECALLCONFLICT to the client, which indicates to
       the client that there is a pending recall.

   LAYOUTGET seqid=1 ->

   <- LAYOUTGET result seqid=2

   <- CB_LAYOUTRECALL seqid=3

The CB_LAYOUTRECALL arrives at the client before the
LAYOUTGET reply. 3 - 1 = 2.

> 		seqid in the CB_LAYOUT_RECALL. (I _do_ see when they

  2.  The client sent the LAYOUTGET after processing the
       CB_LAYOUTRECALL, but the LAYOUTGET arrived before the
       LAYOUTRETURN and the response to CB_LAYOUTRECALL that completed
       that processing.  The "seqid" in the layout stateid of LAYOUTGET
       is equal to or greater than that of the "seqid" in
       CB_LAYOUTRECALL.  The server has not received a response to the
       CB_LAYOUTRECALL, so it returns NFS4ERR_RECALLCONFLICT.

Server sends recall.

<- CB_LAYOUTRECALL seqid=2 range [0..9]

client processes recall and sends:

-> LAYOUTRETURN seqid=2 range [0..9]

-> CB_LAYOUTRECALL

client also sends a LAYOUTGET

-> LAYOUTGET seqid=2 range [20..29]

Server receives above LAYOUTGET before it receives
LAYOUTRETURN and CB_LAYOUTRECALL.
Server processes above LAYOUTGET (because
the range does not overlap) and replies:

<- LAYOUTGET seqid=3

Client sends another LAYOUTGET, and
server has still not received
LAYOUTRETURN and CB_LAYOUTRECALL.

-> LAYOUTGET seqid=3 range [0..19]

Server says "enough" because the requested range
overlaps with what the client was told to return.

<- LAYOUTGET NFS4ERR_RECALLCONFLICT

Server finally receives LAYOUTRETURN and CB_LAYOUTRECALL.

-> LAYOUTGET seqid=3 range [0..19]
<- LAYOUTGET seqid=4

-- 
Mike Eisler, Senior Technical Director, NetApp, 719 599 9026,
http://blogs.netapp.com/eislers_nfs_blog/

---

From: Rick Mesta <[email-masked]>
Date: Tue, 14 Oct 2008 11:13:11 -0500

Hi Mike,

	 Thx for the prompt reply; comments/questions in-line...

On Tue, Oct 14, 2008 at 04:29:10AM -0700, Mike Eisler wrote:
| On Mon, October 13, 2008 2:15 pm, Rick Mesta wrote:
| >
| > 	Hi folks,
| >
| > 	 In my first implementation attempt of the spec's layout race
| > 	detection, I find that I'm having a hard time deciphering the
| > 	spec, namely cases 1 and 2 of 'server considerations' section
| > 	(12.5.5.2.1.3).
| >
| > 	 Given the seqid incrementation rules in section 12.5.3, I do
| > 	not see how...
| >
| > 	case 1) the seqid of the LAYOUTGET could possibly be two less
| > 		than that of the CB_LAYOUT_RECALL.
| 
|    1.  The client sent the LAYOUTGET before processing the
|        CB_LAYOUTRECALL.  The "seqid" in the layout stateid of LAYOUTGET
|        is two less than the "seqid" in CB_LAYOUTRECALL.  The server
|        returns NFS4ERR_RECALLCONFLICT to the client, which indicates to
|        the client that there is a pending recall.
| 
| 
| 
|    LAYOUTGET seqid=1 ->
| 
|    <- LAYOUTGET result seqid=2
| 
|    <- CB_LAYOUTRECALL seqid=3
| 
| The CB_LAYOUTRECALL arrives at the client before the
| LAYOUTGET reply. 3 - 1 = 2.
| 
  	Wow... that is _so_ not what I got from reading the spec =^(

	12.5.5.2.1.3. "The metadata server has sent a CB_LAYOUTRECALL
	 and receives an overlapping LAYOUTGET for the same file before
	 the LAYOUTRETURN(s) that respond to the CB_LAYOUTRECALL."
  
  	The text in the spec clearly states that the CB_LAYOUTRECALL is
	_already_ in flight (but the client has not processed it yet)
	when the client issues the LAYOUTGET.

              client |                                     | server
              ------ |                                     | ------
                (1)  |<---- CB_LAYOUTRECALL (C) seqid=2 ---| <seqid++>
                  ** |                                     |
                     |------- LAYOUTGET (C) seqid=1 ------>|

             ** - CB_LAYOUTRECALL not yet processed, so no reply to
                  CB_LAYOUTRECALL _or_ LAYOUTRETURN req generated.

  	I floated this internally before posting to the wg alias, and
	(for better or worse) other Sun engineers are interpreting the
	spec in the same way. Do we need to crisp up the text or do we
	have a bonafide issue here ?

| >
| > 	case 2) the seqid of the LAYOUTGET would be greater than the
| > 		seqid in the CB_LAYOUT_RECALL. (I _do_ see when they
| > 		are the same, though).
| 
|   2.  The client sent the LAYOUTGET after processing the
|        CB_LAYOUTRECALL, but the LAYOUTGET arrived before the
|        LAYOUTRETURN and the response to CB_LAYOUTRECALL that completed
|        that processing.  The "seqid" in the layout stateid of LAYOUTGET
|        is equal to or greater than that of the "seqid" in
|        CB_LAYOUTRECALL.  The server has not received a response to the
|        CB_LAYOUTRECALL, so it returns NFS4ERR_RECALLCONFLICT.
| 
| 
| 
| Server sends recall.
| 
| <- CB_LAYOUTRECALL seqid=2 range [0..9]
| 
| client processes recall and sends:
| 
| -> LAYOUTRETURN seqid=2 range [0..9]
| 
| -> CB_LAYOUTRECALL
| 
| client also sends a LAYOUTGET
| 
| -> LAYOUTGET seqid=2 range [20..29]
| 
| Server receives above LAYOUTGET before it receives
| LAYOUTRETURN and CB_LAYOUTRECALL.
| Server processes above LAYOUTGET (because
| the range does not overlap) and replies:
| 
| <- LAYOUTGET seqid=3
| 
| Client sends another LAYOUTGET, and
| server has still not received
| LAYOUTRETURN and CB_LAYOUTRECALL.
| 
| -> LAYOUTGET seqid=3 range [0..19]
| 
| Server says "enough" because the requested range
| overlaps with what the client was told to return.
| 
| <- LAYOUTGET NFS4ERR_RECALLCONFLICT
| 
| Server finally receives LAYOUTRETURN and CB_LAYOUTRECALL.
| 
| -> LAYOUTGET seqid=3 range [0..19]
| <- LAYOUTGET seqid=4
| 

	Aha ! Okay, case 2 makes sense now (or at least, I can see how
	the seqid in the LAYOUTGET can be greater than that in the
	CB_LAYOUTRECALL =^) Thx for the additional context, Mike.

	I still am having trouble groking case 1, though.

		rick

---

From: Karen Rochford <[email-masked]>
Date: Tue, 14 Oct 2008 10:41:34 -0600

> LAYOUTGET reply. 3 - 1 = 2.
This confused me too.   Specifically, when I read "The server returns 
NFS4ERR_RECALLCONFLICT to the client, which indicates to the client that 
there is a pending recall", I was assuming this was what was returned 
for the status of LAYOUTGET, however your example indicates that the 
result to LAYOUTGET is successful, and a layout stateid with an 
incremented seqid is returned.  So, what operation would receive the 
NFS4ERR_RECALLCONFLICT?

> -> LAYOUTGET seqid=3 range [0..19]
This makes more sense than the documentation in the draft, as the draft 
does not mention, but implies, the additional LAYOUTGET.

---

From: "Mike Eisler" <[email-masked]>
Date: Tue, 14 Oct 2008 10:36:07 -0700 (PDT)

It apparently was too early in the morning for me. Let me try
case 1 again.

> | > 	spec, namely cases 1 and 2 of 'server considerations' section

client sends:

LAYOUTGET seqid=1 ->

server sends:

<- LAYOUTGET seqid=2

before the client gets the above response, it sends:

LAYOUTGET seqid=1 ->

before the server gets the second LAYOUTGET (the above) it sends

now the server receives the second LAYOUTGET. seqid in
LAYOUTGET is 1 and two less than the seqid in the recall.

---

From: Rick Mesta <[email-masked]>
Date: Tue, 14 Oct 2008 13:02:19 -0500

On Tue, Oct 14, 2008 at 10:36:07AM -0700, Mike Eisler wrote:
| It apparently was too early in the morning for me. Let me try
| case 1 again.
| 

--8<-- snip --8<--

| > |
| > |    1.  The client sent the LAYOUTGET before processing the
| > |        CB_LAYOUTRECALL.  The "seqid" in the layout stateid of LAYOUTGET
| > |        is two less than the "seqid" in CB_LAYOUTRECALL.  The server
| > |        returns NFS4ERR_RECALLCONFLICT to the client, which indicates to
| > |        the client that there is a pending recall.
| 
| client sends:
| 
| LAYOUTGET seqid=1 ->
| 
| server sends:
| 
| <- LAYOUTGET seqid=2
| 
| before the client gets the above response, it sends:
| 
| LAYOUTGET seqid=1 ->
| 
| before the server gets the second LAYOUTGET (the above) it sends
| 
| <- CB_LAYOUTRECALL seqid=3
| 
| now the server receives the second LAYOUTGET. seqid in
| LAYOUTGET is 1 and two less than the seqid in the recall.
| 
	Aaaaaaaaaaaaaah... okay, I see now the scenario and agree that the
	CB_LAYOUTRECALL seqid is indeed two more than the seqid in the
	retransmitted LAYOUTGET request.

	Could we possibly crisp up the text in case 1 ? I definitely wasn't
	even thinking of retransmissions and am obviously not the only one
	that fell into that rats nest.

	Perhaps rewording each case separately such that it's explicitly
	stated in case 1 that the comparison should be done between the
	retransmitted LAYOUTGET and the CB_LAYOUTRECALL ?

		rick

---

From: Fredric Isaman <[email-masked]>
Date: Tue, 14 Oct 2008 14:30:55 -0400 (EDT)

> 	retransmitted LAYOUTGET and the CB_LAYOUTRECALL ?

Just to clarify, based on your phrase "retransmitted LAYOUTGET".  The 
likely scenerio is the client sending two different LAYOUTGETS for two 
different ranges, not a retransmission of the same LAYOUTGET.

Fred

---

From: Rick Mesta <[email-masked]>
Date: Tue, 14 Oct 2008 14:05:51 -0500

On Tue, Oct 14, 2008 at 02:30:55PM -0400, Fredric Isaman wrote:
| On Tue, 14 Oct 2008, Rick Mesta wrote:
| >On Tue, Oct 14, 2008 at 10:36:07AM -0700, Mike Eisler wrote:

--8<-- snip --8<--

| >|
| >| client sends:
| >|
| >| LAYOUTGET seqid=1 ->
| >|
| >| server sends:
| >|
| >| <- LAYOUTGET seqid=2
| >|
| >| before the client gets the above response, it sends:
| >|
| >| LAYOUTGET seqid=1 ->
| >|
| >| before the server gets the second LAYOUTGET (the above) it sends
| >|
| >| <- CB_LAYOUTRECALL seqid=3
| >|
| >| now the server receives the second LAYOUTGET. seqid in
| >| LAYOUTGET is 1 and two less than the seqid in the recall.
| >|
| >	Aaaaaaaaaaaaaah... okay, I see now the scenario and agree that the
| >	CB_LAYOUTRECALL seqid is indeed two more than the seqid in the
| >	retransmitted LAYOUTGET request.
| >
| >	Could we possibly crisp up the text in case 1 ? I definitely wasn't
| >	even thinking of retransmissions and am obviously not the only one
| >	that fell into that rats nest.
| >
| >	Perhaps rewording each case separately such that it's explicitly
| >	stated in case 1 that the comparison should be done between the
| >	retransmitted LAYOUTGET and the CB_LAYOUTRECALL ?
| >
| >		rick
| 
| Just to clarify, based on your phrase "retransmitted LAYOUTGET".  The 
| likely scenerio is the client sending two different LAYOUTGETS for two 
| different ranges, not a retransmission of the same LAYOUTGET.
| 
| Fred

	Point taken ! So, again to clarify, if the client is sending the
	second LAYOUTGET (for a diff range), it's sending seqid=1 simply
	because it hasn't seen the reply to the first LAYOUTGET, yes ?
	Would the same indeed hold for a retransmission ? 

		rick

---

From: "William A. (Andy) Adamson" <[email-masked]>
Date: Tue, 14 Oct 2008 15:56:56 -0400

>        because it hasn't seen the reply to the first LAYOUTGET, yes ?

Yes, it hasn't seen a reply to any LAYOUT operation sent.

>        Would the same indeed hold for a retransmission ?

Retransmissions are different. A retransmission would not even look at
the stored layout stateid value, it would retransmit the already
encoded data which in this case happens to be the same (a layout
stateid seqid of 1). Note that the server detecting a retransmission
via looking at the sequence operation seqid,  would not bump the
layout stateid seqid at the server, it would resend the cached value,
and so the response to the 2nd LAYOUTGET (if it was a retransmission)
would have a layout stateid seqid of 1 not 2.

-->Andy

---

From: Fredric Isaman <[email-masked]>
Date: Tue, 14 Oct 2008 15:27:33 -0400 (EDT)

> 	because it hasn't seen the reply to the first LAYOUTGET, yes ?

Basically yes, but note that the client may have seen the reply, but not 
"fully processed" it.

> 	Would the same indeed hold for a retransmission ?

For a literal retransmission, I would expect SEQUENCE processing to handle 
it.  If you just mean a second LAYOUTGET that happens to have the same 
parameters as the first, then yes, I would expect the same rules to apply.

Fred

---

From: Rick Mesta <[email-masked]>
Date: Tue, 14 Oct 2008 14:36:41 -0500

On Tue, Oct 14, 2008 at 03:27:33PM -0400, Fredric Isaman wrote:
| On Tue, 14 Oct 2008, Rick Mesta wrote:

--8<-- snip --8<--

| >	Point taken ! So, again to clarify, if the client is sending the
| >	second LAYOUTGET (for a diff range), it's sending seqid=1 simply
| >	because it hasn't seen the reply to the first LAYOUTGET, yes ?
| 
| 
| Basically yes, but note that the client may have seen the reply, but not 
| "fully processed" it.

	Ack !
| 
| >	Would the same indeed hold for a retransmission ?
| >
| >		rick
| 
| For a literal retransmission, I would expect SEQUENCE processing to handle 
| it.  If you just mean a second LAYOUTGET that happens to have the same 
| parameters as the first, then yes, I would expect the same rules to apply.
| 
| Fred

	Yup... I meant a second LAYOUTGET that has the same args as the
	first (not necessarily a sessions replay). Thx for the clarification,
	Fred.

	 At any rate, hopefully this discussion is proof that some of us
	were definitely not on the same page when reading thru the case 1
	text.

		rick

---

From: "William A. (Andy) Adamson" <[email-masked]>
Date: Tue, 14 Oct 2008 15:59:12 -0400

> layout stateid seqid at the server, it would resend the cached value,

The above is true. This last statement is not! sorry.