Subject: RE: [nfsv4] Delegation recall with truncate set

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 5 Aug 2003 13:02:46 -0700

I don't remember all the details but my understanding 
was that when we didn't return DELAY, we could get into
a situation with the Solaris client in which the recall 
callback was not processed and that arranging to return 
DELAY solved that.  That's what I was referring to when
I said some clients "depend on" the DELAY response.

-----Original Message-----
From: Spencer Shepler [mailto:[email-masked]]
Sent: Tuesday, August 05, 2003 3:54 PM
To: [email-masked]
Subject: Re: [nfsv4] Delegation recall with truncate set

On Tue, Noveck, Dave wrote:
> Right, but how might a server know that?  The only way
> I can see is that he simply does not return DELAY when 
> such a SETATTR is done.  The problem with a server 
> deciding to do that is that some clients depend on 
> getting that DELAY return, don't they?

I don't know of a client that depends on a response of DELAY.  The
calling context on the client will certainly be stuck waiting for a
response; with our rules about retransmissions, the client should be
careful in retransmitting (and is required to drop the TCP connection
to do so).  Therefore, it may be okay.  Obviously, the server would
need to take care in consuming resources for such an instance.  For a
traditional server where it may have a service thread tied to the
request, blocking the incoming while awaiting for the delegation
recall to occur may be too much of a resource burden.

> The server cannot adopt a strategy of returning DELAY
> only if the recall is takes too long, because, at the
> time he does the recall, he can only do the recall for 
> truncation if he knows he wil never return DELAY, and it 
> doesn't appear that he can know that.

That is correct; it is not deterministic.

> I think the problem here is the semantics of the DELAY
> return.  Although the spec text isn't very clear on this,
> I think it is reasonable to assume that when you get back
> DELAY you are being told that the truncate/REMOVE did not 
> get done and that if you don't redo them, they wil not be
> done.  Maybe we need DELAYED_AND_MAYBE_DONE which allows
> the operation requested to be completed.  The problem is
> that this doesn't fit the RPC re-issue model since a 
> re-issued REMOVE might get NOENT.  Sigh!

Right. I would not have interpreted DELAY in the "might-have-done"
way.  However, you are correct that there isn't enough descriptive
text for NFS4ERR_DELAY to make its semantics clear to a new reader.  I
will put this on the list of interpretation items to be covered.

Spencer

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 7 Aug 2003 13:16:26 -0500

> I said some clients "depend on" the DELAY response.

Okay.  Thanks for reminding me of what we discussed during
Connectathon.  I knew I should have written it down.  With some help
to remember the scenario encountered at Connectathon, this is the case
where the server returning DELAY can help:

	client				server
	------				------
	<assume same openowner for all ops>

(1) OPEN(ACCESS_READ) ->
(2) 				<-	OPEN is okay; provides READ delegation
(3) CLOSE of OPEN in (1) ->
(4)				<-	CLOSE is okay
(5) local open for read
(6) local open for write
(7) OPEN(ACCESS_WRITE) ->
(8)				<-	CB_RECALL of delegation from (2)
(9) OPEN(ACCESS_READ) ***

*** The client is attempting to OPEN the file as a result of the
    delegation recall processing (ensuring all open state is at the
    server)

For the client to do the OPEN in (9) is must have the "next" sequence
id and it can not obtain it since the response to the OPEN in (7) has
not been received. So the client was serialized on access to the
sequence id for the openowner and was waiting for the response from
(7); the client was hung.

When the server returns DELAY to the OPEN in (7), then the client will
be able to push the OPEN for ACCESS_READ to the server, return the
delegation and then resend the OPEN for ACCESS_WRITE.

I suppose the alternative is the client could have sent an OPEN for
both ACCESS_READ and ACCESS_WRITE but it seemed safer to have the
server return DELAY.

Note that our server current employs a heuristic to wait a short
amount of time after sending the delegation recall.  It the delegation
is not returned, then the DELAY response is sent to the original
operation that caused the conflict.  This works in a few cases to
overcome additional requests.

Given this particular scenario, it seems best to have the server
return DELAY to the conflicting operation (possibly after a very short
wait).

I have also changed my opinion about the server's use of CB_RECALL
with truncate == TRUE.  In most cases, the scenario of the client
not retransmitting the request that caused the CB_RECALL to occur,
seems like an edge case and it is more desirable to avoid the
unnecessary transmit of file data when it is likely to be discarded
once the delegation is returned.

Spencer