Subject: Re: [nfsv4] Question on Delegation CB_RECALL and DELEGRETURN

From: Rick Macklem <[email-masked]>
Date: Sat, 1 Jun 2013 07:44:44 -0400 (EDT)

Arnab Bakshi wrote:
> Thanks Rick..my comments inline...
> 
> On Jun 1, 2013 5:03 AM, "Rick Macklem" < [email-masked] > wrote:
> >
> > Arnab Bakshi wrote:
> > > Hi,
> > >
> > >
> > > Was going through the RFC5661 and particularly focusing on the
> > > Delegation part. I have couple of questions which I would like to
> > > put
> > > here:
> > >
> > >
> > > Scenario: Suppose a NFS client (client A) has been granted
> > > Read/Write
> > > delegation by a NFS server for a particular file. Now after
> > > sometime
> > > another client (client B) requests conflicting Delegation on the
> > > same
> > > file. Ideally in this case the server will send a CB_RECALL to
> > > client-A to get back the delegation. Suppose the client behaviour
> > > is
> > > that it sends a CB_OK response and does actual IO to flush its
> > > state
> > > to the server.
> > >
> > >
> > > Question1: Client-A keeps sending IO which say is suppose very big
> > > and
> > > time consuming therefore delaying DELEGRETURN. What steps does the
> > > server take? Will it try to revoke the delegation forcefully after
> > > some elapsed time and how?.
> > >
> > Well, I'm not sure there is a well defined correct answer for these
> > questions, but here is what the FreeBSD server does.
> >
> > Since Client-A keeps sending IO requests, it is not crashed/network
> > partitioned and, as such, the FreeBSD server will return
> > NFS4ERR_DELAY
> > to conflicting Open requests until Client-A does DELEGRETURN. (A
> > client
> > that never sends a DELEGRETURN is broken and since there are only
> > 3-4
> > clients out there, I don't believe any of them are broken this way?)
> >
> The implementation makes sense. To put it more specifically if client
> closes the transport connection before it sends DELEGRETURN then
> hopefully server clears the state for the client-id and the associated
> state-id,is it so...? The clients available are surely not broken but
> I am interested more from protocol definition perpective. ?.

Nope. As far as I know, there is nothing in the RFCs that states a client
can't do a fresh connection whenever it wants to. (A client is required to
use a new connection whenever it is retrying an RPC after it fails to
receive a reply to it.)

I once suggested tying state to TCP connections, but others didn't like
the idea at the time, as I recall.

I can see an argument for eventually revoking the delegation, even if
the client is sending other RPCs to the server but no DELEGRETURN. To
be honest, I can't remember if I did that for the FreeBSD server or not.
(I'd have to look at the code. Just because I wrote it doesn't mean I
 can remember what it does.;-)

rick

> > >
> > > Question2: Client-A sends IO for sometime and then somehow becomes
> > > idle and doesn't send the DELEGRETURN. What does the server do in
> > > this
> > > case, will it revoke and how?
> > >
> > If Client-A has crashed or is network partitioned, the lease will
> > expire
> > and after that happens the FreeBSD throws away all state (including
> > all
> > delegations) for that client and returns NFS4ERR_EXPIRED for the
> > clientid
> > and all associated stateids. After the delegation has expired, the
> > conflicting
> > Open succeeds. (The conflicting Open gets NFS4ERR_DELAY for all
> > attempts
> > until lease expiry.)
> >
> > I haven't looked at WANT_DELEGATION yet, so I am not sure if I will
> > handle
> > that like a conflicting Open or simply return something like the
> > extended
> > delegate none with resource as the reason for not issuing the
> > delegation.
> >
> > rick
> >
> > >
> > > I know this is a negative scenario but would be helpful to know
> > > what
> > > should be the behavior from the server side.
> > >
> > >
> > > Thanks in advance...
> > >
> > >
> > > Regards
> > > Arnab
> > > _______________________________________________
> > > nfsv4 mailing list
> > > [email-masked]
> > > https://www.ietf.org/mailman/listinfo/nfsv4
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Arnab Bakshi <[email-masked]>
Date: Mon, 3 Jun 2013 16:12:04 +0530

Thanks Rick.

As per my understanding the client may incorrectly close a TCP connection
gracefully without returning Delegation (this behavior is unlikely but
possible), and RFC doesn't prevent it to be so. Servers should be able to
handle this scenario by internally Revoking the granted delegations as soon
as the TCP connection is down.

For failover/resiliency case maybe the TCP connection is not shutdown
gracefully and it resumes after the outage is recovered. Tying the state to
TCP connection would have worked unless maybe required some handling for
failover scenarios.

I am sorry but I did not get the argument part. Is it the FreeBSD code you
are talking about or the concept of revoking after providing sufficiently
long timeout..

Regards
Arnab

---

From: Rick Macklem <[email-masked]>
Date: Mon, 3 Jun 2013 07:45:50 -0400 (EDT)

Arnab Bakshi wrote:
> Thanks Rick.
> 
> 
> As per my understanding the client may incorrectly close a TCP
> connection gracefully without returning Delegation (this behavior is
> unlikely but possible), and RFC doesn't prevent it to be so. Servers
> should be able to handle this scenario by internally Revoking the
> granted delegations as soon as the TCP connection is down.
> 
A client can close/create new TCP connections whenever they please.
It has no effect on what the server does for NFSv4.0. For NFSv4.1,
I'm not sure if a server can discard a session once the connection(s)
bound to it are all closed or not? (I've just done the basics of a server
and haven't yet thought throw exactly when a session can be discarded.
At this point my server only discards a session when it gets a
DESTROY_SESSION from the client.) I think it can still bind another
connection to the session. Maybe someone else can answer this.

As for the second part, I believe it is up to the server implementor
as to whether you choose to eventually revoke the delegation when a
client fails to DELEGRETURN it in a reasonable time (at least a lease
duration) or only if the lease expires (a lease can be renewed by
other operations, so waiting a lease duration doesn't imply expiry of
the lease).

> gracefully and it resumes after the outage is recovered. Tying the
> state to TCP connection would have worked unless maybe required some
> handling for failover scenarios.
> 
As far as I know, the only rule is that a client must create a new TCP
connection to retry a compound RPC. (Retry means "sending the same
RPC with same xid in header, etc". A client will normally make another
attempt to perform a compound after receiving NFS4ERR_DELAY, but it
will use a fresh RPC header with a different xid, so this isn't a retry
for the above rule.)

> 
> 
> I am sorry but I did not get the argument part. Is it the FreeBSD code
> you are talking about or the concept of revoking after providing
> sufficiently long timeout..
> 
Just ignore my comment w.r.t. tying state to connections. It isn't a
part of the protocol.

rick

> 
> Regards
> Arnab
> 
> 
> 
> On Sat, Jun 1, 2013 at 5:14 PM, Rick Macklem < [email-masked] >
> wrote:
> 
> 
> 
> 
> Arnab Bakshi wrote:

---

From: "J. Bruce Fields" <[email-masked]>
Date: Mon, 3 Jun 2013 15:33:40 -0400

> connection to the session. Maybe someone else can answer this.

The Linux server will hang onto the session until a lease period lapses
without getting any rpc's referencing any of the client's state.

As you say, it would be incorrect to throw away state on last close of a
tcp connection, over 4.0 or 4.1.

> other operations, so waiting a lease duration doesn't imply expiry of

The Linux server revokes the delegation after a lease period even if the
client is still renewing.

I thought that was what the spec mandated, but I can't find a citation
on a quick skim--maybe I misremember.  You're probably right that this
could only happen given a buggy (or malicious?) client, but it still
seems to me a sensible safeguard....

--b.

---

From: Arnab Bakshi <[email-masked]>
Date: Tue, 4 Jun 2013 13:47:16 +0530

Hi,

   Thanks all.

    I found this in the spec in section 10.4.5 page 215:

   Whether the backchannel is functioning or not, it may be that the
   recalled delegation is not returned.  Note that the client's lease
   might still be renewed, even though the recalled delegation is not
   returned.  In this situation, servers SHOULD revoke delegations that
   are not returned in a period of time equal to the lease period.  This
   period of time should allow the client time to note the backchannel-
   down status and re-establish the backchannel.

     I hope this paragraph above clears the things we are discussing.
Even if there is renewing from client side (after the client
acknowledgement to CB_RECALL RPC) the server should revoke the
delegation if lease timer expires

just to safeguard from malicious client or anything that may have gone
wrong on client side. I hope I understood correctly..

Regards

Arnab

---

From: "J. Bruce Fields" <[email-masked]>
Date: Tue, 4 Jun 2013 13:32:07 -0400

>    down status and re-establish the backchannel.

Yep, that's what I was thinking of but couldn't find before, thanks.

> wrong on client side. I hope I understood correctly..

Yes, I think so.

--b.