Subject: RE: [nfsv4] Callback race(s) and v4/sessions

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 3 Aug 2005 10:51:02 -0400

Tom Talpey wrote:
> This is just a sketch of the idea, to see what folks think and
> see if it's appropriate to dig further. 

I definitely think it is worth proceeding with this to see if there
are any gotcha's.

> In yesterday's WG meeting
> there was some belief that adding a sequence number to the
> layout operations was a solution, I believe that's the wrong
> approach because it's specific to the layout ops, and is an
> upper layer bandaid over a lower layer issue.

Sequence numbers would work but I think a general approach is
much to be preferred. 

> I would greatly
> prefer to fix this problem right.

I think the main benefit of a general fix is going to be in 
further uses of callbacks.  Sessions, by allowing callbacks
on the same connection are going to mean that callbacks will
be generally available and when we design new features, they
are going to be used, since they are often the best choice
for a lot of purposes.

Sessions mean that we don't have to put individual feature-
specific sequencing into normal ops, as we did (sadly) in
v4.0.  It would be a shame if, while solving that problem,
we had to live with an analogous callback sequencing problem
until some v4.x with x > 1. 

-----Original Message-----
From: Talpey, Thomas 
Sent: Wednesday, August 03, 2005 10:28 AM
To: [email-masked]
Subject: [nfsv4] Callback race(s) and v4/sessions

After the working group meeting here in Paris yesterday, some
of us were discussing the callback races that pNFS layouts
(as well as ordinary v4 delegation recalls) can fall prey to.
I have a new proposal using sessions.

Briefly, this is where a client requests a delegation or layout,
and the reply from the server is delayed in the network. If a
conflicting event occurs, causing the server to send a recall
on the callback channel, the client may reply to the callback
with an error, rather than an acknowledgement. This forces
the server to perform a timeout/retry of unknown duration,
which may result in a forcible revocation.

For pNFS, it was thought at first that the presence of a v4
session on the channels would prevent this, however this is
not the case. Sessions cannot impose ordering across multiple
connections, though they do provide reliable semantics on them.

Here's the idea: add an identifier to the sessions's CB_SEQUENCE
op which carries the { slotid, sequenceid } pair associated with
the operation that requested the layout or delegation, if it is still
"active" (not retired in the server's session DRC). These values
uniquely identify each request in the session, and are provided by
the client - in effect, each request is named. The client can inspect
them to determine whether they match a request which it knows
to be outstanding, and can delay its reply to the server, or return
NFS4ERR_DELAY perhaps.

The presence of the session's reliable DRC at the server will
also prevent any issues of retries affecting this across client
reconnect (which is often associated with this type of scenario),
so the session is important for this reason as well. The unique
naming of each request by the { slotid, sequenceid } pair
is what guarantees that the server will know when the reply
to the conflicting requests are seen by the client, as the client
explicitly drains the server's DRC entries during normal activity (by
reusing the slot, or reducing max_slot).

This is just a sketch of the idea, to see what folks think and
see if it's appropriate to dig further. In yesterday's WG meeting
there was some belief that adding a sequence number to the
layout operations was a solution, I believe that's the wrong
approach because it's specific to the layout ops, and is an
upper layer bandaid over a lower layer issue. I would greatly
prefer to fix this problem right.

Reactions? And, are there other callbacks that would benefit
from having additional context such as this (other uses of this
field)?

Tom.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "Talpey, Thomas" <[email-masked]>
Date: Wed, 03 Aug 2005 11:24:30 -0400

>are going to be used, since they are often the best choice

That's a good point to add: if the client uses the session to bind the
callback channel to the same connection as the operations channel,
this race cannot happen, because of the ordering of bytes on the
TCP stream.

Sessions solve the callback problem in this case, without requiring
the extra field. On multiple connections, for example multi-NIC
trunking or even across a reconnect, the field maintains the
guarantee without sequencing heroics.

Tom.

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 3 Aug 2005 14:40:12 -0400

> this race cannot happen, because of the ordering of bytes on the

Yes but that would mean that when processing a callback I must ensure
that all previously-received reply messages must have been processed,
or else I can't be sure that I am processing the callback in the proper
environment.  That could be onerous for the implementer.  If the 
cb-sequence declares any dependency, things are easier.

-----Original Message-----
From: Talpey, Thomas 
Sent: Wednesday, August 03, 2005 11:25 AM
To: [email-masked]
Subject: RE: [nfsv4] Callback race(s) and v4/sessions

Tom.

---

From: [email-masked]
Date: Wed, 24 Aug 2005 19:14:07 -0400

> cb-sequence declares any dependency, things are easier.

I agree with Dave that having some explicit sequencing at the op level
is better (the former layout stateid appears to have been sufficient) -
if an RPC implementation dispatches replies and callbacks to different
threads of execution, reconstructing the order in which things happened
before the dispatch could be painful.

Also, Section 3.1.1 of the blocks draft contains a discussion of this
race and how to cope with it, followed by some material on concurrency
and races in the other direction in Section 3.1.2.  The conclusions reached
on these issues will need to be reflected in the next version of the ops
draft.

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------