Subject: Re: [nfsv4] seqid-based 4.0 replays

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 8 Sep 2011 18:37:12 -0400

On Mon, Aug 29, 2011 at 05:46:02PM -0400, bfields wrote:
> Dumb basic 4.0 question:
> 
> While reviewing the linux 4.0 server code, I was reminded that it
> treats the 4.0 seqid-based reply cache purely at the level of a single
> compound op: if you send a completely different compound with a just one
> of the ops a replay of a previous one, it returns a reply to that one op
> from cache and processes all the other ops in the compound normally.
> 
> I'm pretty sure that's wrong.  Or at least more complicated than
> necessary.
> 
> But looking at the rfc I'm not completely clear what I can assume: can I
> assume that any replay will have the same xid as the original?  (So is
> it enough just to create an entry in a traditional DRC, and just make
> sure that cache entry lasts until the stateowner is destroy or the seqid
> incremented?  Rick's old connectathon talk suggests that's what he
> does.)

After talking this over with Trond: we both thought a server should be
prepared to handle a resend in a new rpc call (so, possibly with new
xid), but should otherwise be able to assume that the compound is
identical.

Rick, is that how your implementation works?

--b.

---

From: Rick Macklem <[email-masked]>
Date: Thu, 8 Sep 2011 21:26:50 -0400 (EDT)

> Rick, is that how your implementation works?
My DRC cache code assumes that a retry will use the same xid, but it
recognizes "same" by not just "same xid" but also "same request length
for the xdr" and same checksum on the first K bytes.

I don't see why a retry (only done after a TCP connection breaks due
to something like a network partitioning, heals and a new TCP connection
is established) would use a different xid? (Sun RPC has never done that.)

Otherwise, yes, if the first N bytes generate the same hash and there are
the same # of xdr bytes, it assumes
that it is a retry of exactly the same compound (all same bits from xid on).
Since CPUs have gotten sooo fast, I'm not sure there's an argument for not
doing the hash on the entire request, but I currently only do the first N bytes.
It holds onto the reply until the next request in sequence (seqid for open_owner4
or lock_owner4) is received or when the open/lock owner goes away. (lock_owner4's
go away when the server sees a ReleaseLockOwner or when the associated open stateid
is closed. An open_owner4 goes away some time after there are no longer any opens
for it.
> --b.
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 08 Sep 2011 21:49:33 -0400

> is established) would use a different xid? (Sun RPC has never done that.)

That is irrelevant. The spec trumps implementations...

I agree that most clients out there will reuse the XID, but there is
nothing in the NFSv4 spec that requires you to do so. OTOH, there are
very strict requirements for when the client replays a stateful
operation.

> is closed. An open_owner4 goes away some time after there are no longer any opens

We had a question about this in the Linux NFS concall earlier today. As
far as I can see, the issue of what happens if a Linux client replays an
OPEN/LOCK/LOCKU/CLOSE/... operation incorrectly (i.e. a different
compound) is undefined. We don't have any equivalent to
NFS4ERR_SEQ_FALSE_RETRY in NFSv4 minor version 0.

Cheers
  Trond
-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 9 Sep 2011 05:35:52 -0400

> very strict requirements for when the client replays a stateful

One minor practical advantage to requiring servers to detect replays
with different xid's: I can write an easy test for it.

Without that I have to have some server-specific magic to disable or
somehow DOS the server's xid-based reply cache.

Grepping through the latest 3530bis:

	9.1.8: "When any one of the sequence values matches a previous
	request, for a state-owner, it is treated as a retransmission
	and not re-executed."

That appears to me to fall on the "same xid not required" side of the
argument.

> NFS4ERR_SEQ_FALSE_RETRY in NFSv4 minor version 0.

3530bis 9.18. does say "When the server can determine that the request
differs from the original it may return NFS4ERR_BADSEQ."

--b.

---

From: Saul Tamari <[email-masked]>
Date: Fri, 9 Sep 2011 17:09:59 +0300

Hi,

What about retries of CREATE or REMOVE? They don't have any sequence state
associated with them.
How do you know when to stop caching them?

Thanks,
Saul

---

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 9 Sep 2011 11:17:25 -0400

> How do you know when to stop caching them?

There's not really a standard 100%-reliable solution (except for
upgrading to 4.1 and sessions).

Rick gave a talk with some ideas here:

	http://nfsv4bat.org/Documents/ConnectAThon/2006/macklem.pdf

--b.

---

From: Rick Macklem <[email-masked]>
Date: Mon, 12 Sep 2011 14:23:11 -0400 (EDT)

J. Bruce Fields wrote:
> > > > > treats the 4.0 seqid-based reply cache purely at the level of
> > > > > a
> > > > > single
> > > > > compound op: if you send a completely different compound with
> > > > > a just
> > > > > one
> > > > > of the ops a replay of a previous one, it returns a reply to
> > > > > that
> > > > > one op
> > > > > But looking at the rfc I'm not completely clear what I can
> > > > > assume:
> > > > > can I
> > > > > assume that any replay will have the same xid as the original?
> > > > > (So
> > > > > is
> > > > > it enough just to create an entry in a traditional DRC, and
> > > > > just
> > > > > make
> > > > > sure that cache entry lasts until the stateowner is destroy or
> > > > > the
> > > > > seqid
> > nothing in the NFSv4 spec that requires you to do so.
Yep, but I'd argue that it is the Sun RPC RFCs that define this. I know
that the NFSv4 specs have special requirements, like the one below and
the requirement that the Compound be retried on a different TCP connection,
but I don't think anything in the NFSv4 RFCs states that you should violate
the Sun RPC RFCs, is there?

(Now I have to go look at the newest version of the Sun RPC RFC and see
 what it states w.r.t. retries and XID.)

rick

> OTOH, there
> > are

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 12 Sep 2011 14:38:40 -0400

>  what it states w.r.t. retries and XID.)

It says:

   All messages start with a transaction identifier, xid, followed by a
   two-armed discriminated union.  The union's discriminant is a
   msg_type that switches to one of the two types of the message.  The
   xid of a REPLY message always matches that of the initiating CALL
   message.  NB: The "xid" field is only used for clients matching reply
   messages with call messages or for servers detecting retransmissions;
   the service side cannot treat this id as any type of sequence number.

IOW: Sun RPC completely punts on the issue.

Cheers
  Trond

NetApp