Subject: [nfsv4] What error to return if destination server fails to READ within cnr_lease_time

From: "Adamson, Andy" <[email-masked]>
Date: Thu, 17 Dec 2015 20:42:00 +0000

From draft-ietf-nfsv4-minorversion2-39 Section 15.3.3.  DESCRIPTION of COPY_NOTIFY:

   If this operation succeeds, the source server will allow the
   cna_destination_server to copy the specified file on behalf of the
   given user as long as both of the following conditions are met:

      The destination server begins reading the source file before the
      cnr_lease_time expires. 

So on an inter-SSC the source server starts the cnr_lease_time upon the reply to COPY_NOTIFY, and 
if the cnr_lease_time expires prior to the beginning of the READ from the source
 server, the source server fails the READ.

1) What is the error returned by the source server on the READ? 

It’s not NFS4ERR_EXPIRED as this refers to the clients lease and will promt the client to recover the stateid.  

2) What is the error returned by the destination server on the COPY?

I would hope it is the same error as returned by READ. 

Do we need a new error code?

Suggestions? 

—>Andy

---

From: David Noveck <[email-masked]>
Date: Thu, 17 Dec 2015 22:46:46 -0500

> It’s not NFS4ERR_EXPIRED

I would not be so quick to dismiss this.  See below.

> as this refers to the clients lease

Normally it does but since there is no NFS4ERR_CNR_LEASE_EXPIRED, I think
it is a reasonable accommodation.

> and will promt the client to recover the stateid.

It would if the client got it, but the client is not going to get it in
this case.  The destination server is going to get it and he could
reasonably conclude that either:

   - The client's lease has expired
   - cnr_lease_time has expired

It would be nice if he could know which of these two occurred but it is not
essential.  In either case, the COPY has to be failed and the client will
find soon out enough whether his lease for the source server has expired or
not.  If it has, he is in a position to re-establish it.

Another possibility is NFS4ERR_ADMIN_REVOKED if you want to distinguish
this from a true lease expiration.

> 2) What is the error returned by the destination server on the COPY?

One possibility is NFS4ERR_PARTNER_NO_AUTH.  That's an exact fit if you
know cnr_lease_time expired.  It's kind of a rough ft if either lease could
have expired.

If you are unsure of the specfic lease expiration causing the failure,
NFS4ERR_OFFLOAD_DENIED seems like it would do what you want.

On Thu, Dec 17, 2015 at 3:42 PM, Adamson, Andy <[email-masked]>
wrote:

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: "Adamson, Andy" <[email-masked]>
Date: Fri, 18 Dec 2015 15:08:29 +0000

> On Dec 17, 2015, at 10:46 PM, David Noveck <[email-masked]> wrote:
> 
> Normally it does but since there is no NFS4ERR_CNR_LEASE_EXPIRED, I think it is a reasonable accommodation.

What if the stateid has actually expired, and the source server is saying ‘recover the stateid’ when it returns NFS4ERR_EXPIRED on the READ? In other words, there is no way to add the ‘this NFS4ERR_EXPIRED means that the cnr_lease_time not the stateid.

> It would if the client got it, but the client is not going to get it in this case.

?? The destination server is a client (referred to as dst-client) - it loads the client module, and the dst-client mounts and reads the data from the source server with _no_ change to the client code. The READ is a normal READ. So if the dst-client gets an NFS4ERR_EXPIRED on the READ, it must assume that the stateid is bad, and try to recover it. But the stateid is not bad, the clientID is not expired. This has nothing to do with the cnr_lease_time and in my opinion, should not be returned.

>  The destination server is going to get it and he could reasonably conclude that either:
> 	• The client's lease has expired:

	in which case the dst-client will try to recover the stateid, and if that indicates that the clientID has expired, then recover the clientID which is nuts as that the dst-client has _just_ established the clientID, as all that dest-client does is mount, READ, umount.

> 	• cnr_lease_time has expired

	in which case the destination server would just fail the COPY.

> It would be nice if he could know which of these two occurred but it is not essential.
> In either case, the COPY has to be failed and the client will find soon out enough whether his lease for the source server has expired or not.

Really? How does the client know if the lease for the source server has expired?

>  If it has, he is in a position to re-establish it.

Well, a new COPY needs to be started.

> 
> Another possibility is NFS4ERR_ADMIN_REVOKED if you want to distinguish this from a true lease expiration.

This also implies a stateid problem, not a cnr_lease_time problem and will prompt stateid recovery.
> One possibility is NFS4ERR_PARTNER_NO_AUTH.  That's an exact fit if you know cnr_lease_time expired.  It's kind of a rough ft if either lease could have expired.

Ah! This is the error I was looking for.

Why not add NFS4ERR_PARTNER_NO_AUTH as an error on a READ? This seems a simple addition to the protocol and would be 

—>Andy

> 
> If you are unsure of the specfic lease expiration causing the failure, NFS4ERR_OFFLOAD_DENIED seems like it would do what you want.
> 
> On Thu, Dec 17, 2015 at 3:42 PM, Adamson, Andy <[email-masked]> wrote:

---

From: David Noveck <[email-masked]>
Date: Fri, 18 Dec 2015 13:14:33 -0500

> ?? The destination server is a client (referred to as dst-client)

Yes it is a client but there is another client as well.  I was referring to
that as "the client" which didn't help clarify things.

> it loads the client module, and the dst-client mounts and reads the data
from the source server with _no_ change to the client code.

I can why one might want that, but the situation is one that typical client
code does not deal with, particularly when lease considerations are
concerned.  Normally, one issue a read with a stateid that was gotten in
the context of the client issuing the request.  If lease expiration occurs,
it is clear which lease has expired.  Here we have two clients and the
stateid was gotten by one and is used by a different one.  If
NFS4ERR_EXPIRED means a lease has expired, the obvious question is "which
one?" and RFC5661 is not very clear about that since the situation doesn't
occur in  v4.1.  I was assuming that the lease expiration would be for the
original client while you seem to be assuming it would be for the
dst-client.

> The READ is a normal READ.

It's normal in form,but it is done with a stateid that it didn't get with
an OPEN that it did, which makes it kind of "non-standard", if "abnormal"
seems inappropriate.

> So if the dst-client gets an NFS4ERR_EXPIRED on the READ, it must assume
that the stateid is bad, and try to recover it.

But there would be no way for it do that.  The normal way to do that is to
do an OPEN, but the dst-client doesn't have the information to do that.

> But the stateid is not bad, the clientID is not expired. This has nothing
to do with the cnr_lease_time and in my opinion, should not be returned.

OK, but I want to explain my logic, which I think still works, if one
assumes, as I did, that the client code will have to be as the server code
will have to be, aware of the the non-standar nature of what is going on.
Since, there are two leases involved, I was assuming NFS4ERR_EXPIRED would
mean that one or more of the following had occurred:

   1. The original client's lease has expired.  In this case, the best
   thing to do is to fail the copy immediately and let the original client do
   any necessary lease recovery on his own
   2. The dst-client's has expired.  As you point out, this case is
   unlikely to arise, since the clientid has just been established, but it is
   still (just barely) possible that a communication break at just the wrong
   time can cause this to occur.  the dst-client can test any stateid's that
   it got itself to see if this is the case that occurred.  If it didn't, it
   can just fail the copy since it doesn't matter whether 1 or 3 occurred.  If
   2 has occurred, the dst-client can only recover stateids it got for itself.
    it has no ability to recover the others and so in this case as well it
   should fail the copy.  In this case, the original client might see that its
   stateid is usable and so reissue the copy immediately
   3. The cnr_lease_time has expired.  In this case also, the best thing to
   do is to fail the copy immediately and let the original client do
   any necessary lease recovery on his own

On Fri, Dec 18, 2015 at 10:08 AM, Adamson, Andy <[email-masked]>
wrote:

> What if the stateid has actually expired, and the source server is saying
> ‘recover the stateid’ when it returns NFS4ERR_EXPIRED on the READ? In other
> words, there is no way to add the ‘this NFS4ERR_EXPIRED means that the
> cnr_lease_time not the stateid.
>
> >
> this case.
> ?? The destination server is a client (referred to as dst-client) - it
> loads the client module, and the dst-client mounts and reads the data from
> the source server with _no_ change to the client code. The READ is a normal
> READ. So if the dst-client gets an NFS4ERR_EXPIRED on the READ, it must
> assume that the stateid is bad, and try to recover it. But the stateid is
> not bad, the clientID is not expired. This has nothing to do with the
> cnr_lease_time and in my opinion, should not be returned.
>
>         in which case the dst-client will try to recover the stateid, and
> if that indicates that the clientID has expired, then recover the clientID
> which is nuts as that the dst-client has _just_ established the clientID,
> as all that dest-client does is mount, READ, umount.
>
> > In either case, the COPY has to be failed and the client will find soon
> out enough whether his lease for the source server has expired or not.
>
> Why not add NFS4ERR_PARTNER_NO_AUTH as an error on a READ? This seems a
> simple addition to the protocol and would be
>
> —>Andy
>
>
>
> >

---

From: "Adamson, Andy" <[email-masked]>
Date: Fri, 18 Dec 2015 19:09:17 +0000

> On Dec 18, 2015, at 1:14 PM, David Noveck <[email-masked]> wrote:
> 
> Yes it is a client but there is another client as well.  I was referring to that as "the client" which didn't help clarify things.
> 
> > it loads the client module, and the dst-client mounts and reads the data from the source server with _no_ change to the client code. 
> 
> I can why one might want that, but the situation is one that typical client code does not deal with, particularly when lease considerations are concerned.  Normally, one issue a read with a stateid that was gotten in the context of the client issuing the request.  If lease expiration occurs, it is clear which lease has expired.  Here we have two clients and the stateid was gotten by one and is used by a different one.  If NFS4ERR_EXPIRED means a lease has expired, the obvious question is "which one?" and RFC5661 is not very clear about that since the situation doesn't occur in  v4.1.  I was assuming that the lease expiration would be for the original client while you seem to be assuming it would be for the dst-client.  
> 
> It's normal in form,but it is done with a stateid that it didn't get with an OPEN that it did, which makes it kind of "non-standard", if "abnormal" seems inappropriate.
> 
> > So if the dst-client gets an NFS4ERR_EXPIRED on the READ, it must assume that the stateid is bad, and try to recover it. 
> 
> But there would be no way for it do that.  The normal way to do that is to do an OPEN, but the dst-client doesn't have the information to do that.
> 
> > But the stateid is not bad, the clientID is not expired. This has nothing to do with the cnr_lease_time and in my opinion, should not be returned.
> 
> OK, but I want to explain my logic, which I think still works, if one assumes, as I did, that the client code will have to be as the server code will have to be, aware of the the non-standar nature of what is going on.  Since, there are two leases involved, I was assuming NFS4ERR_EXPIRED would mean that one or more of the following had occurred:
> 	• The original client's lease has expired.  In this case, the best thing to do is to fail the copy immediately and let the original client do any necessary lease recovery on his own
> 	• The dst-client's has expired.  As you point out, this case is unlikely to arise, since the clientid has just been established, but it is still (just barely) possible that a communication break at just the wrong time can cause this to occur.  the dst-client can test any stateid's that it got itself to see if this is the case that occurred.

Um - the stateid in question did not come from the dest-client. it came from the client. This means that the stateid used by the READ to the source server is not associated with the clientID established by the mount of the dest-client. We already have special code on the source server to allow for a lookup of a stateid against a different clientID than the SEQUENCE operation resolves to - special for the COPY READ. How can it test the stateid!! We would need more special code on the server. This is IMHO not the direction to go.

>  If it didn't, it can just fail the copy since it doesn't matter whether 1 or 3 occurred.  If 2 has occurred, the dst-client can only recover stateids it got for itself.  it has no ability to recover the others and so in this case as well it should fail the copy.  In this case, the original client might see that its stateid is usable and so reissue the copy immediately
> 	• The cnr_lease_time has expired.  In this case also, the best thing to do is to fail the copy immediately and let the original client do any necessary lease recovery on his own

All of the above complexity goes away if a READ for the COPY is able to return NFS4ERR_PARTNER_NO_AUTH. 

—>Andy

> 
> 
> On Fri, Dec 18, 2015 at 10:08 AM, Adamson, Andy <[email-masked]> wrote:
>

---

From: Tom Haynes <[email-masked]>
Date: Fri, 18 Dec 2015 12:36:10 -0800

> On Dec 18, 2015, at 11:09 AM, Adamson, Andy <[email-masked]> wrote:
>> 
> 

My only concern here is that if a COPY in not going on and a client gets NFS4ERR_PARTNER_NO_AUTH, what is it supposed to do with the error?

My proposal is that we simply treat that as NFS4ERR_SERVERFAULT.

Objections?

I’ll post a patch soon for all of this.

---

From: Tom Haynes <[email-masked]>
Date: Fri, 18 Dec 2015 12:52:27 -0800

kinslayer:NFSv4.2 loghyr$ git diff
diff --git a/nfsv42_middle_errors.xml b/nfsv42_middle_errors.xml
index 090c197..c39b754 100644
--- a/nfsv42_middle_errors.xml
+++ b/nfsv42_middle_errors.xml
@@ -2457,6 +2457,7 @@
         NFS4ERR_OLD_STATEID,
         NFS4ERR_OPENMODE,
         NFS4ERR_OP_NOT_IN_SESSION,
+        NFS4ERR_PARTNER_NO_AUTH,
         NFS4ERR_PNFS_IO_HOLE,
         NFS4ERR_PNFS_NO_LAYOUT,
         NFS4ERR_REP_TOO_BIG,
@@ -2538,6 +2539,7 @@
         NFS4ERR_OLD_STATEID,
         NFS4ERR_OPENMODE,
         NFS4ERR_OP_NOT_IN_SESSION,
+        NFS4ERR_PARTNER_NO_AUTH,
         NFS4ERR_PNFS_IO_HOLE,
         NFS4ERR_PNFS_NO_LAYOUT,
         NFS4ERR_REP_TOO_BIG,
diff --git a/nfsv42_middle_new_errors.xml b/nfsv42_middle_new_errors.xml
index 6e2d68a..5cf0ebd 100644
--- a/nfsv42_middle_new_errors.xml
+++ b/nfsv42_middle_new_errors.xml
@@ -463,6 +463,7 @@
         NFS4ERR_OLD_STATEID,
         NFS4ERR_OPENMODE,
         NFS4ERR_OP_NOT_IN_SESSION,
+        NFS4ERR_PARTNER_NO_AUTH,
         NFS4ERR_PNFS_IO_HOLE,
         NFS4ERR_PNFS_NO_LAYOUT,
         NFS4ERR_REP_TOO_BIG,
diff --git a/nfsv42_middle_op_copy_notify.xml b/nfsv42_middle_op_copy_notify.xml
index 71c3df3..e011d3c 100644
--- a/nfsv42_middle_op_copy_notify.xml
+++ b/nfsv42_middle_op_copy_notify.xml
@@ -62,7 +62,11 @@
           before the cnr_lease_time expires. If the cnr_lease_time
           expires while the destination server is still reading
           the source file, the destination server is allowed to
-          finish reading the file.
+          finish reading the file. If the cnr_lease_time
+          expires before the destination server uses READ or READ_PLUS
+          to begin the transfer, the source server can use
+          NFS4ERR_PARTNER_NO_AUTH to inform the destination server
+          that the cnr_lease_time has expired.
         </t>
 
         <t>

---

From: David Noveck <[email-masked]>
Date: Fri, 18 Dec 2015 19:02:19 -0500

> All of the above complexity goes away if a READ for the COPY is able to
return NFS4ERR_PARTNER_NO_AUTH.

Not all of it.  Only the complexity associated with situation #3.
Situations #1 and #2 remain.

Still, I think this is a good change to make.

On Fri, Dec 18, 2015 at 2:09 PM, Adamson, Andy <[email-masked]>
wrote:

> > Yes it is a client but there is another client as well.  I was referring
> to that as "the client" which didn't help clarify things.
> >
> > > it loads the client module, and the dst-client mounts and reads the
> data from the source server with _no_ change to the client code.
> >
> > I can why one might want that, but the situation is one that typical
> client code does not deal with, particularly when lease considerations are
> > It's normal in form,but it is done with a stateid that it didn't get
> with an OPEN that it did, which makes it kind of "non-standard", if
> "abnormal" seems inappropriate.
> >
> > > So if the dst-client gets an NFS4ERR_EXPIRED on the READ, it must
> assume that the stateid is bad, and try to recover it.
> >
> > But there would be no way for it do that.  The normal way to do that is
> to do an OPEN, but the dst-client doesn't have the information to do that.
> >
> > > But the stateid is not bad, the clientID is not expired. This has
> nothing to do with the cnr_lease_time and in my opinion, should not be
> returned.
> >
> >       • The original client's lease has expired.  In this case, the best
> >       • The dst-client's has expired.  As you point out, this case is
> it got itself to see if this is the case that occurred.
>
> Um - the stateid in question did not come from the dest-client. it came
> from the client. This means that the stateid used by the READ to the source
> server is not associated with the clientID established by the mount of the
> dest-client. We already have special code on the source server to allow for
> a lookup of a stateid against a different clientID than the SEQUENCE
> operation resolves to - special for the COPY READ. How can it test the
> stateid!! We would need more special code on the server. This is IMHO not
> the direction to go.
>
> >  If it didn't, it can just fail the copy since it doesn't matter whether
> 1 or 3 occurred.  If 2 has occurred, the dst-client can only recover
> stateids it got for itself.  it has no ability to recover the others and so
> in this case as well it should fail the copy.  In this case, the original
> client might see that its stateid is usable and so reissue the copy
> immediately
> >       • The cnr_lease_time has expired.  In this case also, the best
> > What if the stateid has actually expired, and the source server is
> saying ‘recover the stateid’ when it returns NFS4ERR_EXPIRED on the READ?
> In other words, there is no way to add the ‘this NFS4ERR_EXPIRED means that
> the cnr_lease_time not the stateid.
> >
> > >
> >         in which case the dst-client will try to recover the stateid,
> and if that indicates that the clientID has expired, then recover the
> clientID which is nuts as that the dst-client has _just_ established the
> clientID, as all that dest-client does is mount, READ, umount.
> >
> > > In either case, the COPY has to be failed and the client will find
> soon out enough whether his lease for the source server has expired or not.
> >
> > > Another possibility is NFS4ERR_ADMIN_REVOKED if you want to
> distinguish this from a true lease expiration.
> >
> > > One possibility is NFS4ERR_PARTNER_NO_AUTH.  That's an exact fit if
> you know cnr_lease_time expired.  It's kind of a rough ft if either lease
> could have expired.
> >
> >