Subject: [nfsv4] Issue with section 3.2.5.1?

From: "Noveck, David" <[email-masked]>
Date: Tue, 13 Aug 2013 15:27:20 -0400

Although I noticed this in looking at the diff between -19 and -20, I don't think it is related to the wording
changes that caused that section to be different in the two versions.

The issue is that the idea that the destination sever gets the stateid for the source file by doing
a CLAIM_FH OPEN seems inconsistent with the idea that the client does the OPEN and passes in
the stateid in the COPY request.

---

From: "Haynes, Tom" <[email-masked]>
Date: Tue, 13 Aug 2013 20:14:39 +0000

On Aug 13, 2013, at 12:27 PM, "Noveck, David" <[email-masked]<mailto:[email-masked]>> wrote:

Although I noticed this in looking at the diff between -19 and -20, I don’t think it is related to the wording
changes that caused that section to be different in the two versions.

The issue is that the idea that the destination sever gets the stateid for the source file by doing
a CLAIM_FH OPEN seems inconsistent with the idea that the client does the OPEN and passes in
the stateid in the COPY request.
_______________________________________________
nfsv4 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nfsv4

Right, I struggled with that one as well, let me try to explain how I read it.

   struct COPY4args {
           /* SAVED_FH: source file */
           /* CURRENT_FH: destination file */
           stateid4        ca_src_stateid;
           stateid4        ca_dst_stateid;
           offset4         ca_src_offset;
           offset4         ca_dst_offset;
           length4         ca_count;
           netloc4         ca_source_server<>;
   };

ca_dst_stateid is for CURRENT_FH as seen by the client as the open-owner or lock-owner
ca_src_stateid is for SAVED_FH as seen by the client as the open-owner or lock-owner

So if the destination wants to get a stateid on the source, it needs to establish an
open-owner or lock-owner relationship.

I.e., if the client reboots, does that necessitate that when we break the leased state
between the client and source that we also break the leased state between the
destination and source?

If I'm interpreting this wrong, then I'm open to removing the wording about CLAIM_FH OPEN
in that section.

---

From: "Myklebust, Trond" <[email-masked]>
Date: Tue, 13 Aug 2013 21:38:15 +0000

On Tue, 2013-08-13 at 20:14 +0000, Haynes, Tom wrote:
> 
> On Aug 13, 2013, at 12:27 PM, "Noveck, David" <[email-masked]>
> wrote:
> 
> > Although I noticed this in looking at the diff between -19 and -20,
> > I don’t think it is related to the wording
> > [email-masked]

It can't: the client may already have established an exclusive lock on
that file.

> If I'm interpreting this wrong, then I'm open to removing the wording

I've always been assuming that when doing server-server copy using
NFSv4.1, then the destination server will use the " ca_src_stateid" that
was supplied by the client.

IOW: No open, or lookup: just READ with the supplied FH and stateid. Why
would that not work?

Cheers
  Trond
-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: "Noveck, David" <[email-masked]>
Date: Wed, 14 Aug 2013 11:02:28 -0400

> I've always been assuming that when doing server-server copy using NFSv4.1, then 
> the destination server will use the " ca_src_stateid" that was supplied by the client.

I'd been assuming the same thing.

> IOW: No open, or lookup: just READ with the supplied FH and stateid. 
> Why would that not work?

It would work for v4.0 stateids, but in the case of v4.1 (and later) ca_src_stateid values,
the following paragraphs from section 8.2 of RFC5661 pose a difficulty in the inter-server
case:

   All stateids associated with a given client ID are associated with a
   common lease that represents the claim of those stateids and the
   objects they represent to be maintained by the server.  See
   Section 8.3 for a discussion of the lease.

   The server may assign stateids independently for different clients.
   A stateid with the same bit pattern for one client may designate an
   entirely different set of locks for a different client.  The stateid
   is always interpreted with respect to the client ID associated with
   the current session.  Stateids apply to all sessions associated with
   the given client ID, and the client may use a stateid obtained from
   one session on another session associated with the same client ID.

The basic issue is that, in v4.1, stateids (and lockowners and openowners)
are implicitly qualified by the clientid associated with the session that they 
are used on.  If the destination server were to act as a client of the source 
server, and use one of  these types of locking-based objects, it would have 
to pretend to be another client (the one doing the COPY) leading to the 
kinds of lease management issues that Tom was concerned about. 

-----Original Message-----
From: Myklebust, Trond [mailto:[email-masked]] 
Sent: Tuesday, August 13, 2013 5:38 PM
To: Haynes, Tom
Cc: Noveck, David; nfsv4 list ([email-masked])
Subject: Re: [nfsv4] Issue with section 3.2.5.1?

On Tue, 2013-08-13 at 20:14 +0000, Haynes, Tom wrote:
> > I don’t think it is related to the wording changes that caused that 
> > section to be different in the two versions.
> >  
> > The issue is that the idea that the destination sever gets the 
> > stateid for the source file by doing a CLAIM_FH OPEN seems 
> > inconsistent with the idea that the client does the OPEN and passes 
> > in the stateid in the COPY request.
> ca_dst_stateid is for CURRENT_FH as seen by the client as the 
> open-owner or lock-owner ca_src_stateid is for SAVED_FH as seen by the 
> client as the open-owner or lock-owner
> 
> 
> So if the destination wants to get a stateid on the source, it needs 
> to establish an open-owner or lock-owner relationship.

It can't: the client may already have established an exclusive lock on that file.

> I.e., if the client reboots, does that necessitate that when we break 
> the leased state between the client and source that we also break the 
> leased state between the destination and source?
> 
> 
> If I'm interpreting this wrong, then I'm open to removing the wording 
> about CLAIM_FH OPEN in that section.

I've always been assuming that when doing server-server copy using NFSv4.1, then the destination server will use the " ca_src_stateid" that was supplied by the client.

IOW: No open, or lookup: just READ with the supplied FH and stateid. Why would that not work?

Cheers
  Trond

NetApp

---

From: "Myklebust, Trond" <[email-masked]>
Date: Wed, 14 Aug 2013 15:46:51 +0000

> kinds of lease management issues that Tom was concerned about. 

Right, but COPY_NOTIFY is telling the source server both which stateid
that the destination server will use, as well as the network location
associated with that server.

If you are worried about spoofing, then one solution might be to have
COPY_NOTIFY return an SSV that the destination server can use as part of
an SSV GSS authentication for the actual READ and WRITE requests.

Cheers
  Trond

NetApp

---

From: "Haynes, Tom" <[email-masked]>
Date: Wed, 14 Aug 2013 15:59:44 +0000

> associated with that server.

It is possible for the destination to already be acting as a client to the source,
which means it has a client ID and possibly some existent stateids.

What happens when the source gives the client a stateid CSA which has the same
bit pattern as an existent stateid the destination has with the source?

---

From: "Myklebust, Trond" <[email-masked]>
Date: Wed, 14 Aug 2013 16:10:10 +0000

> bit pattern as an existent stateid the destination has with the source?

See my response to Dave. I suggest having COPY_NOTIFY return an SSV for
use by the destination server to tie the use of this stateid in GSS
SSV-authenticated READ requests to this client (and you can even tie it
to this specific COPY operation).

NetApp

---

From: "Myklebust, Trond" <[email-masked]>
Date: Wed, 14 Aug 2013 17:22:08 +0000

On Wed, 2013-08-14 at 13:02 -0400, Noveck, David wrote:
> I accept that it is possible things could be done this way.  But you need, among other things,
>   o A new section about how the destination server constructs READ operations that it uses
>      to implement the COPY, including how it recovers from the client of the COPY rebooting,
>     and the source server rebooting.

I accept that we might want to describe the READ operation. I'm assuming
a very simple model that is similar to that used by pNFS clients to talk
to the DSes, unless someone sees an objection.
So the destination server sets up a session on the source server using
whatever clientid+authentication that it wants (including possibly the
SSV that was provided by the COPY_NOTIFY - although I'd prefer not to
have to do that). The destination server then uses a standard SEQUENCE
+PUTFH+READ in combination with SSV GSS authentication to actually read
the data from the file on the source.

Reboot recovery is not an issue: the COPY operation halts on reboot of
either server and also in the case where the client's lease expires.

>  o A new version of the description of the *READ* operation that specifies how the source
>      server is to distinguish these special on-behalf-of-COPY READ operations and then handle 
>      them.

That's what the SSV is for. The _source_ server would provide the SSV,
and therefore can use that to authenticate the READ operations. The SSV
will actually tie the entire set of READ requests to the specific
COPY_NOTIFY request.

> In addition to having those things written as well as a bunch of ancillary addition and revisions,
> (volunteers anyone?), and getting consensus on the revised spec, I would not be confident 
> about this until there was an actual implementation of inter-server copy that did this and my 
> guess is that we won't see  that in Westford next month.

I agree that an implementation is going to take a while. For one thing,
I'm not aware of any existing implementations of SSVs or SSV GSS,
despite their prominence in RFC5661.

That said, I believe that this is a viable alternative to the
RPCSEC_GSSv3 route, which, while appearing to suffer no lack of
cheerleaders, has seen no volunteers for shepherding a spec through the
IETF.
It may therefore be worth considering for that reason.

> This is sounding like something that will not make it into a draft-ietf-nfsv4-minorversion2-xx
> for any reasonable value of xx.

Possibly, and in that case, we should probably defer inter-server copy
to 4.3. I'd be OK with that solution, since it would not impact our
current intra-server only Linux implementation.

Cheers
  Trond

> Sent: Wednesday, August 14, 2013 12:10 PM
> To: Haynes, Tom
> > >> The basic issue is that, in v4.1, stateids (and lockowners and 
> > >> openowners) are implicitly qualified by the clientid associated 
> > >> with the session that they are used on.  If the destination server 
> > >> were to act as a client of the source server, and use one of  these 
> > >> types of locking-based objects, it would have to pretend to be 
> > >> another client (the one doing the COPY) leading to the kinds of lease management issues that Tom was concerned about.
> > > 
> > > Right, but COPY_NOTIFY is telling the source server both which 
> > > stateid that the destination server will use, as well as the network 
> > > location associated with that server.
> > 
> > It is possible for the destination to already be acting as a client to 
> > the source, which means it has a client ID and possibly some existent stateids.
> > 
> > What happens when the source gives the client a stateid CSA which has 
> > the same bit pattern as an existent stateid the destination has with the source?
> 
> See my response to Dave. I suggest having COPY_NOTIFY return an SSV for use by the destination server to tie the use of this stateid in GSS SSV-authenticated READ requests to this client (and you can even tie it to this specific COPY operation).
> 
> --
> Trond Myklebust

NetApp

---

From: "Noveck, David" <[email-masked]>
Date: Wed, 14 Aug 2013 13:02:10 -0400

-----Original Message-----
From: Myklebust, Trond [mailto:[email-masked]] 
Sent: Wednesday, August 14, 2013 12:10 PM
To: Haynes, Tom
Cc: Noveck, David; nfsv4 list ([email-masked])
Subject: Re: [nfsv4] Issue with section 3.2.5.1?

NetApp

---

From: "Haynes, Tom" <[email-masked]>
Date: Wed, 14 Aug 2013 17:37:52 +0000

If the source is willing to accept the ca_src_stateid from any client ID, then
the issue is making sure that the destination does not already have the
existing bit pattern in the stateid associated with the source.

Couldn't the destination scan its stateids for the source and return
NFS4ERR_BAD_STATEID or make up some new error code to inform
the client of a stateid mixup?

Again, I'm willing to have inter-server in 4.2 with the goal that once
RPCSEC_GSSv3 comes along, we can spin out a separate document
which will enable inter-server SSC and full-blown LNFS without spinning
a new version of NFSv4.2.

---

From: "Myklebust, Trond" <[email-masked]>
Date: Wed, 14 Aug 2013 17:45:15 +0000

> existing bit pattern in the stateid associated with the source.

Again, the point of the SSV is to ensure that the source ties the
ca_src_stateid to the specific COPY_NOTIFY operation that generated it.
IOW: the source can use that to map the ca_src_stateid back to the
client id of the _client_ which issued the COPY_NOTIFY.

> the client of a stateid mixup?

That shouldn't be necessary when using the SSV.

NetApp

---

From: "Noveck, David" <[email-masked]>
Date: Wed, 14 Aug 2013 13:59:31 -0400

> That's what the SSV is for. The _source_ server would provide the SSV, and therefore 
> can use that to authenticate the READ operations. The SSV will actually tie the entire set 
> of READ requests to the specific COPY_NOTIFY request.

Understood, but my point was that someone is going to have to describe that in the spec
in a way that client and source server and destination server implementers can read and
follow, so that they will all interoperate.

-----Original Message-----
From: Myklebust, Trond [mailto:[email-masked]] 
Sent: Wednesday, August 14, 2013 1:22 PM
To: Noveck, David
Cc: Haynes, Tom; nfsv4 list ([email-masked])
Subject: Re: [nfsv4] Issue with section 3.2.5.1?

>     and the source server rebooting.

I accept that we might want to describe the READ operation. I'm assuming a very simple model that is similar to that used by pNFS clients to talk to the DSes, unless someone sees an objection.
So the destination server sets up a session on the source server using whatever clientid+authentication that it wants (including possibly the SSV that was provided by the COPY_NOTIFY - although I'd prefer not to have to do that). The destination server then uses a standard SEQUENCE
+PUTFH+READ in combination with SSV GSS authentication to actually read
the data from the file on the source.

Reboot recovery is not an issue: the COPY operation halts on reboot of either server and also in the case where the client's lease expires.

>      server is to distinguish these special on-behalf-of-COPY READ operations and then handle 

That's what the SSV is for. The _source_ server would provide the SSV, and therefore can use that to authenticate the READ operations. The SSV will actually tie the entire set of READ requests to the specific COPY_NOTIFY request.

> In addition to having those things written as well as a bunch of 
> ancillary addition and revisions, (volunteers anyone?), and getting 
> consensus on the revised spec, I would not be confident about this 
> until there was an actual implementation of inter-server copy that did this and my guess is that we won't see  that in Westford next month.

I agree that an implementation is going to take a while. For one thing, I'm not aware of any existing implementations of SSVs or SSV GSS, despite their prominence in RFC5661.

That said, I believe that this is a viable alternative to the
RPCSEC_GSSv3 route, which, while appearing to suffer no lack of cheerleaders, has seen no volunteers for shepherding a spec through the IETF.
It may therefore be worth considering for that reason.

> This is sounding like something that will not make it into a 
> draft-ietf-nfsv4-minorversion2-xx for any reasonable value of xx.

Possibly, and in that case, we should probably defer inter-server copy to 4.3. I'd be OK with that solution, since it would not impact our current intra-server only Linux implementation.

Cheers
  Trond

> > >> with the session that they are used on.  If the destination 
> > >> server were to act as a client of the source server, and use one 
> > >> of  these types of locking-based objects, it would have to 
> > >> pretend to be another client (the one doing the COPY) leading to the kinds of lease management issues that Tom was concerned about.
> > > 
> > > stateid that the destination server will use, as well as the 
> > > network location associated with that server.
> > 
> > It is possible for the destination to already be acting as a client 
> > to the source, which means it has a client ID and possibly some existent stateids.
> > 
> > What happens when the source gives the client a stateid CSA which 
> > has the same bit pattern as an existent stateid the destination has with the source?
> 

NetApp

---

From: Andy Adamson <[email-masked]>
Date: Fri, 30 Aug 2013 17:29:52 -0400

Since I am deciding whether or not to proceed with finishing
draft-ietf-nfsv4-rpcsec-gssv3, I would like to first summarize where I
think we stand.

AFAICS there are three proposals for SSC: NFSv4.2 SSC with
RPCSEC_GSS_V3, NFSv4.2 SSC without RPCSEC_GSS_V3, and NFSv4.2 SSC
without RPCSEC_GSS_V3 and with proposed SSV. I think I pretty much get
the first two, which I summarize in the attachment.

Since the the third SSV proposal has not really been discussed, I have
some questions to stimulate a discussion :) So if you want to cut to
the quick,  ignore the
 attachment and just look below where I ask some SSV questions.

Hi Trond:

Here are some of your responses in this email stream:

--- From Trond:

and

finally:

----

Not having any SSV use design description other than above, I have
some initial questions. I also must say that I'm just wrapping my head
around SSV, so I may not understand all of it. Anyway, I need to
understand better what is being proposed.

Questions:

1) As described in rfc5661 Section 2.10.9, the establisment of an SSV
is tied to SP4_SSV in EXCHANGE_ID which is the only way to setup a GSS
SSV context. So for an SSV to be used to authenticate the READ
requests from the destination server acting as a client (herafter
called the ds_client), said SSV must be setup via an EXCHANGE_ID from
the ds_client to the source server.

Is the <ds_client,source server> clientid SSV the one that COPY_NOTIFY
will return for use by the ds_client or are you proposing a new way to
setup an SSV?

2) If the <ds_client,source server> clientid SSV is the one
COPY_NOTIFY will return, how does the source server grant permission
for the COPY_NOTIFY to do so, considering the COPY_NOTIFY is
associated with a different clientid :the <client, source server>
clientid?

3) rfc5661 Section 2.10.8.3. "Protection from Unauthorized State Changes" notes:

   The SP4_SSV protection option is intended for the
   situation comprised of a client that has multiple active users and a
   system administrator who wants to avoid the burden of installing a
   permanent machine credential on each client.

and
      Each time a
      new principal uses a client ID for the first time, the client
      SHOULD send a SET_SSV with that principal's RPCSEC_GSS
      credentials, with RPCSEC_GSS service set to RPC_GSS_SVC_PRIVACY.

Are you saying that the <ds_client, source server> clientid
establishment use SP4_SSV?

If so, what happnens if another principal wants to use the session
created under the SSV that the COPY_NOTIFY has grabbbed, and before
the READ operation. SET_SSV has been called - doesn't that change the
current SSV so that the old SSV presented by COPY_NOTIFY is no longer
valid? Or don't I get SET_SSV :) ?

Thanks

-->Andy